Agent-based evacuation simulations have long been a staple of disaster planning. Researchers model thousands of virtual residents moving through a real street network toward shelters, incorporating details such as reduced walking speed in dense crowds to reproduce bottlenecks and congestion. Traditionally, the results of such models are reported through aggregate statistics: how many agents evacuated, how long it took on average, where the choke points formed. These numbers are indispensable for modelers, but they say little about the lived, second-by-second experience of any individual evacuee. As computational power has grown and simulation outputs have become larger and higher-dimensional, the gap between what the models produce and what humans can readily interpret has widened.

Large language models, combined with retrieval-augmented generation, offer a way to close that gap. In a retrieval-augmented generation, or RAG, pipeline, the model does not rely on its internal knowledge alone; instead, relevant external documents—in this case movement logs, route maps, hazard maps, and population attribute data—are retrieved and supplied as context before the model writes its output. Multimodal RAG extends this to images, which matters here because route diagrams and disaster-prevention maps carry spatial information that a plain CSV table cannot easily convey. The Japanese research team built their pipeline around a tsunami-evacuation simulation of the Sanbo district of Sakai City, Osaka Prefecture, where agents follow shortest paths on a GIS-derived road network toward sites such as Asakayama Hospital.

The central methodological innovation is a three-layer decomposition borrowed from prior work on experience-oriented social simulation. The event layer contains the objective facts output directly by the simulation: the sequence of locations each agent passed, the arrival times, and the route. The context layer holds interpretations derived from those facts—in this study, estimates of whether congestion was present at each key location, obtained by comparing a congestion-free travel time with the agent’s actual passage time. The value layer is the only part allowed to vary: a first-person narrative expressing the agent’s subjective viewpoint, emotions, and decision-making rationale. The design goal is explicit boundary management—subjective color may shift, but the factual skeleton of times, locations, and route order must remain invariant.

To make this auditable rather than merely plausible, the researchers fixed every variable they could. The reference corpus, the vector store built with FAISS, the search query templates, and the retrieved documents were all frozen as snapshots, so that across 100 iterative generations per condition the model received identical evidence. Embeddings were produced with the Sentence-Transformers model all-MiniLM-L6-v2, image captioning used the BLIP model supplemented by analysis from a large vision-language model, and narrative generation ran on GPT-5-nano at temperature zero. Even at temperature zero, deployed language models are not fully deterministic across API calls, and the 100-trial design was built precisely to characterize that residual variability rather than to hide it behind a single cherry-picked example.

The quality gate itself is a set of fully automatic, rule-based checks with no tunable thresholds. An output passes only if it mentions all seven key locations, includes all eight expected timestamps in exact form, preserves the chronological order of locations, contains no prohibited internal representations such as agent identifiers or TRUE/FALSE flags, and avoids formatting deviations like bullet lists that would be jarring in a continuous first-person account. The gate is explicitly framed as a diagnostic instrument, not a certification: it detects and quantifies failure modes rather than guaranteeing that a narrative is fit for public deployment.

The results are instructive. For three agents following the same route, location coverage was high, at 95 to 99 percent, and sequence inconsistency was nearly absent, at 0 to 1 percent—evidence that the factual skeleton of the route survives generation. Timestamp coverage, however, lagged at 83 to 91 percent, and a programmatic check confirmed that the missing times were genuinely absent from the text in any form, not merely reformatted. More strikingly, prohibited terms or internal representations leaked into 7 to 23 percent of outputs, with strings like AgtID and TRUE/FALSE appearing in the prose. Overall quality-gate pass rates ranged from 62 to 81 percent across the three agents—far below what unsupervised public-facing use would require, and the authors state this plainly rather than downplaying it.

The team also tested whether the value layer could be steered without disturbing the factual skeleton. Holding the event and context layers fixed, they substituted household-composition data for one agent across three conditions: a baseline five-person household including a teenager, a household replacing the teenager with a member in their 90s, and an all-elderly household of five members in their 80s. The lexical indicators responded in the expected directions—child-related terms fell when the teenager was removed, elderly-related and mutual-aid terms rose with the older households—while location coverage, time coverage, and sequence consistency remained broadly comparable across conditions. No clear disruption of the factual skeleton was detected under any tested substitution, suggesting that the layered architecture can separate what must stay fixed from what is allowed to vary.

The authors trace the dominant leakage path to a specific engineering choice: the raw log vocabulary, including Boolean congestion flags and column names, was exposed to the model without normalization. Their proposed remedy is a label-normalization layer inserted before value-layer retrieval, which would rewrite TRUE/FALSE flags into natural phrases and mask identifiers at the source. For the timestamp bottleneck, they outline interventions including structured skeleton generation validated programmatically before rendering, prompt constraints requiring every event sentence to begin with its timestamp, and post-generation verify-and-repair loops that regenerate only the affected sentences. Notably, prompt-level prohibitions on meta-commentary proved largely effective, appearing in at most 1 percent of outputs—evidence that instruction-following works when the forbidden content is not simultaneously present in the input data.

The study’s limitations are acknowledged with unusual candor. It examines a single tsunami scenario in one district, three agents on an identical non-branching route, and a narrow set of household substitutions within the same speed class and household size. The narratives were generated in Japanese, so the lexical indicators are tied to Japanese dictionaries and would need re-instantiation for other languages. What the authors argue generalizes is not the specific numbers but the audit protocol itself: the three-layer decomposition, the fixed-retrieval iterative-generation design that isolates model variability, and the rule-based quality gates are all portable to other simulation domains. For now, the framework stands as pre-deployment diagnostic evidence—a way to measure, systematically and repeatably, where AI-rendered simulation narratives break down before anyone entrusts them to the public.

Subject of Research: Quality-gated auditing of LLM-based narrative rendering of agent-based tsunami evacuation simulation logs using multimodal retrieval-augmented generation

Article Title: Quality gate framework for auditing narrative rendering of evacuation simulation logs by multimodal retrieval augmented generation

Article References: Hirahata, Y., & Sakahira, F. (2026). Quality gate framework for auditing narrative rendering of evacuation simulation logs by multimodal retrieval augmented generation. Discover Artificial Intelligence, 6(1), Article 1366. https://doi.org/10.1007/s44163-026-02075-5

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02075-5

Keywords: large language models, retrieval-augmented generation, agent-based simulation, tsunami evacuation, quality gates, data-to-text generation, multimodal RAG, hallucination, synthetic population, disaster preparedness, narrative rendering, simulation explainability

QualiGuard was developed by Batın Mert Kocakaplan, Emin Borandağ, Yusuf Özçevik, Fatih Yücalar, and Osman Altay, and is released under an MIT license with its trained model artifacts and derived metric dataset available under Creative Commons CC BY 4.0. The tool is built in Python 3.10 and integrates a remarkable range of established components: the Radon library for static complexity metrics, PyDriller for mining Git commit history, scikit-learn, LightGBM, XGBoost, TensorFlow, AutoGluon, and H2O AutoML for predictive modeling, and the GitHub REST API for large-scale repository collection. Rather than delivering disconnected fragments of analysis, the system assembles the entire chain, from raw repository to actionable verdict, into a single deployable package.

The premise underlying the tool is that software quality risk can be modeled as a supervised learning problem, in which measurable properties of source code and its development history are associated with observed quality outcomes. For every source file, QualiGuard extracts static size and complexity metrics, cognitive-complexity measurements derived from the abstract syntax tree, and process metrics from the Git history, including code churn, author entropy, and commit cadence. Defect labels are generated through the Śliwerski–Zimmermann–Zeller (SZZ) algorithm, a classic repository-mining technique that identifies lines deleted by bug-fixing commits and traces them back to the changes that introduced them. This combination of structural and historical evidence allows the assessment to account not only for how a file looks today but also for how it has evolved.

Beyond defect prediction, the tool detects seven classical code smells, including long methods, large classes, long parameter lists, deep nesting, god functions, high complexity, and low maintainability, using deterministic abstract-syntax-tree analysis. A second, predictive model complements this detector: using 28 code metrics, it estimates the probability that a file belongs to the high-smell-burden group, defined as the files at or above the project-specific 80th percentile of detected smells. This project-relative labeling avoids applying the same absolute smell-count threshold to repositories of very different sizes and structures. The deterministic detector identifies specific known flaws, while the predictive model provides a broader risk signal that can prioritize files for investigation, and the two components deliberately serve distinct, complementary roles.

To select its predictive models, the team benchmarked nine standalone candidates, spanning conventional machine learning, deep learning, and automated machine learning, plus one hybrid stacking configuration, on a dataset of 62,207 files mined from 1,000 public Python repositories. The evaluation used a cross-project five-fold GroupKFold protocol in which repository identity serves as the grouping variable, preventing files from the same project from leaking between training and test folds. This design matters because inflated accuracy from project-level leakage is a well-known pitfall in defect-prediction research. For defect prediction, the winning configuration was a hybrid stacking ensemble combining LightGBM with AutoGluon through an isotonic-calibrated logistic-regression meta-learner, achieving a mean F1 score of 0.64 while reducing fold-to-fold performance variability by roughly 38 percent compared with the strongest single model.

For code smell prediction, the story was different. Stacking yielded no consistent benefit, so the researchers retained a threshold-optimized LightGBM model, which attained a mean F1 score of 0.69. The lesson, the authors note, is that the optimal predictive configuration differs between tasks: defect labels and smell labels carry different class distributions, and assuming one configuration fits both would be unjustified. Operating thresholds were tuned per task, with files flagged as defect-prone at a calibrated probability of 0.42 and as smelly at 0.34. The calibrated probabilities are then mapped onto a three-tier decision policy: files below 0.30 defect probability pass automatically, those between 0.30 and 0.70 are routed to human review, and those at or above 0.70 are blocked from progressing without further inspection.

The user-facing side of QualiGuard is a Flask web application designed for local, single-user execution. A practitioner can paste the URL of a public GitHub repository or upload a compressed project archive of up to 100 megabytes, provided the archive retains its .git directory, since the process metrics and SZZ labeling depend on commit history. The analysis runs asynchronously in a background thread while a progress bar reports each milestone, from cloning through metric extraction, smell detection, and model inference. The resulting dashboard presents project-level health indicators such as defect density per thousand lines of code, refactor ratio, and recent activity, visualizes detected DevOps practices like GitHub Actions workflows, Dockerfiles, and Jenkins pipelines, and lists every analyzed file with its complexity, maintainability index, calibrated probabilities, and color-coded quality-gate tier.

Security received explicit attention, because the tool parses untrusted third-party code. Uploaded archives are validated against their central directory before extraction, with limits on entry count, total uncompressed size, and compression ratio to mitigate decompression bombs, and absolute paths or path-traversal components are rejected. Repository source code is parsed as text only and is never imported or executed. Before any Git command runs, a sanitization step removes repository-local hooks and rewrites the repository configuration using an allowlist, eliminating configuration options that could cause Git to execute external programs such as hooks, diff filters, or credential helpers. GitHub tokens, which are optional and used only to raise API rate limits, are verified before storage, masked in the interface, and excluded from logs. The software is accompanied by 266 automated tests covering metric extraction, labeling, prediction workflows, and web-tier security.

The authors are candid about the limits of the current evaluation. The practical assessment covers predictive performance and execution on a single workstation, and continuous deployment in production environments has not yet been tested. CI/CD integration, execution time across repositories of varying sizes, false-alarm burden, developer acceptance, and the impact of automated gate decisions on real merge outcomes all remain to be empirically assessed. The quality-gate tiers themselves use fixed probability boundaries, making QualiGuard better characterized as a learned, risk-based gate than a fully adaptive one, and the authors flag project-specific threshold adaptation and temporal recalibration as future work. Platform independence is a design characteristic rather than an experimentally validated result, since development and testing took place on Windows 11.

Even with those caveats, QualiGuard represents a meaningful step toward evidence-driven software quality assessment. It bridges two research threads, defect prediction and code smell detection, that are usually studied in isolation but speak to the same underlying notion of file-level risk, and it wraps both in a reproducible pipeline that researchers can reuse to build language-specific metric datasets. By replacing arbitrary fixed thresholds with calibrated, learned estimates of risk, and by making every verdict explainable down to the individual detected smell, the tool offers a glimpse of what continuous quality control in DevOps pipelines could look like when the gate itself has been taught, rather than merely configured. The code, dataset, and trained models are publicly available, inviting both practitioners and researchers to put the approach to the test.

Subject of Research: AI-driven risk-based quality gates for software defect prediction and code smell detection in DevOps pipelines

Article Title: QualiGuard: An AI-driven risk-based quality gate for defect prediction and code smell detection in DevOps pipelines

Article References: Kocakaplan, B. M., Borandağ, E., Özçevik, Y., Yücalar, F., & Altay, O. (2026). QualiGuard: An AI-driven risk-based quality gate for defect prediction and code smell detection in DevOps pipelines. SoftwareX, 36, Article 103090. https://doi.org/10.1016/j.softx.2026.103090

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103090

Keywords: QualiGuard, DevOps, defect prediction, code smells, machine learning, quality gates, SZZ algorithm, LightGBM, AutoGluon, software quality, repository mining, cross-project validation