AI Quality Gate Screens Code for Bugs and Smells Before It Ever Merges

Every time a software developer pushes new code to a shared repository, an invisible judgment is made: should this change be merged, scrutinized by a human reviewer, or rejected outright? In modern DevOps practice, that judgment is increasingly automated through quality gates, rule-based checkpoints that decide whether code is fit to proceed down the delivery pipeline. The trouble is that most existing gates rely on fixed metric thresholds, such as a maximum cyclomatic complexity or a minimum test coverage percentage, that treat every project identically and offer no genuine estimate of risk. A new open-source tool called QualiGuard, described in the journal SoftwareX, takes a fundamentally different approach by training machine-learning models to predict which files are likely to harbor defects or design problems, and then translating those calibrated predictions into concrete pipeline decisions.

QualiGuard was developed by Batın Mert Kocakaplan, Emin Borandağ, Yusuf Özçevik, Fatih Yücalar, and Osman Altay, and is released under an MIT license with its trained model artifacts and derived metric dataset available under Creative Commons CC BY 4.0. The tool is built in Python 3.10 and integrates a remarkable range of established components: the Radon library for static complexity metrics, PyDriller for mining Git commit history, scikit-learn, LightGBM, XGBoost, TensorFlow, AutoGluon, and H2O AutoML for predictive modeling, and the GitHub REST API for large-scale repository collection. Rather than delivering disconnected fragments of analysis, the system assembles the entire chain, from raw repository to actionable verdict, into a single deployable package.

The premise underlying the tool is that software quality risk can be modeled as a supervised learning problem, in which measurable properties of source code and its development history are associated with observed quality outcomes. For every source file, QualiGuard extracts static size and complexity metrics, cognitive-complexity measurements derived from the abstract syntax tree, and process metrics from the Git history, including code churn, author entropy, and commit cadence. Defect labels are generated through the Śliwerski–Zimmermann–Zeller (SZZ) algorithm, a classic repository-mining technique that identifies lines deleted by bug-fixing commits and traces them back to the changes that introduced them. This combination of structural and historical evidence allows the assessment to account not only for how a file looks today but also for how it has evolved.

Beyond defect prediction, the tool detects seven classical code smells, including long methods, large classes, long parameter lists, deep nesting, god functions, high complexity, and low maintainability, using deterministic abstract-syntax-tree analysis. A second, predictive model complements this detector: using 28 code metrics, it estimates the probability that a file belongs to the high-smell-burden group, defined as the files at or above the project-specific 80th percentile of detected smells. This project-relative labeling avoids applying the same absolute smell-count threshold to repositories of very different sizes and structures. The deterministic detector identifies specific known flaws, while the predictive model provides a broader risk signal that can prioritize files for investigation, and the two components deliberately serve distinct, complementary roles.

To select its predictive models, the team benchmarked nine standalone candidates, spanning conventional machine learning, deep learning, and automated machine learning, plus one hybrid stacking configuration, on a dataset of 62,207 files mined from 1,000 public Python repositories. The evaluation used a cross-project five-fold GroupKFold protocol in which repository identity serves as the grouping variable, preventing files from the same project from leaking between training and test folds. This design matters because inflated accuracy from project-level leakage is a well-known pitfall in defect-prediction research. For defect prediction, the winning configuration was a hybrid stacking ensemble combining LightGBM with AutoGluon through an isotonic-calibrated logistic-regression meta-learner, achieving a mean F1 score of 0.64 while reducing fold-to-fold performance variability by roughly 38 percent compared with the strongest single model.

For code smell prediction, the story was different. Stacking yielded no consistent benefit, so the researchers retained a threshold-optimized LightGBM model, which attained a mean F1 score of 0.69. The lesson, the authors note, is that the optimal predictive configuration differs between tasks: defect labels and smell labels carry different class distributions, and assuming one configuration fits both would be unjustified. Operating thresholds were tuned per task, with files flagged as defect-prone at a calibrated probability of 0.42 and as smelly at 0.34. The calibrated probabilities are then mapped onto a three-tier decision policy: files below 0.30 defect probability pass automatically, those between 0.30 and 0.70 are routed to human review, and those at or above 0.70 are blocked from progressing without further inspection.

The user-facing side of QualiGuard is a Flask web application designed for local, single-user execution. A practitioner can paste the URL of a public GitHub repository or upload a compressed project archive of up to 100 megabytes, provided the archive retains its .git directory, since the process metrics and SZZ labeling depend on commit history. The analysis runs asynchronously in a background thread while a progress bar reports each milestone, from cloning through metric extraction, smell detection, and model inference. The resulting dashboard presents project-level health indicators such as defect density per thousand lines of code, refactor ratio, and recent activity, visualizes detected DevOps practices like GitHub Actions workflows, Dockerfiles, and Jenkins pipelines, and lists every analyzed file with its complexity, maintainability index, calibrated probabilities, and color-coded quality-gate tier.

Security received explicit attention, because the tool parses untrusted third-party code. Uploaded archives are validated against their central directory before extraction, with limits on entry count, total uncompressed size, and compression ratio to mitigate decompression bombs, and absolute paths or path-traversal components are rejected. Repository source code is parsed as text only and is never imported or executed. Before any Git command runs, a sanitization step removes repository-local hooks and rewrites the repository configuration using an allowlist, eliminating configuration options that could cause Git to execute external programs such as hooks, diff filters, or credential helpers. GitHub tokens, which are optional and used only to raise API rate limits, are verified before storage, masked in the interface, and excluded from logs. The software is accompanied by 266 automated tests covering metric extraction, labeling, prediction workflows, and web-tier security.

The authors are candid about the limits of the current evaluation. The practical assessment covers predictive performance and execution on a single workstation, and continuous deployment in production environments has not yet been tested. CI/CD integration, execution time across repositories of varying sizes, false-alarm burden, developer acceptance, and the impact of automated gate decisions on real merge outcomes all remain to be empirically assessed. The quality-gate tiers themselves use fixed probability boundaries, making QualiGuard better characterized as a learned, risk-based gate than a fully adaptive one, and the authors flag project-specific threshold adaptation and temporal recalibration as future work. Platform independence is a design characteristic rather than an experimentally validated result, since development and testing took place on Windows 11.

Even with those caveats, QualiGuard represents a meaningful step toward evidence-driven software quality assessment. It bridges two research threads, defect prediction and code smell detection, that are usually studied in isolation but speak to the same underlying notion of file-level risk, and it wraps both in a reproducible pipeline that researchers can reuse to build language-specific metric datasets. By replacing arbitrary fixed thresholds with calibrated, learned estimates of risk, and by making every verdict explainable down to the individual detected smell, the tool offers a glimpse of what continuous quality control in DevOps pipelines could look like when the gate itself has been taught, rather than merely configured. The code, dataset, and trained models are publicly available, inviting both practitioners and researchers to put the approach to the test.

Subject of Research: AI-driven risk-based quality gates for software defect prediction and code smell detection in DevOps pipelines

Article Title: QualiGuard: An AI-driven risk-based quality gate for defect prediction and code smell detection in DevOps pipelines

Article References: Kocakaplan, B. M., Borandağ, E., Özçevik, Y., Yücalar, F., & Altay, O. (2026). QualiGuard: An AI-driven risk-based quality gate for defect prediction and code smell detection in DevOps pipelines. SoftwareX, 36, Article 103090. https://doi.org/10.1016/j.softx.2026.103090

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103090

Keywords: QualiGuard, DevOps, defect prediction, code smells, machine learning, quality gates, SZZ algorithm, LightGBM, AutoGluon, software quality, repository mining, cross-project validation