CodeBERT edges out BERT rivals in controlled defect-detection benchmark, but the gap is not statistically significant

Catching software bugs before deployment has long been a central goal of software engineering, and artificial intelligence models are increasingly being drafted for that job. A new open-access study in the International Journal of Data Science and Analytics offers a controlled head-to-head comparison of four of the most widely used pre-trained transformer models on a single defect-detection benchmark. Researchers Nasraldeen Alnor Adam Khleel and Károly Nehéz fine-tuned BERT Base, DistilBERT, RoBERTa, and CodeBERT on the same dataset, under the same training conditions, and with the same statistical scrutiny. Their conclusion is refreshingly sobering: while CodeBERT posted the best average scores, none of the differences between the models survived statistical testing.

The task at hand is known as software defect prediction, or SDP. The idea is to flag defective functions in source code before the software ever ships, reducing maintenance costs and improving reliability. Traditional approaches relied on classical machine learning methods such as decision trees, naive Bayes, random forests, and support vector machines, fed with hand-crafted features derived from code metrics. Those methods work, but the authors note that they may struggle to capture the deeper semantic and contextual information embedded in the code itself. Transformer models changed that picture by learning rich contextual representations of text and, when applied to source code, of syntax and semantics alike.

The problem, the authors argue, is that previous studies comparing these models are notoriously hard to line up. Different papers use different datasets, preprocessing pipelines, data partitions, hyperparameters, and evaluation protocols, making it difficult to tell whether one model genuinely outperforms another or simply benefited from a friendlier experimental setup. Worse, many studies report results from a single training run, leaving open the possibility that the outcome was shaped by a lucky random initialization rather than by any real architectural advantage. The new study was designed specifically to address both gaps: a common experimental configuration for all models, and repeated runs to measure stability.

The empirical arena was the Devign dataset from CodeXGLUE, a widely used benchmark containing function-level C source-code samples labeled as defective or non-defective. The researchers retained the predefined training, validation, and test partitions and applied a character-level filter that excluded functions of 512 or more characters before tokenization. That filtering step matters enormously: it shrank the dataset from 27,318 original samples to 8,500 retained ones—6,794 for training, 866 for validation, and 840 for testing. The retained data contained a moderate class imbalance, with 57.6 percent non-defective and 42.4 percent defective samples. The authors are careful to note that their results should not be read as equivalent to official CodeXGLUE leaderboard numbers, precisely because of this preprocessing.

Each of the four models was fine-tuned for binary classification with an identical recipe: five training epochs, an effective batch size of 16 achieved through gradient accumulation, a maximum sequence length of 512 tokens, the Adafactor optimizer, and mixed-precision training. Crucially, every model was trained and evaluated five times using five different random seeds—42, 52, 62, 72, and 82—yielding twenty training runs in total. Performance was measured with Accuracy, Precision, Recall, and F1-score, reported as mean plus or minus standard deviation, and pairwise differences were tested with the Wilcoxon signed-rank test followed by Holm correction for multiple comparisons.

The headline numbers favor CodeBERT, a model pre-trained jointly on natural language and source code. It achieved the highest mean Accuracy at 72.69 plus or minus 0.88 percent, the highest Precision at 74.33 plus or minus 1.75 percent, and the highest F1-score at 66.60 plus or minus 1.67 percent. BERT Base claimed the highest mean Recall at 61.48 plus or minus 3.07 percent, meaning it caught slightly more true defective functions than its rivals. DistilBERT, the distilled and compact member of the family, delivered an Accuracy of 69.29 plus or minus 0.87 percent and an F1-score of 64.33 plus or minus 1.48 percent—respectable figures that came at a fraction of the computational price. RoBERTa, meanwhile, lagged behind with an Accuracy of 63.67 plus or minus 6.10 percent and a wildly unstable F1-score of 37.92 plus or minus 26.36 percent.

That last number deserves a moment of reflection, because it illustrates why the study’s repeated-run design matters so much. A single unlucky run of RoBERTa could have made it look either brilliant or useless; the five-seed average reveals a model whose behavior swings dramatically depending on initialization. Yet even CodeBERT’s apparent dominance dissolved under statistical scrutiny. After Holm correction, none of the pairwise differences between the four models remained statistically significant at the 0.05 level. With only five paired observations per model, the analysis has limited statistical power, so the differences should be read as descriptive tendencies rather than proof of superiority.

The error analysis adds a layer of nuance that aggregate metrics alone would hide. CodeBERT produced the fewest false positives—a mean false-positive rate of 17.22 percent, compared with roughly 25 percent for BERT Base and DistilBERT—meaning it raised fewer false alarms about clean code. But its false-negative rate of 39.58 percent was slightly higher, so the gain in precision came with a modest cost in defect detection. RoBERTa showed the most troubling profile: the lowest false-positive rate at 9.07 percent, but a false-negative rate of 69.50 percent, with enormous variability across seeds. In defect prediction, false negatives are the dangerous errors—defective code that slips through undetected—so RoBERTa’s apparent selectivity actually reflects a tendency to miss bugs.

The computational trade-offs are equally striking. DistilBERT required an average of only about 17.18 minutes of training time per run, compared with roughly 33.8 to 34 minutes for each of the three larger models—a reduction of approximately 49 percent relative to CodeBERT. For that halved cost, DistilBERT surrendered only about 2.3 points of F1-score. For teams weighing whether transformer-based defect prediction is practical in resource-constrained settings, that trade-off may be the study’s most actionable finding, though the authors caution that training-time comparisons are only valid within their own controlled environment, not against published timings from other hardware setups.

The researchers also peeked inside the models with an attention-based analysis of representative correct and incorrect predictions from a single seed. One defective test example was misclassified as clean by all four architectures, with defective-class probabilities ranging from 0.0057 for CodeBERT to 0.2200 for BERT Base—confident, unanimous, and wrong. The models concentrated attention on different tokens of the same function, yet arrived at the same mistaken conclusion, suggesting that some defective code patterns remain genuinely hard for current transformers. The authors interpret attention weights only as exploratory indicators, noting that differing tokenization schemes—WordPiece for the BERT family, byte-level BPE for RoBERTa and CodeBERT—prevent direct token-by-token comparisons. Their broader message is methodological: benchmark claims in this fast-moving field need repeated runs, statistical tests, class-wise evaluation, and honest error analysis before any model can credibly be crowned the winner.

Subject of Research: Empirical benchmarking of fine-tuned pre-trained transformer models for software defect prediction

Article Title: A structured overview and empirical evaluation of fine-tuning pre-trained transformer models for software defect prediction

Article References: Khleel, N. A. A., & Nehéz, K. (2026). A structured overview and empirical evaluation of fine-tuning pre-trained transformer models for software defect prediction. International Journal of Data Science and Analytics, 22(1), Article 351. https://doi.org/10.1007/s41060-026-01341-0

Image Credits: AI Generated

DOI: 10.1007/s41060-026-01341-0

Keywords: software defect prediction, transformer models, CodeBERT, BERT, DistilBERT, RoBERTa, Devign dataset, CodeXGLUE, fine-tuning, statistical significance, benchmarking, source code analysis