Lineage-Aware Validation Reduces Machine Learning Performance for Database-Observed IncC Plasmids

A machine-learning model developed to classify IncC plasmids belonging to highly similar cross-genus pairs observed in public databases experiences substantial declines in predictive performance when evaluated with lineage-aware validation, according to research published in Frontiers in Microbiology. IncC plasmids contribute to the dissemination of antimicrobial resistance across bacterial hosts. The researchers developed an IncC-specific model to classify plasmids belonging to highly similar cross-genus pairs observed in public databases and evaluated its performance when related lineages were kept together during validation.

The research team assembled two distinct datasets to develop and assess their classification system. The primary development cohort comprised 988 IncC plasmids curated from the PLSDB plasmid database, which included 266 plasmids designated as label-positive for database-observed cross-genus pairing. To evaluate external generalization, the investigators compiled a separately sourced cohort of 250 IncC plasmids from the National Center for Biotechnology Information RefSeq repository. The authors explicitly defined their target label operationally, noting that it reflects the recorded occurrence of homologous plasmid sequences across different bacterial genera in public archives rather than an experimentally measured biological transfer event.

In developing the classification pipeline, the investigators evaluated five standalone machine-learning algorithms before selecting a LightGBM model that utilizes 100 foldwise-selected nucleotide 4-mers, designated LightGBM-100. To translate continuous model predictions into binary classification decisions, the researchers established a fixed decision threshold of 0.306. This cutoff was derived through nested random cross-validation using Youden-index optimization, providing a standardized baseline to compare conventional validation results directly against lineage-aware validation frameworks.

When evaluated using conventional random five-fold cross-validation, the LightGBM-100 model appeared to achieve high classification accuracy, generating a pooled out-of-fold area under the receiver operating characteristic curve of 0.960 and an average precision of 0.928. These performance metrics diminished substantially, however, when the investigators implemented lineage-aware validation by grouping related plasmid lineages together. Under primary Mash-component blocking, the pooled out-of-fold area under the receiver operating characteristic curve fell to 0.742, while the average precision dropped to 0.498. Applying a more permissive blocking threshold led to further performance reductions, yielding an area under the curve of 0.712 and an average precision of 0.412.

The lineage-aware validation also revealed a severe drop in the model’s ability to identify true positive cross-genus plasmids when evaluated at the predetermined decision threshold. Applying the random-cross-validation-derived threshold of 0.306 to the blocked datasets caused the model sensitivity to fall to 0.259 under primary Mash-component blocking. Under the more permissive lineage-blocking condition, sensitivity declined even further to 0.162. These findings demonstrated that standard random partitioning creates an overly optimistic impression of model sensitivity by allowing closely related sequence variants to populate both training and validation splits simultaneously.

The external evaluation on the RefSeq cohort produced initial metrics that seemed to mirror the optimistic results of random cross-validation, with LightGBM-100 achieving an area under the receiver operating characteristic curve of 0.937 and an average precision of 0.894 across the 250 plasmids. A detailed examination of genomic distances revealed an important caveat: 81 of the 84 label-positive plasmids in the RefSeq cohort were situated within a close Mash distance of less than 0.00124 relative to plasmids in the PLSDB development cohort. Furthermore, the operational labels assigned within RefSeq relied partly on cross-genus pairing evidence derived across both repositories, indicating that the strong RefSeq performance was largely driven by representation of nearly identical sequence lineages rather than independent predictive generalization.

In an effort to improve classification robustness across isolated lineages, the researchers tested the integration of biological features alongside nucleotide composition. These biological annotations, however, did not yield reproducible performance improvements over the 100 4-mer baseline once lineage blocking was applied. The investigators observed similar predictive limitations when extending their evaluation to plasmid pairs that crossed broader taxonomic boundaries, including family-crossing and order-crossing outcomes, confirming that feature engineering alone was insufficient to bridge the gap created by sequence divergence across lineages.

The study establishes an IncC-specific, lineage-aware benchmark for predicting database-observed cross-genus occurrence and demonstrates how lineage relatedness and database representation shape predictive performance. The researchers emphasized that LightGBM-100 classifies operational database labels reliably only within well-represented sequence space, and cautioned that the algorithm does not estimate bacterial conjugation, absolute host range, or empirical transfer probability.

Subject of Research: Machine learning evaluation of IncC plasmid cross-genus occurrence

Article Title: Lineage-aware evaluation of machine learning for observed cross-genus occurrence of IncC plasmids in public databases

Article References: Wang, Y., Zhou, X., Shen, Y., Jia, Y., Liu, Z., Fang, L., Chen, R., Sun, J., Yi, N., & Guo, X. (2026). Lineage-aware evaluation of machine learning for observed cross-genus occurrence of IncC plasmids in public databases. Frontiers in Microbiology, 17, Article 1991208. https://doi.org/10.3389/fmicb.2026.1991208

Image Credits: AI Generated

DOI: 10.3389/fmicb.2026.1991208

Keywords: IncC plasmids, antimicrobial resistance, machine learning, LightGBM, lineage-aware validation, cross-genus occurrence, PLSDB, RefSeq, 4-mer features, Mash distance, AUROC, average precision, Youden index, plasmid classification, sequence representation, five-fold cross-validation