New software brings rigorous nested validation to genomic prediction

Genomic prediction has quietly become one of the most consequential technologies in modern agriculture. By training statistical or machine learning models on genome-wide molecular markers paired with phenotypic records from a training population, breeders can forecast how candidate lines will perform before they are ever planted in a field. The approach, known in practice as genomic selection, has accelerated the development of improved crop varieties by allowing selection decisions to be made on predicted genomic estimated breeding values rather than on years of laborious field testing. Yet a persistent bottleneck has remained: rigorously evaluating which prediction model works best for a given trait and population, and tuning that model fairly, has demanded programming skills that many plant breeders and quantitative geneticists simply do not have. A newly released open-source tool called NV4GP, described in the journal SoftwareX, aims to close that gap.

NV4GP, short for Nested Validation for Genomic Prediction, was developed by Paolo Vitale of the International Maize and Wheat Improvement Centre, known as CIMMYT. The software is written entirely in Python and distributed under the permissive MIT licence through GitHub, with a permanent code archive on Zenodo. Its designers describe it as a user-friendly, efficient and reproducible framework for nested cross-validation-based model evaluation in genomic prediction, and its interface guides users through three sequential modules without requiring any command-line interaction or scripting. The motivation, the author explains, is that graduate students and early-career researchers entering plant breeding, biometrics or quantitative genetics often struggle to translate theoretical knowledge into practice because of limited coding experience, and accessible software can play a critical role in easing that transition.

The statistical heart of the software is its handling of hyperparameter tuning, a step that strongly influences model performance and generalization but is frequently mishandled in applied studies. Many prediction models, particularly non-parametric and artificial intelligence-based approaches, include one or more hyperparameters that are not estimated during model training. Nested cross-validation addresses this by defining an outer loop of conventional k-fold cross-validation to assess model performance, and then, within each outer-loop training set, running an inner-loop cross-validation that evaluates every combination of hyperparameters in a user-defined search grid. The combination with the lowest prediction error or the highest predictability is selected and used to fit the corresponding outer-loop model, ensuring unbiased performance estimates by preventing information leakage between model tuning and evaluation.

In an independent validation scenario, the software instead performs an internal k-fold cross-validation on the training population to select the best hyperparameters, then refits the model on the full training set and tests it on a genuinely separate dataset. Predictability is reported as Pearson’s correlation between predicted and observed values in the independent set. This strategy mirrors the real-world breeding context, in which breeders must use a training population to predict the performance of an unknown, genetically related breeding population. Independent validation has historically been under-applied in the genomic prediction literature, despite offering a fairer basis for model comparison than naive cross-validation schemes that inadvertently share information between training and testing data.

NV4GP implements fifteen prediction models spanning parametric, non-parametric and ensemble categories. The parametric set includes ridge regression, Bayesian ridge and LASSO, while the non-parametric range covers kernel ridge regression, support vector regression, elastic net, stochastic gradient descent, partial least squares, nearest neighbours, Gaussian process regression, decision trees, random forests, gradient boosting and multi-layer perceptrons, capped by a voting regressor ensemble. Each model carries its own hyperparameter grid; the multi-layer perceptron alone exposes twenty-one tunable settings, and gradient boosting and stochastic gradient descent each expose seventeen. All implementations draw on custom functions and the widely used Scikit-learn library, and a benchmark GBLUP model can be run in R through the BGLR package for comparison purposes.

Before any modelling begins, a marker filtering module lets users upload genotype data in numeric matrix or HapMap format, following IUPAC nucleotide nomenclature where applicable, and filter markers and genotypes by missing data, minor allele frequency and heterozygosity. Optional imputation by mean or major allele is available, and a detailed log reports exactly how many markers were removed by each filter and the per-marker quality statistics. In the validation module, users upload fully imputed marker matrices and phenotype files, which may be unbalanced because the software internally matches line identifiers and restricts analysis to common genotypes. Users specify the target trait, an optional logarithmic transformation, the hyperparameter grid, the number of cycles and folds for the outer and inner loops, a random seed for reproducibility, and the selection criterion: mean absolute error, mean squared error or predictability.

To demonstrate the software, the author ran five cycles of five-fold nested cross-validation, with four inner folds, on the wheat599 dataset of 599 individuals, 1,279 markers and four grain yield phenotypes recorded across environments. Predictability varied substantially across models: for the first target variable, support vector regression reached roughly 0.6 while decision trees scored 0.0; for another, the classical GBLUP benchmark led with values near 0.5, while Gaussian process regression lagged furthest behind; for a fourth, kernel ridge regression exceeded 0.5. Running times were equally revealing, ranging from about fifteen seconds for ridge regression to nearly twelve hours for the multi-layer perceptron on a conventional 64-bit laptop with 32 gigabytes of RAM. In an independent validation using published wheat data from two consecutive years and two simulated irrigation environments, predictability rankings shifted markedly between environments, ranging from slightly negative values for stochastic gradient descent to 0.32 for support vector regression in one environment and 0.16 for the multi-layer perceptron in the other.

The software was benchmarked against the GBLUP implementation in the BGLR R package using identical datasets and validation strategies, and produced comparable predictability estimates, supporting its correctness. Reproducibility is enforced through user-defined random seeds that fix the random state across all model components and fold partitioning, and the software was independently tested on Linux, Windows and MacOS, yielding identical results. Input validation checks flag common errors, such as misformatted marker files or incompatible cross-validation partitioning, before execution, returning informative error messages rather than silent failures. A result summary module processes output files and generates bar plots with standard error bars for test-set predictability, allowing direct visual comparison across models without additional data harmonization.

Compared with existing tools such as BGLR, rrBLUP, sommer, MegaLMM, ShinyGS, CHiDO and GS4PB, NV4GP is distinctive in combining a graphical interface with true nested cross-validation and independent validation, systematic grid-based hyperparameter tuning, and a broad roster of machine learning and deep learning models. The limitations are candidly acknowledged: exhaustive grid searches become computationally demanding as models, hyperparameter combinations or folds multiply, memory demands grow with large marker matrices, and the current version supports only a single trait or environment per run, deliberately excluding multi-environment and multi-trait modelling to keep the software usable on standard laptop hardware. The recommended ceiling for a conventional laptop is a search grid below fifty hyperparameter combinations on datasets of hundreds of lines with a few thousand markers.

The implications extend beyond convenience. By collapsing what previously required substantial bespoke programming into a uniform graphical workflow, and by standardizing reported metrics including training and test predictability, mean squared and absolute errors, percentage errors and the gap between training and test performance, NV4GP makes cross-study and cross-model comparisons of machine learning methods in genomic prediction more reproducible and more accessible. For breeders, the practical consequence is a shift in how model-selection decisions are made: rather than defaulting to a single familiar model because alternatives are impractical to implement, users can rapidly screen multiple model families and choose the empirically best performer for their specific trait and population, embedding statistically rigorous, leakage-free validation into routine genomic selection pipelines.

Subject of Research: A Python software framework for nested cross-validation and independent validation in genomic prediction for breeding

Article Title: NV4GP: Nested validation for genomic prediction

Article References: Vitale, P. (2026). NV4GP: Nested validation for genomic prediction. SoftwareX, 36, Article 103021. https://doi.org/10.1016/j.softx.2026.103021

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103021

Keywords: genomic prediction, genomic selection, nested cross-validation, hyperparameter tuning, machine learning, plant breeding, quantitative genetics, NV4GP, CIMMYT, wheat, Python software, model benchmarking