Twenty-Five Years of One-Parameter-at-a-Time Testing: New Survey Maps the Field’s Hidden Trade-Offs

Every modern software system is, at its core, a combinatorial puzzle. A single application may accept dozens of configuration parameters, each with multiple possible values, and the number of ways those values can combine grows exponentially. Testing every combination is impossible even for modest systems, yet many of the most damaging software failures arise not from a single bad input but from an unexpected interaction among several parameters. A new systematic review published in Frontiers in Computer Science takes stock of one of the most influential families of algorithms ever developed to solve this problem, and its findings reveal both remarkable progress and a persistent, uncomfortable gap between what the literature claims and what can actually be verified.

The review, led by Muhammad Sabo Yahaya of Universiti Teknologi PETRONAS in Malaysia and Federal Polytechnic Daura in Nigeria, together with colleagues from Malaysia and Nigeria, focuses on the One-Parameter-at-a-Time (OPAT) paradigm for Combinatorial Interaction Testing (CIT). The approach traces its origins to the In-Parameter-Order (IPO) algorithm introduced in 2002 and its generalized successor, IPOG, which extended the technique to arbitrary interaction strengths. Rather than generating complete test cases one at a time, OPAT builds a test suite incrementally by introducing one parameter per stage. Two complementary mechanisms drive this construction: horizontal growth, which extends existing test cases by assigning values to a newly introduced parameter in a way that maximizes coverage of previously unseen interactions, and vertical growth, which adds new test cases only when necessary to cover any interactions that horizontal expansion could not reach. This incremental structure allows OPAT to reuse existing configurations while maintaining strong scalability as systems grow more complex.

Despite the paradigm’s importance, no previous review had ever synthesized the OPAT literature in isolation. Prior surveys examined combinatorial testing from broader perspectives, treating metaheuristics as black-box optimizers or focusing narrowly on pairwise testing. The new study, published on 15 October 2026, addresses this gap through a systematic search of 1,300 publications across four major digital libraries, ultimately identifying 40 primary studies published between 2000 and 2026. The screening process was rigorous: two independent reviewers achieved a Cohen’s Kappa agreement of 0.83 during title and abstract screening and 0.89 during full-text eligibility assessment, with a third senior reviewer arbitrating any persistent disagreements. Quality assessment showed that 82.5 percent of the included studies scored at least 3.0 out of 4.0 on a four-criterion checklist evaluating algorithmic clarity, experimental rigor, quantitative reporting, and baseline comparison.

The review’s central contribution is a unified taxonomy that classifies OPAT research into five categories: deterministic refinements, scaling and distribution strategies, advanced feature-enhanced strategies, metaheuristic and hybrid strategies, and specialized OPAT adaptations. Deterministic refinements improve the original IPO and IPOG algorithms through better heuristics, dynamic programming, and structural modifications. IPOG-D, for instance, introduces a recursive doubling mechanism that dramatically improves scalability: in one reported experiment, it required only 3 percent of standard IPOG’s execution time for 5-way testing of 20 four-valued parameters, though it generated 51 percent more test cases. MIPOG applies enhanced greedy optimization during both growth phases, consistently producing smaller suites than IPOG, Jenny, and ITCH, particularly at interaction strengths of 4 or higher, but at increased computational cost. VIPO simplifies horizontal growth decisions to accelerate generation while maintaining competitive suite sizes across 11 benchmark configurations.

Scaling and distribution strategies address a different bottleneck: the memory and time constraints that emerge when combinatorial problems reach industrial scale. G_MIPOG partitions both memory and computation across a grid of computers, demonstrating logarithmic speedups as coverage strength increased in tests involving 6-way interactions of 7 to 10 five-valued parameters. MC-MIPOG exploits shared-memory multicore architectures, achieving near-linear speedup as parameter counts increased from 5 to 16 at interaction strength 4 on a 2.4 GHz Core 2 Quad processor. TS_OP distributes the workload using MapReduce and Tuple Space across a network of five workstations, significantly reducing per-machine memory utilization, though at the cost of non-deterministic outputs due to network overhead.

Feature-enhanced strategies tackle the practical realities that pure algorithms often ignore. Real systems impose constraints: certain parameter combinations are invalid, infeasible, or meaningless. The ACTS tool, developed by researchers at the National Institute of Standards and Technology, embeds IPOG variants within a comprehensive framework supporting constraint specification, mixed-strength testing, and multiple user interfaces, and has been adopted by over 1,200 organizations. IPOG-C minimizes redundant constraint checks by selectively evaluating only relevant constraints, while IPO-SAT transforms constraint handling into a satisfiability problem amenable to SAT solvers. Seeding mechanisms, implemented in strategies such as SCIPOG and SCIPOG-VS, allow domain knowledge to be injected directly into the generation process, ensuring that critical or high-risk configurations are always included in the final suite.

Metaheuristic and hybrid strategies represent the most recent evolutionary branch, embedding OPAT’s structured construction within stochastic optimization frameworks. The Harmony Search Algorithm dominates this category, appearing in 77.8 percent of the analyzed stochastic OPAT studies, including OPAT-HS, HCATS, OPATHS, PWiseHA, PHOSC, SCHOP, eIPOG, and eHS. The enhanced Harmony Search variant eHS reported a 90.91 percent success rate in reducing test suite sizes across 33 configurations compared to IPOG, but achieved zero percent success in execution time superiority, illustrating the inherent tension between search quality and computational overhead. GAMIPOG combines a genetic algorithm with deterministic OPAT components, achieving scalability up to interaction strength 15. The hybrid HOS strategy applies a genetic algorithm for horizontal growth and Simulated Annealing for vertical growth, achieving optimal sizes for 61 of 71 benchmark configurations.

Specialized adaptations extend OPAT beyond conventional parameter interaction problems. DEO applies the paradigm to sequence covering arrays for event-driven systems, where the order of events matters as much as their combination. PMC automates test data generation directly from source code for C and C++ unit testing. N-IPO leverages Fibonacci-based value selection to reduce suite sizes for large input domains, while ReqOrder and ParaOrder support input-output relations and variable interaction strengths. These adaptations demonstrate the framework’s flexibility but also its fragmentation: each targets a niche, and none has achieved the broad industrial penetration of tools like ACTS.

Perhaps the review’s most sobering finding concerns reproducibility. The authors demonstrate that direct cross-study benchmarking of execution times is mathematically invalid due to highly heterogeneous self-reported hardware environments and the closed-source nature of most academic tools. Numerical results from different studies cannot be combined into a universal performance ranking without controlling for differences in hardware, implementation environments, benchmark configurations, and parameter settings. The limited availability of executable implementations and source code constitutes what the authors describe as a substantive research gap, not merely a methodological limitation of their own review. Without accessible implementations, independent reproduction and controlled head-to-head evaluation remain impossible.

Looking forward, the review identifies several promising directions: unified optimization frameworks that simultaneously balance scalability, optimality, and feature integration; exploration of underutilized metaheuristics such as Particle Swarm Optimization, Tabu Search, and hyper-heuristics; intelligent and adaptive test generation leveraging historical testing data and fault patterns; deterministic parallel algorithms; and large-scale industrial validation on real-world systems rather than synthetic benchmarks. The authors argue that transitioning OPAT research into industrial-grade, open-source software engineering pipelines requires not only more effective optimization mechanisms but also standardized, transparent, and reproducible empirical evaluation practices. For a field that has quietly underpinned software quality assurance for over two decades, the message is clear: the algorithms have matured, but the science of measuring them has not kept pace.

Subject of Research: One-Parameter-at-a-Time strategies for combinatorial interaction testing in software test suite generation

Article Title: Constructing t-way test suites: a systematic survey and qualitative synthesis of the One-Parameter-at-a-Time approach

Article References: Yahaya, M. S., Hashim, A. S., Mamman, H., Abdullahi, S., Umer, U., Adamu Aliyu, D., & Godowoli, I. M. (2026). Constructing t-way test suites: a systematic survey and qualitative synthesis of the One-Parameter-at-a-Time approach. Frontiers in Computer Science, 8, Article 1946723. https://doi.org/10.3389/fcomp.2026.1946723

Image Credits: AI Generated

DOI: 10.3389/fcomp.2026.1946723

Keywords: combinatorial interaction testing, One-Parameter-at-a-Time, software testing, t-way test suite generation, IPOG algorithm, metaheuristic optimization, Harmony Search Algorithm, constraint handling, systematic literature review, reproducibility, parallel computing, software quality assurance