New Open-Source Tool Brings Order to the Chaos of Image Quality Metrics

Every time you stream a video, compress a photo, or marvel at an AI-generated image, an invisible arbitration takes place: algorithms decide whether the picture you see is sharp, artifact-free, and faithful to what it should be. The science behind that judgment is called objective image quality assessment, or IQA, and it underpins research on image compression, restoration, transmission, acquisition, and visual content generation. Yet behind the scenes, researchers who study these algorithms have long wrestled with a surprisingly messy problem. Datasets disagree on how quality scores are stored, metrics disagree on whether higher numbers mean better images, and experimental setups are often stitched together with hand-edited scripts that nobody can fully audit later. A new open-source software package called combHIQA, described in the journal SoftwareX by Piotr Lech and Krzysztof Okarma of West Pomeranian University of Technology, aims to bring engineering discipline to this tangled landscape.

The core of the problem lies in heterogeneity. IQA datasets come in wildly different shapes: some bundle a spreadsheet of Mean Opinion Scores (MOS) with folders of distorted images, others use Differential MOS (DMOS), different numerical ranges, or even reversed conventions in which lower values indicate better perceived quality. Some datasets provide reference images, enabling full-reference (FR) metrics that compare a distorted picture against its pristine original; others support only no-reference (NR), or blind, metrics that must judge quality without any ground truth. In-the-wild datasets collected from real-world photography add authentic distortions, while laboratory datasets emphasize synthetic corruption such as JPEG compression, Gaussian blur, and geometric warping. Before a researcher can compare or combine any objective indicators, all of these conventions must be reconciled, and if those decisions are scattered across notebooks and scripts, the resulting experiment becomes nearly impossible to reproduce or reuse.

CombHIQA tackles this with what its authors call a contract-driven preparation workflow. Instead of letting each experiment define its own assumptions, the software requires the researcher to describe a source dataset through an explicit preparation configuration. The tool then validates the available definitions, transforms all subjective scores into a common higher-is-better direction, builds a validated table linking images to their scores, computes the configured quality metrics, and exports three connected JSON files: a uniform dataset, a metric registry, and an experiment configuration. These files are validated sequentially and form a single portable package. Crucially, the metric registry is the single runtime source of truth for each metric’s type, whether full-reference or no-reference, and its direction, whether higher or lower values indicate better quality. Per-image records contain only metric names and numerical values, so the data can never quietly redefine its own semantics.

Architecturally, the software is split into two subsystems connected by those explicit JSON contracts. The preparation subsystem, which accounts for roughly seventy percent of the implemented functionality, recognizes source-dataset structure, validates MOS values and image paths, builds the metric registry, invokes the metric runner, and exports the prepared package. The experimental subsystem loads the package, normalizes metric values, selects a subset of metrics, optionally optimizes hybrid parameters, evaluates the resulting predictions, and serializes reports. This separation has practical benefits beyond tidiness: computationally demanding metrics can be calculated on a high-performance computing cluster while faster metrics run on an ordinary laptop, and the experimental core remains independent of the quirks of any individual dataset layout. The package is written in Python, built on the pyIQA toolbox and PyTorch, runs on CPU or CUDA hardware under Linux, and is released under the MIT license on GitHub.

The mathematical machinery is deliberately transparent. Raw objective metrics are normalized by min-max scaling, and metrics that follow a lower-is-better convention are inverted after normalization so that larger values consistently indicate better predicted quality. Constant or near-constant columns are mapped to 0.5 to avoid numerical failures, and any clipping of out-of-range data is recorded rather than hidden. For metric selection, the software uses what the authors call a Most Valuable Predictor (MVP) selector, based on a deliberately simple criterion: the Spearman rank-order correlation coefficient (SROCC) between the subjective scores and the mean of the normalized metrics in a candidate subset. A greedy forward algorithm evaluates each remaining metric as the next addition, accepts the best candidate, and requires a minimum improvement of 0.001 in the correlation score at each step. Every accepted addition, candidate score, and resulting subset is stored in a selection history that researchers can inspect before any fancier optimization begins.

Only after this interpretable baseline does an experimental optimizer enter the picture. It constructs a nonlinear hybrid quality score combining additive and multiplicative components, with parameters fitted by differential evolution to maximize the Spearman correlation with human judgments. The authors are careful to label this module experimental: its flexibility may fit the idiosyncrasies of the analyzed data, and using the software does not automatically guarantee that findings will generalize beyond it. The core contribution, they emphasize, lies in the preparation methodology and contract architecture rather than in any claim of universal metric superiority. This candor is refreshing in a field where benchmark tables are often presented with more confidence than the underlying protocols can support.

To validate the complete workflow, the team prepared four demonstration datasets covering both no-reference and full-reference scenarios, with quality scores assigned by a single evaluator and treated as pseudo-MOS values purely for testing the software rather than benchmarking algorithms. The datasets spanned diverse degradation scenarios, including authentic quality variations, artificial noise, JPEG artifacts, Gaussian blur, and geometric distortions. On these demonstration sets, combined metrics showed striking gains over the best single metric: the Spearman correlation rose from 0.3022 to 0.7384 on the AI dataset, from 0.5673 to 0.8758 on the Street dataset, and from 0.6115 to 0.7084 on the Mountain dataset, with the Outdoor dataset improving from 0.8423 to 0.8858. The authors stress that these numbers verify the software workflow and should not be read as a benchmark comparison of metric performance.

Limited end-to-end validation on two widely used public datasets, KonIQ-10k and TID2013, told a more sobering and arguably more realistic story. Using fast no-reference metrics such as NIQE, PIQE, and BRISQUE, the optimized combinations improved the correlation modestly, from 0.5273 to 0.5425 on the full-resolution KonIQ-10k variant, from 0.3688 to 0.3906 on the small-resolution variant, and from 0.3672 to 0.4167 on TID2013. The elementary metric results for TID2013 matched the benchmark values reported in the pyIQA documentation, a reassuring sign of computational correctness. Because metric selection and optimization were performed within each dataset, the authors explicitly caution that these results demonstrate workflow validation, not cross-dataset generalization or benchmark superiority. For KonIQ-10k the selected combination comprised NIQE and BRISQUE, while for TID2013 it comprised BRISQUE and PIQE.

The practical impact of the tool lies in eliminating hidden, dataset-specific manual preparation steps. By requiring an explicit mapping and producing a portable package, combHIQA makes it easier to check exactly which images, subjective scores, metric definitions, directions, and configurations are essential to an experiment. Once the package is generated, selection and analysis become independent of the original dataset’s structure, and the same contracts can be inspected manually, versioned in Git, transferred between machines, and consumed by future modules. The traceability is granular: a researcher can follow a single image from its source MOS row through the validated path table, the intermediate metric table, and the final JSON record, and can trace a metric from its preparation-time definition through direction correction and selection history to any optimized hybrid parameters. The authors also see pedagogical value, since students can prepare a dataset through a small JSON configuration and observe firsthand the distinction between score direction and metric direction.

The developers are equally candid about limitations. Because combHIQA relies on third-party libraries, results depend on those implementations and software environments. The flexibility of the nonlinear hybrid model raises the risk of overfitting when many elementary metrics are combined on a small dataset, so the authors recommend fitting parameters on designated development data and evaluating with cross-validation or an independent test set. The current workflow does not automatically implement train/test partitioning or cross-dataset validation; designing and interpreting those protocols remains the researcher’s responsibility. The frozen version also lacks a public plug-in API, so swapping in a new metric backend or optimizer still requires code changes, though the modular interfaces limit the scope of such edits. Even with these caveats, combHIQA represents a quiet but meaningful shift in how image quality research can be conducted: away from fragile, undocumented pipelines and toward experiments whose every assumption is written down, validated, and ready for the next researcher to pick up.

Subject of Research: Contract-driven software for preparing and combining image quality assessment metrics

Article Title: combHIQA: Contract-driven preparation and experimental combination of image quality assessment metrics

Article References: Lech, P., & Okarma, K. (2026). combHIQA: Contract-driven preparation and experimental combination of image quality assessment metrics. SoftwareX, 36, Article 103117. https://doi.org/10.1016/j.softx.2026.103117

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103117

Keywords: image quality assessment, combHIQA, reproducibility, open-source software, no-reference metrics, full-reference metrics, Spearman correlation, pyIQA, PyTorch, MOS, metric combination, JSON contracts