New Benchmark Harness Puts Recommender Systems to a Statistically Honest Test

Recommender systems decide what billions of people see online, yet the field that builds them has a quietly embarrassing problem: many published claims about which algorithm works best do not survive careful scrutiny. A new open-source benchmark harness called towerbench, described in the journal SoftwareX, tackles that problem head-on by wrapping two of the industry’s most important model families—two-tower retrieval networks and autoencoder recommenders—in a frozen, audited evaluation protocol with proper statistical inference built in. The result is a rare thing in machine learning research: a benchmark whose verdicts come with confidence intervals, audit trails and the ability to reproduce every number from a single configuration file.

The software emerged from an eighteen-month industrial study on a Polish primary-market property portal, where researchers compared an attribute autoencoder against a series of two-tower variants. What they found was sobering. On a single frozen evaluation window with fewer than a thousand evaluable users, no model could be separated from any other: the paired 95 percent bootstrap intervals were wider than the effects under test. The resolution came not from better models but from a better protocol. Pooling five non-overlapping rolling-origin windows—each anchored at a different point in time before the last recorded event—turned an indistinguishable comparison into a decisive one. That protocol, transplanted onto public data so anyone can replicate it, is the software contribution now released under an MIT licence.

Three design requirements set towerbench apart from established frameworks such as RecBole, Elliot, Cornac, DaisyRec, RecPack and BARS. First, a frozen protocol with an audit trail: full-catalogue ranking, one shared metric implementation for every model, chronological splits whose constants are fixed in a single file, and a ledger that records every read of the test split. Second, statistics that match the question being asked: non-parametric bootstrap intervals over users, paired per-user differences against a reference model computed on identical users, several seeds per configuration, and rolling-origin windows that can be pooled to recover statistical power. Third, one switch per technique: every industrial two-tower training trick is a Boolean or scalar in a YAML file, so an ablation is a one-line change and every combination is reproducible from its configuration name alone.

The harness ships with loaders for seven public datasets drawn from five sources, spanning MovieLens-1M, three Amazon Reviews 2023 categories, Gowalla check-ins, Inside Airbnb reviews and Steam. Together they range from dense movie ratings with nearly 166 events per user to the brutally sparse Airbnb data, where users average just 3.7 events across a catalogue of roughly 105,000 listings. Item text is embedded once per dataset with a frozen sentence-transformer and cached, so every model consumes identical features. The model zoo includes a popularity floor, the closed-form linear autoencoder EASE, the variational Mult-VAE, the sequential SASRec, the graph-convolutional LightGCN, and a two-tower model whose item tower combines identifier embeddings with optional text, geographic, price and category features, while the user tower pools the last L history items with multi-head learnable-query attention.

The first large-scale sweep, totalling 1,974 experiments across eleven model configurations, five seeds and six evaluation windows per dataset, produced two regularities that held on every dataset. The closed-form linear autoencoder proved a stubbornly hard baseline: EASE beat Mult-VAE everywhere and crushed the identifier-only two-tower model on the five sparsest datasets, sometimes by an order of magnitude. But item side information changed the verdict entirely. The same tower fed with frozen text embeddings and structured features beat EASE on six of the seven datasets, with relative gains from 12 percent on Amazon Software to a striking 209 percent on MovieLens. The lone exception was Inside Airbnb, the sparsest catalogue and the closest analogue of real-estate leads, where EASE kept a small but statistically resolved lead over every neural model.

Reading the two-tower techniques one switch at a time revealed which industrial tricks actually matter on sparse feedback. Removing the logQ correction—the adjustment that prevents in-batch sampled softmax from penalising popular items—was catastrophic, dropping the model to near-random ranking on the sparse datasets. Hard-negative mining via MixGCF and mixed negative sampling, both validated at web scale, moved results by less than one seed standard deviation or actively hurt. The only intervention with a large, consistent effect was side information, which raised the identifier-only model’s NDCG@10 by factors of 1.3 to 8.6 on sparse data. The entire ablation grid of 245 runs cost just 20 hours of summed compute, reproducible from configuration files alone.

Perhaps the most provocative finding concerns the evaluation protocol itself. When the same model comparison was run under a frozen temporal protocol instead of the conventional leave-last-out split, the sign of the difference between the feature two-tower and EASE flipped on two of seven datasets. On MovieLens and Gowalla, EASE—which exploits a user’s complete history—beat a tower that sees only the last fifty events in every one of ten windows. A benchmark reporting only one protocol would have published opposite conclusions depending on which it chose. The rolling-origin windows also demonstrated the power of pooling: comparisons unresolvable in three of five individual windows became decisive when the per-user difference vectors were concatenated, yielding paired intervals on up to half a million users, while comparisons no window could resolve stayed honestly inconclusive.

The authors stress-tested their own statistics. Cluster bootstraps over users, block bootstraps over windows and DerSimonian–Laird random-effects combinations widened some intervals by up to a factor of 7.5 where windows disagreed, yet preserved nearly every verdict. Sensitivity grids over EASE’s regularisation strength and the two-tower temperature and embedding width showed that the headline orderings largely survive retuning, though the feature tower’s margin on Gowalla held only once the tower itself was widened. A training-noise switch, which replaces a fraction of training events with random items, showed the feature tower degrading least on MovieLens because its side features are untouched by interaction noise. Every result file persists per-user metric vectors, split constants and fit diagnostics, so any comparison can be recomputed offline without retraining a single model.

The implications reach beyond academia. Practitioners at marketplaces, where sparse, one-shot, high-turnover feedback is the norm, can now test a technique under this discipline before committing engineering effort to it. The harness’s first public run already reproduced the industrial study’s central negative result—that an identifier-only two-tower model loses to a simple linear autoencoder on sparse data—on five public datasets, and pinpointed the intervention that reverses it. Extensibility is deliberately cheap: a new dataset is one loader function, a new model is a class with two methods, and SASRec was added in under a hundred lines of code. The package is maintained with continuous integration across Python versions, and any change to a protocol constant or metric definition triggers a new semantic version.

The authors are candid about limitations. Every model ran in one literature configuration, with only EASE and the two towers retuned over small grids; no full hyperparameter search was performed, so absolute numbers are not best achievable. Cold-start users, the cohort that dominates real marketplaces, are excluded from evaluation and left to a planned content-only extension. And the harness remains an offline tool on cleaned public logs: online experiments, where business constraints and user behaviour interact in ways no offline metric captures, remain the final arbiter. Still, in a field repeatedly burned by untuned baselines, sampled metrics and leaked future information, a benchmark that refuses to declare a winner without a confidence interval—and that can show when the scoreboard itself changes the game—may prove as influential as any new model architecture.

Subject of Research: A frozen-protocol benchmark harness for evaluating two-tower retrieval and autoencoder recommender systems with paired bootstrap statistics on public implicit-feedback datasets

Article Title: towerbench: A frozen-protocol benchmark harness for two-tower retrieval and autoencoder recommenders with paired bootstrap statistics on public implicit-feedback data

Article References: Bieniek, M., Kanabus, B., Kozłowski, A., & Kurek, J. (2026). towerbench: A frozen-protocol benchmark harness for two-tower retrieval and autoencoder recommenders with paired bootstrap statistics on public implicit-feedback data. SoftwareX, 36, Article 103129. https://doi.org/10.1016/j.softx.2026.103129

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103129

Keywords: recommender systems, two-tower models, autoencoders, benchmarking, bootstrap statistics, implicit feedback, evaluation protocol, EASE, reproducibility, machine learning, collaborative filtering, open-source software