New Framework Shows Recommendation Accuracy Means Nothing if the Link Is Broken

Recommendation systems have long been judged by a single question: how well do they rank the right products at the top of a list? A new study argues that this question, while important, misses half the story. Researchers at Vellore Institute of Technology in Tamil Nadu, India, have built a platform called LocalMarket and, with it, a dual-metric evaluation framework showing that in open-network commerce, ranking accuracy and serving correctness are two independent, jointly necessary dimensions of quality. Reporting only one, the authors contend, materially overstates the value a recommender actually delivers to users.

The work, published in Frontiers in Big Data, is motivated by the architecture of India’s Open Network for Digital Commerce (ONDC) and the Beckn protocol that underpins it. Unlike closed platforms where one company controls the catalog, the storefront, and the recommendation engine, open networks interconnect independently operated buyer apps, seller apps, and logistics providers through asynchronous protocol messages. A single product search may cross several service boundaries, and each boundary introduces a potential failure that has nothing to do with how intelligently items are ranked.

The researchers formalized this problem by modeling the platform catalog as a piecewise-gathered construction assembled from many provider-local catalogs, each with its own schema and identifier space. For the catalog to function, two maps must succeed for every item: a schema-normalizing map that translates provider payloads into a platform-wide format, and an identifier-resolution map that converts provider-local IDs into one global namespace. From this formalization they derived four recurring failure classes: provider heterogeneity with schema mismatches, sporadic upstream unavailability, cross-service identifier heterogeneity, and catalog volatility that destabilizes state between ranking and rendering.

The critical insight is mathematical as much as architectural. Standard offline ranking metrics such as MAP@10, Recall@10, and NDCG@10 depend only on the identities and ordering of a candidate list. They cannot distinguish a list whose top-ranked item opens a working product page from an identical list whose top item produces a 404 error. The two lists are the same; the difference emerges only when the routing layer receives them. To capture this invisible failure surface, the team introduced two new serving-correctness metrics: Route Resolution Success Rate (RRSR), which measures how often recommended items resolve to navigable pages, and Wishlist Consistency Rate (WCR), which measures whether user state survives identifier mismatches.

To test the framework, the team built LocalMarket as a four-layer system with a Next.js frontend, a FastAPI backend, and an ensemble ranking layer combining neural collaborative filtering, LightGCN-based graph collaborative filtering, a GRU4Rec-style session model, and a content-based prior, fused as a convex combination of normalized scores. Notably, each component maps one-to-one onto a failure class: the NCF branch relies only on interaction identities rather than shared schemas, the graph model recovers signal for sparsely encountered items, the session model remains stable when cross-service identifiers drift, and the content-based component can score brand-new items before any collaborative signal exists. Ensemble weights were selected through a constrained search that maximized NDCG@10 subject to a no-regression condition on serving correctness, landing at roughly 0.44, 0.33, 0.22, and 0.01 for the four components.

Experiments used the RetailRocket e-commerce dataset under a deliberately sparse implicit-feedback protocol with strict leave-one-out next-item prediction over a normalized catalog of 74,743 items. The ensemble improved NDCG@10 by 70.7 percent over a popularity baseline, along with gains of 65.2 percent in MAP@10 and 22.2 percent in Recall@10. The authors caution that absolute metric values in this sparse regime are small by design; the meaningful comparison is the relative lift across configurations evaluated on identical traces.

The reliability results were far more dramatic. Before any intervention, the system achieved an RRSR of just 60.0 percent, meaning forty percent of top-ranked recommendations could not be opened by a user at all, and a WCR of 0.0 percent, meaning wishlist state was entirely inconsistent across identifier representations. Four targeted fixes, including limit-aware fallback generation, full ID-band category mapping, boundary-level identifier normalization, and a corrected invocation contract for the similar-item utility, raised RRSR to 99.8 percent and WCR to a perfect 100.0 percent. A single identifier normalization fix alone lifted WCR from zero to complete consistency.

The study’s central finding came from a controlled within-system ablation comparing Model-only, Reliability-only, and Combined configurations on fixed, reproducible traffic traces of 500 synthetic sessions. When both intervention types were enabled simultaneously, ranking gains and serving-correctness gains were preserved together without interference, demonstrating that the two axes are additive rather than substitutive. In other words, a better model cannot compensate for broken infrastructure, and better infrastructure cannot substitute for a better model; excellence in one cannot remedy deficiency in the other.

The researchers are explicit about the limits of their claims. The experiments ran on a single repository-level implementation with injected faults and synthetic traces, and no personal user data was collected or processed. Absolute RRSR and WCR values depend on the specific faults inserted into the test trace, though the qualitative thesis, that serving correctness and ranking accuracy are independent axes deserving equal reporting, is argued to generalize to Beckn-pattern networks. They also flag governance concerns: cross-boundary identifier normalization creates a canonical join key that could, in a live network, correlate user activity across participants without proper access controls.

The broader message is likely to resonate well beyond India’s open commerce initiative. As federated and decentralized marketplaces proliferate, the study suggests a quiet flaw in how the field measures itself: a recommender can be judged brilliant in the lab while delivering a broken experience in production, and no amount of ranking sophistication will fix a dead link. The authors have released the full platform, preprocessing scripts, and an experiment-replay procedure under an MIT license, hoping the dual-metric discipline becomes standard practice for anyone building recommenders that must survive the messy reality of open networks.

Subject of Research: Dual-metric evaluation of ranking accuracy and serving correctness in ensemble recommender systems for open-network digital commerce

Article Title: Reliability-aware ensemble recommendation for open-network commerce: a dual-metric framework for jointly evaluating ranking accuracy and serving correctness

Article References: S, R. P., & Rajasekar, V. (2026). Reliability-aware ensemble recommendation for open-network commerce: a dual-metric framework for jointly evaluating ranking accuracy and serving correctness. Frontiers in Big Data, 9, Article 1911540. https://doi.org/10.3389/fdata.2026.1911540

Image Credits: AI Generated

DOI: 10.3389/fdata.2026.1911540

Keywords: recommender systems, ONDC, Beckn protocol, serving correctness, ranking metrics, ensemble learning, neural collaborative filtering, LightGCN, GRU4Rec, reliability engineering, e-commerce, dual-metric evaluation