The problem the tool addresses is one that any developer in a large codebase will recognise. Wikimedia’s existing Codesearch tool, powered by the Hound engine, indexes repositories using regular expressions and pattern matching. That works well when you know exactly what a function is called, but it fails when vocabulary diverges from intent. A volunteer searching for permission validation logic may never find a function named checkUserLoginStatus, because the words simply do not overlap. With two decades of evolving and inconsistent naming conventions across thousands of repositories, these lexical gaps hinder volunteer onboarding and slow down even the Foundation’s core developers.

Code2Code Search, developed by Francesco Tosoni and described in the journal SoftwareX, takes a fundamentally different approach. Instead of matching strings, it transforms code snippets into vector representations, high-dimensional numerical encodings of a snippet’s semantic content. Queries and indexed code are embedded independently into a shared vector space and compared by distance, following the bi-encoder retrieval paradigm familiar from modern text search. The result is intent-based retrieval: submit a code sketch, and the system returns the functions, classes, and templates whose computational purpose most closely matches, even when they are written in a different programming language or use entirely different identifiers.

The scale of the undertaking is considerable. The team enumerated 2,568 repositories, cloned the main branch of each, and submitted all of them to Software Heritage, the universal archive of source code, ensuring persistent, citable identifiers known as SWHIDs for every artefact. Using the Tree-sitter parsing library, the pipeline extracted structural entities, functions, types, and templates, across twelve programming languages, from PHP and JavaScript to Go, Rust, and Lua. After filtering out repositories containing only documentation or assets, the final index covers 2,242 active repositories and roughly 1.29 million code entities, about 81 percent of them functions.

The neural engine at the heart of the system is Qwen3-Embedding-0.6B, chosen for its permissively licensed, open-source status and its strong showing in benchmarks of code-to-code recall. Each snippet is encoded into a normalised 1024-dimensional vector, a process that takes about 31 minutes on a single NVIDIA H200 GPU. But the clever part of the architecture lies in what happens next. The system follows a build-heavy, serve-light design: the expensive vectorisation runs offline on GPU hardware, while the deployed service operates on Toolforge, Wikimedia’s shared, CPU-only cloud platform with a strict 6 GiB memory limit.

To squeeze a million-vector index into that budget, the team turned to FAISS, Facebook’s similarity-search library, and product quantisation. The index is compressed into an IVF-PQ structure with 128 sub-quantisers at 8 bits per sub-vector, shrinking the memory footprint from roughly 4.9 GiB for a flat float32 baseline to just 168.6 MiB, a reduction of 96.6 percent. On a commodity laptop with no discrete GPU, the system answers queries with a median latency of 1.85 seconds. On the live Toolforge deployment, where the embedding model runs on throttled shared CPUs, latency rises to an average of about 31 seconds, and the team has an infrastructure request pending with the Wikimedia Foundation to bring GPU capacity to the service.

Does semantic search actually beat keywords? To find out, the researchers built a 27-query benchmark spanning four categories: canonical textbook algorithms, name-obfuscated patterns that hide their intent behind unusual identifiers, cross-language tasks, and niche Wikimedia-specific patterns. Because no human gold standard exists for code search in this ecosystem, they adopted the increasingly popular LLM-as-judge paradigm, assembling a jury of three large language models from different families, Claude Opus 4.8, Google Gemini 3.8 Flash, and Moonshot Kimi K2. Each judge graded a blinded, anonymised pool of results from both systems, never seeing which system produced which ranking, and inter-judge agreement reached a Krippendorff’s alpha of 0.74.

The verdict was unambiguous. On the headline rank-aware metric, nDCG@10, Code2Code Search achieved a jury mean of 0.81 against 0.63 for a BM25 keyword baseline over the same 1.29 million entities, with the confidence interval on the improvement excluding zero under all three judges. The advantage was largest precisely where lexical matching is weakest: in the name-obfuscated category, the neural system scored 0.83 versus 0.53, and in cross-language retrieval 0.81 versus 0.62. In one striking example, a query for SHA-256 file hashing pulled the keyword engine toward irrelevant path utilities because of a shared token, while the neural system returned the actual hashing functions at the top ranks. The baseline did retain an edge in a few niche, domain-specific cases where distinctive vocabulary dominates, and the authors note that dense retrievers can be misled by textual similarity between adjacent but unrelated functions, motivating future hybrid ranking approaches.

Beyond raw performance, the project makes a deliberate statement about values. Unlike some other AI initiatives within Wikimedia, which have relied on proprietary embedding models and commercial vector databases, every component of Code2Code Search, including the Qwen3 model, is released under the Apache Software License 2.0, in keeping with the movement’s commitment to free software. The interface, built on Wikimedia’s Codex design system, is localised in 30 languages at the time of writing, 14 of them Indian, maintained by volunteer translators through translatewiki.net and redeployed automatically with each merge. By attaching SWHIDs to every indexed entity, the system also makes each retrieved function permanently citable in academic work, operationalising the UNESCO Recommendation on Open Science at the granularity of individual software artefacts.

The implications reach well beyond Wikipedia. For the 1.5 billion monthly visitors to Wikimedia projects, the tool is invisible infrastructure, but for the developer community it tackles the cold-start problem of navigating an unfamiliar 2,500-repository corpus, and its RESTful API opens the door to integration with IDE plugins, bots, automated code review, and retrieval-augmented code generation. Because only the repository-discovery script is ecosystem-specific, the entire pipeline, parsing, archival, vectorisation, indexing, and serving, can be retargeted to other forges, research institutions, or enterprises by swapping a single component. Future work will explore cross-encoder reranking for higher precision, incremental reindexing for near-real-time updates, and natural-language queries. For now, the tool stands as a demonstration that state-of-the-art neural retrieval can live entirely on open infrastructure, compressing a million snippets of humanity’s collaborative code into a space small enough to run on a shared virtual machine, and finding the right one in under two seconds.

Subject of Research: Neural code-to-code retrieval for semantic search across the MediaWiki open-source software ecosystem

Article Title: MediaWiki Code2Code Search: Neural retrieval for the semantic discovery of open-source software entities

Article References: Tosoni, F. (2026). MediaWiki Code2Code Search: Neural retrieval for the semantic discovery of open-source software entities. SoftwareX, 36, Article 103127. https://doi.org/10.1016/j.softx.2026.103127

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103127

Keywords: MediaWiki, code search, neural embeddings, Qwen3, FAISS, Software Heritage, Wikimedia, semantic retrieval, open-source software, Tree-sitter, BM25, LLM-as-judge

The problem the researchers identified is deceptively simple. Existing validation efforts for AI in surgical assessment tend to focus on classification accuracy, meaning how reliably an algorithm can distinguish a novice from an expert. But classification is not feedback. A trainee does not benefit from being sorted into a category; they benefit from language that accurately reflects their level of performance and that responds sensibly when their performance changes. Until now, no widely adopted framework existed to measure whether the tone of AI-generated commentary appropriately mirrors how well a resident actually performed, or whether that commentary shifts consistently in the right direction when scores rise or fall.

To fill that gap, Mohamed S. Baloul, Jonathan D’Angelo and their colleagues proposed two complementary metrics. The first is Calibration, expressed as the squared correlation between the sentiment of the feedback and the resident’s actual performance scores. In psychometric terms, it functions as an analogue of validity evidence, asking whether the emotional register of the machine’s commentary matches the quality of the work being described. The second is the Feedback Robustness Index, or FRI, which measures the percentage of times the direction of the feedback correctly follows a controlled change in performance scores. This acts as an analogue of reliability, probing whether the system behaves consistently rather than drifting arbitrarily. Sentiment was quantified using two established natural language processing tools, TextBlob and VADER, providing a reproducible, algorithmic measure of how positive or negative each piece of feedback was.

The experimental design was unusually systematic. The team drew assessments from nine second-year general surgery residents per station, deliberately selected to span the full range of performance, across five standardized simulation stations: FLS Knot Tying, FLS Circle Cutting, Vascular Anastomosis, Endostitch, and Imaging Interpretation. These stations were scored with rubrics ranging from minimal, essentially time and accuracy alone, to detailed, behaviorally anchored descriptors that spell out what specific levels of skill look like. Feedback was generated by Qwen3 4B, an open-source large language model running locally, a choice that keeps sensitive trainee data on institutional hardware and makes the entire pipeline transparent and reproducible.

The crucial manipulation came next. For each assessment, the researchers perturbed the resident’s scores in five controlled ways: a global improvement, a global decline, a targeted correction of a weakness, a targeted deterioration of a strength, and a minimal change that should barely register. In total, 225 baseline-to-perturbed comparisons were generated, each producing a fresh piece of AI feedback that could be scored for sentiment and compared with what the score changes should have implied. This perturbation approach transforms a vague worry about AI quality into a measurable stress test, revealing exactly where the model’s commentary tracks reality and where it loses the plot.

The results were striking in their variation, and they point to a single dominant factor: the design of the assessment rubric. The two stations equipped with detailed, behaviorally anchored rubrics produced the strongest AI feedback. FLS Knot Tying achieved a calibration of R-squared 0.40 with a Feedback Robustness Index of 84 percent, while Vascular Anastomosis reached R-squared 0.37 and an FRI of 73 percent. At those stations, the language model’s commentary genuinely rose and fell with performance, and did so most of the time the scores changed. For trainees, that means the feedback they read bore a meaningful relationship to what they actually did with the needle driver or the anastomosis.

The picture deteriorated sharply where rubrics were thinner. Stations with structured but not behaviorally anchored rubrics showed moderate robustness yet essentially zero calibration: the Endostitch station recorded R-squared 0.00 with an FRI of 64 percent, and Imaging Interpretation posted R-squared 0.00 with an FRI of 69 percent. In plain terms, the AI could often tell when performance had improved or declined, but the overall tone of its feedback bore no relationship to the resident’s actual level. Worst of all was the station with a minimal, two-item rubric, FLS Circle Cutting, where calibration sat near zero at R-squared 0.03 and the Feedback Robustness Index of 49 percent was no better than a coin flip. Across all assessments, the mean FRI of 68 percent versus a mean R-squared of just 0.16 captured the study’s central asymmetry: AI feedback moves in the right direction more often than it matches the resident’s overall performance level.

Why would rubric detail matter so much to a language model? The answer likely lies in how these systems generate text. A behaviorally anchored rubric gives the model rich, specific descriptors of what novice, intermediate, and expert performance actually look like, providing the raw material for commentary that distinguishes a struggling resident from a strong one. A sparse rubric offering only time and accuracy leaves the model with almost nothing to anchor its language to, so it defaults to generic phrasing that cannot differentiate performance levels even when the numbers change. The researchers also situate their findings in a broader concern from the AI literature: sycophancy, the well-documented tendency of language models to tell users what they want to hear. A sycophantic feedback generator could shower praise on weak performers or fail to sharpen its criticism when scores plummet, which is precisely the failure mode the FRI is designed to detect.

The practical implications extend well beyond one simulation lab. Surgical education has long wrestled with feedback that is inconsistent, delayed, or influenced by unconscious biases, and large language models promise scalable, immediately available narrative commentary at a fraction of the cost of faculty time. But this study demonstrates that the quality of that commentary is not a fixed property of the model; it is a joint product of the model and the assessment instrument feeding it. A program that invests in a capable language model but pairs it with a skeletal rubric may be deploying a system whose praise and criticism are essentially uncorrelated with trainee performance, potentially misleading learners at the most formative stage of their development.

The authors’ recommendation is therefore twofold. Programs implementing AI-generated feedback should evaluate those systems with calibration and robustness metrics, alongside conventional expert review, before putting them in front of trainees. And they should ensure that the assessment rubrics underlying those systems contain detailed performance criteria, because the framework developed here shows that rubric design is the variable most tightly associated with whether AI feedback is appropriate and consistent. As institutions increasingly hand narrative evaluation to machines, this study offers something the field has lacked: a rigorous, replicable way to audit what the machine is actually saying about a surgeon’s hands before anyone trusts it enough to learn from it.

Subject of Research: Psychometric evaluation of AI-generated feedback quality in surgical simulation assessment

Article Title: A psychometric framework for AI feedback systems in surgical simulation assessments

Article References: Baloul, M. S., Giri, O., Mehta, A., Cui, D., & D’Angelo, J. (2026). A psychometric framework for AI feedback systems in surgical simulation assessments. Global Surgical Education – Journal of the Association for Surgical Education, 5(1), Article 183. https://doi.org/10.1007/s44186-026-00588-2

Image Credits: AI Generated

DOI: 10.1007/s44186-026-00588-2

Keywords: artificial intelligence, large language models, surgical education, psychometrics, simulation assessment, feedback calibration, Feedback Robustness Index, sentiment analysis, rubric design, medical training, Qwen3, surgical residents