New vocabulary tests reveal how word frequency shapes semantic models

Every year, thousands of research papers in psychology, cognitive science, and natural language processing rely on some form of word vector—a numerical representation of a word’s meaning as a point in high-dimensional space. These vectors underpin everything from computational models of human language acquisition to search engines and large language models. But how do researchers actually verify that these vectors capture meaning in the way they claim to? The dominant answer has long been the vocabulary multiple-choice question, or VMCQ, a test format borrowed directly from human vocabulary assessment. In a study published in Behavior Research Methods, Joseph P. Levy, formerly of the University of Roehampton, and John A. Bullinaria of the University of Birmingham’s School of Computer Science demonstrate that the field has been doing this suboptimally, and they provide a rigorously engineered alternative.

The premise of a VMCQ is deceptively simple: a test-taker—human or machine—is shown a target word and must identify its closest synonym from among several distractor words. For distributional semantic vectors, which are built directly or indirectly from word co-occurrence statistics extracted from large text corpora, answering the test amounts to computing distances between vectors. If the vector for the correct synonym is closer to the target vector than the vectors for the distractors, the model “passes” that question. The proportion of questions answered correctly becomes a benchmark score. This technique traces its lineage back to the foundational work on Latent Semantic Analysis by Landauer and Dumais in 1997, and it has since become the standard yardstick for comparing vector sets across the field.

What Levy and Bullinaria argue, however, is that most existing VMCQs were designed for humans, and humans and vectors fail in characteristically different ways. Human test designers select distractor words to challenge specific aspects of human lexical knowledge, exploiting psychological subtleties that have no bearing on how a computational vector space organizes meaning. When those same tests are applied to vectors, the questions inherited from human-oriented design may probe dimensions of performance that are irrelevant or, worse, may introduce systematic noise into the measurement. The result is that comparisons between vector sets—say, between a classic counting method and a modern prediction-based one—can be distorted by properties of the test itself rather than genuine differences in semantic quality.

To address this, the researchers first rebuilt the scoring machinery. Standard practice is to report the raw proportion of correct answers across a test set, a statistic whose reliability depends heavily on the number of questions and the difficulty distribution across them. Levy and Bullinaria developed an improved VMCQ scoring method that provides considerably more statistically reliable results, allowing finer distinctions between vector sets to be drawn with confidence. This matters because the differences between competing vector methods are often small; without a reliable score, the field risks drawing conclusions about which models are “better” based on differences that are statistically indistinguishable from chance variation. With the improved scoring in place, the authors then introduced a series of new evaluation methods designed specifically to probe one of the most important variables in lexical semantics: word frequency.

Word frequency is arguably the most powerful single predictor of linguistic behavior. Since Zipf’s classic observations in the 1940s, researchers have known that frequent words tend to have more meanings, are processed faster, and are learned earlier. Frequency effects are ubiquitous in first language acquisition, in skilled and unskilled reading, and in lexical decision tasks across the psycholinguistic literature. For distributional semantic vectors, frequency matters in a second, purely computational sense: words that appear rarely in a training corpus receive vectors based on sparse co-occurrence statistics, which tend to be noisier and less informative than vectors for frequent words that enjoy rich contextual evidence. Any serious evaluation of a vector set therefore needs to understand how performance degrades, or shifts, as a function of frequency.

The new evaluation framework allows exactly that kind of exploration. By manipulating the details of VMCQ questions—selecting targets, synonyms, and distractors with controlled frequency profiles—the researchers could systematically separate the influence of target word frequency from that of answer and distractor frequency. Previous VMCQ designs, being inherited from human testing, confounded these factors. The new series of analyses revealed, step by step, how each design choice affects the sensitivity and reliability of the resulting scores, eventually converging on an improved general VMCQ approach purpose-built for exploring frequency effects in vector spaces rather than in human populations.

To demonstrate that their redesigned tests generalize, the authors applied the final approach to a representative set of nine previously published distributional semantic vector sets, spanning three generations of methodology. The oldest family is based on Latent Semantic Analysis: the “BigLSA” vectors were generated using the LSAfun R package from a 2.8-billion-word corpus combining the British National Corpus, the ukWaC corpus, and a 2009 Wikipedia dump, with dimensionality reduced to 300. The second family relies on direct co-occurrence counting: the PPMI vectors tally word co-occurrences within a window of one word on either side across the 2-billion-word ukWaC corpus, converting raw counts into positive pointwise mutual information values over the 50,000 most frequent targets and contexts—a configuration previously shown to be optimal in Bullinaria and Levy’s own earlier work. The BL-Caron variant then applies singular value decomposition to that PPMI matrix, reducing dimensionality to 5000 with Caron-weighted components.

The remaining vector sets represent the prediction-based era. The GloVe family—Glove6, Glove42, and Glove840—uses regression modelling to learn vectors such that the dot product between two word vectors approximates the logarithm of their co-occurrence probability, trained on corpora ranging from 6 billion words of Wikipedia to 840 billion tokens of Common Crawl. The word2vec family includes Word2V, trained with the CBOW algorithm on 100 billion words of the Google News dataset; FastText, trained on roughly 3 billion words of Wikipedia with position-dependent weighting and phrase tokenization; and BigFast, trained without sub-word information on 600 billion tokens from Common Crawl. Each of these vector families embodies a different theoretical bet about how meaning emerges from usage, and each has its advocates. The ability to compare them under a purpose-built, statistically reliable testing regime is precisely what the field has lacked.

The findings carry practical implications well beyond academic benchmarking. Vocabulary tests are increasingly proposed as benchmarks for evaluating large language models, as a 2024 PLoS ONE study by Martínez and colleagues argued, and VMCQ performance remains a standard evaluation for embedding quality in applied natural language processing. If the tests themselves encode human-oriented design assumptions, then model rankings derived from them may mislead developers and researchers alike. The Levy–Bullinaria framework offers a cleaner instrument: tests whose items are constructed with the statistical properties of vector spaces in mind, scored in a way that maximizes reliability, and interpretable along the frequency dimension that dominates lexical behavior.

The study also speaks to a deeper theoretical question in cognitive science: the relationship between human vocabulary knowledge and distributional models of meaning. Decades of research, from Landauer and Dumais’s solution to “Plato’s problem” onward, have treated VMCQ scores as evidence that human semantic knowledge could, in principle, be learned from co-occurrence statistics alone. By showing that human-designed tests measure something different from what vector-based evaluation requires, the new work sharpens the interpretive lens on that entire literature. It does not invalidate prior findings, but it supplies the methodological foundation for asking more precise questions—for instance, whether the frequency at which a vector’s semantic quality degrades mirrors the frequency profile of human vocabulary acquisition, a comparison that is now methodologically feasible.

For researchers wanting to build on the work, the authors have made the materials openly available. The VMCQ word set, all nine vector sets used in the analyses, and the data underlying every figure in the paper are deposited on the Open Science Framework. The authors note that the analyses were not preregistered, but they argue that the released data and files are sufficient for independent examination and replication of the reported results without additional code. The nine vector sets themselves—spanning counting, matrix-factorization, and neural-prediction approaches—constitute a ready-made testbed for anyone developing the next generation of semantic representation methods.

As language technologies continue their rapid evolution, the humble multiple-choice vocabulary test turns out to be anything but humble: it is the measuring instrument on which claims about meaning in machines ultimately rest. Levy and Bullinaria’s contribution is a reminder that instruments matter. A test designed for a human student answering questions about synonyms is not the same object as a test designed to interrogate the geometry of a 300-dimensional embedding space. By rebuilding that instrument from first principles—reliable scoring, frequency-controlled item design, and validation across nine heterogeneous vector sets—the study gives both cognitive scientists and machine learning researchers a sharper tool for answering the question that matters most: not simply whether a model knows words, but how well, and where it begins to fail.