The review, conducted by Andrea Bazzoli of Baruch College and the City University of New York Graduate Center, Thao T. Vo, Ana Campillo-Martínez, Alexis N. Downey, Laura Pires Gifford, and Brian F. French of Washington State University and the University of Murcia, examined 63 empirical articles published between 2021 and 2025 in five leading journals: Applied Psychology: An International Review, the Journal of Applied Psychology, the Journal of Business and Psychology, the Journal of Occupational and Organizational Psychology, and Personnel Psychology. Every article included in the analysis either introduced a new work-relevant instrument or provided validation evidence for an existing one. The team then used a theory-guided coding protocol to map what kinds of evidence each study actually reported, drawing on three complementary frameworks: the Standards for Educational and Psychological Testing, Michael Kane’s interpretation and use argument, and a justice-oriented antiracist validity framework developed by Jill Randall and colleagues.

The conceptual starting point matters here. Under the modern Standards, validity is not a property of a test itself but of the interpretations and uses of its scores. Kane’s argument-based approach formalizes this by breaking validation into a chain of inferences that must each be supported: domain description, which concerns how well the construct is defined and represented; scoring, which concerns the rules that convert responses into numbers; generalization, which concerns whether scores are consistent across samples and occasions; extrapolation, which concerns whether scores relate to external criteria such as job performance; and implications, which concerns the real-world decisions and consequences that flow from test use. Each link in that chain requires its own evidence, and a gap anywhere weakens the whole argument.

What the researchers found was a striking imbalance. Evidence supporting generalization and extrapolation appeared in nearly every study, with 61 of the 63 articles reporting reliability information such as internal consistency or test-retest coefficients, and the same number reporting relationships with other variables, most often convergent, discriminant, concurrent, or predictive validity correlations. Scoring evidence, typically factor analyses and measurement invariance tests, appeared in 60 articles, and domain description evidence, usually expert review of item content, appeared in 56. But the implications inference, the one that asks what actually happens when scores are used to make decisions about people, was addressed in only four articles, and even those studies often presented evidence that mapped more naturally onto other parts of the argument. Under a stricter reading of Kane’s framework, the authors note, it is possible that none of the reviewed studies provided direct evidence about the consequences of score use.

The gap extends to how respondents actually experience and interpret the instruments. Only 12 of the 63 articles, roughly 19 percent, reported any evidence based on response processes, and most of that consisted of simple ratings of item clarity rather than deeper investigations of how people understand questions, map their experiences onto response options, and arrive at judgments. Cognitive interviews and think-aloud procedures, which can reveal whether respondents interpret items the way test developers intended, were rare. This matters because when validation relies predominantly on factor analyses and correlations among similar constructs, there is a risk that constructs become defined after the fact by their statistical covariation patterns rather than by a well-articulated theoretical account of what they represent, for whom, and under what conditions.

The justice-oriented findings were equally revealing. Only 19 of the 63 articles reported evidence that could inform a justice-oriented validity argument, and when they did, it almost always took the form of statistical comparability across groups, such as measurement invariance tests or differential item functioning analyses. Sixteen studies addressed issues related to construct definition or interpretation, but twelve of those simply evaluated whether constructs retained similar meaning across demographic groups. Far fewer examined whose experiences and values were embedded in the construct in the first place, whether the language of items privileged certain cultural norms, whether assessment processes themselves demand alignment with dominant group discourse, or how structural inequality might shape results. No study in the review explicitly articulated how a construct was intended to disrupt oppressive narratives, and none documented the involvement of marginalized stakeholders in defining what was being measured.

The review also highlights how the field’s traditional approach to fairness, rooted in adverse impact analysis and the so-called diversity-validity dilemma, may leave important questions unexamined. That tradition, shaped partly by legal frameworks governing personnel selection, has generally understood validity as evidence of statistical association with future job performance. Demonstrating that predictor-criterion relationships hold across groups is valuable, the authors argue, but it says little about who participated in defining the predictor and the criterion, whose experiences those definitions represent, or how historical and organizational inequalities may shape access to the opportunities the measures capture. A justice-oriented approach would also ask whether alternative constructs or criteria might better represent performance, whether workers had meaningful input into the assessment process, and whether resulting decisions reproduce existing inequities.

The timing of these findings is significant because the measurement landscape is changing rapidly. A small but growing subset of the reviewed studies, about 10 percent, used technology-mediated formats such as automated video interviews, game-based assessments, and natural language processing scoring systems. Three articles incorporated machine-learning components, treating algorithms as scoring agents whose outputs must be evaluated for consistency and agreement with human ratings. But neither Kane’s framework nor the justice-oriented validity framework was designed for contexts in which algorithms generate or supplement scores. The authors warn that generative AI models can be prompted to explain their ratings, yet such explanations may be post hoc rationalizations that require independent verification and should not be treated as validity evidence. More troubling still, when AI models learn from incumbent employees or prior hiring decisions, they may encode an organization’s existing assumptions about what competence looks like, creating an echo chamber in which historically favored profiles are used to select applicants who resemble those already rewarded.

Geographic and cultural narrowness compounds these problems. More than half of the reviewed studies, 52 percent, drew samples exclusively from the United States, and 79 percent relied on self-report Likert or rating scales. Translation and adaptation work was limited, and when it occurred, it generally targeted semantic correspondence rather than investigating whether apparently equivalent language carries different historical or cultural meanings across populations. The authors argue that back-translation alone is inadequate and that item-level cultural context should be documented when constructs are defined and response processes investigated, not after measurement noninvariance has already been detected.

The path forward, according to the review, requires several shifts happening simultaneously. Researchers should specify the intended use of scores in advance and collect evidence appropriate to that use, since evidence supporting the use of scores for theory testing does not automatically support their use in personnel decisions. Validation studies should routinely incorporate cognitive interviews, think-aloud protocols, and web-based probing, ideally with purposively selected subsamples drawn from groups for whom differences in interpretation are theoretically plausible. Affected stakeholders, including employees and applicants, should be involved throughout item generation and content validation rather than consulted only after constructs have been defined. And the consequences of score use, from selection rates to the distribution of classification errors, should be treated as integral components of validity rather than afterthoughts. A revised version of the Standards expected in 2027 will likely include guidance on artificial intelligence across the entire assessment process, offering a starting point for updating validity practice. The authors’ conclusion is measured but pointed: psychometric rigor remains essential, but it must be extended to the places it currently does not reach, including how constructs are defined, how respondents interpret items, how scores are used, and whether those uses mitigate or reproduce inequity.

Subject of Research: Validity evidence and justice-oriented practices in industrial-organizational psychology scale validation research

Article Title: Building justice-informed validity arguments in organizational psychology: Where the evidence is (and isn't)

Article References: Building justice-informed validity arguments in organizational psychology: Where the evidence is (and isn't). (n.d.). https://doi.org/10.3389/forgp.2026.1861006

Image Credits: AI Generated

DOI: 10.3389/forgp.2026.1861006

Keywords: industrial-organizational psychology, validity, scale validation, psychometrics, justice-oriented validity, measurement invariance, interpretation and use argument, algorithmic assessment, automated video interviews, test fairness, consequences of testing, construct definition