The study, conducted by John Alexander Calderón and Nelson Andrés Rodríguez of Colombia’s National Institute for Educational Evaluation (Icfes) and Víctor H. Cervantes of the University of Illinois at Urbana-Champaign, examined whether items on the Mathematics test of SABER 11 functioned differently for the two testing populations. This question is central to a psychometric property called differential item functioning, or DIF, the phenomenon in which examinees of equal underlying ability have different probabilities of answering an item correctly depending on which group they belong to. If items show DIF between groups, then score comparisons between those groups may reflect bias rather than genuine differences in the measured trait, undermining the fairness of any conclusions drawn from the results.

DIF has been a concern in testing since at least the 1960s, but interest intensified after the 1984 “Golden Rule” settlement in the United States, which pushed the testing industry to distinguish statistically between real group differences and bias against particular groups. Under item response theory, the framework most large-scale assessments use for scoring, DIF is defined precisely: for an item scored correct or incorrect, a two-parameter logistic model assigns each item a discrimination parameter, describing how sharply the item separates high-ability from low-ability examinees, and a difficulty parameter, locating the ability level at which an examinee has a fifty percent chance of answering correctly. If either parameter differs across groups, the item’s characteristic curves diverge. Uniform DIF corresponds to a difference in difficulty alone, producing a constant shift between the curves; non-uniform DIF involves a difference in discrimination, so the curves cross and the group advantage varies with ability; and mixed DIF involves both.

Because SABER 11 already uses item response theory for its scaling and scoring, the researchers chose an IRT-based DIF procedure, the non-compensatory DIF (NCDIF) index from Raju’s Differential Functioning of Items and Tests framework, complemented by the widely used Mantel–Haenszel procedure. The NCDIF index quantifies the area between the two groups’ item characteristic curves, weighted by the distribution of ability in the focal group, meaning the group of special interest. This weighting has an attractive property: it emphasizes parameter differences where they matter most for the focal group’s actual scores. The statistical test on the NCDIF index is conducted through parametric bootstrap, known in the DFIT framework as item parameter replication, in which the item parameters are repeatedly re-estimated from simulated data to build the distribution of the statistic under the null hypothesis of no DIF.

Following a framework laid out by Sireci and Rios for tailoring DIF analyses to specific testing contexts, the team had to make a series of practical decisions: which detection method to use, how to define the comparison groups and their sample sizes, how to construct the matching variable on which the two groups are compared, whether to incorporate effect size measures, and at what level to analyze the results. Most of these choices could be settled from the existing literature. But two could not, because no published research had explored the performance of the NCDIF index under conditions that match SABER 11: sample sizes reaching roughly 40,000 examinees in the majority group against about 1,500 in the minority group, a ratio of up to 1:25, combined with a moderate gap in mean ability between the two populations.

To resolve this, the researchers ran a series of simulation studies in which they generated response data for test forms mirroring the real structure of SABER 11, with 40-item forms sharing an anchor of half their items, using item parameters drawn from the actual operational pool of SABER 11 mathematics items rather than from artificially clean, well-distributed parameter sets. They manipulated sample size and ratio, the impact between groups (shifting the focal group’s mean ability from zero up to plus or minus 0.8 standard deviations), the purification of the matching variable, and the use of effect size classifications, with 200 replications per condition. The purification step matters because the matching variable itself must be free of DIF contamination: the two groups’ ability scales are linked using common items, and if items with DIF are included in the linking, the resulting bias can masquerade as or mask genuine DIF. The researchers used a two-stage purification in which the linking is first performed with all items, suspect items are removed, and the linking is repeated with the purified set.

The simulation results overturned a piece of conventional advice. Previous studies had suggested that DIF analyses work best when the two groups being compared have similar sample sizes, and it had been recommended that the larger group be subsampled to match the smaller one. But in these simulations, the Type I error rates, meaning the rate of incorrectly flagging items as showing DIF when they truly do not, were no worse for the extreme 1,500-versus-40,000 condition than for equal-sized groups of 40,000 each. In fact, the error rates remained less well controlled precisely in the conditions with equal sample sizes at the 40,000 level. The contrast between the 1,500-to-1,500 and 1,500-to-40,000 conditions showed no statistically significant difference in either Type I error or power after purification. The practical conclusion was clear: there was no reason to discard data and subsample the larger group.

The second major finding concerned the interplay of impact and effect sizes. Before purification, Type I error rates ballooned in the largest samples, especially when there was a true difference in mean ability between the groups, an asymmetry that also depended on whether the focal group was favored or disadvantaged. After purification, the NCDIF index’s error rates approached nominal levels, but the inflation caused by impact remained. Crucially, applying effect size guidelines, which classify flagged items by the practical magnitude of the difference rather than statistical significance alone, drove the false positive rates to nearly zero across almost all conditions, while barely reducing power for detecting moderate or large DIF given the enormous sample sizes involved. For the NCDIF procedure, power was near-perfect for items with genuinely non-negligible DIF. An analysis of variance confirmed that nearly all interactions among the experimental factors were significant, with the largest effects tied to the interplay of sample size ratio, the number of DIF items, and the use of effect sizes.

With these design decisions settled, the team applied the full protocol to actual SABER 11 data: the Mathematics test form administered on the second date of 2018, with 39,377 examinees as the reference group, and the form from the first date of 2019, with 1,508 examinees as the focal group. After purification, the item parameter replication test flagged eight to ten of the 22 common items, and the Mantel–Haenszel procedure flagged three to four. But when the effect size classifications were applied, only a single item, item 18, was classified as showing non-negligible DIF. Its NCDIF value was 0.03096 and its Mantel–Haenszel delta was 1.6386, placing it in the “large DIF” category. Once this item was removed from the common set, the Stocking–Lord scale linking between the two forms yielded the transformation constants needed to place both groups’ abilities on a single scale, with the residual mean difference of 0.59 favoring the August-cohort group, consistent with the historical advantage of that subpopulation.

The content review of item 18 proved instructive. The item asks students to judge whether three different algebraic procedures for solving the equation (x + 2)(x + 3) = 5(x + 3) were performed correctly, with each student’s work shown step by step. The researchers found that although two of the procedures were on track toward the correct solution, none of them actually reached a final answer, and the point at which the student Nelson’s work stopped could appear incorrect to examinees from the September cohort more frequently than to those from the March cohort. For an examinee of ability 1.0 on the scale where the March group’s abilities were standardized, the probability of a correct response was roughly 0.62 for the March group but below 0.35 for the September group. The item’s characteristic curves crossed at an ability value of about −0.824, meaning the difference was small near the average of the focal group but grew rapidly with ability. Icfes’s mathematics team has since revised the item, making the three procedures more explicit and ensuring each reaches a final answer.

The study’s implications extend well beyond Colombia. The authors emphasize that detecting DIF is only the first step toward fairness, and that content review, think-aloud protocols with students, teacher interviews, and curricular analyses should follow to understand why an item functions differently. They also caution that future simulation studies of DIF statistics, whether NCDIF, Mantel–Haenszel, or any other index, should draw item parameters from realistic operational pools rather than sanitized, evenly distributed sets, since the distribution of item difficulties within a test interacts with detection behavior in ways that clean simulated pools fail to capture. In their data, the average difficulty of common items was shifted by 0.66 relative to non-common items, an asymmetry that may itself interact with detection rates. For practitioners running large-scale assessments anywhere in the world, the message is that statistical significance alone, with samples in the tens of thousands, can manufacture bias where none exists, and that effect size classification, scale purification, and realistic item parameter pools are the practical safeguards against turning fairness checks into false alarms.

Values, as fundamental guiding principles in human behavior and cognition, serve as anchors for decision-making, social interactions, and individual life trajectories. However, retrieving reliable, valid measurements of these abstract constructs in non-Western cultures presents a significant methodological challenge. The PVQ series, developed within Schwartz’s theory of basic human values, has been a cornerstone in this endeavor, yet its application remains under-explored in diverse cultural fabrics like Turkey’s, which blends Eastern tradition with Western modernity. This study embarks on a rigorous psychometric evaluation to fill that critical gap.

At the core of this work is the comparative analysis of three different versions of the Portrait Values Questionnaire. The PVQ-40, a 40-item tool, offers a comprehensive, nuanced mapping of Schwartz’s ten value types, enabling detailed exploration of subtle nuances in value priorities. The PVQ-21 condenses these items into a more manageable format, potentially enhancing participant engagement and data collection efficiency. Meanwhile, the recently revised PVQ-RR, an augmented and theoretically refined iteration, promises heightened sensitivity and clarity in value distinctions.

The researchers employed a methodologically robust design, drawing upon a demographically diverse Turkish sample. Given Turkey’s unique geopolitical position bridging Europe and Asia, with a rich tapestry of cultural, religious, and social influences, the dataset provides fertile ground for psychometric scrutiny. The study incorporated thorough statistical analyses including confirmatory factor analysis (CFA), internal consistency assessment via Cronbach’s alpha, and measurement invariance testing to discern whether the instruments maintained construct equivalence across subpopulations within Turkey.

One of the landmark findings of this investigation lies in the demonstration that while all three instruments generally replicate the theoretical value structures proposed by Schwartz, significant variations emerge in their psychometric robustness and cultural fit. The PVQ-RR notably surfaces as the instrument with superior model fit indices, underscoring its enhanced sensitivity to cultural semantics intrinsic to Turkish society. This suggests the revision process successfully tuned the tool for detecting and distinguishing value dimensions within non-Western contexts that frequently challenge Western-developed scales.

Delving deeper, the study highlights subtle cultural variations that affect specific value priorities. For example, notions tied to collectivism versus individualism manifest distinctly in response patterns, reflecting Turkey’s collectivist social fabric influenced by family-centric and community-oriented values. The flexibility and adaptability of the PVQ-RR allowed for better accommodation of such cultural idiosyncrasies without sacrificing theoretical coherence. Conversely, the more concise PVQ-21, while easier to administer, showed limitations in capturing these nuanced contrasts, potentially leading to oversimplification in cross-cultural value assessments.

Moreover, the research underscores the critical importance of measurement invariance – the degree to which a psychological construct is assessed equivalently across groups. The PVQ-RR emerged superior in achieving partial invariance across gender and age groups in Turkey, reassuring researchers of its robustness and utility for comparative studies within the population. These findings carry profound implications for both theoretical psychology and applied social research, suggesting that culturally sensitive tools can sharpen the precision of value measurement and interpretation.

Beyond technical psychometric evaluations, this work also contributes to broader theoretical implications concerning the universality of value structures. While Schwartz’s theory enjoys widespread acceptance, this study demonstrates that cultural calibration is indispensable. Universal frameworks must be flexibly interpreted and adapted to each societal context to maintain analytic integrity and avoid ethnocentric biases. Thus, advances in culturally attuned measurement instruments like PVQ-RR pave the way for more equitable and meaningful cross-cultural comparisons.

This research further informs policy-making, organizational behavior, and intercultural communications by clarifying which value facets resonate most strongly within Turkish society. For multinational corporations, NGOs, and governmental bodies operating in or partnering with Turkey, these insights provide actionable intelligence to tailor messaging strategies, motivate personnel, and craft culturally consonant interventions. Understanding value priorities enables the design of programs that better align with intrinsic cultural motivations, thereby enhancing effectiveness and social acceptance.

The study also calls attention to the evolving nature of human values in an increasingly interconnected world. By providing a replicable evaluation framework applicable to other non-Western populations, it encourages future research to adopt similar methodological rigor. Capturing the fluid dynamics of value change over time—amid globalization, technological diffusion, and sociopolitical transformations—requires instruments that are both theoretically grounded and culturally attuned, a balance well embodied by the PVQ-RR.

Intriguingly, this study opens avenues for further exploration into the psychological underpinnings of culture-specific value expressions and their neurological correlates. Future interdisciplinary collaborations might probe how cultural context modulates neural circuits linked to value-based decision-making, employing tools like functional MRI alongside sophisticated psychometric instruments. Such integrative endeavors promise a more holistic comprehension of human values bridging psychology, neuroscience, and anthropology.

Methodologically, the researchers emphasize the necessity of ongoing refinement and validation cycles. Psychological measurement is far from static; evolving cultural contexts, language nuances, and emergent sociocultural phenomena require instruments to remain adaptable and receptive to feedback from diverse populations. The demonstrated superiority of the PVQ-RR in this study exemplifies the benefits of iterative enhancement rooted in empirical data and theoretical rigor.

Ethically, this research underscores the imperative for cross-cultural psychologists to engage deeply with the populations they study, fostering mutual respect and collaborative dialogue. Developing and validating measurement tools should move beyond surface translation towards meaningful cultural immersion and consultation to safeguard against misinterpretation and cultural bias. These ethical commitments enrich scientific validity while honoring participants’ lived experiences.

In conclusion, the study by Karadag, Ergin-Kocaturk, and Nasir represents a landmark contribution to cross-cultural value psychology. By rigorously assessing three prominent value measurement instruments within Turkey, it offers compelling evidence favoring the enhanced PVQ-RR as a culturally sensitive, psychometrically robust tool capable of capturing the rich, multifaceted nature of human values in a uniquely complex society. This research not only advances methodological sophistication but also enriches our collective understanding of how values shape and are shaped by cultural contexts.

As societies evolve and interconnect at ever-accelerating rates, reliable and culturally aligned measurements of values will remain foundational for research, policy, and practice worldwide. The pioneering work presented here sets critical new standards and inspires continued innovation toward harmonizing global psychological constructs with local cultural realities—heralding a future where science genuinely embraces and interprets the diversity of human value systems.

Subject of Research:
Cross-cultural evaluation of human value measurement instruments in Turkey using PVQ-RR, PVQ-40, and PVQ-21.

Article Title:
Evaluating value measures across cultures: study on PVQ-RR, PVQ-40, and PVQ-21 in Turkey.

Article References:
Karadag, E., Ergin-Kocaturk, H. & Nasir, S. Evaluating value measures across cultures: study on PVQ-RR, PVQ-40, and PVQ-21 in Turkey. BMC Psychology (2025). https://doi.org/10.1186/s40359-025-03725-6

Image Credits:
AI Generated