New Digital Test Measures How Students Internalize Digital Citizenship Values

What does it actually mean for a teenager to be a good digital citizen? A research team from Universitas Lambung Mangkurat in Banjarmasin, Indonesia, has taken a serious run at answering that question with numbers rather than slogans. In a study published in Discover Education, Rabiatul Adawiah and colleagues report the first psychometric evaluation of a new digital assessment tool called IDBEL, short for Innovative Digital-Based Learning Evaluation. The instrument is designed to measure not whether students know the rules of responsible online behaviour, but whether they have genuinely absorbed the values behind them. With 661 high school students, five expert reviewers, and a battery of statistical tests, the study offers one of the most transparent early-stage validations of a digital citizenship attitude instrument to date, and its findings are as instructive for what they reveal about measurement as for what they reveal about students.

The conceptual heart of IDBEL is a decades-old idea from educational psychology: the affective domain. Drawing on the classic taxonomy of Krathwohl, Bloom, and Masia, the researchers treat digital citizenship attitudes as a process of progressive value internalisation with five levels. Receiving captures a student’s initial awareness and acceptance of digital norms. Responding reflects active engagement with those norms. Valuing indicates genuine appreciation of responsible digital behaviour. Organization describes the integration of digital values into a coherent personal value system, and characterization represents consistent, value-driven conduct online. Each of the five dimensions was translated into five Likert-scale statements, producing a 25-item questionnaire delivered entirely through a digital application, so that the mode of assessment mirrors the digital construct being measured.

The technical groundwork began before any student saw the instrument. Five experts in educational assessment, language, civic education, and information technology rated the relevance of each of the 25 initial items on a five-point scale, and the researchers quantified agreement using Aiken’s V, a content-validity coefficient. The scores ranged from 0.70 to 0.95, with a mean of 0.83, and the team applied an operational threshold of 0.80. Twenty items cleared the bar; five, specifically Items 3, 5, 10, 13, and 20, fell short and were revised in response to expert feedback. Notably, the researchers acknowledge that the revised versions were never re-rated, so the content-validity evidence describes the initial drafts rather than a guarantee that every final item meets the criterion. That kind of candour is rare in instrument-development papers and sets the tone for the entire study.

Data collection ran from October 2025 to February 2026 through the IDBEL application, drawing Grade X and XI students from seven public senior secondary schools in Banjarmasin, selected purposively to span urban and peri-urban contexts. Grade XII students were excluded because of final examinations. The final analytical matrix contained 661 students, with school contributions ranging from 34 to 142 participants. The descriptive results told a familiar story with an uncomfortable twist: students overwhelmingly reported positive digital citizenship attitudes, with combined high and very high responses ranging from 66.0 percent to 97.4 percent across items. One item, Item 12, drew a maximum-category endorsement from 61.4 percent of respondents, while another, Item 23, drew only 14.5 percent. That spread matters, because when responses pile up at the top of a scale, the instrument loses its ability to distinguish between students who are good and students who are exceptional.

Before any structural modelling, the team tested whether the data were even suitable for factor analysis, and the answer was emphatically yes. The Kaiser-Meyer-Olkin measure of sampling adequacy reached 0.963, a value considered excellent, and item-level adequacy statistics ranged from 0.930 to 0.980. Bartlett’s test of sphericity was overwhelmingly significant, χ²(300) = 9,751.236, p < 0.001, confirming that the correlation matrix contained enough shared variance to justify extracting factors. In plain terms, the items were related to one another in exactly the way a coherent attitude construct should be. The question then became whether the relationships followed the five-dimensional architecture the theorists had prescribed.

The answer, delivered through a prespecified second-order confirmatory factor analysis, was a qualified no. Standardized item loadings were respectable, ranging from 0.567 to 0.776, and the second-order loadings linking the five dimensions to the overarching attitude construct were extremely high, between 0.929 and 0.971. But the overall model fit fell short of accepted standards: the chi-square ratio of 5.567 was high, RMSEA came in at 0.083, just above the conventional 0.08 guideline, and the incremental fit indices CFI, TLI, and IFI all landed below 0.90, at 0.871, 0.857, and 0.872 respectively. Only SRMR, at 0.054, was comfortably within target. The researchers also flagged a methodological caveat: they used normal-theory maximum likelihood on Pearson correlations, which is suboptimal for ordinal items concentrated in upper response categories, and they call for an ordinal CFA using polychoric correlations and a weighted estimator in future work.

Perhaps the most intriguing technical finding concerns what those very high second-order loadings imply. Together with a McDonald’s omega hierarchical of 0.940, they suggest that a single dominant general factor accounts for most of the reliable variance in scores. In other words, the five affective dimensions may be so conceptually intertwined, progressive stages of the same internalisation process, that they cannot yet be interpreted as distinct subscales. Modification diagnostics reinforced this concern, identifying strong local dependencies between Items 1 and 2 and between Items 23 and 24, with additional cross-dimensional overlap for pairs such as Items 4 and 8 and Items 15 and 25. Crucially, the team refused the temptation to patch the model post hoc by correlating error terms or deleting items, arguing that such changes require content justification and independent-sample cross-validation.

Reliability, by contrast, was stellar. Cronbach’s alpha reached 0.953 for the total scale, McDonald’s omega total was 0.958, and corrected item-total correlations ranged from 0.489 to 0.727, with no single item’s removal materially improving consistency. Domain-level alpha coefficients ranged from 0.792 to 0.854 and composite reliability from 0.806 to 0.854, all comfortably above the 0.70 threshold. Yet the convergent validity picture was murkier. Average variance extracted, a measure of how much variance a latent dimension explains in its own indicators, ranged from 0.456 to 0.540 and crossed the conventional 0.50 criterion only for the Organization dimension, with Characterization sitting essentially at the threshold. The lesson the authors draw is an important one for anyone who reads assessment research: very high internal consistency does not by itself establish that an instrument measures what it claims, and it may partly reflect item similarity and response concentration rather than construct strength.

So where does IDBEL stand? The researchers position it deliberately as initial psychometric evidence, not a finished, validated scale. The study’s limitations are laid out with unusual precision: the sample covers one Indonesian city and excludes vocational schools and Grade XII students; the data are entirely self-reported and vulnerable to social desirability bias, since digital citizenship items describe morally desirable behaviour; and the validity argument remains incomplete, lacking evidence on relations to external variables, measurement invariance, test-retest reliability, and the consequences of test use. Future refinements they propose include scenario-based items presenting realistic digital dilemmas, carefully constructed reverse-coded items to counter acquiescence, and Item Response Theory or Rasch analysis to map item difficulty and discrimination, particularly at the upper end of the attitude continuum where current items blur together.

For educators, the practical message is twofold. First, digital citizenship can indeed be measured as an affective construct, not merely as technical skill, and a digital platform is a natural home for such assessment in civic education classrooms. Second, caution is warranted: IDBEL is suitable as a formative, low-stakes tool and a foundation for further development, but it should be paired with teacher observations, student reflections, and scenario-based tasks rather than used for high-stakes judgements. In an era when schools everywhere are being asked to certify not just what students know but who they are online, this study offers both a promising template and a sobering reminder that building a trustworthy measure of character is harder, and more honest, work than it looks.

Subject of Research: Psychometric validation of a digital instrument assessing students' digital citizenship attitudes

Article Title: Initial psychometric evidence of an innovative digital-based learning evaluation instrument for assessing students’ digital citizenship attitudes

Article References: Adawiah, R., Ruchliyadi, D. A., & Nugroho, D. A. (2026). Initial psychometric evidence of an innovative digital-based learning evaluation instrument for assessing students’ digital citizenship attitudes. Discover Education, 5(1), Article 997. https://doi.org/10.1007/s44217-026-02201-6

Image Credits: AI Generated

DOI: 10.1007/s44217-026-02201-6

Keywords: digital citizenship, psychometrics, affective assessment, instrument validation, confirmatory factor analysis, digital assessment, civic education, reliability, convergent validity, Likert scale, Indonesia, educational measurement