AI Panel Helps Build a Readiness Test for Adaptive Moodle Courses

Researchers in Ukraine have built and validated a new diagnostic instrument that measures whether a Moodle course is structurally prepared to support adaptive learning, and the results reveal a striking gap between what learning management systems can technically do and what universities actually deploy. The Adaptive Learning Readiness Assessment Framework, or ALRAF, was developed by Serhiy Semerikov, Pavlo Nechypurenko, Tetiana Vakaliuk, Iryna Mintii, Liliia Fadieieva and colleagues, and applied to 985 real courses at Kryvyi Rih State Pedagogical University. The work, published open access in the Journal of New Approaches in Educational Research, arrives at a moment when generative artificial intelligence is transforming what personalized learning means, and when most institutions remain structurally unprepared for that transformation.

Adaptive learning systems adjust content, instructional sequences and assessment to individual learners, drawing on decades of theory from constructivism, scaffolding and self-regulated learning. Moodle, the world’s most widely used open-source learning management system, was never designed as an adaptive platform, yet its modular architecture, conditional activities, quiz engine and plugin ecosystem make it capable of supporting adaptive approaches when thoughtfully implemented. Prior studies have shown that Moodle outperforms other open-source systems in adaptivity features, and recent plugins and AI-based integrations have demonstrated measurable engagement and performance gains. Nevertheless, surveys of student opinion consistently report a perceived lack of personalization in Moodle courses, and existing readiness frameworks target institutions or platforms rather than individual courses. ALRAF was designed to fill precisely that gap: a course-level, quantitative diagnostic operationalizable entirely from standard Moodle reporting data.

The framework rests on the ICAP framework of Chi and Wylie, which distinguishes Passive, Active, Constructive and Interactive modes of cognitive engagement and predicts better learning as engagement deepens. Each candidate dimension had to satisfy two constraints: it must be scoreable from observable course components alone, without classroom observation or instructor interviews, and it must map onto a recognized ICAP engagement category. Literature synthesis initially yielded five dimensions: Content Variety, capturing the breadth and quantity of resource types; Interaction Diversity, covering the range of collaborative and individual activities; Assessment Flexibility, reflecting varied and formative assessment options; Learning Path Personalization, anchored in conditional access and branching components such as Lesson and SCORM; and Feedback Mechanisms, spanning forums, surveys and dedicated feedback tools. Each dimension is scored on a 0-to-20 interval using explicit formulas combining breadth, capped quantity and diversity terms, producing a total readiness score that can be normalized to a 100-point scale divided into Low, Moderate, High and Very High bands.

The most methodologically novel element of the study is its validation protocol. Rather than convening a conventional Delphi panel of human experts, the team introduced the Multi-LLM Synthetic Expert Consensus, or MLSEC, procedure: a pre-registered, two-round content-validity exercise conducted with a stratified panel of 40 synthetic experts, generated by pairing eight large language models from eight different providers with five expert personas, including an adaptive-learning researcher, an instructional designer, a psychometrician, a Moodle developer and an AI-in-education specialist. Items were rated on relevance, clarity, comprehensiveness and theoretical alignment, with falsifiable retention thresholds locked in advance: an item-level content validity index of at least 0.78, Aiken’s V of at least 0.70 and modified kappa of at least 0.74. A deliberately off-topic poison-pill item, concerning font sizes in PowerPoint files, served as a quality-control audit, and the panel unanimously rejected it, demonstrating that synthetic raters did not endorse items uncritically.

The synthetic panel did more than ratify the researchers’ initial design. In free-text responses, 33 of 38 panelists, spanning all eight base models, independently proposed a sixth dimension belonging to the artificial intelligence and learning-analytics family. The team formalized this as AI and Data-Driven Adaptivity Integration, or ADAI, which scores whether a course contains the infrastructure needed to plug into the contemporary adaptive ecosystem: LTI external tool gateways, learner-data collection tools, and SCORM or xAPI-compatible content. The new dimension achieved a perfect content validity index of 1.00 for its definition and component mapping in the second round. A runner-up candidate on learner agency and self-regulation attracted only three endorsements and was rejected, confirming that the pre-registered decision rules were genuinely falsifiable rather than rigged toward a predetermined outcome. The authors are careful to frame MLSEC as a transparent, ordinal-ranking validation step rather than a substitute for human expertise, explicitly citing the cautionary literature on synthetic respondents, including their tendency to under-represent variance relative to human samples.

When the validated six-dimensional framework was applied to 985 Moodle 3.8.2 courses delivered between 2020 and 2022, the institutional profile proved conservative. No course reached the Very High readiness band and only 14, or 1.4 percent, reached the High band, while roughly 54 percent fell in the Low band. The dimensional breakdown was even more revealing. Content Variety dominated with a mean of 10.76 out of 20, driven largely by structurally light resources such as URLs, pages and labels, yet only 4 percent of courses used all five common resource types. Feedback Mechanisms followed at 7.90, almost entirely on the strength of near-universal forum presence rather than richer channels. Assessment Flexibility was moderate at 5.90, Interaction Diversity weak at 3.57, while Learning Path Personalization averaged just 0.28 and ADAI an almost nonexistent 0.02. Only 22 of 985 courses used any branching or sequencing component, and just 7 contained any LTI external tool.

Perhaps the most counterintuitive finding concerns grades. The total readiness score correlated negatively with the proportion of high grades, Pearson r equal to minus 0.22, and positively with the proportion of failing grades, r equal to plus 0.22, both highly significant. The authors resist any suggestion that adaptive structures harm learning. Instead, they argue, the pattern confirms that ALRAF measures structural capability rather than pedagogical enactment: courses rich in Moodle scaffolding may assign more demanding, interactive work that spreads grade distributions, and instructors who invest in infrastructure may grade more strictly. The correlation, they contend, strengthens the case for treating the framework as a capability-surface index rather than a predictor of student success. A multiple regression controlling for educational level, form of education and faculty fixed effects explained 18 percent of variance in high-grade share, with Assessment Flexibility the only individually significant dimension, retaining the negative sign.

Disciplinary differences were robust. A one-way analysis of variance across nine faculties yielded F of 10.26 with a small-to-medium effect size, eta squared of 0.078. The faculties of Geography, Tourism and History, and Pedagogical Education led the readiness distribution, while the Faculty of Arts trailed consistently, a pattern the authors attribute partly to studio-based pedagogy that Moodle component counts intrinsically fail to capture. Robustness checks comparing the original five-dimensional score with the validated six-dimensional version showed rank-order correlations above 0.9 and band agreement in over 80 percent of courses, confirming that the substantive conclusions do not depend on the specific dimensional structure, even though the added ADAI dimension and quality-weighted scoring shift absolute values systematically downward.

The practical implications are direct. The near-zero Learning Path Personalization scores point to an urgent need for faculty development on conditional activities, restriction sets and the Lesson module, the structural prerequisites for branching pathways. The near-zero ADAI scores expose a widening institutional gap on the AI frontier, at precisely the moment Moodle’s newer releases support LTI-Advantage AI plugins, connections to OpenAI, Gemini and self-hosted models, and machine-learning-driven adaptive assessment. The authors recommend a phased implementation strategy that builds on existing strengths in content and feedback before tackling personalization and AI integration, alongside discipline-sensitive judgments about what reasonable readiness looks like in fields where pedagogy is not naturally mediated by the LMS. They caution that readiness scores should diagnose capability gaps, not forecast grades.

The study’s limitations are candidly enumerated: a single-institution sample, a single Moodle version, missing conditional-restriction metadata that forced a proxy measure, correlational rather than causal design, course-level grade aggregation, heteroskedastic regression residuals, and the inherent caveats of a synthetic expert panel. Future priorities include multi-institutional validation, cross-version testing against Moodle 4.x and 5.x, direct querying of restriction data through the Web Services API, and pairing structural scores with behavioral learning-analytics variables such as time on task and navigation paths. Within those bounds, the researchers deliver something the field has lacked: a transparent, replicable, theory-anchored index of where, exactly, an institution’s adaptive learning infrastructure stands, and where the next investment should go as artificial intelligence redraws the map of personalized education.

Subject of Research: A validated course-level framework for assessing the adaptive learning readiness of Moodle courses

Article Title: Development and validation of an adaptive learning readiness assessment framework for Moodle courses

Article References: Semerikov, S., Nechypurenko, P., Vakaliuk, T., Mintii, I., & Fadieieva, L. (2026). Development and validation of an adaptive learning readiness assessment framework for Moodle courses. Journal of New Approaches in Educational Research, 15(1), Article 19. https://doi.org/10.1007/s44322-026-00069-w

Image Credits: AI Generated

DOI: 10.1007/s44322-026-00069-w

Keywords: adaptive learning, Moodle, readiness assessment, higher education, large language models, synthetic expert panel, content validity, AI integration, personalized learning, learning management systems, Development, validation