Rule-based AI chatbot associated with higher short-term sustainability test scores in Nigerian university study

A three-week quasi-experimental study at a Nigerian public technical university found that students using the rule-based educational chatbot “Ovayo” achieved significantly higher post-test scores in sustainability concepts than peers receiving traditional instruction or no additional intervention. The result was associated with the chatbot-supported condition rather than established as a causal effect, because intact course sections were not individually randomized.

The study included 120 purposively selected first-year students from 29 departments enrolled in the compulsory Environment and Sustainable Development course, GST 005. Of the participants, 57 were male and 63 were female. The researchers assigned three pre-existing course sections of 40 students each to Ovayo-supported learning, traditional lecture-based instruction, or a control condition with no additional intervention. The quantitative phase used a pre-test–post-test comparison design, while a qualitative phase examined students’ perceptions through semi-structured interviews.

All participants completed a pre-test in the first week to assess their prior knowledge of sustainability concepts. During the second week, the chatbot group completed modules on topics including climate change, energy conservation, renewable energy transitions, waste management, and sustainable consumption through Ovayo. The traditional-instruction group received lectures, textbooks, static presentations, and worksheets on the same general topics, while the control group received no additional intervention for that week. The post-test was administered in the third week, and longer-term retention was not assessed.

Ovayo was not a generative large-language-model system. It used a web-accessible, predefined decision-tree architecture in which learner inputs were matched to keyword clusters and semantic patterns. Diagnostic responses at the start of each module selected among three explanatory tiers: foundational clarification, intermediate elaboration, or advanced integrative extension. All instructional responses were pre-authored, and the system adjusted follow-up prompts and explanatory depth according to the conceptual specificity of students’ answers rather than through autonomous learner modeling.

Conceptual knowledge was measured with a researcher-developed Sustainability Achievement Test containing 20 multiple-choice and short-answer questions. Subject-matter experts reviewed the items for content validity, and the study reported a Cronbach’s alpha of 0.866 and a Kaiser–Meyer–Olkin coefficient of 0.903. Statistical analysis included a one-way analysis of variance, Tukey’s HSD post-hoc comparisons, and calculation of eta squared. The three groups did not differ significantly at baseline, F(2,117) = 1.347, p = 0.264.

Post-test scores differed significantly across the three instructional conditions, F(2,117) = 78.241, p < 0.001. The reported effect size was large, with eta squared equal to 0.57, meaning that approximately 57% of the variance in post-test scores was associated with instructional condition in the study analysis. Students in the Ovayo group scored significantly higher than both the traditional-instruction and control groups. Traditional instruction and the control condition did not differ significantly from each other. These assessments measured short-term conceptual understanding rather than sustainability attitudes, behavior, ethical reasoning, or long-term knowledge retention.

Twenty students from the chatbot group were selected purposively for one-on-one interviews lasting approximately 30 to 40 minutes. Thematic analysis identified generally positive experiences with conversational interaction, responsiveness to learners’ needs, clarity of feedback, intuitive communication, perceived personalization, and conceptual clarification. Students also proposed adding multimedia resources, expanding topic coverage, and improving the chatbot’s handling of complex or abstract questions. These self-reported experiences provide qualitative context for the score differences, but the study did not measure engagement as a formal construct or directly test which mechanism produced the gains.

Although interaction logs confirmed use and completion of assigned modules, the researchers did not retain detailed pathway-transition data for quantitative analysis. Evidence that students received differentiated instruction therefore came from the documented rule-based design and students’ reports rather than reported activation frequencies for the three explanatory tiers. The authors also noted that novelty could partly account for favorable engagement perceptions. Interviews included only chatbot users, preventing comparative analysis of how students in the other instructional conditions experienced the learning process.

The principal limitation is the use of intact sections without individual random assignment, which permits association but not definitive causal inference even though baseline scores were statistically equivalent. Unmeasured contextual factors may have influenced outcomes. The three-week design, immediate post-test, purposive single-institution sample, and reliance on a researcher-developed test also limit generalization and leave retention and transfer untested. The findings therefore support Ovayo’s association with better short-term test performance within this structured course, but they do not establish long-term learning gains, comprehensive sustainability competence, or a general benefit from replacing educators with AI chatbots.

Subject of Research: A quasi-experimental evaluation of a rule-based educational chatbot for short-term sustainability learning in first-year university students

Article Title: Cultivating a sustainable future through an interactive AI chatbot for personalized learning of sustainability concepts

Article References: Ade-Ibijola, A., Olugbade, D., & Oyelere, S. S. (2026). Cultivating a sustainable future through an interactive AI chatbot for personalized learning of sustainability concepts. Discover Education, 5(1), Article 925. https://doi.org/10.1007/s44217-026-02129-x

Image Credits: AI Generated

DOI: 10.1007/s44217-026-02129-x

Keywords: sustainability education, educational chatbots, personalized learning, higher education, Nigeria, conceptual learning, quasi-experiment, rule-based AI, formative feedback, Cultivating, sustainable, future