Timing beats volume: AI study shows how learners align with course cues

When students log into an online course, the most revealing information may not be how much they click, but when. A new study published in Discover Artificial Intelligence argues that the temporal rhythm of learner behaviour — how quickly and consistently students respond to machine-generated events such as assessment releases and feedback notifications — offers a more meaningful signal of engagement than raw activity counts. Drawing on interaction logs from 10,452 learners on a large-scale online platform, the researchers modelled what they call behavioural responsiveness and found that timing-based features outperformed traditional engagement metrics in distinguishing learners who stayed aligned with course structure from those who drifted toward withdrawal.

The study, led by Patrick O. Akinwumi, Meihua Qian and Oyinkansola A. Babatope, is grounded in a long-standing puzzle from the learning sciences: students in self-regulated online settings frequently miss crucial cues, including feedback, pacing misalignments and emerging performance trends. The theory of noticing holds that learning is shaped not merely by what students experience but by what they attend to and act upon, yet novices often lack the developed schemas needed to recognise relevant signals. Rather than trying to measure that internal, metacognitive process directly, the team took a deliberately conservative approach, operationalising noticing as a strictly behavioural proxy: whether a learner clicked at least once within 24 hours of a machine-generated pedagogical event.

To capture this responsiveness at scale, the researchers engineered a suite of novel temporal features anchored to precise system timestamps. Among them was the Human–Machine Collaboration Index, or HMCI, defined as the proportion of all of a learner’s clicks occurring within three days of system-generated events. Higher values indicate stronger temporal alignment with pedagogical structure. The team also computed delta clicks, the change in activity in a symmetric window around each event; response lag, the time between an event and the learner’s next interaction; burstiness, a measure of whether activity is clustered or evenly paced; volatility, the standard deviation of weekly click counts; and Shannon entropy across activity types, capturing attentional diversity.

The data came from the Open University Learning Analytics Dataset, a publicly available collection of demographic, registration, assessment and virtual learning environment interaction records from more than 30,000 students. The researchers included only learners with complete assessment records, at least five distinct logged interactions and enrolment in one of seven modules with consistent event structures, yielding the final analytic sample of 10,452. All data were fully anonymised before release, and the study involved no collection of new human data. Sensitivity analyses testing alternative response windows of 12, 48 and 72 hours confirmed that the 24-hour threshold best separated responsive from non-responsive learners.

Four machine learning models — logistic regression, random forest, XGBoost and a feedforward neural network — were trained on two binary outcomes: course withdrawal, serving as a proxy for disengagement, and behavioural responsiveness, the trace-based proxy for noticing. Class distributions were imbalanced, with roughly 31 percent of learners withdrawing and about 18 percent classified as behaviourally responsive, so the team applied class weighting rather than resampling. Each model underwent fivefold stratified cross-validation with hyperparameter tuning, and performance was evaluated using accuracy, precision, recall, F1 and AUC metrics.

XGBoost delivered the strongest results, achieving an AUC of 0.857 for withdrawal and 0.997 for behavioural responsiveness, while the random forest reached an AUC of 1.000 on the responsiveness task. The neural network was competitive at 0.967, and logistic regression offered interpretability with somewhat lower discrimination. But the authors are emphatic that these figures should not be read as evidence of deployable predictive power. Because several input features, such as release_delta_mean and HMCI, are conceptually close to the outcome definition, the high scores partly reflect construct overlap rather than generalisable foresight. The models, the team stresses, are diagnostic validators of the internal coherence of the behavioural proxy, not real-time prediction tools.

Interpretability analyses using SHAP values and permutation importance revealed a striking asymmetry between the two outcomes. For withdrawal, academic achievement variables — weighted assessment scores, number of assessments attempted and average score — dominated the feature rankings, consistent with prior findings that performance is a strong dropout predictor. Yet temporal features also contributed: high burstiness and low early engagement were associated with increased dropout risk, and lower HMCI values were linked to higher withdrawal risk, reflecting reduced alignment with course pacing. For behavioural responsiveness, by contrast, the strongest features were micro-temporal — mean reaction timing to assessment releases and score postings — while traditional engagement measures such as total clicks and active days contributed minimally.

Taken together, the results sketch a latent behavioural continuum anchored by temporal coordination. At one pole, withdrawal is characterised by declining academic performance, irregular engagement and weak alignment with system events. At the other, behavioural responsiveness reflects strong temporal alignment, rapid changes in click activity around events and proactive reactions to feedback cues. Learners with stable, rhythmically coordinated behaviour appear engaged in sustained self-regulation, while erratic, delayed responders show patterns the authors interpret as consistent with motivational breakdown or cognitive overload. The opposing directions of core features such as HMCI and burstiness across the two outcomes support this continuum framework.

The conceptual contribution lies in approximating one measurable dimension of what has been theorised as human–AI co-agency. The authors are careful not to overclaim: HMCI captures one-directional temporal coordination with fixed pedagogical events, not interpretive or dialogic partnership, and click data cannot reveal learners’ cognitive states, perceptions or intent. Still, the finding that reactive timing, rather than activity volume, best distinguishes responsive learners challenges the aggregate-usage assumptions embedded in many learning analytics dashboards, which often present static summaries and implicitly equate more activity with deeper engagement. Systems such as OnTask and SARA deliver timely nudges, the paper notes, but rarely capture the behavioural shifts that follow those cues.

The authors acknowledge significant limitations. Beyond construct circularity, generalisability may be constrained by reliance on a single platform’s data and limited contextual variables, and near-perfect classification scores raise concerns about overfitting or conceptual tautology. Future work, they suggest, should integrate qualitative data such as self-reports and think-aloud protocols to clarify how behavioural timing relates to metacognitive engagement, extend the framework to synchronous and collaborative environments, and evaluate the ethics of algorithmic nudging and feedback timing. For now, the study offers designers of adaptive educational AI a concrete, interpretable target: not who interacts the most, but who recognises and responds to the system fastest and most adaptively — a shift that could help educators identify when learners are falling out of step with course pacing before they disengage entirely.

Subject of Research: Modelling learner behavioural responsiveness to machine-generated pedagogical events in online learning using temporal interaction features

Article Title: Tracing learner responsiveness through temporal interaction patterns

Article References: Akinwumi, P. O., Qian, M., & Babatope, O. A. (2026). Tracing learner responsiveness through temporal interaction patterns. Discover Artificial Intelligence, 6(1), Article 949. https://doi.org/10.1007/s44163-026-01917-6

Image Credits: AI Generated

DOI: 10.1007/s44163-026-01917-6

Keywords: learning analytics, educational AI, behavioural responsiveness, Human-Machine Collaboration Index, self-regulated learning, machine learning, OULAD, withdrawal prediction, temporal alignment, SHAP interpretability, adaptive feedback, noticing