The core idea behind AI-TEACH is what the researchers call a co-teacher paradigm. Rather than replacing educator judgment, the system acts as a silent, objective partner that augments it. Strategically positioned surveillance cameras with synchronized audio capture stream classroom data to an edge computing layer, where parallel pipelines analyze behavior and speech in near real time. On the video side, frames are preprocessed with OpenCV and passed through YOLO-NAS, a neural architecture search-optimized object detector that isolates each student in the room. A tracking algorithm called ByteTrack then assigns each detected student a persistent identity across frames, preserving the spatial-temporal coherence needed to study individual behavior over time. MediaPipe Pose extracts skeletal landmarks—nose, eyes, shoulders, wrists, hips—while MediaPipe’s facial model tracks 468 landmark points across the mouth, eyes, and jawline, providing the geometric substrate from which distraction cues are inferred.

From those landmarks, the system derives a structured taxonomy of behavioral distraction. A neck-tilt angle computed between the head axis and torso axis in three-dimensional skeletal space reveals slouching or simulated sleep, especially when combined with prolonged eye closure measured through the eye-aspect ratio and sustained stillness. Wrist-to-hip proximity paired with a downward gaze estimate flags mobile phone use. Rapid arm extension combined with a fast-moving optical flow contour, extracted via the Farnebäck method, identifies thrown objects. Exaggerated facial expressions are caught by measuring the standard deviation of facial landmark displacements across a sliding window of roughly ten frames, and an abrupt displacement of the hip keypoint toward a mapped door location signals a student bolting from class. Each detected incident is logged with a timestamp, an indicator type, and the responsible modality.

The audio pipeline complements the cameras by capturing the verbal disruptions that vision alone cannot see. Incoming sound is first segmented using Silero Voice Activity Detection, a lightweight neural model that separates speech from background noise with low latency. Detected speech segments are then passed through Wav2Vec2, a self-supervised speech model that produces rich acoustic embeddings, which a Support Vector Machine classifies into specific distraction categories: off-task back-talk occurring outside the teacher’s directional audio profile, loud or chaotic sound events such as shouting or whistling identified through amplitude envelope tracking and spectral flatness, rhythmic tapping detected via autocorrelation and short-time energy, and abusive or offensive language flagged by a toxicity classifier applied to transcribed utterances. The design consciously draws on recent advances in multimodal emotion recognition, including per-sample modality equilibration and adversarial fusion techniques, to keep weaker signals from being drowned out by dominant ones when audio and video streams arrive asynchronously.

Fusing these streams is the job of a Bidirectional Long Short-Term Memory network, a recurrent architecture that reads each multimodal feature sequence in both temporal directions, capturing escalating patterns of disruption that a frame-by-frame classifier would miss. A softmax layer over the pooled hidden states assigns each incident a distraction category, and every category carries an empirically calibrated severity weight reflecting its actual disruptive impact on the class—from a single slouching student to thrown objects or hostile verbal outbursts. Over the course of a session, the fog layer multiplies each indicator’s occurrence count by its severity score and sums the results into a single Class Distraction Score, alongside timestamped behavioral metadata. Only this structured metadata, not the raw recordings, is transmitted over encrypted channels to the cloud layer, where an interactive dashboard visualizes the score, breaks down each indicator with its frequency and severity, and plots a temporal heatmap of when distraction clustered during the lesson.

The engineering choices are deliberately pragmatic. The fog layer runs on edge hardware comparable to an NVIDIA Jetson Xavier NX, sustaining frame-level video analytics and concurrent audio processing at 25 to 30 frames per second, while the cloud tier requires only a server-class GPU and at least 10 Mbps of upstream bandwidth. Under these conditions the system achieves an average end-to-end latency of roughly 350 milliseconds from frame capture to logged distraction event, consuming about 15 watts at the edge during continuous operation. Teachers need no technical training beyond an estimated one to two hours of onboarding, because the system is designed for post-session reflection rather than in-the-moment alarms: educators review the dashboard, add contextual notes where needed, and adjust pacing, format, or grouping in subsequent lessons.

The validation combined benchmark testing with a live classroom experiment. The team assembled a curated dataset of more than 5,350 labeled classroom images drawn from three open-source repositories, spanning behaviors such as phone use, object throwing, slouching, yawning, gaze aversion, and head-on-desk posture. The video module achieved 90% overall accuracy and a weighted average recall of 92%, with a macro-average F1 score of 92% across nine evaluation rounds, indicating balanced performance on both frequent and rare distraction categories. The audio module performed comparably, with accuracy between 88% and 94% and a macro-average F1 of 92%, maintaining reliable classification despite overlapping speech and ambient noise. In head-to-head comparisons, AI-TEACH’s overall F1 score of 93.5% substantially outperformed conventional baselines, including a convolutional neural network at 74.14%, a standalone LSTM at 76.04%, HOG plus SVM at 80.52%, OpenPose plus SVM at 83.26%, and even a BiLSTM used alone at 85.95%—evidence that the advantage comes from the fusion of complementary modalities and severity weighting rather than any single component.

The pedagogical test was a controlled experiment with 200 students randomly assigned to experimental and control groups. Control classrooms received traditional instruction, while teachers in the experimental group worked with AI-TEACH’s real-time feedback. After baseline pretests, the experimental group gained an average of 9 points on post-tests compared with 5 points for the control group, and a two-sample t-test confirmed the difference was statistically significant at the 0.05 level. The calculated effect size, Cohen’s d of approximately 0.90, qualifies as large by conventional standards—a striking result for an intervention that changes nothing about the curriculum itself and everything about how quickly teachers can perceive and respond to disengagement. The finding aligns with earlier research showing that objectively measured engagement predicts academic performance better than self-reported distraction.

The authors are candid about the ethical terrain. Because several monitored behaviors—fidgeting, repetitive movement, gaze aversion—can be natural self-regulation strategies for neurodivergent students with autism or ADHD, the system is explicitly designed as a class-level tool rather than an individual diagnostic instrument, and future versions will let educators suppress specific indicators for particular students. Algorithmic bias across skin tones, body types, cultural behavioral norms, and languages remains a recognized limitation, as does the single-institution setting of the 200-student trial and the absence of physiological signals such as heart rate variability. Raw audio and video never leave the edge; only encrypted metadata is stored remotely under role-based access control, with informed consent and periodic bias audits recommended as deployment prerequisites. Framed against the United Nations Sustainable Development Goals, particularly SDG 4 on quality education, AI-TEACH represents a broader shift toward classrooms where attention itself becomes measurable, and where teachers—armed with evidence instead of intuition—can reach drifting students before the drifting becomes permanent.

Subject of Research: Multimodal AI-based real-time assessment of student distraction and engagement in classrooms

Article Title: Artificial Intelligence Based Framework for Student Engagement Assessment in Classroom Environments

Article References: Saini, M., Sharma, H., & Sengupta, E. (2026). Artificial Intelligence Based Framework for Student Engagement Assessment in Classroom Environments. Cognitive Computation, 18(1), Article 108. https://doi.org/10.1007/s12559-026-10629-z

Image Credits: AI Generated

DOI: 10.1007/s12559-026-10629-z

Keywords: artificial intelligence, classroom engagement, distraction detection, computer vision, speech recognition, BiLSTM, fog computing, education technology, multimodal fusion, smart education, SDG 4, student behavior