New Benchmark Exposes Inflated Scores in Robot Autism Therapy AI

Social robots are increasingly finding a place in therapy rooms for children with autism, delivering predictable, repetitive and motivating interactions that many children find easier to navigate than human-only sessions. But the artificial intelligence that allows these robots to sense whether a child is actually engaged has a hidden problem: much of the published performance data may be quietly inflated. A new benchmark dataset called QAMQOR, described in Frontiers in Robotics and AI, shows that the way researchers split their data can shift reported results more than the choice of algorithm itself, and in some cases by roughly twice as much.

The dataset was collected during real robot-assisted autism therapy sessions with 34 children with Autism Spectrum Disorder aged 3 to 12 years, in cooperation with a rehabilitation center. Over 187 therapy sessions totaling 47 hours of video, children interacted with the humanoid NAO robot while a therapist provided guidance. Sessions included dances, songs, storytelling, tactile games called Touch Me, imitation exercises, emotion recognition activities, and social acts such as handshaking and hugging. Each session lasted roughly 10 to 20 minutes, and a wide-angle camera recorded everything for later annotation.

To turn those recordings into machine learning data, the team built a six-step pipeline: extracting frames at one frame per second, detecting and cropping the child with a YOLOv3-based detector, discarding frames without the child, assigning engagement labels, parsing poses with two pose-estimation frameworks, and compiling the results into a structured dataset. OpenPose contributed 70 face and 25 body keypoints per frame, while MediaPipe Holistic extracted 468 face, 33 pose and 42 hand keypoints. The one-frame-per-second sampling is a deliberate design choice: engagement is a behavioral state that persists over seconds, so denser sampling would mostly produce redundant frames while inflating storage and computation.

Annotation followed a Likert-style coding scheme adapted from earlier studies, compressed to five levels. Level 1 marks non-compliance, where a child ignores the therapist entirely; level 5 describes spontaneous interaction with the robot and completed tasks. Three trained annotators labeled every second of footage, and their agreement on a shared subset, measured with an intraclass correlation coefficient, indicated good reliability. Labels were assigned at one-second intervals rather than fixed segments to capture the rapid behavioral shifts common in autistic children. The resulting labels are naturally imbalanced, with higher engagement occurring far more often than disengagement, and the team deliberately preserved that clinical distribution in the benchmark.

The core innovation lies not in the data alone but in the four evaluation protocols released alongside it. The Random split, standard in prior work, shuffles all samples freely, meaning near-identical frames from the same child and even the same session can land in both training and test sets. The Child-Independent split holds out entire children, forcing models to generalize to people they have never seen. The Session-Independent split holds out whole therapy sessions, and the Activity-Independent split tests whether models trained on some therapeutic activities transfer to others. QAMQOR is, according to its authors, the first robot-assisted autism therapy dataset to ship with predefined, leakage-controlled splits.

The benchmarking exercise covered seven models, from classical classifiers such as XGBoost, LightGBM, CatBoost and Logistic Regression to a Temporal XGBoost variant, a Long Short-Term Memory network and a Temporal Convolutional Network, all evaluated under identical windows, protocols and untuned hyperparameters across 336 single-partition experiments. The headline result is stark: in the multi-class task, mean macro-F1 varied by 0.12 across the four protocols but only 0.07 across the discriminative classifiers. In other words, evaluation design moved the needle roughly twice as far as algorithm choice. In the binary task, the two effects were comparable at 0.07 each. The Child-Independent split proved substantially harder than the Random split, and performance under it often collapsed toward the majority-class baseline.

The deep learning results were humbling. Under matched, untuned conditions, both the LSTM and the more recent Temporal Convolutional Network converged toward predicting the dominant class in several settings, including complete collapse under the Random split across all six visual modalities. Critically, the team ruled out class imbalance as the sole explanation: when every model received identical inverse-frequency class weighting, all imbalance-sensitive metrics improved, but the model ranking stayed exactly the same, with the balanced tree ensembles still on top. The authors stress this reflects the difficulty of training deep sequence models on small, imbalanced clinical datasets with heterogeneous participants, and should be read as a lower bound rather than a verdict on the architectures themselves.

Gradient-boosted trees emerged as the strongest family, with XGBoost and Temporal XGBoost statistically indistinguishable at the top, though the margins were modest. Interestingly, temporal aggregation of pose statistics over 10-frame windows helped most precisely when models faced unseen children, sessions or activities, while offering no advantage on the leak-prone Random split. Modality mattered too: body keypoints outperformed facial features for fine-grained engagement classification, suggesting that posture, movement initiation and gross-motor responses carry more diagnostic signal than facial cues alone. MediaPipe features edged out OpenPose for binary recognition, but the two tools were equivalent for multi-class work once the struggling deep baselines were excluded.

The practical implications reach beyond academia. A robot that appears 70 percent accurate under random splitting may perform barely better than chance when facing a new child in a clinic, so current keypoint-based classifiers are not yet reliable enough to autonomously adapt therapy for unseen children. The authors recommend that Child-Independent evaluation become a required reporting standard, that imbalance-aware metrics accompany any accuracy figure, and that Random split results never be published in isolation. They also note that lightweight models remain preferable in this data regime: LightGBM trained in about 26 seconds on a CPU, roughly one-seventh of the time the TCN required, while matching or beating it on every metric.

Limitations remain, and the team is candid about them. The cohort of 34 children, skewed toward boys, is small for modern deep learning, single-site data from one robot platform limits generalizability, and the one-fps sampling may miss fast behavioral cues. Future work points toward larger multi-site cohorts, audio and physiological modalities, transformers and graph neural networks over skeletons, and cross-child domain adaptation. For now, QAMQOR offers the field something it has lacked: a public, reproducible reference point where future engagement recognition methods can be compared on equal terms, and a vivid demonstration that honest evaluation, not fancier algorithms, is the fastest route to robots that genuinely understand the children they work with.

Subject of Research: A benchmark dataset and evaluation protocols for engagement recognition in robot-assisted autism therapy

Article Title: QAMQOR: a benchmark dataset for engagement recognition in robot-assisted autism therapy

Article References: Rakhymbayeva, N., Tungatarova, A., Telisheva, Z., & Sandygulova, A. (2026). QAMQOR: a benchmark dataset for engagement recognition in robot-assisted autism therapy. Frontiers in Robotics and AI, 13, Article 1868211. https://doi.org/10.3389/frobt.2026.1868211

Image Credits: AI Generated

DOI: 10.3389/frobt.2026.1868211

Keywords: robot-assisted autism therapy, engagement recognition, QAMQOR dataset, child-robot interaction, machine learning benchmarks, data leakage, pose estimation, temporal convolutional network, XGBoost, class imbalance, autism spectrum disorder, human-robot interaction