Real-time multimedia CPR training feedback using pose estimation and action recognition

A team of researchers in Taiwan has built an artificial intelligence system that watches people perform cardiopulmonary resuscitation and tells them, instantly and out loud, exactly what they are doing wrong. The system, described in a new study published in Multimedia Tools and Applications, combines human pose estimation with skeleton-based action recognition to deliver real-time audio-visual feedback during CPR training, and in a controlled trial with 60 participants it produced markedly better skill performance, lower cognitive load and higher learner satisfaction than conventional instruction alone.

Cardiac arrest remains one of the most time-critical emergencies in medicine. Survival depends overwhelmingly on what happens in the first minutes, and high-quality CPR, delivered with correct posture, depth and sequence, is one of the few interventions bystanders can provide before emergency services arrive. Yet training outcomes vary widely, and studies cited by the authors point to persistent challenges in nursing education and public awareness campaigns across Europe. Traditional CPR courses rely on instructors observing multiple trainees at once, which makes fine-grained, continuous correction of every compression and hand position practically impossible. The new system was designed to fill that gap by acting as a tireless, always-attentive digital assistant that evaluates every movement the moment it happens.

The technical core of the system is a two-stage pipeline. The first stage uses HRNet, a deep neural network architecture for human pose estimation that maintains high-resolution representations throughout its processing stages. Unlike architectures that downscale images and then attempt to recover spatial detail, HRNet preserves fine-grained information about body joints, which is essential for CPR, where subtle deviations in arm angle or shoulder alignment can mean the difference between effective and ineffective chest compressions. The network converts a standard camera feed into a skeletal representation of the trainee, a stick-figure abstraction that captures where the head, shoulders, elbows, wrists, hips and knees are in each frame.

The second stage takes those skeleton sequences and classifies what the trainee is actually doing. The researchers employed ST-GCN++, an improved version of the Spatio-Temporal Graph Convolutional Network, a class of models built specifically for skeleton-based action recognition. A human skeleton is naturally a graph: joints are nodes, and bones are edges. A graph convolutional network propagates information across that graph, learning patterns both in space, how the joints relate to each other in a single frame, and in time, how those relationships evolve across a sequence of frames. The “plus-plus” refinement adds improved backbone design and multi-stream inputs, boosting accuracy without requiring enormous computational resources.

To make the system work in real time, the team segmented the streaming video into short clips, classified the CPR actions on the fly, and compared the recognized action sequence against the standard CPR procedure. This is where the “process-aware” component comes in. The system does not merely ask whether a given movement resembles a chest compression; it tracks whether the actions are happening in the correct order, whether the posture matches the protocol, and whether the trainee is hesitating. When execution order or posture deviates from the standard procedure, the system immediately triggers audio-visual prompts, essentially coaching the learner in the moment rather than after the fact. The architecture is deliberately lightweight and runs on commodity hardware, a design decision that opens the door to deployment in schools, community centers, and remote or resource-constrained settings where expensive manikin sensor systems are unavailable.

The machine-learning performance figures are strong. Evaluated with randomized segment-level splits, the action classifier achieved an accuracy of 0.93 and a recall of 0.93, with a balanced accuracy of 0.91 and a macro-averaged F1-score of 0.91. The use of balanced accuracy and macro-averaged F1 indicates the team was attentive to class imbalance, ensuring the model performs well across all CPR action categories rather than excelling only on the most common ones. The researchers framed the entire work as a controlled feasibility study, and were careful to note that the model evaluation used segment-level randomization, a detail that matters because action-recognition models can sometimes exploit temporal redundancy between adjacent clips if data splitting is not handled properly.

The team then took the system out of the lab’s benchmark datasets and into a quasi-experimental training study. Sixty participants were divided into two groups of thirty. One group received traditional instructor-led CPR training; the other trained with the intelligent feedback system. The entire experiment, including participant briefing, a pre-test, instructor-led instruction, group-based practice, and post-test assessment, was completed over approximately eight hours. Performance after training was assessed by a blinded CPR instructor, who scored each participant on action sequence correctness and hesitation duration without knowing which group the participant had belonged to, a design choice that reduces evaluator bias.

The results were unambiguous. The group trained with the intelligent system significantly outperformed the traditional training group in skill execution, with a mean score of 89 versus 72, a difference that was highly statistically significant with p less than 0.001 and a large effect size, reflected in a rank-biserial correlation of 0.606. In practical terms, trainees who received instant AI feedback not only performed better but also hesitated less, suggesting the system helped them internalize the correct procedural sequence rather than forcing them to consciously recall it under pressure. The experimental group also reported lower cognitive load, measured with established psychological instruments, an outcome the authors link to the way immediate, automated feedback relieves learners of the burden of self-monitoring while performing a physically and mentally demanding task.

Usability and satisfaction metrics told a similar story. The system achieved a System Usability Scale score of 78.6, placing it in the upper range of the widely used 0-to-100 usability instrument, and participants expressed greater satisfaction with their training experience compared with the traditional group. The authors suggest that skeleton-based action recognition coupled with real-time multisensory feedback can support practical skill acquisition and learner confidence in a controlled feasibility setting, and that the reduction in cognitive load may be a key mechanism behind the improved learning outcomes, since overload during skill practice is known to impair retention.

The implications extend well beyond CPR. The study’s authors, Ming-Chuan Chiu and Zi-Heng Huang of National Tsing Hua University and Meng-Chun Kao of Yuanpei University of Medical Technology, note that the same skeleton-based approach could be extended, after further validation and runtime profiling, to other skill-based domains, from industrial assembly and physical rehabilitation to surgical technique and workplace safety training. Because the system needs only a camera and commodity computing hardware, it avoids the cost barrier of sensor-instrumented manikins, which could make high-quality, feedback-rich training accessible in low-resource environments and remote learning scenarios where certified instructors are scarce.

The research was supported by the National Science and Technology Council, Taiwan, and approved by the Research Ethics Review Committee of National Tsing Hua University. The team emphasizes that the work remains a feasibility study: larger samples, longer-term retention testing and real-world deployment studies will be needed before the system can be recommended as a replacement for, rather than a supplement to, certified CPR instruction. But the combination of near-human-level action recognition accuracy, measurable improvements in trainee performance, and a lightweight, hardware-agnostic design makes a compelling case that the next generation of life-saving training may not need a human eye on every compression, only a camera, a skeleton, and a graph neural network watching the bones move.