Visual Transformer Algorithm Scores Athletic Movement from Ordinary RGB Video

A new study reports a Visual Transformer-based algorithm that assigns quantitative technical scores to athletic movements using ordinary RGB video, addressing a persistent gap between action recognition and reliable movement evaluation. The work, published open access in Discover Artificial Intelligence by Chunyan Gong, Haiyang Zhang, and Lianghua Gong, targets the difference between recognizing what an athlete did and judging how well the movement was executed. The authors argue that existing visual recognition models capture global action patterns but often ignore subtle local deviations such as landing instability, trunk tilt, incomplete limb extension, and shoulder–elbow–wrist coordination errors, and that a single regression objective may not preserve grade-boundary discrimination. The proposed framework integrates temporal dependency enhancement, local detail attention, and joint regression–classification optimization to produce scores that are both precise and grade-consistent.

The system follows a four-level closed-loop pipeline running from video capture to score output. RGB cameras record motion at 1920 by 1080 resolution at frame rates between 25 and 60 frames per second, with raw clips cut to 6 to 10 seconds and 8 to 12-second action samples extracted. Preprocessing uniformly resamples sequences to 32 or 48 frames, converts them to 224 by 224 frames, and applies a sliding-window strategy with a temporal overlap of 0.25 to 0.50 to organize keyframe tokens. A composite saliency function combines inter-frame difference, joint displacement magnitude, and local feature variance to rank frames, and the top 8 or 12 frames are selected under saliency peaks and temporal spacing constraints. Before any sliding window is generated, each video is assigned an immutable anonymized athlete identifier, a source-video identifier, and a SHA-256 hash, and a group-stratified split with a fixed random seed of 2025 partitions the data by athlete identity, stratified by sport category and grade label. Every derived subclip inherits its parent partition label, and automated integrity checks confirmed zero repeated athlete identifiers, source-video identifiers, or file hashes across the training, validation, and test sets.

Feature extraction uses a Vision Transformer encoder of the ViT-B/16 type. Each frame is divided into 16 by 16 patches, producing 196 patch tokens per frame plus one learnable classification token, and linear projection maps each patch into a 768-dimensional embedding. Spatial and temporal position embeddings are added so the model can represent both where a posture deviation occurs on the body and when it occurs in the sequence. The encoder applies multi-head self-attention with 12 heads across 12 layers, and the final classification token of each frame is averaged to yield a global action feature vector. The backbone was initialized from ImageNet-21K pretrained weights, while the temporal modules and task-specific heads used Xavier initialization.

A Temporal Dependency Enhancement Module captures phase transitions and movement rhythm by decomposing each frame-level feature into three components: a global pose component, a motion-difference component computed between adjacent frames, and a second-order temporal curvature component reflecting rhythmic fluctuation. A sigmoid gate fuses these components into an intermediate representation, after which dependency attention with an exponential temporal decay term weights the contribution of nearby and distant frames. This design encodes action amplitude variation, smooth phase transitions, and the location of rhythmic peaks, capabilities that matter most in multi-phase movements such as the run-up, takeoff, flight, and landing of track-and-field jumps.

A Local Detail Attention Module focuses on fine-grained posture deviations. A 17-keypoint human-pose estimator is applied to every frame, and three body regions are defined: an upper-limb mask from shoulder, elbow, and wrist keypoints, a trunk mask from the shoulder–hip quadrilateral, and a lower-limb mask from hip, knee, ankle, and supporting-foot keypoints. When keypoint confidence falls below 0.50 because of rapid movement or self-occlusion, masks from the previous frame are warped forward by an affine transformation estimated with RANSAC, with propagation limited to three consecutive frames. Each mask is projected onto a 14 by 14 patch grid, and region-level attention compares pooled features against a class-wise prototype built from high-score training samples, with a normative offset derived from the local pose distance. This mechanism directs the model’s attention to the body regions where technical errors actually occur, such as foot placement on the balance beam or racquet-arm angles during a serve.

The scoring head jointly optimizes continuous regression and grade classification. Each of the 3160 videos was independently scored by three certified coaches or judges with more than five years of professional experience, and inter-rater reliability was verified using a two-way random-effects, absolute-agreement, average-measures intraclass correlation coefficient for the 0–100 scores and quadratic-weighted Cohen’s kappa for the ordinal grades before the ratings were averaged. The label system weights four dimensions—motion normativity, completion, stability, and rhythm consistency—at 0.40, 0.25, 0.20, and 0.15, values tuned on the validation set and explicitly not presented as universal judging standards. The composite score maps to five grades from A (90–100) to E (0–59), and the joint loss combines mean squared error for the continuous score, cross-entropy for grade classification, and a ranking-consistency term applied to pairs within the same action subcategory whose true scores differ by more than 5 points.

The dataset comprised 1200 gymnastics videos, 960 track-and-field jumping videos, and 1000 racquet-sport videos covering badminton forehand clears and tennis serves, split 8:1:1 into 2528 training, 316 validation, and 316 held-out test videos. On the held-out test set, the full model achieved a root mean square error of 3.49, a mean absolute error of 2.72, a Grade Accuracy of 84.0 percent, and a Spearman rank correlation of 0.908. Under the same split and local reimplementation protocol, these figures represent modest improvements over included action-recognition and video-Transformer baselines, with overall RMSE decreasing from 3.57 for ViViT-B/16 and Grade Accuracy rising from 83.2 to 84.0 percent. Racquet sports proved easiest to score, with an RMSE of 3.21 and Grade Accuracy of 86.3 percent, while track-and-field jumping remained hardest at 3.85, which the authors attribute to the limits of RGB video in representing takeoff force, flight posture, and landing-buffer information.

Ablation experiments quantify each module’s contribution. Removing the Temporal Dependency Enhancement Module raised the overall RMSE from 3.49 to 4.28, an increase of 0.79 points, although a category-level paired test across the three sport groups yielded a 95 percent confidence interval of −0.03 to 1.55 points and did not reach the conventional 0.05 significance level. Removing the Local Detail Attention Module caused the largest deterioration, with RMSE rising to 4.41 and Grade Accuracy falling to 76.9 percent, a 0.92-point RMSE increase. Removing the joint optimization loss raised RMSE to 3.97 and cut Grade Accuracy to 80.5 percent, an increase of 0.48 points supported by a statistically significant category-level test with a confidence interval of 0.23 to 0.69 points. Stratified error analysis showed that severe occlusion, oblique viewpoints, high-speed actions, and borderline scores between 60 and 79 points all inflated errors, with the borderline band showing the lowest Grade Accuracy at 78.2 percent.

An interpretability analysis compared the model’s regional attention with expert-annotated fault regions across all 316 test videos. In the high-score range, the model weighted the lower-limb region most heavily at 37.2 percent, while in the low-score range upper-limb attention rose to 42.7 percent. Mean spatial overlap between the top 20 percent of model attention and expert-consensus masks reached 81.3 percent for the lower-limb region in gymnastics and 76.9 percent for the upper-limb region in racquet sports, but only 58.7 percent for upper limbs in track-and-field jumping, a value the authors identify as a limitation of the current visual explanation rather than reliable biomechanical localization. The authors emphasize several constraints on their claims: the results support scoring potential only under the current dataset, annotation protocol, and RGB-video setting and do not establish cross-dataset or cross-sport generalizability. On a single NVIDIA A100 GPU, inference took 36.8 milliseconds per 32-frame clip, a hardware-specific benchmark positioned for offline scoring and post-training analysis rather than unrestricted real-time deployment, and future work should add multi-view video, skeleton sequences, kinetic priors, and external validation against official judging records.

Subject of Research: Visual Transformer-based automatic scoring of sports motion quality from RGB video

Article Title: Sports motion scoring algorithms using visual transformers

Article References: Gong, C., Zhang, H., & Gong, L. (2026). Sports motion scoring algorithms using visual transformers. Discover Artificial Intelligence, 6(1), Article 1617. https://doi.org/10.1007/s44163-026-02345-2

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02345-2

Keywords: Vision Transformer, sports motion scoring, RGB video, temporal dependency enhancement, local detail attention, joint optimization, action recognition, movement quality assessment, gymnastics, track-and-field jumping, racquet sports, machine learning