The field of human activity recognition, or HAR, has evolved dramatically since its beginnings in the early 2000s, when researchers first strapped accelerometers to volunteers and applied classical algorithms such as decision trees and k-nearest neighbors to the resulting signals. Wearable sensor systems offer privacy and continuous tracking but often struggle with limited contextual information and sensitivity to sensor placement. Camera-based systems, by contrast, capture rich visual detail through RGB video, depth maps, and skeletal data, achieving high accuracy at the cost of privacy concerns, fixed fields of view, and vulnerability to lighting and background changes. Each modality has its own blind spots: skeleton data encodes human motion but omits the objects a person interacts with, while RGB captures both the person and the scene but falters when illumination shifts. Combining modalities has therefore become the dominant strategy for building systems that can cope with the messiness of the real world.

Deep learning accelerated this trend considerably. Convolutional and recurrent architectures, including CNN-LSTM hybrids, automated feature extraction and pushed recognition accuracy well beyond what handcrafted statistical features could achieve. Yet as datasets grew in scale and diversity, these models revealed intrinsic limitations. Their narrow receptive fields and strictly sequential processing make it difficult to capture long-range temporal dependencies and the complex interactions between modalities. Recognition accuracy, studies have shown, increases substantially with longer context windows, which is precisely where recurrent networks struggle. Transformers, with their self-attention mechanism, have emerged as the natural successor, dynamically focusing on the most salient features across entire sequences and processing them in parallel rather than one step at a time.

The new work builds on this Transformer backbone but attacks a different part of the pipeline: the loss function that guides training. The authors’ multimodal architecture processes four data streams—RGB video, depth maps, Kinect skeletal coordinates, and inertial sensor readings—through modality-specific encoders. Visual streams pass through 3D ResNet-18 backbones that extract spatiotemporal features, skeletal data flows through a graph convolutional network that respects the joint-and-bone structure of the human body, and inertial signals are handled by an LSTM. The resulting features are projected into a common 128-dimensional embedding space, enriched with learnable positional encodings, and fused by a Transformer encoder with two layers and four attention heads before a classification layer produces the final activity prediction.

The innovation lies in what happens during optimization. Standard training treats all parameters as free vectors updated by gradient descent in flat space. The proposed Riemannian loss adds a regularization term that measures the gradient of the cross-entropy loss with respect to a metric tensor, encouraging the model’s fusion projection matrices and classification weights to remain on a Stiefel manifold—the set of matrices with orthonormal columns. In practice, the Euclidean gradient computed by backpropagation is projected onto the tangent space of the manifold, and parameters are updated via a QR-based retraction that keeps them feasible. Only about three percent of the network’s parameters, those in the fusion and classification layers, are constrained this way; everything else trains with standard Adam. This two-track scheme keeps the method compatible with ordinary deep learning frameworks and requires no second-order gradient computation.

The empirical results are striking. On the UCI-HAR smartphone dataset, the Transformer’s accuracy rose from 97.75 percent with a standard loss to 99.80 percent with the Riemannian loss. On the more challenging, class-imbalanced Opportunity dataset, accuracy climbed from 93.01 to 95.10 percent. On the multimodal UTD-MHAD dataset, which contains synchronized RGB, depth, skeleton, and inertial recordings of 27 actions performed by eight subjects, fused multimodal accuracy improved from 98.84 to 99.42 percent. Notably, among single modalities, depth data achieved the highest unimodal accuracy at 97.68 percent while inertial data lagged at 85.55 percent, underscoring how much complementary information the fusion stage has to work with—and how much the geometric regularizer helps exploit it.

Crucially, the authors went beyond the favorable evaluation protocols that can inflate reported accuracy. Under stratified random splits, windows from the same person can appear in both training and test sets, allowing the model to exploit within-subject regularities in posture and gait. When the researchers repeated their experiments with subject-independent splits, in which test subjects are entirely unseen during training, accuracy dropped as expected—but the Riemannian loss preserved a clear advantage, holding 95.18 percent versus 92.03 percent for the standard loss. An eight-fold leave-one-subject-out cross-validation confirmed the pattern, with mean accuracies of 94.9 plus or minus 0.6 percent for the proposed method against 92.1 plus or minus 0.8 percent for the baseline. In zero-shot cross-dataset transfer from UTD-MHAD to the NTU-RGB+D 120 benchmark, the Riemannian model reached 83.9 percent against 78.6 percent, and on free-living inertial data from the RealWorld-HAR dataset it achieved 86.4 percent F1, outperforming a CNN-LSTM baseline at 82.7 percent.

Statistical rigor backs the headline numbers. Across ten random seeds on a fixed stratified partition, the Riemannian variant averaged 99.04 plus or minus 1.11 percent accuracy, beating both the standard cross-entropy baseline at 97.77 plus or minus 1.31 percent and an L2-regularized variant at 96.91 plus or minus 1.74 percent. One-sided Wilcoxon signed-rank tests yielded p-values of 0.002 and below 0.001 with large effect sizes, and the results survived Holm correction. An ablation study reinforced the point: swapping the manifold-aware term for a conventional L2 penalty dropped accuracy from 99.42 to 99.08 percent, showing that explicitly respecting the geometry of the data provides regularization that flat-space penalties cannot match. Reducing the Transformer’s depth or attention width also degraded performance, confirming that both architectural capacity and geometric constraints matter.

Perhaps the most surprising demonstration of generality came from an appendix experiment with GPT-2, an autoregressive language model with a fundamentally different architecture from the multimodal Transformer. When fine-tuned on sensor windows converted for sequence modeling, GPT-2’s accuracy on UCI-HAR rose from 84.03 to 87.60 percent with the Riemannian loss, and on UTD-MHAD multimodal fusion jumped from 78.49 to 85.67 percent. The loss function, in other words, is model-agnostic: geometric regularization helps whether the backbone is an encoder-style Transformer or a generative language model, hinting at a broader role for manifold-aware optimization as large language models enter the activity recognition arena.

Deployment considerations matter just as much as accuracy for real-world systems, and here the news is also good. Because only the small fusion matrices live on the manifold, the computational overhead is modest: training throughput on an RTX A6000 dropped about 11 percent relative to Adam, with under one percent extra memory, yet the Riemannian optimizer reached 95 percent of its final accuracy in 25 percent fewer epochs, keeping wall-clock training time comparable. On edge hardware, the isolated Transformer blocks run at 54 frames per second on a Jetson Xavier NX and sustain 32 frames per second on a Jetson Orin Nano with INT8 quantization, comfortably meeting the 25 to 30 frames per second of typical camera streams. The authors envision applications ranging from fall detection and routine monitoring for elderly care to workplace safety and rehabilitation feedback, while cautioning that fully unconstrained in-the-wild deployments—with occluded views, adversarial lighting, and shifting domains—will demand stronger adaptation techniques. For now, the message is clear: sometimes the shortest path to better AI is not a straight line at all, but a geodesic.

Subject of Research: Riemannian optimization for multimodal human activity recognition with Transformers

Article Title: Activity recognition via Riemannian optimization and multimodal Transformers

Article References: Soleimani, F., Khodabandelou, G., Chibani, A., & Amirat, Y. (2026). Activity recognition via Riemannian optimization and multimodal Transformers. Machine Learning with Applications, 26, Article 101021. https://doi.org/10.1016/j.mlwa.2026.101021

Image Credits: AI Generated

DOI: 10.1016/j.mlwa.2026.101021

Keywords: human activity recognition, Riemannian optimization, Transformers, multimodal fusion, deep learning, wearable sensors, Stiefel manifold, GPT-2, edge computing, machine learning, sensor data, loss function