EgoTokens: Language Models Learn to Rebuild 3D Human Bodies From Head-Mounted Cameras

Imagine strapping on a virtual reality headset, glancing down at your own body, and having a computer instantly rebuild your entire form in three dimensions — hands, face, posture and all — simply by reading a single frame of video. Now imagine typing “make the belly fatter” or “give the person shorter legs” and watching that digital double reshape itself on command. That is the ambition behind EgoTokens, a new framework from researchers at the University of Hull and collaborating institutions, published in the journal Machine Learning with Applications, that turns a vision-language model into a surprisingly capable 3D human body artist.

The problem the team attacked is one of the most stubborn in computer vision: reconstructing a full 3D human body from a first-person camera. Head-mounted devices see the world from a perspective no studio camera ever captures — steep downward angles, extreme self-occlusion, and constant scale changes as limbs swing toward and away from the lens. When you look down at yourself, your own torso blocks your legs; your hands appear and vanish; body parts blur together in ways that make standard 3D estimation techniques falter. Yet this egocentric viewpoint is exactly what healthcare monitoring, robotics, and augmented reality increasingly demand, because machines need to understand human behavior from the observer’s own visual perspective.

Existing systems typically handle this by regressing the parameters of SMPL-X, a widely used statistical body model that encodes pose, shape, hand articulation, facial expression, jaw and eye orientation in a high-dimensional parameter space. A single dense regression head predicts all these continuous values at once, but the parameters are strongly coupled through the body model’s kinematics, and the model receives little structural guidance about how the components depend on each other. Under occlusion, truncation, and unusual viewpoints, such approaches rely heavily on broad training coverage and often produce implausible results — joints that hyperextend, limbs that drift, or bodies that penetrate themselves.

The EgoTokens team, led by Gulraiz Khan and Maneeha Rani with Kevin Pimbblet and Kenneth Y. Wertheim among the authors, took a radically different route. Instead of regressing continuous numbers, they reformulated the entire reconstruction task as sequence generation — the same paradigm that powers modern chatbots. Each of the 122 SMPL-X parameter dimensions is quantized into one of 1000 discrete bins, and each bin becomes a dedicated token in an extended vocabulary. The model is then trained to autoregressively generate a 122-token parameter sequence from an image-text prompt, treating the human body as a sentence to be written rather than a set of numbers to be fitted.

This tokenization choice is more technically loaded than it first appears. The team deliberately avoided learned codebooks — the vector-quantization approach used by earlier systems such as VQ-VAE and VQ-HPS — because learned codebooks suffer from codebook collapse, where many entries go unused, and from error propagation, where mistakes made during codebook training contaminate the generation stage. Uniform quantization, by contrast, maps each parameter value directly to an interpretable bin with a known numerical meaning, making the process transparent and directly reversible. The maximum quantization error at 1000 bins is just 0.0032 radians (0.18 degrees) for angular parameters and 0.005 meters for translation — precision comparable to the margin of error in multi-camera studio setups.

But the real breakthrough came from a training innovation the researchers call a distance-aware loss. Standard cross-entropy treats every wrong token prediction identically: predicting bin 500 when the true bin was 501 incurs exactly the same penalty as predicting bin 900, even though the former is nearly correct and the latter is catastrophically wrong. The team solved this by introducing a soft-argmax operator that converts the model’s raw logits into a probability distribution over all 1000 bins, then computes the probability-weighted expected value and penalizes the L2 distance from the true parameter. Combined with cross-entropy in a 0.3-to-0.7 weighting, this distance-aware objective slashed reconstruction error dramatically — from 36.2 mm to 28.20 mm PA-MPVPE on the EgoBody benchmark for the InternVL2-8B backbone.

The results are competitive with, and in some cases superior to, far larger specialized pipelines. On EgoBody, the largest egocentric dataset with ground-truth SMPL-X parameters, the fine-tuned InternVL2-8B achieved 29.49 mm PA-MPJPE and 28.20 mm PA-MPVPE, outperforming the recent SMPLest-X-H40+ (33.5 mm PA-MPVPE) despite that model being trained on fourteen additional datasets. Zero-shot evaluation on the EHF studio dataset — never seen during training — yielded 33.85 mm PA-MPVPE, beating OSX (48.7 mm), PyMAF-X (50.2 mm), and Hand4Whole (50.3 mm), and approaching the performance of SMPLer-X-L32 (37.1 mm). The Qwen2.5-VL-7B backbone produced similarly strong numbers, confirming that the tokenization and loss design generalize across model architectures.

Perhaps the most striking capability is language-guided body shape editing. The team built an augmentation pipeline that generates eight semantically controlled shape variants for each training image — belly fat or slim, legs fat or slim, face fat or slim, height tall or short — by manipulating the shape coefficients of the SMPL-X model. Each variant is paired with a natural-language instruction such as “Reconstruct the SMPL-X body parameters with a fat belly” or “Predict the SMPL-X body parameters with thinner legs.” This expanded the dataset from roughly 150,000 to more than 1.3 million instruction-parameter pairs. Anthropometric validation confirmed the edits produce anatomically correct changes: belly enlargement increased hip circumference by 21.7 percent, leg thickening increased thigh circumference by 22.5 percent, and height scaling changed body height by 10 percent, all with statistically significant differences and no cross-talk between unrelated body regions.

At inference time, the model automatically decides whether to perform plain reconstruction or semantic editing based on the instruction in the prompt — no explicit body-part segmentation or post-processing required. The team demonstrated robustness to paraphrasing: instructions like “increase belly size,” “make the person’s abdomen fat,” and “make the person look heavier in the belly region” all map to the same geometric edit. This suggests the model learns genuine semantic understanding between language and body geometry rather than memorizing fixed instruction templates.

The implications ripple outward. A single fine-tuned vision-language model can now serve as a competitive alternative to specialized 3D reconstruction pipelines, while simultaneously offering interactive, language-driven control over the reconstructed body. The framework is model-agnostic, demonstrated across two distinct backbones, and the training cost — three days on a single NVIDIA A40 GPU for the base model, fifteen days for the augmented version — scales linearly with dataset size and can be reduced through multi-GPU distribution. Future work, the authors suggest, could explore adaptive loss weighting, extend quantization to represent temporal information in video, and add further semantic editing directions such as athletic build or broader shoulders. For now, EgoTokens stands as a vivid demonstration that the same models trained to write essays and answer questions can learn to sculpt the human form — one token at a time.

Subject of Research: EgoTokens: Egocentric 3D human reconstruction and language-guided shape editing via quantized position tokens in vision-language models

Article Title: EgoTokens: Egocentric 3D human reconstruction and language-guided shape editing via quantized position tokens in vision-language models

Article References: Khan, G., Rani, M., Ahmed, W., Pimbblet, K., & Wertheim, K. Y. (2026). EgoTokens: Egocentric 3D human reconstruction and language-guided shape editing via quantized position tokens in vision-language models. Machine Learning with Applications, 26, Article 101042. https://doi.org/10.1016/j.mlwa.2026.101042

Image Credits: AI Generated

DOI: 10.1016/j.mlwa.2026.101042

Keywords: EgoTokens, Egocentric, human, reconstruction, language-guided, shape, editing, quantized, position, tokens, vision-language, models