AI learns to read dogs: new 4D dataset captures human–dog interactions in motion

Every dog owner knows the silent choreography of a shared moment with a pet: a raised hand sends a dog into a sit, a verbal command straightens its posture, a called name sends it sprinting across the park. These exchanges are among the most familiar interactions in human life, yet they have remained largely invisible to artificial intelligence. While researchers have built rich datasets for human–human and human–object interactions, the movements of dogs responding to human cues have never been captured at scale. A team at Institute of Science Tokyo, working with collaborators including Carnegie Mellon University, has now changed that with InterPet4D, the first large-scale multimodal 4D dataset of natural dog movements in response to human actions, gestures and speech.

The dataset is described as 4D because it records three-dimensional information over time, allowing researchers to study not only what humans and dogs look like during an interaction, but how their bodies move through space second by second. InterPet4D comprises 6.83 million synchronized frames drawn from 161 recording sessions involving 23 human participants and 13 dogs representing 11 breeds. Twelve synchronized third-person cameras captured the interactions from multiple viewpoints, while an egocentric camera recorded the scene from the human participant’s own perspective. Aligned audio and text captions accompany the visual data, and the recordings were processed to estimate human body and hand motion alongside the dog’s 3D pose.

Collecting such data is harder than it might sound. Close-range interactions between a person and a dog frequently cause one participant to occlude the other, blocking the camera’s view and complicating detailed motion reconstruction. Preserving the timing and context of each exchange, which is essential if a machine is to learn the causal link between a cue and a response, requires precise synchronization across many sensors. By combining multi-view video, egocentric video, audio and 3D motion in a single curated resource, the team aimed to overcome both problems at once. “By capturing human–dog interactions, we wanted to create a dataset that reflects their multimodal nature and supports more systematic research,” says Yichen Peng, Specially Appointed Assistant Professor in the Department of Computer Science at Science Tokyo, who led the work with Professor Hideki Koike.

The interactions in InterPet4D are organized into four categories: petting, commanding, calling, and free-form activities such as fetch, tug-of-war and chase. This structure reflects the reality of human–dog communication, in which physical touch, hand gestures, verbal commands and distance calls each elicit different changes in the dog’s posture, position and movement. Having labeled categories spanning both contact-based and distance-based interaction gives machine learning models a way to learn how different types of human cues map onto different canine responses, something no previous public dataset has offered at this scale.

The dataset served as the foundation for a second contribution: InterPetMoGen, abbreviated IPMG, an AI framework that generates plausible dog motion conditioned on human body and hand movements and accompanying audio cues. The system first converts complex motion sequences into compact motion tokens, discrete representations that neural networks can process efficiently. Generation then proceeds with an autoregressive transformer, which produces the dog’s movement sequentially while attending to the context of the ongoing interaction, so that each predicted frame is consistent with both the human cue and the motion generated so far.

Two further components shape the quality of the output. A PetVAE module, a variational autoencoder, learns a compact latent representation of dog movements, effectively compressing the enormous variety of canine motion into a smooth space from which realistic sequences can be sampled. Meanwhile, modality-aware attention allows the model to weigh how strongly each input type, body motion, hand motion or audio, should influence the generated response at each moment. A spoken command might dominate during a calling interaction, whereas hand trajectories matter more during petting, and the attention mechanism lets the network learn those differences from data rather than through hand-crafted rules.

The researchers evaluated the generated motion using established metrics. Fréchet Inception Distance, or FID, measures how closely the distribution of generated motion resembles real motion, with lower values indicating greater realism. Retrieval Precision assesses whether generated dog movements actually correspond to the human cues given to the model, and diversity scores capture the variety of movements produced. IPMG achieved a kinetic FID of 11.21, compared with 21.22 for a Seq2Seq-Transformer baseline, a 47.2 percent reduction. It also improved hand-motion alignment from 0.41 to 0.63 and body-motion alignment from 0.38 to 0.59, while increasing motion diversity from 5.01 to 5.93.

Human judgment reinforced the quantitative results. In a user study, 12 participants rated the naturalness and appropriateness of the generated dog responses on a seven-point scale. The full IPMG model received scores of 6.58 for naturality, 6.55 for responsiveness and 6.63 for overall quality, compared with 4.04, 3.67 and 3.67 respectively for the Seq2Seq-Transformer. “The results suggest that modeling different modalities and broader interaction context can help AI-based frameworks predict and generate dog movements that are both more realistic and consistent with human cues,” Peng says. In other words, an AI that listens to the voice, watches the hands and tracks the body produces dogs that behave like dogs, rather than generating generic four-legged animation.

The implications extend well beyond the laboratory. A standardized foundation for human–pet interaction research could support behavioral analysis, helping scientists quantify how dogs respond to different handlers, commands and environments. In animation and games, the technology points toward virtual animals that react believably to player gestures and speech instead of following scripted paths. In robotics, plausible dog-motion models could inform socially aware machines that interpret and respond to animal behavior, a capability relevant to service animals, veterinary settings and human–animal interaction studies. The dataset’s multimodal design, pairing video, audio, text and 3D pose, makes it a reusable benchmark for any system that must understand two very different bodies acting in concert.

The authors are candid about the current limits. The present work focuses exclusively on dogs, does not model the physical forces involved in contact interactions such as petting or tug-of-war, and generates fixed-length clips of ten seconds. Longer sequences, richer physical simulation and extension to other species are outlined as future directions. Even so, the arrival of InterPet4D marks a shift in how machines can learn from the oldest partnership between humans and animals: for the first time, the everyday language of gestures, commands and wagging tails has been recorded in enough detail, and in enough quantity, for artificial intelligence to begin speaking it back. The findings were presented at the 19th European Conference on Computer Vision (ECCV) 2026, one of the leading international conferences in computer vision, held on September 9, 2026.

Subject of Research: Multimodal 4D datasets of human–dog interactions for AI-based pet motion generation

Article Title: InterPet4D: capturing human–dog interactions to generate AI-based pet motion generation

Article References: InterPet4D: capturing human–dog interactions to generate AI-based pet motion generation. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: InterPet4D, human–dog interaction, motion generation, artificial intelligence, computer vision, 3D pose estimation, multimodal dataset, autoregressive transformer, variational autoencoder, animal behavior, ECCV 2026, Science Tokyo