The toolkit, called the MultiSOCIAL Toolbox, short for Multimodal timeSeries Open-SourCe Interaction Analysis Library, was developed by a multidisciplinary team of psychologists and computer scientists at Colby College and the University of Connecticut. It packages three gold-standard open-source technologies, MediaPipe for pose estimation, OpenSMILE for acoustic feature extraction, and Whisper for automatic speech transcription, into one Python-based application that runs entirely on a researcher’s own laptop. No data ever leaves the user’s machine, a design decision that both protects participant privacy and reassures institutional ethics boards, and no internet connection is required once the software is installed. The team designed the system to run on ordinary computers with around 16 gigabytes of memory, sidestepping the expensive graphics processors that many state-of-the-art AI pipelines demand.

Each modality is handled by a well-tested engine. For movement analysis, the Toolbox combines a lightweight YOLOv5s person detector with MediaPipe’s BlazePose model, which tracks 33 body landmarks, from nose to ankles, in three-dimensional coordinates frame by frame. The choice of MediaPipe over the popular OpenPose alternative was deliberate: benchmarks cited by the team show it detects roughly 10 percent more body points while running faster. For people detected in standard 30-frames-per-second commercial video, the system produces a time series of 132 values per person per frame, with each coordinate accompanied by a visibility score that acts as a proxy for the algorithm’s confidence. Because many interaction experiments involve multiple people, the Toolbox includes a custom tracking pipeline that assigns each individual a consistent identity across frames, so each exported CSV file contains the trajectory of a single participant suitable for longitudinal analysis.

Speech and language are covered just as thoroughly. The audio module draws on OpenSMILE’s ComParE 2016 feature set, extracting 65 low-level acoustic features such as pitch, energy, jitter, and shimmer, sampled every 10 milliseconds, which amounts to a 100-hertz sampling rate. The transcription module runs Whisper Large V3 Turbo, a pruned and fine-tuned version of OpenAI’s speech recognition model that trades a small amount of accuracy for substantial speed, and can optionally apply PyAnnote-based speaker diarization to label who said what and when. Users can even extract audio directly from their video files, and an alignment step aggregates acoustic features over the time interval of each transcribed word, producing word-level datasets that researchers can later integrate with movement data. The authors are candid that audio and movement streams are sampled at different densities and must be handled carefully during integration, a warning aimed especially at newcomers to multimodal work.

Several design touches reflect hard-won practical experience. An Embed Pose option overlays the extracted skeletal keypoints onto the raw video, serving both as a compelling visual for science communication and as a check that the extraction actually captured what the researcher intended. A Verify Pose Match function goes further, comparing embedded videos against their companion CSV files to catch workflow errors such as stale outputs, mismatched person identifiers, or frame-index shifts introduced by frame skipping. Users can also downsample frames or reduce video resolution to trade modest accuracy losses for faster processing, making the toolkit viable on machines that would choke on full-resolution pipelines. The modular architecture means researchers can extract just one modality or all three, and the team invites the community to contribute new tools through the project’s open GitHub repository.

To demonstrate the system in action, the authors present a study of interpersonal movement coordination conducted with undergraduate students, several of whom had no prior technical training, as part of a one-semester seminar course. The study involved 206 participants aged 18 to 24, randomly paired into 103 dyads, of whom 44 reported knowing each other beforehand and 59 arrived as strangers. Each pair sat roughly four feet apart in small private rooms and spent six to eight minutes discussing which book or movie everyone should experience in their lifetime. Three synchronised Microsoft LifeCam webcams recorded the conversations, one capturing both participants from above and two facing each participant directly, using the MultiRec application to ensure the recordings stayed time-locked.

For the analysis, the team used the Toolbox to extract movement from the face-on camera videos, focusing on the nose and on a computed neck point midway between the shoulders. They then applied cross-recurrence quantification analysis, a nonlinear time-series technique that measures not just how much two people’s movements align but how stable and how complex that alignment is over the course of a conversation. After z-scaling each person’s movement within their own series and determining optimal phase-space reconstruction parameters, they compared friends and strangers using Mann-Whitney U tests, with the de-identified data and full analysis code published in an Open Science Framework repository for anyone to scrutinize or reuse.

The results were striking and consistent across both body regions. Friends showed significantly higher recurrence rates than strangers, meaning their head and torso movements were more coordinated overall: nose recurrence averaged 6.09 for friends versus 4.97 for strangers, and neck recurrence averaged 9.80 versus 7.73. Friends also scored higher on maximum line length, indicating their coordination was more stable over time, with nose values of 98.91 versus 82.81 and neck values of 121.91 versus 97.92. Perhaps most intriguingly, friends’ coordination was also more complex, with higher entropy values suggesting less predictable, more dynamically rich patterns than the simpler coupling seen in strangers. In other words, friendship seems to produce bodily coordination that is simultaneously stronger, more stable, and more intricate, a combination that suggests deeper interpersonal attunement rather than mere imitation.

These findings matter because most synchrony research has focused on strangers, whether naive pairs, confederates, or experimenters, leaving friendship itself surprisingly understudied in psychological science despite its central role in human wellbeing. Prior work had hinted at differences: one study found synchrony predicted later closeness ratings only for strangers, and another observed an inverse relation between synchrony and perceived support among people who identified merely as friends rather than close friends. The new results suggest that researchers who pool friends and strangers together, or who study only strangers, may be missing fundamentally different coordination dynamics. The authors argue that asking participants about their prior relationships should become standard practice in interaction research.

The team closes with hard-earned practical guidance. Automated methods inherit the conditions of their training data, so pose estimation suffers when bodies overlap, lighting is poor, or camera angles are oblique, and the authors recommend well-lit setups with full bodies visible and no frequent occlusion. Overlapping speech recorded on a single channel can confound transcription and diarization, so separate microphones and separate channels per speaker are strongly advised. Multi-person recordings can occasionally produce detection errors or sudden identity swaps between tracked individuals, as the team discovered when testing on videos of dyads doing yoga, so well-separated seating and even a post-hoc data-rescue strategy of cropping and reprocessing can help. The authors also emphasize that the Toolbox is a means of data extraction, not a substitute for theoretical care: researchers must still bring discerning eyes to their data, verify outputs, and adapt as needed. If the toolkit succeeds in its ambition, the conversation between body, voice, and language that animates every human interaction may finally become something any curious scientist can measure.

Subject of Research: An open-source graphical toolkit for multimodal extraction of body movement, acoustic-prosodic, and speech data from interaction videos, applied to comparing bodily coordination between friends and strangers.

Article Title: The MultiSOCIAL Toolbox: An open-source toolkit for advancing multimodal interaction research

Article References: Romero, V., Chowdhury, T., Paxton, A., & Nafees, M. (2026). The MultiSOCIAL Toolbox: An open-source toolkit for advancing multimodal interaction research. Behavior Research Methods, 58(10), Article 293. https://doi.org/10.3758/s13428-026-03176-w

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03176-w

Keywords: MultiSOCIAL Toolbox, multimodal interaction analysis, open-source software, pose estimation, MediaPipe, OpenSMILE, Whisper speech recognition, interpersonal coordination, movement synchrony, friendship, cross-recurrence quantification, Behavior Research Methods