Measuring body language

Is it possible to decode how we feel from our movements? How can emotions be studied “from the outside” by using empirical methods? To answer these questions, a large international and interdisciplinary research team led by the Max Planck Institute for Empirical Aesthetics (MPIEA) in Frankfurt am Main, Germany, has developed an integrative scientific methodology. Using artistic and digital means such as motion capture technology, the researchers developed the EMOKINE software to measure the objective kinematic features of movements that express emotions. The results of the study have recently been published in the journal Behavior Research Methods.

The team had a professional dancer repeat short dance choreographies in front of a green screen. She was asked to express different emotions through her movements: anger, contentment, fear, happiness, neutrality, and sadness. To capture the dance movements as “data,” the scientists dived into the MPIEA’s technology pool: the dancer wore a full-body motion capture suit from XSENS®, equipped with a total of 17 highly sensitive sensors. In combination with a film camera, the dynamic body movements were measured and recorded. The researchers then extracted the objective kinematic characteristics (movement parameters) and programmed the software EMOKINE, which provides these movement parameters from data sets at the touch of a button.

Computerized Tracking for Whole-Body Movement

A total of 32 statistics from 12 movement parameters were compiled and extracted from a pilot dance dataset. The kinematic parameters recorded were, for example, speed, acceleration, or contraction of the limbs.

“We identified 12 kinematic features of emotional whole-body movements that have been discussed separately in the literature about previous research. We then extracted all of them from one same data set, and subsequently fed the features into the EMOKINE software,” reports first author Julia F. Christensen of the MPIEA.

Movement tracking has been used in many areas in recent years because the objective recording of movement parameters can provide insights into people’s intentions, feelings and state of mind. However, this research requires a theory-based methodology so meaningful conclusions can be drawn from the recorded data.

“This work shows how artistic practice, psychology, and computer science can work together in an ideal way to develop methods for studying human cognition,” says co-first author Andrés Fernández of the Max Planck Institute for Intelligent Systems in Tübingen, Germany.

The methodological framework that accompanies the software package, and which explicitly uses dance movements to study emotions, is a departure from previous research approaches, which have often used video clips of “emotional actions,” such as waving hands or walking.

“We are particularly excited about the publication of this work, which involved so many experts, for example from the Goethe University Frankfurt am Main, the University of Glasgow, and a film team from WiseWorld Ai, Portugal. It brought together disciplines from psychology, neuroscience, computer science, and empirical aesthetics, but also from dance and film,” summarizes senior author Gemma Roig, Professor of Computer Science, Computational Vision, and AI Lab at Goethe University.

The Open-Source Software Package

EMOKINE is freely available on ZENODO and GitHub and can be adapted to other motion capture systems with minor modifications. These freely available digital tools can be used to analyze the emotional expression of dancers and other groups of artists, and also everyday movements.

The researchers now hope that the EMOKINE software they have developed will be used in experimental psychology, affective neuroscience, and in computer vision—especially in AI-assisted analysis of visual media, a branch of AI that enables computers and systems to extract meaningful information from digital images, videos, and other visual inputs. EMOKINE will help scientists answer research questions about how kinematic parameters of whole-body movements convey different intentions, feelings, and states of mind to the observer.