AI Assistant That Reads Your Voice Makes Virtual Lab Work Feel More Human

A conversational agent that listens not only to what users say but to how they say it has shown in a controlled experiment that it can make guided work in mixed reality feel noticeably more usable and supportive, even though it slows the task down. The finding comes from researchers at the University of Salerno, who built a mixed-reality laboratory assistant powered by a large language model and equipped it with a parallel pipeline that infers emotional cues from the prosody of a user’s voice. In a study of forty participants performing a guided chemical procedure while wearing a mixed-reality headset, those who interacted with the emotion-aware version of the agent rated the system significantly higher on perceived usability than those who received a flat, non-adaptive version.

The research, published open access in the Journal of Ambient Intelligence and Humanized Computing, addresses a gap that has grown as extended reality applications increasingly call on conversational agents for real-time guidance. Large language models have transformed what spoken assistants can do, replacing rigid dialogue trees with flexible, open-ended exchange. But in immersive, task-oriented environments, an agent must satisfy demands that go well beyond answering correctly: it must stay grounded in the task and the surrounding scene, respond within the tight timing of natural turn-taking, and offer help without breaking the user’s sense of presence. Menus, panels and controller commands pull attention away from the hands-on work; speech promises a more direct channel, particularly when assistance is needed mid-action rather than before or after it.

Whether an assistant should also read emotion from speech has remained an open question. Human voices carry information far beyond their linguistic content, and acoustic-prosodic features such as pitch, intensity, rhythm and pausing patterns offer partial evidence about a speaker’s affective state. In principle, an agent that detects frustration, hesitation or elevated effort could modulate its tone, phrasing and level of detail to better match the user’s needs. Previous work has examined conversational agents in extended reality, LLM-based dialogue and affect-aware systems largely in isolation or in pairs, but fully integrated evaluations combining spoken dialogue, language-model generation and real-time prosody-based adaptation in task-oriented immersive settings have been rare, and the empirical consequences of such adaptation for both experience and task execution were unclear.

To test the idea, the Salerno team designed a modular client-server system built around three layers. The client, developed in Unity 6 and running on standalone Meta Quest-class headsets, handles rendering, speech acquisition through the Meta Voice SDK, transcription via Wit.ai, gesture-based interaction through Meta’s hand-tracking SDKs, and voice playback through Meta Text-to-Speech. The backend, implemented as Python micro-services, hosts the computationally heavy work: a Dialog Orchestrator that queries GPT-4o-mini using the transcription and the retained dialogue history, and a speech emotion recognition model based on Whisper Large V3 that has been fine-tuned for emotion classification. Communication between the layers relies on JSON messages and streamed audio, with asynchronous exchanges deliberately chosen to avoid blocking operations and preserve conversational continuity.

The architectural elegance lies in running semantic and affective processing in parallel, so that emotional inference never delays a response. Raw audio is accumulated until roughly twenty-five seconds of speech are available, and the resulting affective estimate, complete with a confidence score, is stored in a short-lived memory valid for up to ninety seconds. When the orchestrator processes a dialogue request, it retrieves the most recent valid estimate if one exists; if not, the reply is generated without affective conditioning at all. Crucially, the emotional descriptor serves only as an additional conditioning signal. It shapes the wording, interpersonal tone, reassurance, response length and explanatory detail of the language model’s output, while leaving the procedural facts, the task objective, the agent’s voice and even its visual behavior untouched, so that affective conditioning is the only difference between the two experimental configurations.

The agent itself appears as a deliberately non-anthropomorphic symbolic sphere anchored in the workspace, whose brightness and surrounding animation signal whether it is idle, listening or speaking. Prior research on embodied agents suggests that less humanlike representations can strike a better balance between recognizability, expressiveness and comfort, especially when the system is not meant to emulate a person. Participants, all university students with little prior exposure to immersive technology and almost no laboratory experience, wore a Meta Quest 3 headset and carried out a nine-step chemistry procedure: placing a flask on a stand, measuring and transferring resorcinol, adding ethanol, heating the mixture, preparing a nitrating mixture from sulfuric and nitric acid, and observing the final color change. The chemistry framing was chosen not to study chemistry education but because such a procedure demands the coordination of physical manipulation, sequential decision making, spatial attention and continuous spoken dialogue that characterizes procedural assistance scenarios generally.

The results revealed a striking trade-off. On the System Usability Scale, the adaptive condition scored an average of 88.6 against 81.4 for the non-adaptive one, a statistically significant difference with a large effect size, and both ratings fall within what standardized guidelines describe as the excellent range. Yet participants guided by the empathetic agent took substantially longer to finish, averaging 693 seconds against 503 seconds, and engaged in more conversational turns, averaging 20.3 against 16.3, both differences statistically significant with large effects. Overall workload, measured with the NASA Task Load Index, showed no significant difference between conditions, though the subscales told a subtler story: the adaptive group reported lower physical and temporal demand but higher frustration, suggesting that the emotional adaptation softened the felt pressure of the task even as it lengthened it, and occasionally irritated users when responses seemed more elaborate than the moment required.

The qualitative interviews added texture to the numbers. Participants in the adaptive group frequently described the agent as more human, friendlier and more attentive, with remarks that the interaction felt natural, like dealing with someone trying to help rather than a device issuing commands, and that the agent’s manner made the procedure less stressful during its most complex phases. Others found the empathy excessive for a short task, noting that the agent sometimes explained too much or offered comfort when none was needed. The non-adaptive group, by contrast, described the interaction as simple, clear and direct, and offered notably fewer spontaneous comments, which the authors suggest may itself reflect lower engagement with an assistant perceived mainly as a functional tool.

The researchers are careful about interpretation. The longer completion times and extra turns cannot, on their own, distinguish productive support from unnecessary verbosity, and the authors argue the pattern is best read as a genuine trade-off between concise execution and a richer conversational experience, one whose desirability depends on context: extra dialogue may be an asset in education and training but a liability in time-critical procedures. They recommend treating affect-aware adaptation as a context-dependent design strategy rather than a default upgrade, triggering it selectively during hesitation, repeated clarification requests or errors, and regulating not just emotional tone but response length and detail. Limitations include the student-only sample, the single procedural domain, and reliance on vocal prosody alone; future work should explore multimodal affect recognition combining gestures, gaze, task progress and physiological signals, while addressing the privacy and transparency questions that such sensing inevitably raises. What the study establishes is that emotional attunement measurably reshapes how people experience machine guidance in immersive environments, making the interaction feel more supportive at the price of speed, and giving designers an evidence-based lever for deciding when that price is worth paying.

Subject of Research: Affect-aware conversational adaptation using LLM-driven dialogue and prosody-based emotion recognition in mixed-reality procedural tasks.

Article Title: Affect-aware conversational adaptation in mixed reality procedural tasks

Article References: Cantone, A. A., Ercolino, M., Sebillo, M., & Vitiello, G. (2026). Affect-aware conversational adaptation in mixed reality procedural tasks. Journal of Ambient Intelligence and Humanized Computing. https://doi.org/10.1007/s12652-026-05125-z

Image Credits: AI Generated

DOI: 10.1007/s12652-026-05125-z

Keywords: mixed reality, conversational agent, large language model, affect-aware adaptation, speech emotion recognition, extended reality, GPT-4o-mini, human-computer interaction, procedural guidance, System Usability Scale, vocal prosody, immersive training