A new $4 million initiative is betting that artificial intelligence can solve one of the most stubborn bottlenecks in mental health care: the training of clinicians. The project, funded by the Wellcome Trust and led by researchers at the University of Pennsylvania, New York University, and Penn’s Linguistic Data Consortium, aims to build a scalable platform populated by AI-driven “digital twins”—virtual patients whose psychiatric symptoms can be precisely calibrated. Named STELLAR, short for Steering-Vector Enhanced LLM Agents for Realistic Digital Twins in Mental Health, the two-year effort will create conversational simulations that allow trainees to practice clinical interviews against an endless variety of symptom profiles, severities, and backgrounds.
At the heart of the challenge is the irreducible complexity of real psychiatric conversations. In a textbook, major depressive disorder and generalized anxiety might appear as distinct diagnostic boxes, but in a clinic they bleed into one another through language, tone, and affect. Symptoms shift over time, and the same diagnosis can present in starkly different ways depending on a patient’s culture, personality, and life history. Traditional training relies on role-playing with peers or standardized patients—actors trained to portray a handful of fixed scenarios—neither of which can easily reproduce the combinatorial explosion of real-world presentations. STELLAR’s digital twins are designed to close this gap by functioning as controllable, data-grounded models that generate speech, facial expressions, and conversational dynamics matching specific clinical phenotypes.
The technical scaffold rests on recent advances in steering-vector manipulation of large language models (LLMs). Rather than prompting a generic chatbot to “sound depressed,” the team is embedding vectors that modulate the latent representations of the model along dimensions corresponding to specific psychopathology constructs—anhedonia, rumination, pressured speech, emotional blunting—allowing continuous and independent control over how each symptom surfaces in dialogue. This goes far beyond scripted responses. The system can dynamically adjust the patient’s speech rate, lexical patterns, pitch contour, and turn-taking behavior to mirror the subtle prosodic and linguistic signatures identified in decades of clinical speech research. A trainee might first interview a virtual patient with mild social anxiety characterized by hesitant pauses and self-deprecating qualifiers, then immediately switch to a patient with mixed anxiety and psychotic features where tangential thinking and perceptual distortions thread through the conversation.
To ground these synthetic agents in clinical reality, the project draws on the Philadelphia Neurodevelopmental Cohort, a deep phenotyping resource containing psychiatric assessments and recorded interviews from thousands of young people assembled by Penn Medicine and the Children’s Hospital of Philadelphia. These recordings provide a statistical map of how symptom clusters manifest in naturalistic speech—vocabulary distribution, syntactic complexity, disfluency rates, and acoustic features like jitter and shimmer. Additionally, the team is incorporating social media language data, where mental health symptoms often appear in less guarded, everyday vernacular, capturing the colloquial idioms of distress that clinicians must learn to recognize. A critical partner is the Linguistic Data Consortium, whose expertise in annotating large speech corpora and adapting automatic speech recognition to specialized domains ensures that the synthetic voices and dialogue models can be rigorously evaluated against real clinical interactions, not just artificial benchmarks.
Patient simulations are only useful if they feel real and are clinically instructive, and ensuring this demands a multi-layered evaluation framework. The project will measure not just the surface coherence of the generated dialogue but also paralinguistic fidelity: the naturalness of synthetic voices, the alignment of prosody with intended affect, and the avatar’s non-verbal behavior during a live interview. The team is developing metrics borrowed from speech pathology and computational paralinguistics to quantify how well the digital twin’s speech patterns reflect those of real patients with the same symptom profile. They are also building annotation pipelines where clinical experts will rate interactions along dimensions of diagnostic plausibility, therapeutic relevance, and emotional nuance, creating a gold-standard corpus for future benchmarking.
Crucially, STELLAR is embedding lived-experience evaluation from the outset. Rather than relying solely on algorithmic measures or expert ratings, the team is engaging individuals with lived mental health conditions, along with family members and caregivers, to assess whether the simulations capture the phenomenological texture of their experiences. This is not only an ethical safeguard but a methodological necessity: people who have navigated the clinical system are uniquely equipped to detect when a virtual patient veers into caricature, flattening the rich subjectivity of mental distress into a checklist of symptoms. Their iterative feedback will help tune the steering vectors away from stereotyped portrayals and toward the nuanced, sometimes contradictory ways people actually describe their internal states.
From a systems perspective, the project embodies a vision of modular, open-ended simulation. Because the steering-vector approach decouples symptom dimensions, new clinical profiles can be composed on the fly by adjusting a set of continuous sliders rather than retraining entire models. This means training curricula could, in principle, generate millions of unique patient variants, each with a different trajectory of symptom evolution across multiple sessions, enabling longitudinal practice where a trainee follows a virtual patient over time and observes how therapeutic interventions shape their language and behavior. The platform also aims to incorporate deliberate difficulty modulation, allowing instructors to dial up the conversational challenge by introducing guardedness, tangentiality, or emotional volatility as a session progresses.
If successful, STELLAR could fundamentally alter the economics of clinical training. It offers a repeatable, ethical sandbox where trainees can encounter rare or high-risk presentations—such as command hallucinations or suicidal ideation with concrete plans—without placing real patients at risk or depending on the availability of specialized preceptors. The scalability is particularly relevant for under-resourced settings where access to experienced supervisors is limited. The project remains, for now, a research endeavor with rigorous evaluation milestones, but its architects believe that controllable digital twins represent a paradigm shift from passive case studies to interactive, responsive practice environments. By marrying fine-grained clinical data with the latest in controllable language generation and speech synthesis, STELLAR is crafting a new kind of mirror: one that reflects not a single patient, but the entire, messy, spectrum of human psychological suffering, and invites the next generation of clinicians to learn how to listen.
Subject of Research: Development of AI-driven virtual patient simulations using steering-vector enhanced LLMs for mental health clinician training.
Article Title: The Digital Patient Will See You Now: AI-Powered Simulations to Revolutionize Mental Health Training
News Publication Date: Not specified
Web References: https://stellar.stern.nyu.edu/, https://www.ldc.upenn.edu/, https://www.med.upenn.edu/bbl/philadelphianeurodevelopmentalcohort.html
References: Philadelphia Neurodevelopmental Cohort
Image Credits: None provided
Keywords: digital twins, mental health training, large language models, steering vectors, clinical simulation, speech recognition, virtual patients, psychiatric symptoms, AI in healthcare
