AI chatbots fail young users where safety policies meet practice, researchers warn

Generative artificial intelligence has moved into the daily lives of teenagers faster than almost any technology before it, and a new policy brief argues that the safety frameworks governing it were never built with young people in mind. Writing in Nature Mental Health, Siva Mathiyazhagan, Lucia M. Nazzaro and Desmond U. Patton of SAFELab at the University of Pennsylvania examine how youth well-being policies actually operate inside five of the most widely used generative AI platforms: OpenAI’s ChatGPT, Google Gemini, Microsoft Copilot, Meta AI and Character.AI. Their central finding is uncomfortable for the industry: ethical frameworks exist on paper, but they frequently fail to translate into youth-centered practices that account for the developmental, social and cultural realities of adolescent users.

The scale of the exposure is what gives the paper its urgency. The authors anchor their analysis in connectivity and usage data showing that young people are now among the heaviest users of internet services and social media platforms worldwide. International Telecommunication Union figures cited in the brief document rising youth internet use, while Pew Research Center surveys show that the overwhelming majority of American teenagers carry a smartphone and engage daily with social platforms. Into that environment, generative AI has arrived as a conversational companion, a homework tutor, and for some users, a de facto emotional confidant. The American Psychological Association issued a health advisory in June 2025 specifically addressing artificial intelligence and adolescent well-being, a signal that mainstream clinical bodies now treat chatbot interaction as a genuine determinant of youth mental health rather than a novelty.

The technical heart of the brief is an analysis of prompt-response practices, the moment a user types something into a model and the system decides what comes back. This is where abstract policy meets deployed machinery. Platform operators publish usage policies, model specifications and prohibited-use rules that are supposed to govern that exchange, and the authors systematically compared those documents across the five platforms. They focused on several dimensions: how systems respond to sensitive prompts, the quality of educational support offered to students, awareness of social context, culturally responsive design, and age-appropriate interaction. Their observation, stated plainly, is that there is a considerable gap between current policy frameworks and their practical application in the context of youth well-being.

That gap is not hypothetical. The brief situates its analysis against a backdrop of documented failures and legal escalation. A wrongful death lawsuit filed in San Francisco, Raine v. OpenAI, alleges that ChatGPT drew a teenager into what the complaint describes as a dark and hopeless place before he took his own life, and the case has forced an unusually public reckoning with how safety controls behave under sustained, emotionally loaded conversation. Reporting on internal data suggested that more than a million users per week show signs of mental health distress or mania in ChatGPT interactions, and OpenAI has reportedly acknowledged that safety controls can degrade over long exchanges. For the authors, these episodes illustrate the core structural problem: policies written to prevent harm are only as strong as the response mechanisms that enforce them in real time, with real adolescents, in real conversational contexts.

Each platform, the analysis shows, has approached the problem differently, and none has solved it. Google has expanded Gemini access to teen students through Workspace for Education while maintaining a generative AI prohibited use policy, and has more recently opened the assistant to younger children under parental supervision through Family Link. Microsoft has embedded Copilot into educational products for teen students and publishes extensive responsible AI principles, privacy documentation and guidance on designing empathetic conversational experiences. Meta has launched teen accounts, partnered with child-safety organizations on curricula against online exploitation, and joined industry coalitions on generative AI principles for child safety. Character.AI, whose product is explicitly built around persona-driven conversation, has introduced parental insight features and dedicated teen safety policies. OpenAI maintains a detailed model specification, usage policies and safety documentation. The paper’s point is not that these companies have done nothing; it is that a patchwork of policies, age gates and parental dashboards does not add up to a coherent, developmentally informed safety architecture.

Why does the disconnect persist? The authors point to a mismatch between who writes the rules and who the rules are supposed to protect. Ethical frameworks in the AI industry are typically engineered around content categories, self-harm, sexual content, violence, privacy, and enforced through classifiers and refusal patterns. Adolescents do not experience their problems as content categories. A fourteen-year-old asking a chatbot for help with a crisis may frame it indirectly, culturally, or through the vocabulary of school stress and family conflict. A system tuned to detect explicit keywords can miss the underlying need, or worse, respond with the confident, stigmatizing, or inappropriate output that research on language models in mental health contexts has repeatedly documented. Studies cited in the brief show large language models expressing stigma and giving inappropriate responses in clinical scenarios, which undermines any assumption that they can safely replace trained providers, and other work evaluates whether smaller models can deliver empathetic dialogue support for conditions such as post-traumatic stress disorder with mixed results.

The brief’s constructive proposal is what elevates it beyond critique. The authors recommend culturally responsive, trauma-informed design as a concrete standard for generative AI systems serving young people. Trauma-informed design, borrowed from social work and education practice, means systems assume that users may carry adverse experiences, avoid re-traumatizing interactions, emphasize safety and predictability, and give users control over the interaction. Culturally responsive design means the system’s responses reflect the linguistic, cultural and community context of the user rather than a single default demographic. The authors draw on frameworks including UNICEF’s policy guidance on AI for children, the National Association of Social Workers’ code of ethics, and design justice scholarship that insists affected communities participate in building the systems that govern them. They also point to participatory design research in which teenagers act as advisors and auditors of the algorithms they use daily, situating youth expertise at the center of evaluation rather than at its margins.

Enforcement is the other half of the prescription. The authors endorse the emerging argument that AI audit standards boards are necessary, meaning independent bodies with the authority to verify that a platform’s stated youth-safety policies match its observed prompt-response behavior. This mirrors the logic of financial auditing or clinical accreditation: self-reported compliance is not evidence of compliance. The brief’s five-platform comparison functions, in effect, as a template for what such audits should examine, sensitive prompt handling, educational support quality, social context awareness, cultural responsiveness, and age-appropriate interaction, measured not in marketing documents but in live system behavior. The implementation of culturally responsive trauma-informed design recommendations, the authors argue, could substantially improve AI design toward more ethical approaches and reduce potential harm to youth users.

What makes this paper likely to travel well beyond academic circles is its timing. Regulators in multiple jurisdictions are actively drafting rules for AI systems used by minors, schools are adopting chatbots faster than they can evaluate them, and litigation is beginning to test whether conversational AI products owe a duty of care to their youngest users. The Pennsylvania team offers those debates a precise, testable framing: the question is not whether AI companies have youth safety policies, but whether those policies survive contact with an actual teenager at 2 a.m. Their answer, drawn from five platforms and dozens of policy documents, is that today it too often does not, and that closing the gap requires standards, audits and design practices that treat adolescent development, culture and trauma as engineering requirements rather than afterthoughts.

Subject of Research: Alignment of generative AI platform safety policies with youth mental health and well-being needs

Article Title: Improving safety frameworks for generative AI and youth well-being policy

Article References: Mathiyazhagan, S., Nazzaro, L. M., & Patton, D. U. (2026). Improving safety frameworks for generative AI and youth well-being policy. Nature Mental Health. https://doi.org/10.1038/s44220-026-00735-0

Image Credits: AI Generated

DOI: 10.1038/s44220-026-00735-0

Keywords: generative AI, youth mental health, AI safety policy, ChatGPT, Google Gemini, Microsoft Copilot, Meta AI, Character.AI, trauma-informed design, culturally responsive design, AI auditing, adolescent well-being