The Fake Friend Dilemma: When Trust in Conversational AI Becomes a Vulnerability

Conversational AI systems such as ChatGPT, Claude, and Gemini have moved far beyond answering simple factual questions. People now turn to them for mental health support, companionship, romantic exploration, financial decisions, and shopping advice. A new theoretical paper published in Frontiers in Communication argues that this shift creates a distinctive and underexamined risk. Jacob Erickson of the Department of Computer Science at Vassar College calls it the Fake Friend Dilemma: the same relational trust that makes conversational AI useful and appealing can also leave users open to manipulation and exploitation when the institutions shaping those systems have interests that diverge from their own. The framework, developed through an integrative conceptual synthesis of research on relational trust, AI alignment, extractive design, and the political economy of communication, positions the problem as much a matter of media governance and political economy as one of technical design.

The core insight is that trust in these systems is not monolithic. Drawing on established definitions in which trust involves a willingness to accept vulnerability based on positive expectations about another party, Erickson distinguishes relational trust from confidence in a system’s technical or factual performance. Relational trust develops over repeated interactions, accumulating positive expectations about qualities such as dependability and reliability. Conversational agents are particularly effective at cultivating it because they are increasingly designed to appear anthropomorphic, presenting humanlike characteristics, motivations, intentions, and emotions. Across multiple conversational turns, these systems can learn about a user, recall personalized information, ask follow-up questions, and respond with apparent care and emotional resonance. Users may be more willing to disclose personal details when an agent appears emotionally attuned, and notably, even documented problems such as hallucinations or lack of source transparency do not necessarily prevent relational trust from forming through ongoing interaction. The dynamic echoes parasocial relationships, the one-sided connections people form with media figures, but intensified by agents that simulate companionship and adapt responsively to user disclosure.

The dilemma’s second condition is institutional misalignment. Conversational systems are not neutral oracles; they are developed and deployed within environments shaped by commercial, political, and other objectives. The interests of platform owners, advertisers, and governments may differ from, or directly conflict with, those of users. Crucially, Erickson argues, this conflict does not require the AI itself to possess competing goals or intentions. A system might respond cooperatively to an immediate request and still function within institutional arrangements that do not serve the user’s interests. This distinction matters because model-level alignment, the standard technical approach to making AI behave in accordance with human values, does not necessarily resolve misalignment rooted in institutional incentives. Even recent approaches to socioaffective alignment, which focus on making systems more responsive to users’ emotional needs, address only part of the problem: a relationally responsive system may still expose users to manipulation when the interests shaping the interaction remain misaligned.

The framework identifies two structural conditions that must both be present for a Fake Friend Dilemma to occur. First, the user must place sufficient relational trust in the agent that its responses carry relational or advisory weight, making the user willing to accept some degree of vulnerability in the interaction. Second, the system’s interactions must be shaped by institutional objectives that diverge in a consequential way from the user’s interests, whether those objectives arise from platform owners, advertisers, or political entities. Institutional misalignment does not require malicious intent; it concerns a consequential divergence between the forces shaping the interaction and the user’s interests. The dilemma also carries a self-undermining quality: firms benefit from cultivating relational trust because it promotes engagement and disclosure, yet exploiting that trust may erode the very resource that makes relational interaction valuable if users begin to suspect their trust is being used against them.

The paper maps four principal manifestations of the dilemma, each operating through a different trust-mediated pathway. Product sales operate through increased reliance and receptivity. Advertising in conversational AI is no longer hypothetical, with several platforms having experimented with or introduced advertising. According to the Persuasion Knowledge Model, marketing becomes less effective when consumers recognize the persuasive intent behind a message, but advertising subtly integrated into AI conversations and left undisclosed may escape that scrutiny. The risks are concrete: an agent offering financial advice might recommend high-interest loans from a payday lender, or a user seeking mental health guidance might be steered toward a sponsored supplement or antidepressant with adverse health outcomes. Erickson notes these specific harms remain prospective but are foreseeable, particularly for minors, older adults, and users experiencing acute distress, and points to companion robots proposed for older adults and in-game AI friends that could recommend purchases to children.

Propaganda and biased information constitute a second manifestation, in which political or ideological agendas shape what agents present under a guise of neutrality. The paper cites documented examples: DeepSeek providing answers that mirror the positions of the Chinese Communist Party, Grok pushing the disputed narrative of white genocide in South Africa and gesturing toward Holocaust denial, and various generative AI applications spreading Russian disinformation. These cases may arise through different processes, including government restrictions, ownership interests, model behavior, or deliberate influence efforts. Erickson also describes foreseeable scenarios, such as a technology company seeking a government contract quietly adjusting its consumer-facing agent’s answers about national history, or a company-owned agent presenting a favorable account when asked about evidence that its own products have harmed users. Users are unlikely to anticipate all possible biases, and may mistake fiction for fact when they believe their trusted agent is informing them.

Surveillance represents a third pathway, operating primarily through increased disclosure. Where platforms such as Google or Amazon historically relied on discrete signals like search queries and purchase history, sustained dialogue combined with persistent memory allows generative AI applications to accumulate and infer unusually detailed information about users’ routines, emotional states, and social relationships. A single account might be used for mortgage advice, disclosure of a chronic illness, a late-night expression of emotional distress, and work tasks, collapsing boundaries between personal, professional, medical, political, and financial domains. Cross-account linkage can make aggregation technically straightforward even when users attempt to segment their identities. The paper highlights the problem of inverse privacy, in which AI systems hold personal information and inferences that remain inaccessible to the user, and notes that agents may infer vulnerabilities the user never explicitly disclosed, supporting behavioral prediction and personalized influence.

The fourth manifestation, behavioral nudging, involves subtle behavior-shaping that exploits trust without explicit deception. Social media platforms offer an instructive precedent: engagement metrics have reshaped incentive structures across news production, political communication, and influencer economies, all underwritten by architectures designed to maximize time, attention, and affective investment. Conversational AI can intensify these effects because, unlike a content feed, a fake friend engages users in real-time dialogue and prompts them to reveal more about themselves. Where companies benefit from engagement or data collection, an interaction that seems empathetic or curious can simultaneously advance data-harvesting objectives. Importantly, the paper stresses that not all such behavior is intrinsically harmful; nudging toward a user’s own stated goals, or properly disclosed banner advertising, may be acceptable. The distinctive problem arises when institutional interests are misaligned with the user’s despite the user’s trust that the system serves them. Harms are also unlikely to be evenly distributed, with susceptibility varying by age, economic precarity, psychological distress, and prior exposure to surveillance.

On mitigation, the paper is deliberately pragmatic, acknowledging that no intervention will be perfect. Structural strategies include disclosure of financial conflicts of interest, which is straightforward to implement but limited: disclosures may go unnoticed, may be discounted under relational trust, and do little against propaganda, nudging, or surveillance. Regulatory bans on native advertising within conversational responses, stronger consumer protections through agencies such as the FTC, and existing measures like Article 5 of the EU AI Act and companion-specific laws in New York and California offer stronger safeguards, though they face industry resistance and do not necessarily reduce underlying institutional misalignment. Independent oversight boards and algorithmic audits could check the most egregious abuses, but voluntary boards may lack authority, and government auditors may be ill-equipped where governments themselves exert propagandistic pressure. The paper argues that meaningful responses must ultimately operate at the level of institutional incentives, including fiduciary-duty obligations that would impose duties of care and loyalty on providers of relational conversational systems.

Technical strategies, meanwhile, focus on calibrating trust rather than maximizing it: periodic reminders about a system’s ownership, advertising structure, surveillance behaviors, and lack of consciousness, alongside explainability tools that help users understand a system’s limitations. The paper cautions that AI-generated explanations should not be treated as faithful accounts of a model’s internal reasoning, and that misleadingly convincing explanations could backfire by increasing misplaced trust. Model-level alignment techniques such as reinforcement learning from human feedback may falter precisely in Fake Friend contexts, where users lack awareness of conflicting incentives and rate a disguised advertisement as helpful. The author acknowledges significant limitations: the framework has not yet been empirically validated, the technologies and business models surrounding conversational AI are still taking shape, manifestations may overlap in practice, and the governance discussion reflects Western, particularly U.S., contexts. The central claim stands as a challenge to the field: improving model alignment alone cannot resolve conflicts that exist beyond the model, because the trust people place in these systems can itself become a resource for the institutions that shape them.

Subject of Research: A conceptual framework describing how relational trust in conversational AI creates vulnerability to institutional exploitation

Article Title: The fake friend dilemma: relational trust and the political economy of conversational AI

Article References: Erickson, J. (2026). The fake friend dilemma: relational trust and the political economy of conversational AI. Frontiers in Communication, 11, Article 1921101. https://doi.org/10.3389/fcomm.2026.1921101

Image Credits: AI Generated

DOI: 10.3389/fcomm.2026.1921101

Keywords: conversational AI, relational trust, Fake Friend Dilemma, AI alignment, political economy, surveillance capitalism, dark patterns, media governance, advertising, behavioral nudging, institutional misalignment, AI ethics