The research team, led by Maggie Liao, an assistant professor at the University of Georgia who completed her doctoral work at Penn State, and co-author S. Shyam Sundar, Evan Pugh University Professor and James P. Jimirro Professor of Media Effects at Penn State, set out to isolate a specific variable: the degree to which a chatbot communicates conversationally. Both researchers have conducted several prior studies on how people use and trust AI chatbots, and they were well aware that the technology can produce inaccurate information. What they wanted to know was whether the conversational delivery itself, independent of the content, makes users more likely to accept false claims as true.

“We tested whether the conversational nature of GenAI chatbots is driving users to trust misinformation coming from them,” Liao said. “And if that is the case, the natural next question is: What can we do about it?” The question has taken on considerable urgency as chatbots become a mainstream source of information on topics ranging from personal finance to health. Recent survey data cited by the researchers indicate that Americans are increasingly turning to AI chatbots as trusted sources, including for medical questions, at the same time that these systems are becoming more human-like in the way they communicate. The combination, the researchers suggest, creates a potentially problematic dynamic in which the medium’s style overwhelms users’ critical judgment.

The theoretical backbone of the study is a concept the researchers call the “negative machine heuristic,” the ingrained tendency to believe that machines lack the human touch when it comes to intuition and subjective decision-making. Historically, that heuristic has served as a protective filter: people have tended to discount what machines tell them precisely because machines were perceived as mechanical, impersonal, and lacking judgment. The researchers found that the more conversational or personable a chatbot is, the more users curb this negative machine heuristic, effectively lowering their psychological defenses. Strikingly, this happened even when the information being delivered was, in the researchers’ words, “ridiculously wrong.”

Part of what makes conversational AI so persuasive, Sundar explained, is the specificity of its delivery. “The conversational delivery of information by generative AI is hyper-specific to your particular question, so it seems very tailored,” he said. “This is the first time in the history of technology where a machine can actually hold forth a conversation that has not been predicted by a programmer or has not been pre-scripted.” Unlike earlier information technologies, where scripted responses or pre-programmed menus reminded users that a machine was on the other end, generative AI produces spontaneous, unscripted dialogue. That spontaneity signals human-like agency, and users appear to transfer the trust they would extend to a knowledgeable human interlocutor onto the machine.

To test these ideas empirically, the researchers conducted an online experiment with 477 participants. Each participant was told they were helping to test a new AI health assistant and was asked to submit one prompt from a pool of sample prompts to the chatbot. During the interaction, the AI deliberately provided incorrect information. Participants were then asked to indicate how credible they believed the chatbot was. The experiment varied how conversational the AI was, with some participants interacting with a low-conversational version and others with a high-conversational version. The design allowed the researchers to measure whether conversational style alone shifted credibility judgments, holding the accuracy of the information constant at deliberately wrong.

The results were sobering. Participants who received highly conversational AI responses were more likely to find the information credible than those interacting with less conversational versions, and this effect persisted even when the information was blatantly inaccurate. “Some of the misinformation we used in the study was ridiculous,” Liao said. “One example was that ginger can be more effective than chemotherapy in treating cancer. It sounds ridiculous, but if it is given by a very conversational AI, people trust it more. I think that’s worrisome.” The finding suggests that the stylistic polish of a chatbot’s delivery can override even basic common-sense scrutiny, at least in the context of a health-related interaction where users might otherwise be expected to be cautious.

The second part of the study tested a potential remedy. The researchers designed a verification option modeled after the standard human fact-checking process, giving users the opportunity to confirm whether the AI’s responses were backed by evidence. The idea was to counter a common assumption among users that AI is always more objective and accurate than humans, an assumption that can make people suspend the skepticism they would naturally apply to a human source. The experiment compared four conditions: a simple cue consisting of a small icon warning that information may be incorrect; a required check with a button that users had to click in order to proceed; an optional check with a button that users could click if they chose to verify the information; and no verification option at all.

The findings on verification were nuanced and, in some ways, counterintuitive. Users who had the ability to verify the information were more skeptical of the AI’s responses overall, but a passive verification cue alone was not enough to change minds. Trust in the AI system only decreased for users who actively clicked on the verification buttons. In other words, the act of verification itself, rather than the mere presence of a warning icon, appears to be what rebuilds skepticism. When given the option, nearly half of the participants chose to verify the information they received, and those who did were less likely to accept false claims as accurate. The behavioral engagement seems to matter: clicking a button and checking evidence forces a moment of critical evaluation that a passive icon does not.

“The verification option is a useful antidote to the conversational persuasiveness of GenAI,” Sundar said. “That’s what we say in the title of our study: ‘Chat, but verify.’ Enjoy the conversation by all means, but don’t be lulled into believing all that you’re getting.” The researchers hope the results serve both as a warning and as a practical roadmap for the AI industry. As Sundar put it, research showing that verification tools help combat misinformation should persuade AI platforms to take such mechanisms seriously and to develop more checks on information generated by AI. For a technology whose persuasive power grows with every conversational improvement, the study suggests that the solution lies not in making chatbots less engaging, but in building active, user-driven verification into the conversation itself, so that enjoying the chat no longer means surrendering judgment to the machine.

Subject of Research: How conversational AI chatbots increase user trust in misinformation and whether verification tools reduce that effect

Article Title: Chatty chatbots can mislead users, but verification tools help, researchers find

Article References: Chatty chatbots can mislead users, but verification tools help, researchers find. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: AI chatbots, misinformation, generative AI, verification tools, human-computer interaction, media effects, health information, credibility, Penn State, Journal of Computer-Mediated Communication, negative machine heuristic, fact-checking