ChatGPT Label Makes People More Likely to Accept Answers, Study Finds

When an answer carries the ChatGPT label, people are more likely to accept it as correct. That is the central finding of a new experimental study from the University of Messina, published in Frontiers in Psychology, which examined how source attribution shapes evaluative judgment in a reasoning task. Across 103 participants judging pre-generated answers to logic and arithmetic problems, overall acceptance was 68.0 percent when answers were attributed to ChatGPT, compared with 63.4 percent when the same kinds of answers were attributed to a generic computer, a paired difference of about 4.6 percentage points that was statistically significant.

The study adapted the Cognitive Reflection Test, a classic paradigm in reasoning research. CRT problems are built around a conflict between an intuitively compelling but incorrect response and a less obvious, mathematically correct solution. In the famous bat-and-ball problem, for example, many people instantly answer 0.10 euros, while the correct answer is 0.05 euros. Rather than solving such problems themselves, participants in this study were shown each problem together with a supplied answer and asked simply to judge whether that answer was correct or incorrect. This design shifts the question from problem solving to answer evaluation, a situation that is increasingly common as generative AI systems supply answers in everyday, academic, and professional contexts.

The experiment used a within-subjects design, meaning every participant saw answers under both source labels. Thirty trials presented three categories of answers: CRT problems paired with the reflective, correct solution; CRT problems paired with the intuitive, incorrect solution; and simple arithmetic control problems paired with objectively correct answers. Each source condition contained five trials per category. Two parallel batteries of fifteen matched problems, identical in reasoning structure but differing in numerical values and surface context, were randomly assigned to the ChatGPT or Computer label between participants, and trial order was randomized. The displays were visually matched, differing only in the source label and a small corresponding icon, to minimize perceptual confounds.

An independent pretest with 35 participants, conducted without any source labels, supported the comparability of the two batteries. Mean judgment accuracy for reflective answers was 0.510 in one battery and 0.504 in the other, and for intuitive answers 0.620 and 0.650, with neither difference significant and cross-battery correlations very strong. Control accuracy was perfect in both batteries. The main experiment was administered online, and participants afterward completed questionnaires measuring trust in AI, general attitudes toward artificial intelligence, and dispositional critical thinking. At the end of the session, they were told that all answers had been prepared by the researchers and that the labels had been assigned for experimental purposes.

The headline result came from a participant-level analysis: answers attributed to ChatGPT were accepted more often overall. A trial-level logistic mixed model, which included crossed random intercepts for participants and for the fifteen parallel-problem pairs, revealed a significant source-by-category interaction. The source association was particularly pronounced for the arithmetic control answers, where the odds of acceptance were more than six times higher under the ChatGPT label, with an odds ratio of 6.08 and a 95 percent confidence interval ranging from 2.70 to 13.69. By contrast, the source contrasts within the two CRT categories were not significant, with odds ratios of 1.17 for both reflective-correct and intuitive-incorrect answers.

The control-answer contrast is especially informative because both conditions presented objectively correct arithmetic answers in matched displays, and judging them did not require resolving any intuitive-reflective conflict. Yet acceptance still differed by label: correct control answers were accepted 98.6 percent of the time under ChatGPT versus 92.4 percent under Computer, meaning participants rejected correct arithmetic nearly 1.4 percent of the time in one condition and 7.6 percent in the other. The authors caution that this pattern shows a contribution of attributed source identity to answer evaluation within this task, but it does not identify whether participants held different expectations about the two sources or verified particular answers differently.

Signal-detection analysis added an important nuance. Treating reflective-correct CRT answers as signal trials and intuitive-incorrect answers as noise trials, the researchers separated sensitivity, the ability to discriminate correct from incorrect answers, from response criterion, the willingness to say correct. Mean sensitivity was close to zero under both labels, and neither the sensitivity comparison nor the criterion comparison reached statistical significance, although the criterion was descriptively more liberal under ChatGPT. In other words, the study found greater source-related endorsement, but no evidence that the ChatGPT label improved participants’ actual discrimination between right and wrong CRT answers.

Evaluation times told a similarly qualified story. Raw mean latencies were about one second longer under the ChatGPT label, roughly 12,558 milliseconds versus 11,515 milliseconds, a difference that reached significance in the raw-time analysis. However, because the screened response times were strongly positively skewed, the researchers also ran the analysis on log-transformed times, and there the source difference was not significant. The authors note that longer evaluation could reflect scrutiny, uncertainty, attention to the source label, or hesitation, and that longer elapsed time can still end in the endorsement of an incorrect answer, so latency alone does not demonstrate successful verification.

The exploratory individual-difference analyses detected no significant associations between source-related differences in acceptance, accuracy, or latency and any of the measured dispositions. Trust in AI correlated positively with positive attitudes toward AI and negatively with negative attitudes, but neither trust nor attitudes nor critical-thinking disposition tracked how much more strongly a given participant endorsed ChatGPT-labeled answers. The authors stress that these correlations are descriptive and that the behavioral difference scores showed very low internal consistency, so the null findings cannot establish that these characteristics are unrelated to source effects.

The study has clear limitations. Participants were recruited by convenience, five trials per source-by-category cell is a brief assessment, and answer category and objective truth were not fully crossed, since reflective and control answers were correct while intuitive answers were incorrect. The comparison involved two technological sources, a named conversational AI and a generic computer, so the results do not show how either would fare against a human expert or an unlabeled answer, and prior work has documented both algorithm appreciation and skepticism toward AI-labeled content depending on task and domain. Even so, the findings sharpen an increasingly urgent question: as generative AI becomes embedded in contexts where verification carries real consequences, a simple source label can measurably raise the odds that people accept an answer, without any measurable improvement in their ability to tell correct answers from wrong ones.

Subject of Research: The effect of ChatGPT source attribution on answer acceptance in a cognitive reflection and arithmetic evaluation task

Article Title: The ChatGPT source cue shapes answer acceptance in a reasoning task

Article References: Fabio, R. A., Rantuccio, E., & Bruno, F. (2026). The ChatGPT source cue shapes answer acceptance in a reasoning task. Frontiers in Psychology, 17, Article 1923236. https://doi.org/10.3389/fpsyg.2026.1923236

Image Credits: AI Generated

DOI: 10.3389/fpsyg.2026.1923236

Keywords: ChatGPT, source attribution, Cognitive Reflection Test, evaluative judgment, answer acceptance, dual-process reasoning, trust in AI, signal detection, algorithm appreciation, critical thinking, generative AI, cognitive psychology