A comprehensive review evaluating artificial intelligence in communication studies reveals that while large language models can substantially optimize early questionnaire formulation and open-ended response processing, employing synthetic silicon samples introduces acute methodological limitations that threaten empirical rigor. Researchers Sebastián Valenzuela, Stephan Winter, and Sebastián Rivera synthesized current evidence across the survey research cycle, examining how computational systems interact with established empirical protocols. Their assessment emphasizes that although language models rapidly evaluate phrasing clarity and automate text standardization, they cannot reliably replace human respondents in substantive opinion research. The authors stress that maintaining scientific integrity requires treating artificial intelligence as a supportive analytical instrument rather than an uncritical substitute for empirical fieldwork.
Surveys have long served as a methodological cornerstone within communication scholarship, tracing directly back to foundational audience studies conducted by Paul Lazarsfeld and colleagues. By gathering structured self-reports, surveys systematically map communication frequencies, media behaviors, and theoretical correlations across defined populations. However, traditional survey administration faces mounting modern obstacles, including rising participant recruitment costs, declining response rates in high-quality sampling frames, and pervasive self-reporting inaccuracies regarding media exposure. As social scientists experience a broader computational turn, machine learning and natural language processing tools—historically applied to extract semantic meaning from broadcast and digital texts—are increasingly deployed to examine the beliefs, opinions, and behaviors of media consumers themselves.
In the questionnaire formulation phase, the authors document clear operational efficiencies when utilizing models to detect structural flaws before pretesting. Writing effective survey items is inherently complex, as double-barreled phrasings, ambiguous wording, or omission of critical response categories can compromise measurement validity. The review highlights evidence from consumer research where automated querying successfully identified vague and double-barreled items. In an empirical demonstration conducted with GPT-4o, researchers evaluating a categorical survey question regarding trusted news platforms found that the model identified missing options, recommending the addition of major platforms such as YouTube and TikTok, specialized networks like Reddit, Snapchat, and LinkedIn, as well as an essential open specification category.
Beyond structural error detection, language models facilitate linguistic and cross-cultural item adaptation, though they demand strict structural constraints. Relying on an established operational framework composed of an action verb, focus, context, alignment, and constraints, researchers can refine questions to avoid Western-centric framing. In testing focused on Latin American youth aged 18 to 24 regarding health trust, the authors observed that Gemini 2.5 Pro tailored phrasing to incorporate locally dominant platforms, specifically highlighting Instagram, TikTok, and WhatsApp. Furthermore, models provide cost-effective alternatives to preliminary cognitive pretesting by evaluating sentence complexity and generating alternative phrasings. However, because generated variations frequently mirror existing published literature, researchers face the risk of reproducing historical biases, requiring repeated prompting iterations to evaluate output stability.
The review identifies the deployment of synthetic data and silicon samples as the most methodologically fraught application in modern survey science. Proponents argue that querying language models parameterized with demographic personae—such as age, gender, and political ideology—can simulate human public opinion and bypass recruitment costs. Nevertheless, comparative empirical investigations cited in the review expose profound deficiencies in synthetic datasets. When Bisbee and colleagues queried ChatGPT to reproduce feeling thermometer evaluations from the American National Election Studies, the model produced comparable mean ratings across political groups but drastically compressed variance, failing to represent the substantive influence of specific predictor variables that form the core of explanatory research.
Wider empirical assessments reinforce these fundamental limitations regarding simulated survey participation. A systematic literature review conducted by Sarstedt and colleagues revealed that across early studies comparing silicon samples to genuine human samples in social science experiments and surveys, outputs matched human benchmarks in less than one-third of evaluated comparisons. These divergences stem from architectural realities: current commercial architectures operate as probabilistic black boxes prone to hallucination, lacking the transparency required for scientific replication. Furthermore, training corpuses remain predominantly concentrated within American and European contexts, severely undermining generalizability across non-Western societies and failing to capture rapid, contemporary shifts in public opinion or specialized media evaluations.
In contrast to synthetic generation, the processing and thematic coding of open-ended responses represents an area where language models offer validated utility. Open-ended survey questions provide rich descriptive qualitative insight but routinely generate considerable textual noise, including typographical errors, abbreviations, and informal colloquialisms that confound traditional frequency counts. Using data from the Chilean Youth, Media, and Participation project surveying urban residents aged 18 to 29 regarding personal role models, the authors demonstrated that Claude 3.7 Sonnet accurately standardized varied Spanish expressions—such as categorizing both colloquial and formal maternal references as identical conceptual units—while generating an English translation and documenting its recoding rationale.
For substantive analysis, language models significantly lower computational barriers for tasks traditionally requiring specialized scripting in statistical programming environments. Researchers can employ these systems to conduct automated descriptive analyses, classify textual narratives into predefined codebooks, execute emergent unsupervised categorization analogous to structural topic modeling, or standardize multilingual sentiment across diverse international samples. However, empirical studies caution that model-assigned thematic labels exhibit high sensitivity to minor alterations in prompt wording. Consequently, the authors mandate that any automated textual classification protocol must incorporate rigorous inter-coder reliability testing comparing artificial intelligence annotations against validated human-coded reference sets.
The authors also examine future survey possibilities alongside profound systemic risks, noting potential applications in downstream findings dissemination and interactive survey design. Integrated chatbots could conceivably introduce dynamic questionnaires that clarify items or offer real-time personalized stimuli, though personalized instruments inevitably reduce statistical comparability across respondents. More critically, uncritical adoption risks intellectual homogenization; an overreliance on automated suggestions may steer survey designs toward mainstream linguistic consensus while marginalizing novel theoretical hypotheses. Because closed proprietary platforms obscure training composition and algorithm updates impede exact computational replication, the research team emphasizes that silicon samples cannot supersede genuine human respondents.
Ultimately, the review asserts that navigating the adoption of generative artificial intelligence requires balancing logistical cost reductions against rigorous measurement validity. Communication scholars occupy a distinctive vantage point, uniquely situated to interrogate message design, audience reactions, and algorithmic transparency across digital media systems. By applying continuous critical assessment, strict prompt iteration, and empirical verification, survey researchers can successfully leverage language models to streamline questionnaire preparation and textual analysis without sacrificing the scientific standards, population representativeness, and empirical integrity that define modern social research.
Looking closer at questionnaire preparation, researchers face specific operational choices when translating theoretical constructs into measurable survey items. Beyond generating alternative wording, computational systems can assist in developing research stimuli, such as generating images based on textual prompts or suggesting response mechanisms suitable for Likert-type scales and open-ended formats. Pre-testing remains essential because cognitive interviews can reveal interpretive processes and uncover measurement error prior to deployment, but their resource demands often restrict implementation. By querying language models about question complexity and requesting simpler wording alternatives, researchers can implement a preliminary diagnostic filter before engaging human panels. Cross-cultural adaptations also demonstrate clear advantages, as seen in evidence where respondents evaluated survey questions adapted by language models more favorably than traditional versions when translating items across Western and non-Western populations.
The analytical lifecycle also extends into post-survey modeling, where computational tools offer functional support for complex statistical designs. Contemporary communication scholarship routinely applies advanced techniques, including structural equation modeling, multilevel modeling, mediation, moderation, and conditional process analyses, to capture conditional communication effects across different populations. Chatbots can assist scholars by recommending statistical workflows aligned with specific hypotheses and variables, or by translating analytical code across programming languages, such as converting Stata commands directly into R scripts. These automated analytical supports reduce technical friction, allowing investigators without deep computational backgrounds to execute data cleaning, missing value imputations, and systematic descriptive evaluations more efficiently.
Despite these efficiencies, the review highlights specific circumstances where synthetic samples might still offer bounded value. While unsuitable for substituting human populations in substantive communication research, synthetic respondents can assist preliminary pilot tests assessing study materials, support agent-based simulations of collective group behavior, or simulate historic populations where living survey respondents no longer exist. Scholars have proposed training models on historical texts and archives to capture the mentality of ancient populations during major events, addressing an inherent limitation of traditional survey polling that restricts measurement to contemporary subjects. However, the authors caution that applying synthetic datasets to historical or present inquiries demands extensive benchmarking against human samples and transparent acknowledgment of model limitations.
Broader professional challenges also accompany the integration of artificial intelligence into academic practice. Methodological stratification poses a tangible concern, as disparities in computational skills and access to advanced language models may disadvantage certain researchers. Furthermore, distinctions between closed commercial architectures and more accessible open-source systems highlight ongoing trade-offs between proprietary capabilities and methodological transparency. The authors observe that scholars often overlook the downstream dissemination phase, where models could assist in tailoring academic findings for stakeholders, designing visual figures, or drafting policy briefs across varied discourse communities. Because models trained on existing literature risk reinforcing prevailing orthodoxies, the authors conclude that intentional, transparent methodological choices remain indispensable to preserving scientific rigor across communication research.
Subject of Research: Methodological opportunities and limitations of large language models in survey research within communication science
Article Title: Using large language models for survey research in communication: opportunities and challenges
Article References: Valenzuela, S., Winter, S., & Rivera, S. (2025). Using large language models for survey research in communication: opportunities and challenges. Communication and Change, 1(1), Article 14. https://doi.org/10.1007/s44382-025-00014-z
Image Credits: AI Generated
DOI: 10.1007/s44382-025-00014-z
Keywords: large language models, survey research, communication science, synthetic samples, silicon samples, questionnaire design, open-ended coding, prompt engineering, measurement validity, inter-coder reliability, cross-cultural adaptation, data cleaning, methodological bias, reproducibility, algorithmic opacity, public opinion polling
Cite Scienmag News
Courtney Benton. (November 21, 2026). Language Models Offer Survey Efficiency but Pose Substantial Methodological Risks. Scienmag. https://scienmag.com/language-models-offer-survey-efficiency-but-pose-substantial-methodological-risks/
Courtney Benton. "Language Models Offer Survey Efficiency but Pose Substantial Methodological Risks." Scienmag, 21 November 2026, https://scienmag.com/language-models-offer-survey-efficiency-but-pose-substantial-methodological-risks/. Accessed 21 November 2026.
Courtney Benton. "Language Models Offer Survey Efficiency but Pose Substantial Methodological Risks." Scienmag. November 21, 2026. https://scienmag.com/language-models-offer-survey-efficiency-but-pose-substantial-methodological-risks/








