In the rapidly evolving landscape of artificial intelligence, large language models (LLMs) have taken center stage not only as tools for text generation but also as potential annotators capable of evaluating and classifying nuanced political discourse. A recent study by Vallejo Vera and Driggers delves into a compelling and somewhat disconcerting aspect of these models’ behavior: their susceptibility to party cues and the subsequent biases they introduce in labeling political statements. While humans bring their own spectrum of biases to political annotation, LLMs’ biases are embedded in the way they internalize and utilize political contextual information during text evaluation — an effect that may shape the future use of these models in social sciences and political communication research.
At the heart of this study lies a critical question: how do pre-existing political cues embedded in text influence the decisions made by LLMs compared to human coders? The researchers approached this question by systematically varying the prompts given to several LLMs, including instructions to answer as if they were an ‘average citizen,’ a ‘low-information citizen,’ or explicitly ignoring any known party affiliations mentioned in the text. Intriguingly, despite these modifications, LLMs consistently demonstrated bias, particularly favoring left-leaning party cues such as those from the Green Party and Austria’s Social Democratic Party (SPÖ). This indicates that these models rely heavily on political context encoded during their vast pretraining, rather than solely on the neutral evaluation of statement content.
To understand the magnitude and nature of this bias, the researchers aggregated responses from multiple LLMs and contrasted them with human coder outputs. The resulting analysis revealed a strong predisposition in LLMs towards interpreting statements with party cues differently than statements without. Perhaps counterintuitive, asking LLMs to consider themselves as less informed or average citizens actually amplified this bias, suggesting the models view less informed actors as even more reliant on the political signals embedded within text. This insight challenges assumptions that removing explicit party information or simulating ignorance might neutralize bias — the models appear unable or unwilling to disregard contextual party labels fully.
Moreover, the persistence of bias even when LLMs are explicitly instructed to ignore party cues underlines the profound influence of their original training data and architecture. Party labels embedded in text are not merely metadata but imbued with encoded political signals that the models decode and utilize, consciously or unconsciously, in their labeling output. This mechanism seems analogous, though distinct, to how human coders incorporate political context into their judgments about policy statements. Human annotation is inevitably colored by political leanings and expectations; LLMs appear to mirror this behavior, though differently calibrated and more rigidly tied to their training priors.
Diving deeper, Vallejo Vera and Driggers tested how these biases manifest across different policy domains by introducing an alternative dataset encompassing less salient political topics from Austrian parliamentary debates and party press releases spanning nearly two decades. Unlike the polarizing immigration-related data used initially, these statements generally lacked well-known party stances, providing a more ambiguous testing ground. Results here were revealing: LLMs exhibited little to no bias related to party cues on these less divisive issues. This suggests that the models’ partisan biases are context-dependent, surfacing most strongly when there exist clear, polarized party positions intricately woven into the political narrative. It aligns with emergent research showing that LLMs respond differentially to politically charged versus neutral topics, revealing how model behavior fluctuates with topic polarization intensity.
An additional layer of analysis focused on the ‘temperature’ setting— a parameter controlling the randomness of an LLM’s text generation. Generally, a lower temperature yields more deterministic outputs, while higher temperatures introduce more variability in responses. The team reasoned that elevated randomness could unpredictably amplify or dilute bias by changing the reliance on party cues. Experiments with temperature settings of 1 and 1.25 demonstrated mixed findings. Higher temperatures tended to reduce the probability of positive evaluations in general, yet the interaction with party cues yielded nuanced effects. For instance, party cues associated with the Green Party maintained positive bias even at higher temperatures, reflecting a resilience of certain biases under increased model variability.
Interestingly, statements associated with the Freedom Party of Austria (FPÖ), traditionally viewed as more right-wing and polarizing, showed increased likelihood of negative labels as temperature rose, suggesting that model randomness might exacerbate negative biases tied to controversial parties. Moreover, individual model differences emerged prominently: ChatGPT’s variants and LLaMa models produced divergent interactions between temperature and party cues, reflecting inherent architectural or training divergences. Across all models, higher temperature settings typically lowered inter-coder reliability, underscoring that increasing randomness may introduce noise complicating consistent political text annotation.
Crucially, these findings have profound implications for the use of LLMs as political annotators in research, policy analysis, and media monitoring. The entanglement of party cues with model outputs reveals inherent vulnerabilities: despite the appealing automation potential, LLMs may inadvertently reproduce or amplify partisan biases, especially in polarized domains. This challenges researchers to design more sophisticated prompting techniques, debiasing strategies, or hybrid human-machine approaches to ensure that automated annotations remain trustworthy and impartial.
The persistence of bias under conditions explicitly attempting to negate party influence also raises philosophical questions about the nature of ‘neutrality’ in machine judgment versus human evaluation. Are LLMs merely reflecting the political realities present in their training data, or does their architecture inherently fuse text with associated metadata to an extent exceeding human-like contextualization? This complex interplay between learned political knowledge and evaluation strategy makes disentangling bias a nontrivial task and calls for further exploration into model interpretability and fairness.
Furthermore, the contextual dependency of partisan bias—strong in polarized topics but muted in neutral domains—offers potential strategies for applying LLMs selectively, respecting their strengths and limitations. In less divisive arenas, LLMs might provide relatively unbiased assistance, while in highly contentious issues, human oversight or tailored intervention could be indispensable. This nuanced understanding moves beyond simplistic assumptions of LLM omnipotence toward a balanced appraisal of their role in political analysis.
Beyond politics, these findings resonate more broadly with concerns about AI systems reproducing societal biases embedded in training corpora. Political partisanship is but one vector by which models can inherit and perpetuate systemic prejudices, underscoring the critical need for transparency, rigorous evaluation, and proactive mitigation efforts across AI applications. The political domain, given its high stakes and visibility, serves as an ideal proving ground for developing best practices that might generalize to other sensitive fields such as healthcare, criminal justice, or education.
The study from Vallejo Vera and Driggers also elegantly ties into emerging literature on LLMs’ differential behavior in polarized versus neutral contexts. This growing body of work affirms that political complexity and social dynamics profoundly shape model outputs, encouraging future research into dynamic prompting, context-aware adjustment, and domain adaptation. It invites interdisciplinary dialogue bridging AI, political science, cognitive psychology, and ethics to better understand and harness the potentials of large language models responsibly.
As the development of LLMs continues apace, their increasing deployment in automated content moderation, opinion mining, and political sentiment analysis makes these insights highly timely. Stakeholders ranging from academic researchers to policymakers and journalists must account for the nuanced biases unearthed by this study to avoid misleading conclusions or unintended amplification of polarization. Implementing frameworks that monitor model bias, encourage transparency in annotation protocols, and involve human-in-the-loop designs can help navigate the challenges illuminated by this research.
Ultimately, the work underscores a key lesson: artificial intelligence systems are neither inherently neutral nor purely objective evaluators. Instead, they constitute socio-technical constructs deeply embedded in political and cultural contexts that shape their interpretive lenses. Recognizing and addressing their susceptibilities to party cues and other contextual influences is essential to unlocking their full potential as aids rather than adversaries in understanding the politically charged world around us.
Subject of Research: Investigation of biases in large language models (LLMs) resulting from political party cues influencing their annotation of political textual data.
Article Title: LLMs as annotators: the effect of party cues on labelling decisions by large language models.
Article References:
Vallejo Vera, S., Driggers, H. LLMs as annotators: the effect of party cues on labelling decisions by large language models. Humanit Soc Sci Commun 12, 1530 (2025). https://doi.org/10.1057/s41599-025-05834-4
Image Credits: AI Generated
