Combining Language Models and Knowledge Graphs Accurately Maps News Viewpoints

Computational researchers investigating media pluralism have developed an automated pipeline that couples fine-tuned large language models with structured background knowledge to classify political viewpoints in news reporting. Evaluating their architecture on a benchmark derived from the debate surrounding immigration in the United Kingdom, the research team demonstrated that combining fine-tuned language processing with semantic actor descriptions drawn from the open knowledge graph Wikidata achieved a peak classification performance of 91.74 percent F1 score. The investigation, reported by Massimiliano Fadda, Enrico Motta, Francesco Osborne, Diego Reforgiato Recupero, and Angelo Salatino in EPJ Data Science, addresses the technical challenge of evaluating whether modern media landscapes provide fair and balanced public debate. News sources fulfill a central democratic function by elevating specific topics, viewpoints, and voices, yet automated systems historically struggle to interpret complex, nuanced political assertions without contextual grounding. The findings demonstrate that while domain fine-tuning and knowledge graph representations each improve classification accuracy on their own, their joint integration delivers the highest classification performance, particularly when implemented with models capable of processing substantial contextual input windows.

The conceptual framework underpinning this research builds upon earlier work that established an ontology for capturing viewpoint diversity across public communication channels. In this analytical architecture, political discourse is decomposed into distinct, structured components rather than unstructured text snippets. A claim is formally defined as a sextuple encompassing an utterance, the specific actor making the statement, the news item reporting it, the publishing news source, the date of publication, and the overarching topic. Within this formalization, viewpoints are explicitly characterized as coherent collections of claims that are semantically and ideologically congruent, such as arguments asserting that immigration provides net economic advantages. By separating claims, topics, and viewpoints into distinctive ontological categories, the methodology enables analysts to trace how ideological arguments circulate across media venues, determine which political factions endorse specific perspectives, and assess overall media balance over time without collapsing complex political arguments into crude binary classifications.

This structured definition addresses key technical limitations observed across previous computational attempts to track media perspectives and stance. Earlier text mining pipelines frequently treated viewpoint identification as equivalent to claim extraction or relied exclusively on unigram distributions and topic models, which often generate noisy summaries or depend on explicit user-to-user conversational interactions on social networks. In addition, existing stance detection approaches generally restrict analytical categories to binary favor-or-against classifications regarding a particular controversy. The authors note that such binary structures fail to accommodate nuanced public discourse, where a given perspective may articulate logistical or procedural arguments without endorsing a polar positive or negative stance. Other contemporary perspective-discovery frameworks infer viewpoints directly from isolated claims without establishing an explicit topic representation, leaving the epistemological distinction between individual assertions and broader viewpoints ambiguous. Furthermore, while traditional political communication research relies on manual content analysis to study media framing and actor representation, human annotation remains inherently limited in scale. The hybrid pipeline was designed to bridge this divide, scaling analysis across large textual collections while preserving interpretable units aligned with political science methodologies.

To evaluate the upgraded classification architecture, the investigators focused on an empirical case study examining the United Kingdom immigration debate across news stories published between June 1 and August 31, 2023. The underlying news corpus was assembled using the Aylien News API by retrieving articles pre-tagged with the keyword immigration or containing closely related terms, such as immigrants, within their headline or body text. Using GPT-4o guided by structured prompts, the authors extracted 766 candidate claims attributed to specific speakers from the corpus, spanning both direct quotations and indirect paraphrases provided by journalists. The extracted statements ranged from concise atomic remarks about border enforcement to elaborate political assertions regarding parliamentary maneuvering and policy disputes. From this initial extraction, a subset of 402 claims was selected to form the foundation of a rigorous, standardized claim classification benchmark.

The identification of the thematic viewpoint space relied on a hybrid human-machine workflow originally introduced in the team’s prior baseline studies. Utterances were initially partitioned into smaller textual batches and ingested by GPT-4 Turbo, which extracted candidate perspectives for each segment before a subsequent consolidation prompt synthesized a coherent master list. Domain experts subsequently evaluated and refined the generated viewpoint definitions to ensure conceptual distinctness, appropriate granularity, and comprehensive coverage of the national debate. Through this review, human specialists contributed one entirely new category that the automated workflow had failed to identify, resulting in nine distinct viewpoints: immigration as a management issue, which represents a neutral administrative perspective focusing on logistical practicalities; immigrants as victims emphasizing humanitarian concerns; immigrants as potential criminals or security threats; enhancing or maintaining immigration pathways; restricting immigration pathways; economic benefits of immigration; economic costs of immigration; integration policies celebrating multiculturalism; and anti-integration policies emphasizing cultural preservation.

A core architectural innovation introduced in the study involves augmenting textual claim representations with semantic information describing the associated speakers, extracted from the Wikidata knowledge graph. Because political declarations are intrinsically shaped by an actor’s political history, institutional affiliations, and social identity, interpreting statements in isolation often leads automated classifiers to misunderstand nuanced or coded rhetoric. The researchers systematically mapped the 52 distinct actors responsible for the 402 benchmark claims to their corresponding Wikidata entity records. Through targeted programmatic queries, the system extracted standard properties including canonical labels, biographical descriptions, gender, occupations, historical positions held, political party affiliations, and, when recorded, political ideologies and philosophical or religious stances. These structured attributes were subsequently translated via simple text templates into concise biographical profiles, exemplified by detailed descriptions of public figures such as Prime Minister Keir Starmer, which could be appended directly to the input prompts supplied to the language models.

Claim classification was formalized as a supervised binary decision task: given an input claim consisting of an utterance and speaker name, an operational definition of a candidate viewpoint, and contextual data, the model outputs a binary flag indicating whether the claim aligns with the specified viewpoint. To rigorously evaluate the source of performance gains, the researchers designed three distinct contextual input configurations. The first configuration, designated as surrounding text, incorporated only the sentences immediately preceding and following the utterance within the original news article. The second configuration, designated as knowledge graph context, provided solely the templated actor biographical profile generated from Wikidata triples without surrounding journalistic text. The third configuration combined both surrounding news sentences and the structured actor profile into a single extended prompt. While the combined context introduces additional computational processing overhead, it delivers comprehensive background information that captures both the immediate conversational framing and the structural background of the speaker.

The empirical evaluation utilized the Immigration-3K dataset, constructed by re-annotating the 402 news claims against the nine defined political viewpoints. Five human annotators initially evaluated each claim-viewpoint combination, yielding 10,854 binary annotation pairs. An initial agreement assessment revealed a moderate average Cohen’s kappa score of 0.42 across the nine viewpoints, reflecting the inherent linguistic ambiguity and interpretive complexity of real-world political debate. To establish a dependable experimental ground truth, the authors applied a targeted filtering rule that removed 310 contentious claim-viewpoint pairs where isolated annotators disagreed sharply with their peers. This filtering process yielded 9,924 robust annotation instances and elevated the inter-annotator agreement to a substantial Cohen’s kappa score of 0.66. Applying majority voting across the remaining annotations produced the final Immigration-3K corpus containing 3,308 distinct evaluation instances, which was pre-split into standardized training, validation, and test subsets according to a 70 percent, 10 percent, and 20 percent ratio.

The benchmark experiments evaluated seven distinct instruction-tuned language models encompassing proprietary and open-weight architectures of varying scale: OpenAI’s GPT-4o mini, Mistral Nemo 12B, Google’s Gemma2 27B and Gemma2 9B, Meta’s Llama 3.1 8B and Llama 3.2 3B, and Microsoft’s Phi 3.5 mini. Each architecture was evaluated across all three contextual settings under two operational paradigms: zero-shot classification using structured prompting and supervised fine-tuning utilizing the Immigration-3K training and validation splits. Fine-tuning for GPT-4o mini was conducted through proprietary platform application interfaces, while the open-weight models were adapted using the open-source Unsloth training library. In total, the experimental matrix comprised 42 separate comparative trials, measuring model performance through precision, recall, overall F1 score, and chance-corrected Cohen’s kappa agreement against the human majority labels.

Across all evaluated architectures, supervised fine-tuning produced substantial performance gains over zero-shot baselines, confirming the essential value of task-specific training data. The highest overall performance was achieved by the fine-tuned version of GPT-4o mini operating on the combined context of surrounding news text and Wikidata actor profiles, which attained an F1 score of 91.74 percent and a Cohen’s kappa agreement of 0.835. This result marked a decisive improvement over the previous zero-shot baseline of 77.80 percent F1 obtained when using GPT-4o mini solely with surrounding textual context. Notably, fine-tuned open-weight models also demonstrated competitive analytical capability: fine-tuned Mistral Nemo 12B reached an 85.21 percent F1 score, while Gemma2 9B and Gemma2 27B achieved 84.78 percent and 84.13 percent F1 scores, respectively. These results confirm that accessible, open-weight language models fine-tuned with parameter-efficient techniques offer practical, inspectable alternatives for long-term academic research where proprietary model transparency and stability cannot be guaranteed.

The experimental outcomes revealed critical nuances regarding the integration of structured knowledge graphs and textual context across different model sizes. In zero-shot experiments, every evaluated architecture achieved higher classification agreement whenever Wikidata actor profiles were included, either alone or alongside textual context, outperforming setups that relied exclusively on surrounding news text. However, in fine-tuned settings, only GPT-4o mini successfully maximized performance by concatenating surrounding text with knowledge graph profiles. The remaining six models exhibited divergent behaviors: three models achieved peak accuracy using only Wikidata profiles, while three performed best using only surrounding article text, with the combined input producing a slight decline in classification metrics. The authors suggest this divergence indicates that smaller or context-constrained models may encounter difficulties reconciling disparate information modalities or processing expanded token lengths, even when nominal context windows support extensive inputs.

Performance breakdowns across the nine individual viewpoints revealed additional operational challenges rooted in real-world data distributions. While model accuracy aligned closely with aggregate metrics across most perspectives, emotionally complex viewpoints such as humanitarian emphasis presented distinct difficulties, showing marked performance drops among smaller models. More critically, severe class imbalance within the news corpus hindered performance on heavily under-represented perspectives, specifically those focusing on the economic benefits of immigration and the value of multicultural integration. The authors note that while filtering ambiguous claims elevated benchmark consistency, it intentionally removed real-world linguistic ambiguity, potentially yielding optimistic performance estimates compared to messy operational discourse. Furthermore, knowledge graph contextualization carries inherent risks of inheriting data biases from crowdsourced repositories or over-attributing historical political stances to actors making atypical statements. Future research plans include compiling multi-topic benchmarks covering domains such as climate change and health policy, exploring synthetic data generation to support rare viewpoints, and building full-scale regulatory analytics applications for social and political scientists.

Subject of Research: Automated political viewpoint classification in news media using language models and knowledge graphs

Article Title: Integrating Large Language Models and knowledge graphs to capture political viewpoints in news media

Article References: Fadda, M., Motta, E., Osborne, F., Reforgiato Recupero, D., & Salatino, A. (2026). Integrating Large Language Models and knowledge graphs to capture political viewpoints in news media. EPJ Data Science, 15(1), Article 68. https://doi.org/10.1140/epjds/s13688-026-00678-7

Image Credits: AI Generated

DOI: 10.1140/epjds/s13688-026-00678-7

Keywords: natural language processing, large language models, knowledge graphs, political communication, news analytics, text classification, wikidata, media discourse, Integrating, Large, Language, Models