PaperlayerView exact revision on arXiv

Source-linked AI summary

From Entity Mentions to Tone: An LLM-Based Pipeline for Media Bias Analysis

Klesti Hoxha, Olti Qirici

arXiv:2608.17454v1cs.CL

TL;DR

Media-bias analysis traditionally relies on costly, potentially biased manual investigation that is difficult to apply continuously. This paper develops and evaluates an LLM-based pipeline for Albanian news, finding useful semantic signals and practical trade-offs between annotation consistency, speed, and coverage.

  • Problem

    Manual news-provider investigations are costly, potentially biased, and difficult to apply continuously across individual providers or articles.

  • Method

    The paper builds a local-LLM pipeline for Albanian news that compares annotations with GDELT and analyzes framing, tone, and event coverage across sources.

  • Results

    The systems show moderate agreement while the LLM identifies additional person mentions; stricter validation improves consistency but reduces processing speed and annotation coverage.

  • Takeaways & Limitations

    The resulting source-, person-, and event-level views provide a practical basis for comparing coverage differences, especially in low-resource media settings.

Abstract

from arXiv · show

This paper presents a pipeline for analyzing media bias and framing in online news. The pipeline groups articles into topics and events, adds named-entity and sentiment annotations, and compares news sources through people mentions, source-level tone, and event-level coverage patterns. We apply it to 8,358 Albanian news articles collected from GDELT and compare the resulting annotations with GDELT's automated annotations. The results show moderate agreement for sentiment and entity extraction, as well as additional person-entity pairs that can potentially support the bias analysis. We compare two annotation prompts and find that stricter sentiment-validation rules remove label-score inconsistencies but increase execution time and reduce annotation coverage. Based on these results, the simpler prompt is used for the rest of the analysis. We have provided sample analysis on source-level framing pro les, person-level tone differences across sources, and event-level gatekeeping and coverage indicators. These outputs show how the same news collection can be used to examine what sources cover, how they describe public figures, and where coverage is concentrated. The approach is particularly useful in settings where manually verified datasets or specialized language tools are limited.

I. INTRODUCTION

The introduction motivates continuous, multi-level media-bias analysis for Albanian news, where manual investigations and language-specific resources are limited. It presents a local-model pipeline that compares annotation strategies and GDELT outputs while examining source-, person-, and event-level patterns.

  • Motivation: Online news and recommender systems can expose readers to disinformation and selective representations through topic, entity, and framing choices.The introduction links selection to individuals, organizations, or topics and framing to the tone used in coverage.
  • Problem: Manual media investigations are costly, heavyweight, potentially biased, and difficult to apply continuously to individual providers or articles.
  • Low-resource setting: Albanian news presents a low-resource setting with limited ready-made tools for entity recognition, sentiment analysis, and source comparison.These limitations complicate consistent questions about event coverage, public-figure mentions, and differences in tone across outlets.
  • Analytical framework: Media-bias analysis should separate source-level tendencies, person-level tone, and event-level coverage rather than rely on a single score.These signals respectively describe provider favorability or criticism, descriptions of public figures, and the breadth and concentration of event coverage.
  • Approach and contributions: The study implements a Kedro pipeline with a local Gemma model for Albanian news and tests whether local open-source models can support structured sentiment and entity analysis without prior language-specific adaptation.
  • Approach and contributions: 8,358 Albanian news articles are compared with GDELT annotations, alongside a comparison of two prompts and analyses of source-, person-, and event-level coverage.The stricter prompt removes label-score inconsistencies but slows processing and reduces annotation coverage.

II. BIAS AND FRAMING CATEGORIES

The paper organizes media bias and framing into recurring categories that distinguish selection of coverage, narrative description, and attention allocated to actors or events. It treats bias as deviation from neutrality and framing as the structure imposed after an event is selected for coverage.

  • Categories of Bias and Framing: The reviewed literature identifies recurring categories separating what outlets cover, how they describe it, and how much attention they give actors or events.These categories distinguish coverage selection, narrative description, and allocation of attention.
  • Media Bias: Media bias is commonly defined as a deviation from neutrality, with the paper’s considered categories summarized in Table I.Table I is titled “Media Bias Categories.”
  • Media Framing: Media framing describes how a narrative is structured after an event has been selected for coverage, using categories listed in Table II.The framing categories characterize these narrative choices.

III. RELATED WORK

Prior work uses LLMs and machine learning to measure news bias, framing, sentiment, and coverage at multiple levels. It also shows that bias-detection systems require careful mitigation because both datasets and LLM classifications can carry bias.

  • LLM-based bias and framing analysis: Recent studies use LLMs to detect bias and framing in news outlets.These studies motivate automated analysis of tone, topics, political lean, and framing.
  • LLM-based bias and framing analysis: Wang et al. developed a real-time dashboard combining LLM-based tone, topic, and political-lean metrics with event-level browsing and publisher-level aggregation.The system was evaluated with communication experts and crowdsourced participants and showed high statistical correlation with human-based evaluations.
  • LLM-based bias and framing analysis: Kumar et al. used LLMs to compare story framing, perform context-based sentiment analysis, and compare coverage across providers reporting the same event.Their framework supports side-by-side story comparison and highlights possible polarization in framing.
  • Pre-LLM and machine-learning approaches: Earlier systems ranked 50,000 global news sources using reporting bias, peer reputation, and social-media popularity, with good agreement against independent expert scorecards.Ye and Skiena’s pre-LLM framework focused on source-level rankings rather than LLM-based annotation.
  • Limitations and mitigation: LLM-based bias analysis remains challenging because identical phrases can receive different sentiment labels when political targets are substituted.Elbouanani et al. found that replacing politician names with fictitious ones reduced bias during sentiment classification.

IV. METHOD

The method introduces an LLM-based pipeline for Albanian media-bias analysis that groups news into topics and events, enriches articles with entity and sentiment annotations, and measures source-level framing and coverage patterns.

  • Pipeline design: The pipeline tracks people mentions and sentiment polarization, or tone, in Albanian news articles to analyze media bias.It is designed for a low-resource language setting.
  • Data and processing: 8,358 Albanian news articles from GDELT, published in April 2026, form the experimental dataset.The dataset is described in Table III.
  • Data and processing: TF-IDF and k-means group incoming news into topics, while TF-IDF, cosine similarity, and time windows identify events within topics.Each news item is identified by a unique URL.
  • Annotation and implementation: Gemma 41, a local LLM, enriches each news item with named-entity recognition and sentiment-analysis outputs.The pipeline was developed using the Kedro2 Python framework.
  • Bias and framing measures: Bias and framing are measured through tone variation in topic and event coverage, tonal bias in people mentions, and repeated or omitted coverage across sources.The latter produces possible gatekeeping indicators for people, topics, and events.

V. EXPERIMENTS AND RESULTS

The section presents experiments and their results to demonstrate the feasibility of implementing the pipeline through potential use cases.

  • The experiments and results assess the feasibility of implementing the pipeline.
  • The section presents potential use cases as evidence of the pipeline’s practical implementation.
  • The section combines experimental evaluation with use-case demonstrations.

A. NER and Sentiment Enrichment

The pipeline enriches Albanian news articles with deterministic LLM-based named-entity and sentiment annotations, comparing two prompts and evaluating outputs against GDELT. Prompt v2 improves label-score consistency but is slower and offers no substantial evaluation benefit, so Prompt v1 is used for analysis.

  • Annotation setup: The enrichment uses Gemma 4 with deterministic generation to extract sentiment and named entities from each article’s full text.Inference used temperature 0.0, a 250-token generation limit, a 90-second timeout, and one retry.
  • Prompt comparison: 8.53% was the Prompt v1 inconsistency rate between sentiment labels and their corresponding scores.Prompt v2 adds explicit validation rules requiring score ranges and label-score alignment.
  • Evaluation against GDELT: The LLM and GDELT annotations showed moderate alignment, with Prompt v2 achieving slightly higher sentiment label agreement and the LLM identifying more person-entity pairs.The comparison used articles labeled by both systems; NER evaluation focused on PERSON entities and sentiment agreement measured matching labels.
  • Evaluation against GDELT: Higher recall than precision indicates that the LLM recovered most GDELT person names while also extracting additional names requiring manual validation.These additional entities may support media-bias analysis when better alternatives are unavailable, but the comparison with GDELT is not a definitive gold-standard evaluation.
  • Prompt comparison: Prompt v2 was slower, produced fewer labeled articles, and did not substantially improve evaluation results, so the analysis continued with Prompt v1.The gain in consistency did not justify the doubled execution time; remaining inconsistencies could be handled during cleaning.

B. Media Bias and Framing Analysis

The analysis compares source-level framing, person-level tone differences, and event-level coverage concentration using sentiment, bias, and gatekeeping measures. These outputs support cross-source comparisons while remaining a baseline because models may introduce biases when processing political mentions.

  • Source-level framing: Source profiles summarize positive, neutral, and negative framing alongside mean sentiment, average topic bias, and deviation from the corpus baseline of −0.192.Avg. bias averages per-topic positive-minus-negative rates, while Bias vs. corpus indicates whether coverage is more favorable or critical than the corpus average.
  • Person-level tone: Person-tone analysis displays the lowest and highest tone-balance rows across sources to show how the same individuals receive more favorable or critical portrayals.Tone balance is the positive-minus-negative mention rate, ranging from −1 for uniformly negative coverage to +1 for uniformly positive coverage.
  • Limitations: Pre-trained models may bias politician mentions, but the results remain a useful baseline and can be mitigated through name anonymization or user-provided feedback.The limitation concerns model behavior when processing mentions of politicians in news articles.
  • Event-level coverage: Event-level indicators measure article volume, distinct covering sources, omitted sources, gatekeeping, and framing polarity to identify broad or concentrated coverage.Gatekeeping Score is defined as 1 − covering sources total active sources, while Framing Polarity is the event-level positive-minus-negative rate.

VI. DISCUSSION

The discussion emphasizes that GDELT comparisons and person-level tone differences require caution because both automated annotations and model biases affect interpretation. It also identifies Prompt v1 as the more practical continuous-pipeline choice and highlights LLMs’ usefulness for low-resource, messy news data.

  • Annotation comparison: GDELT is another automated annotation system rather than a human-labeled gold standard, so differences do not automatically establish which system is wrong.The systems show moderate agreement while also capturing different parts of the text.
  • Annotation comparison: Many person mentions found by the LLM were absent from GDELT, which can support bias analysis by strengthening later source and tone comparisons.Missing a public figure can weaken downstream comparisons.
  • Prompt selection: Prompt v2 reduced sentiment label-score inconsistencies but processed fewer articles and took longer, whereas Prompt v1 offered broader coverage and faster execution.The paper therefore uses Prompt v1 for the continuous pipeline, with remaining inconsistencies handled by simple rule-based checks.
  • Data preparation: LLM-based annotation can reduce preparation needs by normalizing inconsistent names, spellings, formatting, and mixed-language forms in multilingual, low-resource news data.Traditional processing would require substantial manual cleaning before reliable entity and sentiment analysis.
  • Interpretation and limitations: Large person-tone differences may reflect editorial framing or model bias, especially for political figures, motivating mitigation through neutral placeholders and human feedback.The model may have learned associations before seeing the analyzed article.

VII. CONCLUSION

The paper presents an end-to-end local-LLM pipeline for media-bias, framing, and gatekeeping analysis, demonstrated on 8,358 Albanian news articles in a low-resource setting. Its outputs support source, person, and event comparisons while motivating human validation, lighter-weight models, and integration into monitoring systems.

  • Conclusion: The pipeline uses local LLMs and Kedro to analyze media bias, framing, and gatekeeping in 8,358 Albanian news articles.The study shows that Gemma 4 can extract useful semantic signals without a large local training corpus.
  • Conclusion: Future work will train lighter-weight models using LLM annotations and crowdsourced data, addressing low coverage rates and the computational cost of NER and sentiment extraction.The authors also plan an MCP server and continuous monitoring integration for news aggregators.
  • Conclusion: Source-level profiles, person-level tone balances, and event-level indicators support comparisons of framing, public-figure treatment, and coverage breadth.These views can help readers inspect coverage differences more easily.
  • Conclusion: LLM enrichment reduces the material requiring manual inspection by converting large article collections into clearer signals, especially where language-specific NLP tools are limited.The pipeline is intended to support, not replace, editorial judgment or expert review.
  • Conclusion: LLM annotations provide a fast bootstrapping layer and baseline, while later editorial validation can focus on extracted names, aliases, and disputed tone assignments.This approach is particularly useful when no manually normalized dataset is available.
Loading 2608.17454v1…