PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Judges the Frame? Auditing Multimodal LLM Judges for News Framing Across Event-Level Perspectives

Antonela Tommasel, Markus Schedl

arXiv:2610.00071v1cs.CY

TL;DR

LLM judgments of news framing may reflect model, prompt, and contextual assumptions as well as content, making them measurement outputs to audit. This study combines event-level embeddings and LLM assessments across modality and metadata conditions. It finds modality is the strongest source of variation and recommends reporting input conditions alongside framing analyses.

  • Problem

    LLM assessments may depend on model tendencies, prompt design, and social, political, cultural, linguistic, or modality-specific assumptions, complicating their use as measurements of news framing.

    Source: §1 · p. 1

  • Method

    The study combines within-event embedding comparisons for left-, center-, and right-oriented news articles with LLM assessments of framing constructs across modality and metadata conditions.

    Source: §1 · pp. 1–2

  • Results

    Modality was the strongest source of variation: representations tracked modality more than ideology, while framing scores were more stable for text-only versus multimodal than text-containing versus image-only inputs.

    Source: §4 · p. 8

  • Takeaways & Limitations

    Modality and metadata visibility should be reported as part of the measurement setup for LLM-generated framing assessments.

    Source: §4 · p. 8

    The audit covers one LLM and one dataset, does not test alternative prompts, and does not validate model-generated Likert scores against human or expert annotations.

    Source: §4 · p. 8

Abstract

from arXiv · show

News coverage of major world events is shaped not only by what is reported, but also by how events are framed through text, images and their combination. At the same time, Large Language Models (LLMs), including multimodal LLMs, are increasingly used as scalable instruments for analysing framing, sentiment, ideological slant and perspective differences in multimodal media datasets. This creates a methodological challenge. When used as measurement instruments, LLM outputs may reflect not only content properties, but also model-specific tendencies, prompt design choices, and social, political, cultural, linguistic or modality-specific assumptions. This work audits LLMs as instruments for large-scale framing and perspective analysis in multimodal news coverage. Using an event-centered dataset of 2025--2026 news coverage, where each event includes left-, center- and right-oriented articles about the same headline, we combine embedding-based measures of within-event viewpoint similarity with model-based assessments of framing constructs across modality-specific and metadata-visible input conditions. Rather than treating either dataset labels or model outputs as ground truth, our goal is to examine the usefulness and limitations of LLM-based media analysis. The study contributes an audit protocol that highlights the need to report modality effects, metadata sensitivity, and prompt-induced artifacts alongside substantive claims about news framing and ideological viewpoint differences.

1 Introduction

LLMs offer scalable ways to analyze framing and perspective in multimodal news, but their judgments may reflect model, prompt, and contextual assumptions as well as content. This study audits how those judgments vary with modality and metadata in event-centered news coverage.

1 Introduction

  • LLM judgments may reflect model tendencies, prompt design, and social, political, cultural, linguistic, or modality-specific assumptions, so they are situated outputs rather than neutral observations.The concern is especially relevant when high-impact events are represented through different headlines, summaries, images, actors, and source contexts.Source: §1 · p. 1
  • The study tests whether assessments of the same event-level perspectives remain stable or change across modality and metadata conditions, without treating LLMs or ideological labels as ground truth.It aims to examine LLM-based analysis rather than replace human perception studies or establish definitive ideological ground truth.Source: §1 · p. 1
  • The event-centered dataset pairs left-, center-, and right-oriented articles covering the same headline, allowing perspective comparisons while holding the topic fixed.The ideological categories are reference labels for analysis, not absolute ground truth.Source: §1 · p. 1
  • The audit combines embedding-based perspective similarity with LLM assessments of four framing constructs across modality and metadata conditions.The constructs are emotional intensity, conflict framing, polarization framing, and sensationalism; the embeddings provide a non-generative baseline for contextualizing model judgments.Source: §1 · pp. 1–2
  • The study argues that framing analyses should report not only model judgments but also how the models and protocols shape those judgments.Its contribution is an audit-oriented approach to LLM-based framing and perspective analysis in multimodal news datasets.Source: §1 · p. 2

2 Experimental Methodology

The methodology separates representation-level differences among event-level perspectives from model judgments about framing. It applies embeddings and controlled LLM assessments to a matched news dataset, varying modality and metadata visibility.

2 Experimental Methodology

  • The three research questions distinguish event-level viewpoint divergence from framing judgments that may vary with modality or visible metadata.RQ1 provides a non-generative baseline; RQ2 and RQ3 test model assessments under different input conditions.Source: §2 · p. 2

2.1 Data Collection

  • The final paired multimodal subset contains 915 headlines with left-, center-, and right-oriented articles, each with at least one usable image.The dataset covers 2025–2026 news; articles are matched to a shared headline, and retained text and images are not preprocessed.Source: §2.1 · p. 2

2.2 Model Selection

  • The study uses one locally runnable multimodal LLM as a controlled case study, prioritizing consistent application over performance maximization or representativeness.The selected model has 5.12B parameters; calls use temperature zero, a fixed evaluation prompt, and structured JSON output.Source: §2.2 · p. 2

2.3 RQ1. Event-level Viewpoint Divergence

  • For event-level divergence, the study compares pairwise cosine similarities among left–center, center–right, and left–right article embeddings across text, image, and combined representations.Higher similarity indicates lower viewpoint divergence; the analysis also summarizes average within-event similarity, left–right similarity, and center asymmetry.Source: §2.2 · pp. 2–3

2.4 RQ2. Modality Sensitivity in Framing Assessments

  • For modality sensitivity, the same article-level prompts assess four framing constructs using text-only, image-only, or multimodal inputs and 1–5 Likert scores.The scores are compared across conditions, including text-only versus multimodal to estimate the change associated with adding images to article text.Source: §2.4 · p. 3

2.5 RQ3. Metadata Sensitivity in Framing Assessments

  • For metadata sensitivity, the study compares content-only scores with scores when source information and ideological reference labels are visible, measuring rank-pattern correlations and signed shifts.The task, construct definitions, scale anchors, and output format stay fixed across conditions; shifts are summarized by construct, modality, and reference category.Source: §2.4 · p. 3

3 Experimental Results

Across the three analyses, embedding structure and model judgments vary with modality; metadata changes assessments most in image-only conditions. Within-event ideological pairs do not show a simple endpoint separation.

3.1 RQ1. Event-level Viewpoint Divergence

  • The embedding projection is organized more clearly by modality than by ideological reference category, with multimodal representations between text- and image-based regions.Left-, center- and right-oriented articles are substantially intermixed in the projection.Source: §3.1 · pp. 3–4, Figure 1 · p. 5
  • Within-event left–center, right–center and left–right similarity distributions broadly overlap; left–right pairs are not consistently less similar.Similarity varies across topic and modality, rather than showing a simple linear separation between ideological endpoints.Source: §3.1 · p. 4, Figure 2 · p. 5, Figure 3 · p. 6

3.2 RQ2. Modality Sensitivity in Framing Assessments

  • Across four constructs, text-only and multimodal score distributions were broadly similar, whereas image-only scores were substantially lower for most constructs.This pattern suggests that multimodal judgments were largely anchored in text when full article text was available.Source: §3.2 · p. 4, Figure 4 · p. 7
  • Text-only versus multimodal Spearman correlations ranged from 0.68 for sensationalism to 0.80 for polarization framing; ICC ranged from 0.70 for sensationalism to 0.83 for conflict and polarization framing.These measures capture agreement in relative score patterns and absolute Likert levels, respectively.Source: §3.2 · p. 5, Table 3 · p. 5
  • Text-only versus image-only agreement was low: Spearman correlations were 0.11–0.19, and ICC values were 0.02–0.04 for three constructs and 0.04 for emotional intensity.Image-only and headline-text-image agreement was similarly low, with Spearman correlations of 0.12–0.17 and ICC values of 0.03–0.05.Source: §3.2 · p. 5, Table 3 · p. 5

3.3 RQ3. Metadata Sensitivity in Framing Assessments

  • For content-only versus source+label, Spearman correlations were 0.70–0.79 for text-only, 0.76–0.84 for multimodal, and 0.00–0.05 for image-only.Text-containing correlations remained high across left-, center- and right-oriented reference categories; image-only correlations stayed near zero across categories.Source: §3.3 · p. 7, Table 4 · p. 7
  • Signed shifts show no simple ideological-label effect; the strongest directional changes occur for image-only inputs, where metadata generally lowers scores.For left-oriented image-only articles, source+label shifts were -0.38 for conflict framing, -0.71 for emotional intensity, -0.16 for polarization framing, and -0.32 for sensationalism.Source: §3.3 · pp. 7–8, Table 5 · p. 8

4 Conclusions

The audit finds that framing judgments depend on input conditions, with modality emerging as the strongest source of variation. Its conclusions are bounded by the models, dataset, prompts, and validation used.

4 Conclusions

  • Article representations varied more by input modality than ideological reference category, while framing scores were more stable between text-only and multimodal than text-containing and image-only inputs.Modality emerged as the strongest source of variation across the embedding and judgment experiments.Source: §4 · p. 8
  • Metadata sensitivity was limited when text was available but more consequential in image-only conditions, so modality and metadata visibility should be reported with measurement setups.Source: §4 · p. 8
  • The findings are a focused audit of one multimodal LLM judge and one event-centered dataset, while embedding-space findings depend on a single embedding model.Source: §4 · p. 8
  • The study did not test prompt robustness or validate model-generated Likert scores against human or expert annotations.Source: §4 · p. 8
  • Because the data concern socially and politically consequential events, over-interpreting model judgments could misrepresent coverage or simplify complex public issues.Source: §4 · p. 8
Loading 2610.00071v1…