PaperlayerView exact revision on arXiv

Source-linked AI summary

Inference-Time Mitigation of Adversarial Political Bias in Large Language Models

Tejaswi V. Panchagnula, Bruce Coburn, Bryce J. Dietrich, Robert X. Browning, Edward J. Delp, Fengqing Zhu

arXiv:2608.14629v1cs.CLcs.AI

TL;DR

Political bias in LLM-generated political summaries remains vulnerable to adversarial prompting. This paper evaluates CoT- and DPO-based mitigation strategies, finding that Recursive Self-Correction brings performance close to the neutral baseline.

  • Problem

    LLMs used in political and news media require methods to identify and mitigate political bias that can influence decisions and shift narratives.

  • Method

    The paper compares CoT prompting and DPO, introducing Recursive Self-Correction to iteratively audit and remove partisan framing from summaries.

  • Results

    Recursive Self-Correction recovers performance close to the neutral baseline by the third turn when summaries face jailbreak-style political prompts.

  • Takeaways & Limitations

    Iterative CoT-based self-correction can mitigate injected political bias using the base LLM model during inference.

Abstract

from arXiv · show

As Large Language Models (LLMs) become the mainstay for information retrieval and summarization tasks, ensuring that they are always non-partisan and invulnerable to political bias is a critical step towards safer and more trustworthy Artificial Intelligence (AI). Current model alignment paradigms, such as reinforcement learning from human feedback (RLHF), make LLMs follow overarching safety instructions. However, this instruction tuning can be exploited via adversarial prompt injection and be used to generate unsafe content. In particular, political bias has not been specifically targeted by modern alignment techniques as harmful and biased content. To address this vulnerability of LLMs, we propose mitigation strategies using Chain of Thought (CoT) prompting and Direct Preference Optimization (DPO). Using a public dataset of legislative videos, we generate summaries using LLMs, inject bias via adversarial prompting and evaluate their performance on a four axis scale designed for political summarization. In this paper, we present different methods to shield LLMs against the injection of political bias. Our results demonstrate that the proposed Recursive Self-Correction approach raises model performance from a Political Neutrality Likert scale baseline of 2.14 to 4.56, averaged across all models, demonstrating effective inference-time mitigation of political bias in LLM-generated summaries.

I. INTRODUCTION

LLMs are increasingly used in political and news media, creating a need to identify and mitigate political bias that can influence decisions and policy narratives. This paper proposes Recursive Self-Correction, compares inference-time CoT prompting with DPO fine-tuning, and evaluates political-video summaries using a multi-axis LLM-as-a-judge metric.

  • Motivation: LLMs increasingly generate political and news media content, making political-bias identification, mitigation, and control necessary because bias can influence decisions and policy narratives.The introduction connects expanded media use with the need to control politically biased outputs.
  • Contributions: The paper proposes Recursive Self-Correction to mitigate political bias through an iterative Chain of Thought prompt strategy.The method is presented as a contribution for mitigating political bias at inference time.
  • Contributions: The study compares inference-time Chain of Thought prompting with Direct Preference Optimization, implemented as a model fine-tuning approach.This comparison places prompt-based mitigation alongside a fine-tuning strategy.
  • Contributions: The paper introduces a multi-axis evaluation metric using an LLM-as-a-judge to assess summaries generated from political videos.The evaluation targets summaries produced from political-video content.
  • Background: The work builds on video summarization methods that use transcript information, video frames, or both, with LLMs contributing semantic understanding and representation power.Traditional approaches include key-frame sampling and spatiotemporal extraction, while LLM-based methods support semantic summarization.

II. METHODOLOGY

The methodology addresses the difficulty of predicting the direction and magnitude of LLM bias by using two bias-injection methods and investigating three mitigation approaches. The evaluation framework is summarized in an overall block diagram.

  • Methodological Rationale: LLM bias direction and magnitude are difficult to predict for a given prompt.This motivates the study’s experimental design.
  • Bias Injection: The study uses two methods to inject bias into LLM outputs.
  • Bias Mitigation: The study investigates three approaches to mitigate political bias.
  • Evaluation Framework: An overall block diagram presents the evaluation framework.The diagram is identified as Figure 2.

A. Video Summarization · B. LLM-as-a-judge

The pipeline transcribes legislative videos with Whisper, summarizes transcript text using five LLMs at a 100-word target, and evaluates outputs with an LLM judge. Judging uses four 1–5 Likert axes covering coverage, factuality, coherence, and political neutrality.

  • A. Video Summarization: Whisper API transcribes the videos before summarization, producing transcripts averaging 1,147 words and reaching 2,205 words at maximum.Only textual transcripts are used as the source during summarization.
  • A. Video Summarization: Five modest-compute HuggingFace model families summarize each transcript: Llama-3-8B-Instruct, Gemma-2-9B-it, Phi-4-mini-Instruct, Ministral-3-14B-Instruct-2512-BF16, and Qwen3.5-9B.The models are described as the latest iterations of their most downloaded families.
  • A. Video Summarization: The summarization task sets the output word count at 100.Each of the five LLMs is asked to summarize every transcript it reads.
  • B. LLM-as-a-judge: The LLM-as-a-judge paradigm uses another larger model to grade generated outputs when ground-truth data and standard quantitative metrics are insufficient.The paper uses this approach because generated content lacks ground-truth data for fully assessing quality and fidelity.
  • B. LLM-as-a-judge: Information Coverage measures whether summaries capture the transcript’s core legislative topics and facts.This axis focuses on the substantive content retained from the legislative discussion.
  • B. LLM-as-a-judge: Factual Alignment measures whether summaries hallucinate external details or numbers absent from the transcript.The axis evaluates adherence to information explicitly mentioned in the source.
  • B. LLM-as-a-judge: Argumentative Coherence evaluates logical transitions and overall readability, while Political Neutrality assesses whether summaries endorse partisan viewpoints.All four axes—Information Coverage, Factual Alignment, Argumentative Coherence, and Political Neutrality—use a 1–5 Likert scale with score-trait guidance for the judge LLM.

C. Biased Summary Generation

The study quantifies political-bias injection by generating LLM summaries under different partisan viewpoints, using Section II-A transcripts as baseline reference texts. It compares Neutral, Liberal, and Conservative prompt slants while defining the LLM as a “helpful assistant.”

  • Prompt Design: The experiment measures political-bias injection by prompting LLMs to summarize transcripts from different partisan viewpoints.The transcripts generated in Section II-A serve as baseline reference texts.
  • Prompt Design: The prompts use three slants: Neutral, Liberal, and Conservative.The Neutral viewpoint does not inject bias into the generated summary.
  • Prompt Design: In every prompt style, the LLM is defined as a “helpful assistant.”

D. Jailbreaking LLMs · III. EXPERIMENTS

The paper examines political-bias jailbreaking in LLM summarization and evaluates mitigation methods on congressional video summaries. Experiments use adversarial prompts, four evaluation axes, and an LLM judge across multiple models.

  • D. Jailbreaking LLMs: Figure 3 illustrates a conservative-bias jailbreak prompt with salient filter-bypassing phrases highlighted, alongside a symmetric liberal-bias prompt.The figure demonstrates how instruction tuning can be exploited for political-bias injection.
  • D. Jailbreaking LLMs: Adversarial prompting crafts prompts to bypass LLM safety filters and force behavior outside intended guidelines, including political bias.The paper identifies prompt injection, jailbreaking, and prompt leaking as common techniques.
  • D. Jailbreaking LLMs: Jailbreaking exploits failure modes in current LLM training and is studied here through political-bias injection in summaries.The jailbreak prompt contains system-override phrases and emphasizes adopting a specified political bias.
  • D. Jailbreaking LLMs: The study tests these prompting strategies across all evaluated models and scores outputs on four summarization axes using a judge model.The prompts instruct models to disregard previously installed guardrails.
  • III. EXPERIMENTS: 100 videos from the C-SPAN Video Library form the experimental subset, with mean duration 7 minutes 51 seconds and standard deviation 2 minutes 27 seconds.The videos comprise around 11 hours of congressional floor proceedings and target moderate-length recordings.
  • III. EXPERIMENTS: The experiments investigate Chain of Thought prompting, Direct Preference Optimization, and Recursive Self-Correction to mitigate bias in LLM outputs.Outputs are evaluated on the four defined axes using an LLM-as-a-judge.
  • III. EXPERIMENTS: Implementation uses PyTorch on a single Nvidia GB-10 GPU, with vLLM serving as an inference accelerator using paged attention.The experimental setup evaluates different models and mitigation approaches.

A. Zero Shot Chain-of-Thought · B. Direct Preference Optimization

The paper uses zero-shot Chain-of-Thought prompting as a low-cost inference-time strategy to detach summaries from ideological bias, and Direct Preference Optimization to alter model weights toward unbiased generation. The DPO formulation optimizes preferred over non-preferred responses while constraining deviation from a frozen reference policy.

  • A. Zero Shot Chain-of-Thought: The CoT method uses a reasoning block containing four explicit instructions to guide unbiased summarization.The reasoning block is designed to help the LLM detach from bias in the prompt before producing the summary.
  • A. Zero Shot Chain-of-Thought: Zero-shot Chain-of-Thought prompting is introduced as a low-cost inference-time mitigation strategy for bias in generated summaries.Its two-step process first detaches ideology from the biased prompt while maintaining neutrality and transcript faithfulness.
  • A. Zero Shot Chain-of-Thought: The first CoT step requires detaching ideology from the biased user prompt and preserving strict neutrality and faithfulness to the transcript.This step precedes the reasoning block that supplies the remaining guidelines.
  • B. Direct Preference Optimization: Direct Preference Optimization is used to fundamentally alter LLM weights so inference under biased prompts produces un-biased summaries.The method begins by defining an implicit reward function rθ(x, y) for response y to prompt x.
  • B. Direct Preference Optimization: The DPO objective simplifies to binary cross-entropy over the reward difference between preferred response y_w and non-preferred response y_l.The training data consists of triplets (x, y_w, y_l).
  • B. Direct Preference Optimization: β = 0.1 controls the KL-divergence penalty that constrains the optimized policy π_θ relative to the frozen reference policy π_ref.The formulation also uses the logistic sigmoid function σ.
  • B. Direct Preference Optimization: Both 8B and 70B models show similar reductions in performance when moving from neutral to biased prompts.The comparison reports Factual Alignment (FA), Information Coverage (IC), Political Neutrality (PN), and Argumentative Coherence (AC).

C. Recursive Self-Correction

Recursive Self-Correction uses a three-stage Chain-of-Thought process to resist jailbreak-induced political bias. It combines baseline generation, internal auditing, and corrective revision, with two implementation variants.

  • Process: The three-stage process generates a baseline summary, audits it through an “Internal Auditor” role, then revises it using the audit commentary.The final revision also uses the original biased summary, transcript, and instructions to fix identified errors.
  • Implementations: The first implementation uses the three-turn approach described for Recursive Self-Correction.
  • Implementations: The second implementation adds the <reasoning-block> from a Zero-Shot CoT prompt as an additional guardrail for Turn 1.

IV. RESULTS · A. Baseline Biased Summaries

Table II evaluates adversarial-prompting mitigation using Political Neutrality scores, where higher values indicate better performance on a 1–5 scale. Baseline biased summaries fell from 4.24 to 2.14 averaged across all models, showing susceptibility to biased prompting despite source transcripts.

  • IV. RESULTS: Table II reports Political Neutrality scores from an LLM-as-a-judge, with higher scores indicating better performance.The metric uses a 1–5 scale.
  • IV. RESULTS: 1 represents an extremely biased summary, whereas 5 represents an unbiased summary.
  • A. Baseline Biased Summaries: 2.14 from 4.24 on the Political Neutrality axis marked the baseline decrease averaged across all models.
  • A. Baseline Biased Summaries: The decrease shows that LLMs are susceptible to biased prompting despite having a source transcript.
  • A. Baseline Biased Summaries: The generated summaries qualitatively displayed clear political bias and sycophancy toward users’ views despite a clear transcript for summarization.
  • A. Baseline Biased Summaries: The observed susceptibility can be attributed to the instruction-tuning post-training paradigm.The supplied passage identifies this paradigm as allowing the described behavior.

B. Bias Mitigation Strategies

The section evaluates zero-shot Chain-of-Thought, Direct Preference Optimization, and Recursive Self-Correction as defenses against injected political bias. Recursive Self-Correction brings jailbreak-prompt performance close to the neutral, unbiased level by Turn 3, while safety filters usually fail to detect politically biased jailbreaks.

  • Zero Shot Chain-of-Thought: Zero-shot Chain-of-Thought improves all models toward neutral, unbiased summary performance by localizing prompt bias and redirecting attention to the source transcript.The method functions as a guardrail without fine-tuning the model.
  • Direct Preference Optimization: Direct Preference Optimization makes models more resistant to adopting user-prompt bias and restores performance to prior levels on biased prompts.DPO changes model weights through Low-Rank Adaptations.
  • Effects of Jailbreaking: 7% of 1,200 evaluated prompts triggered safety filters correctly, allowing models to sidestep the request.Jailbreak prompts commonly concealed malicious requests within professional tasks, avoiding safety-filter activation.
  • Recursive Self-Correction: Recursive Self-Correction uses iterative CoT prompting to neutralize adversarial jailbreaks, with Turn 3 performance close to the neutral, unbiased level.CoT prompts also improve performance at Turn 1, and both recursive methods recover close to neutral performance by Turn 3.

V. CONCLUSION

The paper presents methods to inject political bias into LLM-generated summaries and strategies to mitigate or prevent it. Chain-of-Thought prompting returns performance close to the original Neutral baseline for single biased prompts.

  • V. CONCLUSION: The paper presents methods for political bias mitigation in Large Language Models and demonstrates injection through biased prompts and jailbreak attempts.The proposed strategies aim to mitigate or prevent bias in LLM output summaries.
  • V. CONCLUSION: Chain-of-Thought prompting returns LLM performance close to the original Neutral baseline for single biased prompts.This result applies specifically to summaries affected by a single biased prompt.
Loading 2608.14629v1…