Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback

Mikhail Krasitskii; Alexander Gelbukh; Olga Kolesnikova; Grigori Sidorov

Source-linked AI summary

Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback

Mikhail Krasitskii, Alexander Gelbukh, Olga Kolesnikova, Grigori Sidorov

arXiv:2608.15530v1Aug 16, 2026cs.CL

TL;DR

RLHF can improve summarization while flattening emotional content, raising a gap in understanding why sentiment drifts toward neutrality. The paper introduces Policy Attribution to trace this drift to alignment-objective components and finds KL regularization is the primary driver, while sentiment-aware regularization reduces drift without degrading summary quality.

Problem
RLHF summaries increasingly attenuate sentiment variance and affective markers, threatening emotional fidelity in downstream tasks that depend on sentiment preservation.
Method
The study combines variable-regularization RLHF training, multilingual sentiment evaluation, and Policy Attribution to decompose policy updates by objective component.
Results
Across models, domains, and eight languages, RLHF systematically suppresses affective content, with the KL penalty accounting for ≈82 % of negative attribution on sentiment-bearing tokens.
Takeaways & Limitations
Sentiment-aware objectives can reduce drift by 18-22 % without degrading summary quality, supporting affect-sensitive alignment when sentiment preservation is essential.
Takeaways & Limitations
The evaluation relies on static pretrained sentiment classifiers and lacks explicit correlation analysis between JSD and human judgments of sentiment preservation.

Abstract

from arXiv · show

Reinforcement learning with human feedback (RLHF) aligns LLMs with human preferences, improving summarization fluency and safety, but causes sentiment drift: overly neutral summaries stripped of emotional nuance. We diagnose why RL acts as a sentiment neutralizer and present Policy Attribution, a framework using gradient and logit decomposition to trace drift to reward model (RM) signals and KL (Kullback-Leibler) penalty. Sentiment drift reflects a strategic bias toward "low-risk" tokens maximizing expected rewards under preference uncertainty (Stiennon et al., 2020; Gao, Schulman, and Hilton, 2023). On Reddit TL;DR and CNN/DailyMail, RLHF summaries get higher rewards but show 30-40% lower sentiment variance. Cross-lingual analysis across eight languages shows language-independent drift, with morphologically richer languages more suppressed (Krasitskii et al., 2026). We propose and validate a sentiment-aware regularization technique reducing drift by 18-22% without harming summary quality. The code and toolkit will be public.

1 Introduction

RLHF improves helpfulness, safety, fluency, and factual correctness but increasingly suppresses affective content, producing sentiment drift. The paper introduces Policy Attribution to identify this suppression’s RL drivers and validates a sentiment-aware KL intervention that reduces drift while preserving summary quality.

Motivation and problem: RLHF summaries become increasingly neutral as emotional markers such as intensifiers and evaluative adjectives are attenuated, creating sentiment drift between original and summarized affect.The paper frames this as the Neutrality Trap: avoiding controversial outputs suppresses affective content needed for faithful sentiment conveyance.
Method: Policy Attribution decomposes policy gradients by reward, KL, and clipping to identify causal drivers of affective suppression.The framework moves beyond qualitative explanations of neutrality by attributing drift to specific RL objective components.
Evaluation: Sentiment Drift is formally defined and quantified across major benchmarks, with validation across eight typologically diverse languages.The paper positions multilingual evaluation as part of its empirical validation.
Findings: ≈82 % of negative attribution on sentiment-bearing tokens is accounted for by KL regularization.This result identifies KL regularization as the dominant measured contributor to sentiment suppression.
Findings: Sentiment loss is a strategic low-risk outcome consistent with RLHF dynamics, and suppression persists under Direct Preference Optimization.The analysis extends beyond RLHF to show the pattern across alignment methods.
Intervention: 18-22 % drift reduction is achieved by Sentiment-Aware KL Regularization while maintaining ROUGE-L within 0.5 points of standard RLHF.The intervention reduces sentiment drift without materially sacrificing the reported summary-quality metric.

2 Background and Related Work

The section reviews RLHF’s SFT–RM–RL pipeline, KL-regularized PPO, and direct preference methods, while situating sentiment suppression within reward hacking, policy conservatism, and attribution research. It frames this work as a mechanistic, policy-level approach to diagnosing and preserving sentiment in RL-tuned summarizers.

RLHF Background: RLHF typically combines supervised fine-tuning, reward modeling, and reinforcement learning, with PPO maximizing reward while constraining KL divergence from the SFT model.The KL penalty strength is controlled by β.
Alternative Alignment Methods: Direct Preference Optimization and Kahneman-Tversky Optimization remove the explicit reward model and KL penalty, but their susceptibility to sentiment suppression remains open.The work presents a systematic sentiment-drift comparison across PPO and DPO.
Reward Hacking and Conservatism: Reward hacking in summarization can produce verbosity bias, excessive hedging, and stylistic flattening, while KL-regularized RLHF favors conservative policies under preference uncertainty.The work extends this concern to disproportionate effects on sentiment-bearing tokens, especially across multilingual settings.
Attribution Methods: Gradient-based attribution methods support token-level and component-level analyses of generation dynamics, making them suitable for diagnosing affective suppression.Recent theoretical work clarifies assumptions governing when gradient-based explanations reliably capture model behavior.
Sentiment Preservation: Prior work shows that preserving emotional tone requires explicit modeling constraints, while this work treats sentiment preservation as policy-level optimization and diagnoses objective components driving sentiment loss.The attribution-based approach enables targeted intervention rather than heuristic reward modification.

3 Methodology

The methodology combines variable-regularization RLHF training, multilingual sentiment evaluation across eight languages, and Policy Attribution to identify which objective components drive sentiment drift. It evaluates sentiment preservation with distributional metrics and decomposes policy updates into reward, KL, and clipping contributions.

Methodology overview: The framework has three modules: RLHF training with variable regularization, multilingual sentiment evaluation, and Policy Attribution gradient decomposition.These modules evaluate sentiment preservation and attribute observed drift to specific training-objective components.
RLHF optimization: Models follow PPO, maximizing expected reward while penalizing deviation from the SFT reference model, with β ∈{0,05, 0,1, 0,2}.The study explicitly tracks gradient contributions from each objective term during backpropagation.
Multilingual evaluation: Sentiment preservation is evaluated across eight typologically diverse languages using Sentiment Variance and Jensen-Shannon Divergence between source and summary sentiment distributions.Lower Sentiment Variance indicates convergence toward neutral outputs, while lower JSD indicates better preservation of the original distribution.
Multilingual evaluation: Classifiers remain frozen so measured sentiment shifts are attributable solely to policy updates.The language selection and evaluation protocol follows prior cross-lingual findings, including greater difficulty in morphologically rich languages.
Policy Attribution: Policy Attribution decomposes policy gradients by loss component rather than input feature, covering reward, KL, and PPO clipping terms.Token-level scores quantify component-specific suppression of sentiment-bearing tokens; negative AKL(ti) indicates KL suppression.
Policy Attribution: 50 Riemann steps approximate the Integrated Gradients integral used to compute token-level attribution scores.The analysis targets abstractive generation, while extractive methods serve as sentiment-preserving upper bounds rather than optimization targets.

4 Experimental Setup

The experiments evaluate sentiment drift across two emotionally varied summarization benchmarks, multiple RLHF configurations, and eight languages. They measure quality and sentiment preservation, compare PPO-based RLHF with DPO, and test sentiment-aware KL regularization.

Models and Training: Experiments use Llama-3-8B and Mistral-7B-v0.1 in a TRL-based RLHF pipeline, varying the KL coefficient β ∈ {0,05, 0,1, 0,2}.Figure 1 examines how increasing β affects neutrality.
Datasets and Languages: The evaluation uses Reddit TL;DR and CNN/DailyMail, filtering for clearly polarized source texts to make sentiment drift measurable.Multilingual analysis uses validated parallel subsets in English, Arabic, Finnish, French, German, Hungarian, Italian, and Spanish.
Evaluation Metrics: Sentiment Variance, Jensen-Shannon Divergence, and ROUGE-L measure sentiment preservation, distributional shift, and summarization quality at aggregate and language-specific levels.Figure 1 shows that increasing β systematically reduces sentiment variance.
Policy Attribution: Integrated Gradients attribute sentiment drift using the top-10 % of tokens with the highest sentiment mass, aggregated over sentiment-bearing tokens identified with multilingual resources.Attribution uses 50 integration steps and a classifier confidence threshold of p > 0,8.
Comparative and Mitigation Experiments: 18-22 % improvement in sentiment drift is achieved by relaxing KL penalties for sentiment-bearing tokens, without ROUGE-L degradation.The mitigation sets γ = 0,3 and keeps other hyperparameters identical to baseline RLHF; DPO is also evaluated on the same models, datasets, and metrics.

5 Results and Analysis

Results show that higher KL regularization and RLHF optimization systematically suppress sentiment, especially in abstractive summaries and morphologically rich languages. Attribution and mitigation experiments identify KL-driven token suppression as the main mechanism and reduce drift while preserving summary quality.

Sentiment drift patterns: Higher KL coefficient β values induce stronger neutrality bias, with abstractive summaries showing the largest sentiment variance reduction and extractive summaries better preserving affective cues.Morphologically rich languages, including Finnish and Hungarian, show more pronounced suppression.
KL effects: Increasing β consistently reduces sentiment variance while marginally improving ROUGE-L, confirming neutrality as a low-risk optimization strategy.This result isolates the KL effect from other components of the training objective.
Policy attribution: 82 % of negative attribution is assigned to KL-divergence, while sentiment-bearing tokens with strong emotional polarity consistently receive negative gradients.The attribution pattern indicates active suppression by the KL penalty.
DPO and Mitigation Experiments: 18-22 % drift reduction from Sentiment-Aware KL Regularization keeps ROUGE-L within 0.5 points of baseline, with greatest effectiveness in morphologically rich languages.Human evaluation rates RLHF summaries as more neutral than SFT (mean=2.1 vs. mean=3.4), while SA-KL partially restores expressivity (mean=2.9; Fleiss’ κ=0.62).

6 Discussion

The discussion attributes sentiment drift primarily to KL-penalty-driven conservative policies, while showing that suppression persists across alignment methods and languages. Sentiment-Aware KL Regularization reduces but does not eliminate drift, motivating affect-sensitive objectives and further reward-model study.

Mechanism: ≈82 % of negative attribution on sentiment-bearing tokens is accounted for by the KL-divergence penalty.The KL term suppresses low-frequency and high-entropy tokens, disproportionately affecting sentiment-bearing expressions.
Across alignment methods: 25-30 % sentiment suppression occurs for DPO versus 30-40 % for PPO, indicating neutrality bias extends beyond PPO-KL.The discussion links this persistence to conservative preference optimization and implicit DPO regularization.
Cross-lingual effects: Cross-linguistic consistency supports sentiment attenuation as a robust effect whose preservation correlates with morphological complexity.The pattern is reported across RLHF-optimized models and is independent of the specific summarization architecture.
Mitigation and limitations: 18-22 % reduction in sentiment drift is achieved by Sentiment-Aware KL Regularization without compromising summarization quality.Mitigation remains incomplete because SA-KL restores only part of the lost variance, implicating the reward model as another possible source of neutrality bias.
Implications: Affective suppression can bias sentiment analysis, obscure urgency in news, hide critical customer-feedback signals, and flatten mental-health content.These risks motivate affect-sensitive objectives in RL-based pipelines.
Reproducibility: The Policy Attribution toolkit will be made publicly available to support reproducibility.The experiments report aggregate and per-language metrics and visualize key trends.

7 Conclusion

RLHF improves summarization quality but systematically suppresses affective content, producing increasingly neutral summaries through an optimization-driven neutrality bias. Policy Attribution identifies KL regularization as the primary driver and motivates a sentiment-aware remedy that reduces drift without degrading quality.

Findings: RLHF improves fluency, safety, and perceived quality while systematically suppressing affective content, creating increasingly neutral summaries that challenge sentiment-sensitive downstream tasks.The conclusion characterizes sentiment drift as a problem for applications where emotional fidelity matters.
Findings: Sentiment drift emerges directly from the optimization objective, with KL regularization discouraging sentiment-bearing tokens and steering policies toward low-variance, neutral outputs across models, domains, and eight languages.The cross-lingual consistency extends prior findings of systematic sentiment attenuation across languages.
Policy Attribution: ≈82 % of negative attribution on sentiment-bearing tokens is accounted for by the KL-divergence penalty, identifying it as the primary driver of neutrality bias.Policy Attribution provides fine-grained analysis of how RL-objective components influence affective expression.
Cross-method analysis: 25-30 % sentiment suppression for DPO versus 30-40 % for PPO shows that neutrality bias persists across alignment methods rather than arising solely from PPO.The reduced magnitude under DPO suggests a broader consequence of preference optimization.
Mitigation: 18-22 % lower sentiment drift is achieved by Sentiment-Aware KL Regularization without degrading summary quality, with ROUGE-L remaining within 0.5 points of baseline.The strategy supports affect-sensitive objectives in RL-based generation where sentiment preservation is essential.

Limitations

The study’s limitations concern task scope, sentiment measurement, human-evaluation scale, generalization beyond summarization, and the computational cost of Policy Attribution. These constraints motivate larger, broader, and more efficient validation of affect-aware alignment.

Limitations: Experiments cover only single-document summarization on Reddit TL;DR and CNN/DailyMail, excluding multi-document, query-focused, and dialogue settings.Affective dynamics may differ because of inter-speaker sentiment interaction or topical framing.
Limitations: Static pretrained sentiment classifiers for eight languages introduce architectural, training-data, and label-schema variance while missing implicit, contextual, or culturally nuanced affect.The classifiers capture explicit sentiment but may overlook irony and understatement, especially in morphologically rich languages.
Limitations: 50 summary pairs evaluated by 3 annotators provide only small-scale evidence, motivating larger studies with more diverse annotator pools.Larger evaluations could more robustly validate perceived neutrality and emotional fidelity.
Limitations: Conclusions are grounded in summarization, leaving open whether the same neutrality bias and “low-risk token” explanation apply to open-ended generation.Chatbots and translators may involve different interactions among length, coherence, and safety.
Limitations: Over 50 integration steps per token make Policy Attribution computationally expensive, motivating efficient approximation methods for real-time diagnostics.The computational overhead is a stated limitation of gradient decomposition.
Limitations: Future work should pursue human-in-the-loop diagnostics and cross-task generalizations of affect-aware alignment.These directions are presented as ways to address the study’s limitations.