PaperlayerView exact revision on arXiv

Source-linked AI summary

You Can't Prefer Emotions You Don't Sample: Intensity Undershoot in DPO-Tuned LLMs

Hyunwoo Kim, Usama Khalid

arXiv:2609.07808v1cs.CLcs.AIcs.LG

TL;DR

Instruction-tuned LLMs often move in the requested emotional direction without matching its intensity. The paper measures this with VA-conditioned DPO and tests whether more extreme targets and candidate responses improve control. Valence gain improves across two backbones, while arousal remains harder and less reliable.

  • Problem

    It is unclear whether continuous affect control preserves the magnitude of extreme emotional targets, beyond getting their direction right.

    Source: §1 · p. 1

  • Method

    The authors measure achieved affect with a frozen regressor while sweeping VA-conditioned targets, then train DPO with uniformly sampled targets and hotter, larger candidate pools.

    Source: §2 · p. 2, §5 · p. 3

  • Results

    Valence gain rises from 0.26 to 0.40±0.02 on Llama and reaches 0.44 on Qwen3-8B, while arousal remains low and seed-unstable.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The evidence indicates that intensity control is bottlenecked by the extremity of the DPO candidate pool rather than by the conditioning format.

    Source: §7 · p. 4

    The evaluation is English-only and fully automatic, with no human study confirming readers perceive the requested intensity.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Ask a language model to respond "very excitedly," and its output is typically only mildly more energetic. We quantify this effect. We condition an instruction-tuned LLM on a continuous Valence-Arousal (VA) target, where valence measures how pleasant a state is and arousal how activated it is, measure the achieved affect with a frozen regressor, and sweep the requested target from -1 to +1. The response moves far less than asked: the gain, the slope of achieved against requested affect, is only 0.26 for valence and 0.13 for arousal on Llama-3.1-8B, where a faithful controller would score 1. The model systematically undershoots requested emotional intensity, which puts a number on the qualitative observation of Fazzi et al. (2025). Our experiments trace this to the preference-learning pipeline. Training targets from natural corpora such as EmoBank are neutral-heavy, and the sampled candidates themselves rarely reach extreme affect, so Direct Preference Optimization (DPO) is left with no extreme exemplar to prefer. If instead we cover the target space uniformly and sample a hotter, larger candidate pool, valence gain rises from 0.26 to 0.40 +/- 0.02 (3 seeds) and extrapolation error drops, at only a modest in-distribution cost (EmoBank-test VA distance 0.092 to 0.107). The same recipe reproduces on Qwen3-8B (gain_v 0.44, with in-distribution accuracy preserved). Arousal is harder and less reliable: its gain barely moves on average and swings across seeds (0.14 +/- 0.07, against valence's tight +/- 0.02), because raising arousal needs candidates the base model is reluctant to generate. The evidence indicates that faithful intensity is bottlenecked by the extremity of the candidate pool rather than by the conditioning format.

1 Introduction

Instruction-tuned LLMs undershoot requested emotional intensity, even when they move in the requested direction. The paper links this gap to training targets and candidate pools that rarely reach extreme affect, and tests a sampling-based remedy.

1 Introduction

  • Continuous Valence–Arousal control represents graded emotional states with a point in a two-dimensional space rather than a discrete label.Valence measures pleasantness and arousal measures activation.Source: §1 · p. 1
  • Gain is 0.26 for valence and 0.13 for arousal on Llama-3.1-8B, showing substantial undershoot when requested targets sweep from −1 to +1.A faithful controller would score 1.Source: §1 · p. 1
  • Natural corpora are neutral-heavy, and sampled candidates rarely reach extreme affect, leaving DPO without extreme exemplars to reward.The introduction identifies candidate-pool extremity as a shared lever for these two effects.Source: §1 · p. 1
  • Uniform target coverage with hotter, larger candidate sets raises valence gain to 0.40±0.02 on Llama and 0.44 on Qwen3-8B, while reducing extrapolation error.The improvement comes with a modest in-distribution cost on Llama and none on Qwen.Source: §1 · p. 2
  • The paper uses gain as an interpretable diagnostic, traces undershoot to candidate-pool extremity, and reports a valence improvement across two backbones alongside harder arousal control.The conditioning interface does not explain undershoot; arousal remains low and seed-unstable.Source: §1 · pp. 1–2

2 Setup

The study trains LoRA-adapted, VA-conditioned policies with DPO using sampled response pairs scored by a frozen affect regressor. Its evaluation measures how closely achieved affect tracks requested intensity, along with extrapolation and in-distribution performance.

2 Setup

  • The policy is conditioned on VA targets and trained with DPO on candidate pairs ranked by a frozen RoBERTa VA regressor’s measured distance from each target.The candidates are sampled from the frozen base model, and only pairs whose distance gap exceeds a margin are optimized.Source: §2 · p. 2
  • The primary backbone is Llama-3.1-8B-Instruct, with Qwen3-8B as a second backbone, and control metrics are also reported under a held-out DeBERTa regressor.The RoBERTa regressor was fit on EmoBank train and had dev valence/arousal CCC of 0.79/0.55.Source: §2 · p. 2
  • Undershoot is identical with a learned soft-token target embedding and under the held-out DeBERTa regressor, supporting a diagnosis not specific to the text-tag interface or one regressor.The authors interpret the embedding comparison as evidence that the problem sits in the training signal.Source: §2 · p. 2
  • Gain is the mean per-prompt least-squares slope of regressor-measured affect against requested affect, with 1 indicating one-for-one tracking and 0 indicating ignored intensity.The study also reports extrapolation MAE at |t| ≥0.9 and in-distribution mean VA distance on EmoBank test.Source: §2 · p. 2

3 Diagnosis: LLMs undershoot requested intensity

Natural-target DPO yields shallow intensity control: the model tracks the direction of requested affect but compresses its magnitude toward neutral.

3 Diagnosis: LLMs undershoot requested intensity

  • Valence gain is 0.26 and arousal gain 0.13 under natural EmoBank targets, with valence extrapolation MAE of 0.75.Figure 1’s leftmost group and Table 1’s first row present this diagnosis.Source: §3 · p. 2
  • Valence target–achieved correlation is 0.94 even as gain remains low, so sign tracking coexists with collapsed magnitude.Valence monotonicity is also high at ρv = 0.84, indicating an ordered but shallow response curve.Source: §3 · p. 2
  • The shallow slope quantifies the controllability analogue of Fazzi et al. (2025)’s observation that LLMs avoid strong affect.The passage presents gain as a single interpretable measure of that effect.Source: §3 · p. 2

4 Mechanism: candidate-pool extremity as the bottleneck

Changing the conditioning interface does not remove undershoot; the evidence instead points to the extremity of sampled candidates as a bottleneck. Arousal candidates are especially compressed.

4 Mechanism: candidate-pool extremity as the bottleneck

  • Standard sampling yields mean maximum candidate extremities of |v| ≈0.22 and |a| ≈0.13, leaving DPO without extreme exemplars to reward.The learned soft-token interface also undershoots, with valence gain ≤0.21, much like the text tag.Source: §4 · p. 2
  • On Figure 1, the baseline valence curve is shallow, while uniform extreme targets and a hotter pool steepen it toward the faithful line.Uniform+hot raises valence gain from 0.26 to 0.40 ± 0.02 on Llama; arousal remains low and seed-unstable.Source: Figure 1 · p. 3
  • Figure 2 relates candidate-pool extremity to achieved gain; the arousal pool stays compressed even with hot sampling.The figure describes pool extremity as the mean, across prompts, of the most extreme achieved affect among sampled candidates.Source: Figure 2 · p. 3

5 Fix and results

Uniform extreme targets with hotter, larger candidate pools raise valence gain across both backbones, while arousal remains a less reliable control axis. The intervention carries a small in-distribution cost on Llama and none on Qwen.

5 Fix and results

  • Uniform targets alone raise valence gain from 0.26 to 0.33; adding a hotter, larger candidate pool raises it further.The intervention samples targets uniformly over [−1, 1]² and uses T=1.2 with N=16 candidates.Source: §5 · p. 3
  • Arousal gain barely moves on average and is seed-unstable at 0.14 ± 0.07 on Llama, versus valence’s ±0.02.The hotter pool still rarely produces high-arousal candidates, with pool max |a| ≈0.15.Source: §5 · p. 3
  • ∼54% higher valence gain on both backbones follows uniform extreme targets plus hotter, larger candidate pools, with a small in-distribution cost on Llama and none on Qwen.Llama uniform+hot is averaged over 3 seeds; the other Table 1 rows are single-seed runs.Source: §5 · p. 3, Table 1 · p. 4
  • Table 2 reports distinct-n diversity on EmoBank-test generations and characterizes the diversity cost of the intensity fix as small.Higher distinct-n indicates more diverse text.Source: Table 2 · p. 4
  • The undershoot is identical under the held-out DeBERTa regressor, while distinct-2 falls only from 0.57 to 0.54 on Llama.These checks address regressor artifacts and whether the intensity fix collapses lexical diversity.Source: §5 · pp. 3–4
  • Matched prompts show the fixed model committing to the requested strong affect where the baseline uses a more measured, neutral register.For a strongly valenced target, the example contrasts “I’m thrilled to hear about…” with “a very encouraging development.”Source: §5 · p. 4

6 Related work

Prior work spans dimensional affect models and multiple approaches to emotion-conditioned generation, often evaluating direction or correlation. This paper connects intensity undershoot to candidate-pool extremity and measures a qualitative observation about strong affect.

6 Related work

  • Emotion-conditioned generation has used intensity parameters, control codes, decoding-time guidance, prompting, style vectors, and preference optimization.The cited approaches include Affect-LM, Emotional Chatting Machine, and later control and preference-optimization methods.Source: §6 · p. 4
  • Direction-of-shift and target-correlation evaluations are insensitive to magnitude collapse, the phenomenon termed undershoot here.Lexicon-based affect signals share this blind spot.Source: §6 · p. 4
  • Fazzi et al. (2025) qualitatively observed resistance to strong affect; this paper measures it with gain and links it to candidate-pool extremity.The reported residual difficulty is specific to arousal and depends on the backbone.Source: §6 · p. 4

7 Conclusion

Instruction-tuned LLMs undershoot requested emotional intensity, and the conclusion locates the bottleneck in the extremity of the DPO candidate pool rather than the conditioning interface. Uniform targets and hotter, larger pools improve valence gain, while the paper proposes sampling and data as distinct future levers.

7 Conclusion

  • The candidate pool’s limited extreme affect is the stated bottleneck to faithful intensity, rather than the conditioning interface.The conclusion says undershoot persists across conditioning formats and points to the base model’s sampling support as a constraint.Source: §7 · p. 4
  • Uniform targets and hotter, larger sampling raise valence gain by ∼54% across two backbones at a small in-distribution cost.Table 3’s matched generations show U+hot landing closer to the requested valence than the natural-target baseline.Source: §7 · p. 4, Table 3 · p. 5
  • Sampling-side fixes can raise pool extremity without new data, while data-side expansion targets the base model’s support for high-affect generation.The paper suggests sampling-side fixes as a cheaper first step and reserves new data for axes such as arousal where base-model support appears binding.Source: §7 · p. 5

Limitations

The evaluation is limited to English and automatic measures, and the authors identify measurement and intervention-design caveats. In particular, regressor bias remains possible, and the intensity fix changes multiple factors at once.

Limitations

  • A single English regressor and corpus constrain evaluation, and held-out DeBERTa mitigates but does not eliminate possible regressor bias.The authors also report weaker arousal than valence measurement, so some arousal instability may reflect measurement noise.Source: Limitations · p. 5
  • The intensity fix changes target distribution, sampling temperature, and candidate-pool size together, so their individual contributions are not isolated.The authors identify disentangling these factors as future work.Source: Limitations · p. 5
  • The study evaluates point targets in English and uses no human study to confirm that readers perceive the requested intensity.Gain’s perceptual validity therefore remains to be established.Source: Limitations · p. 5

Data and reproducibility

The experiments use EmoBank’s official splits and specify the training setup, regressors, and planned release of replication materials.

Data and reproducibility

  • All experiments use EmoBank’s official train, dev, and test split, with no training on the test split.Preference pairs are generated from the base models with logged seeds and sampling temperatures.Source: Data and reproducibility · p. 5
  • Policies are LoRA adapters trained with DPO for three epochs using the stated learning rate, batch size, and single-H100 setup.The passage specifies adapter settings, DPO parameters, cosine scheduling, and bfloat16 training.Source: Data and reproducibility · p. 5
  • The study uses RoBERTa-large as its VA regressor and DeBERTa-v3-large as a held-out check, with code and configurations slated for release.The planned release also includes the preference-construction pipeline.Source: Data and reproducibility · p. 5

A Changing the conditioning format does not fix undershoot: a learned-embedding ablation

The learned-embedding interface maps the requested valence-arousal target to soft control tokens, but changing the conditioning format does not fix undershoot. Its best in-distribution result and valence gain remain behind the text policy.

A Changing the conditioning format does not fix undershoot: a learned-embedding ablation

Loading 2609.07808v1…