PaperlayerView exact revision on arXiv

Source-linked AI summary

VA-DPO: Valence-Arousal Direct Preference Optimization for Controllable Emotion Generation in Language Models

Hyunwoo Kim

arXiv:2608.20374v1cs.CLcs.AIcs.LG

TL;DR

Emotional generation often uses discrete labels that cannot specify fine-grained affective states. VA-DPO instead ranks sampled text by distance to a continuous Valence–Arousal target and trains with unchanged DPO, reducing target distance while preserving broad capability, though it remains limited by regressor and evaluation scope.

  • Problem

    Discrete emotion labels cannot express fine-grained combinations of valence and arousal, motivating continuous affect control.

  • Method

    VA-DPO uses a frozen VA regressor to rank candidate generations by target distance, filters pairs with a margin, and trains a LoRA adapter with standard DPO against a frozen reference.

  • Results

    VA-DPO reduces mean VA distance by 33% versus system-prompting and 25% versus few-shot prompting, with rv = 0.93 and ra = 0.75 on Llama-3.1-8B.

  • Takeaways & Limitations

    Control gains replicate across Qwen3-8B and Llama-3.2-3B, while general capability is retained across MMLU, HellaSwag, and TruthfulQA.

  • Takeaways & Limitations

    The method inherits bias and error from a single English VA regressor, and multi-turn dialogue dynamics remain unevaluated.

Abstract

from arXiv · show

How precisely can we tell a language model how to feel? Most work on emotional generation answers with a discrete label - happy, angry, sad - which cannot express a target like "mildly downcast but calm." We instead specify the desired affect as a continuous point (v*, a*) in the Valence-Arousal plane and train the model to hit it. Our method, VA-DPO, is a small modification to Direct Preference Optimization: a frozen VA regressor scores each sampled generation by its Euclidean distance to the target, we keep only candidate pairs whose distance gap clears a margin tau, and we optimize a LoRA adapter with the ordinary DPO loss against a frozen reference. The DPO objective itself is unchanged; what is new is how the preference data is built. On Llama-3.1-8B-Instruct this cuts mean VA distance to the target by 33% over system-prompting and 25% over few-shot prompting, lifting valence/arousal correlation to r_v=0.93 and r_a=0.75. The gains carry over to Qwen3-8B and Llama-3.2-3B, and they do not come at the usual price: MMLU is unchanged (Delta=+0.0) and HellaSwag and TruthfulQA are preserved. We release the code, configs, and the preference-construction pipeline.

1 Introduction

VA-DPO addresses the limits of discrete emotion labels and inconsistent prompting by specifying affect as a continuous Valence–Arousal target. It builds preference data from target distance while retaining standard DPO training and LoRA-based regularization.

  • Discrete emotion labels cannot distinguish affective intensity or nearby states such as mildly melancholic and calm versus deeply melancholic and agitated.
  • Prompting and few-shot examples provide uneven emotional control that can decay over a conversation, while RLHF requires a separate reward model.
  • VA-DPO scores generations by Euclidean distance from a continuous target (v*, a*) in the Valence–Arousal plane and ranks candidates accordingly.
  • Margin-thresholded selection retains only preference pairs with sufficiently different target distances, reducing near-tie label noise.
  • VA-DPO modifies preference construction while retaining the ordinary DPO objective and applies DPO only to LoRA adapters with frozen base and reference models.

2 Related Work

The paper situates VA-DPO within dimensional affect modeling, preference optimization, and controllable emotion generation. Its focus is text-modal, continuous target control rather than discrete or training-free steering.

  • Dimensional emotion models and corpora: Russell’s dimensional model represents affect with continuous valence and arousal axes, while EmoBank supplies approximately 10K English sentences with human VAD annotations.
  • Preference optimization: DPO reformulates RLHF as a closed-form policy update against a frozen reference without a learned reward model.
  • Preference optimization: VA-DPO keeps the standard DPO loss and changes only preference-pair construction using continuous ranking and margin filtering.
  • Emotion control in generative models: Prior emotion-control work includes discrete audio-modal DPO, training-free hidden-state steering, and evidence that affective control is difficult to maintain across multi-turn dialogue.

3 Method

VA-DPO converts continuous target affect into preference pairs, filters ambiguous comparisons, and trains a LoRA adapter with the unchanged DPO objective against a frozen reference.

  • 3.1 Problem setup: The task conditions a generation policy on a target (v*, a*) in [−1, +1]^2 and seeks outputs with small distance under a fixed VA regressor.
  • 3.1 Problem setup: The target is formatted as a text prefix appended to the input prompt while the response remains fluent and on-distribution.
  • 3.2 Continuous reward and pair construction: For each prompt and target, candidates are sampled from the reference model and scored once by their negative distance to the target.
  • 3.2 Continuous reward and pair construction: Pairs are retained when |d(yw)−d(yl)| > τ, with the smaller-distance candidate labeled winner and the larger-distance candidate labeled loser.
  • 3.3 Training objective: The filtered preference set is optimized with the standard DPO loss, while only LoRA adapter weights receive gradients and the base, reference, and regressor remain frozen.
  • 3.3 Training objective: The paper’s novelty is constructing Dpref from a continuous scalar with an explicit margin filter rather than modifying the DPO objective.

4 Experimental Setup

The experiments use open-weight instruction-tuned models, a frozen VA regressor, text-based target conditioning, and LoRA-based training with reproducible evaluation across control, quality, and general-capability metrics.

  • The primary model is Llama-3.1-8B-Instruct, with Qwen3-8B and Llama-3.2-3B-Instruct used as robustness backbones.
  • The frozen RoBERTa-large VA regressor predicts two affect dimensions after training on EmoBank’s rescaled human VAD ratings.
  • Targets are prepended in the text format [VA: v*=<float>, a*=<float>], adding a fixed 4–6 token overhead per example.
  • VA-DPO sweeps β, τ, LoRA rank, learning rate, batch size, candidate temperature, and candidates per prompt, while optimizing only LoRA parameters.
  • Baselines include system prompting, few-shot prompting, SFT, quadrant-based DPO, and no-margin VA-DPO; lexicon and steering baselines are defined but not run.
  • Evaluation covers mean Euclidean VA distance, per-dimension correlations and CCC, generation quality, MMLU, HellaSwag, TruthfulQA, and three training seeds.

5 Results

VA-DPO achieves the strongest reported emotional control on the primary model and across tested backbones, while retaining general capability; its margin filter is less effective for the smaller 3B model and control involves a modest quality trade-off.

  • 0.092 mean VA distance is VA-DPO’s primary-model result, improving 25% over few-shot prompting and 33% over system-prompting.
  • r_v = 0.93 and r_a = 0.75 are VA-DPO’s primary-model valence and arousal correlations with targets.
  • ΔMMLU = +0.0 indicates no measured change in MMLU relative to the base model.
  • VA-DPO is the best method across Qwen3-8B, Llama-3.1-8B-Instruct, and Llama-3.2-3B-Instruct, but its margin-filter advantage reverses at 3B.
  • 0.080 mean VA distance is achieved near τ ≈ 0.1, compared with 0.097 without filtering and 0.139 at τ = 0.5.
  • VA-DPO preserves lexical diversity and avoids reward collapse, but scores below B0 on fluency, relevance, and naturalness.The reported relevance scores are 3.11 for VA-DPO versus 3.68 for B0.

6 Analysis

The analysis tests evaluator robustness, regularization settings, capability retention, and dimension-wise control. VA-DPO remains strong across these checks, while arousal and overly aggressive pair filtering mark important boundaries.

  • Evaluator robustness: VA-DPO remains best under a held-out DeBERTa evaluator, with mean VA distance 0.097 versus 0.111 for the strongest baseline.The held-out regressor never shaped training, and the ranking is unchanged.
  • Evaluator robustness: A reward swap produces essentially the same result: mean VA distance is 0.093 with DeBERTa versus 0.092 with RoBERTa.
  • Capability retention: VA-DPO preserves general capability: MMLU changes by +0.0, HellaSwag by −0.6, and TruthfulQA by +3.1.The analysis attributes this resilience to the low-rank LoRA update and the DPO β-KL anchor.
  • β and retention: Within the tested range, control is largely insensitive to β, but the regime where β trades control for retention remains uncharacterized.
  • Failure modes: Valence control is stronger than arousal control, with ours reaching r_v=0.93 versus r_a=0.75.The regressor is also weaker on arousal, with dev CCC 0.55 versus 0.79 for valence.

7 Limitations

VA-DPO is bounded by the VA regressor, the English EmoBank training domain, the single-turn setting, and text-prefix target conditioning.

  • VA-DPO inherits systematic errors from its frozen VA regressor, so underestimated arousal remains uncorrected for affected expressions.
  • The regressor uses a single English model trained on the mixed-genre but narrow EmoBank corpus, leaving dialectal and domain-specific generalization untested.
  • The method does not evaluate multi-turn dialogue dynamics, which would require turn-conditioned regression or turn-level reward shaping.
  • Targets are supplied as text prefixes, which may be brittle to prompt-format changes; learned target embeddings are left for future work.

8 Conclusion

VA-DPO adapts DPO for fine-grained continuous emotional control by constructing distance-based, margin-filtered preferences. The control gain transfers across several models without general-capability loss, although the margin filter is less effective at 3B scale.

  • VA-DPO combines a frozen VA regressor, distance-based utility, and margin-thresholded pair selection for continuous Valence–Arousal control.
  • On Llama-3.1-8B, mean VA distance falls 33% versus system-prompting and VA-DPO outperforms few-shot prompting, SFT, and discrete-label DPO.
  • At Llama-3.2-3B scale, the margin-filter component does not transfer outright because the smaller candidate pool supplies fewer usable pairs.

A Hyperparameters and setup

The headline setup uses DPO with eight candidates per prompt, margin filtering, sampled generation, and a LoRA adapter trained for three epochs on one H100.

  • Configuration: The headline configuration uses DPO β=0.1, N=8 candidates per prompt, margin τ=0.2, temperature T=0.9, and top-p=0.95.
  • Configuration: LoRA uses rank r=16 and α=32 with 5×10−5 learning rate, 0.05 dropout, AdamW, and zero weight decay on LoRA.
  • Training: Training runs for 3 epochs with effective batch 16 in bfloat16 on 1× H100 80 GB.
  • Reward and generation: The frozen RoBERTa-large regressor is fine-tuned on EmoBank for 5 epochs, while targets are prepended as text tags and candidates come from the frozen reference model.

B Qualitative samples

Table 5 compares representative Llama-3.1-8B generations from VA-DPO and system-prompting across VA quadrants, showing how each method expresses target affect.

  • VA-DPO produces stronger target affect than B0 in representative positive-high and negative-high examples.The examples show overt elation for positive-high targets and overt frustration for negative-high targets, while B0 remains closer to a neutral assistant register.
  • Table 5 reports one EmoBank-test example per VA quadrant where VA-DPO most improves over system-prompting.
  • For a negative-valence, low-arousal prompt, B0 is more expressive while VA-DPO responds in a comparatively neutral way.
Loading 2608.20374v1…