PaperlayerView exact revision on arXiv

Source-linked AI summary

Do SpeechLMs Hear Their Own Opinions? Diagnosing and Mitigating Previous-Belief Contamination in Streaming Emotion Understanding

Haoyue Liu, Zhichao Wang, Ye Chen, Haonan Deng, Xiaoying Tang

arXiv:2608.20769v1cs.SDcs.AI

TL;DR

Streaming emotion systems reuse previous predictions as context, but this can contaminate perception of unchanged current audio. The paper introduces training-free EmoUpdate, which separates prior-blind acoustic observation from evidence-supported historical revision and provides a decontamination fallback. Across four SpeechLMs and two benchmarks, it achieves the best step accuracy and state-balanced accuracy in all eight settings.

  • Problem

    Previous-belief contamination occurs when a model’s prior prediction re-enters the prompt and changes its perception of identical current audio.

  • Method

    EmoUpdate isolates historical state during perception, applies evidence-shrunk causal revision afterward, and offers a closed-form decontamination repair when isolation is unavailable.

  • Results

    EmoUpdate achieves the best step accuracy and state-balanced accuracy in all eight model–benchmark settings, with gains up to 38.41 and 69.71 percentage points over controlled baselines.

  • Takeaways & Limitations

    Historical state should update a grounded observation rather than participate in creating current acoustic evidence.

  • Takeaways & Limitations

    Emotion recognition remains culturally and contextually uncertain, so these systems should not be used for clinical diagnosis, hiring decisions, or consequential surveillance.

Abstract

from arXiv · show

Streaming emotion understanding uses historical state while continuously interpreting current audio, often feeding the model's previous prediction back as context. We show that this history conditioning can distort current perception. On a balanced CREMA-D-Stream counterfactual diagnostic, changing only the injected previous emotion label while holding the audio fixed reduces current-audio accuracy from 72.50% to 30.42% and flips 65.69% of predictions. The effect is strongly label-asymmetric, with prior pull ranging from 4.76% to 98.20%, revealing a failure we call previous-belief contamination (PBC). To address PBC, we introduce EmoUpdate, a training-free framework that separates current-audio perception from historical state revision through three components: (1) a prior-blind acoustic firewall that prevents historical state from entering perception; (2) an evidence-shrunk causal belief filter that introduces history only after observation formation and retains label-asymmetric transition structure only when supported by observed evidence; and (3) a closed-form decontamination operator derived from the same counterfactual measurements for serving stacks where firewalling is unavailable. Across four SpeechLMs and two streaming emotion benchmarks, EmoUpdate achieves the best step accuracy and state-balanced accuracy in all eight model--benchmark settings, improving S-BAcc by up to 69.71 points and step accuracy by up to 38.41 points over the strongest controlled baselines.

1 INTRODUCTION

The paper shows that feeding a SpeechLM’s previous emotion prediction into current-audio perception can produce previous-belief contamination, then introduces EmoUpdate to separate perception from historical state revision. Controlled interventions quantify the failure, while the framework targets it through prior isolation and evidence-based revision.

  • Motivation: Streaming emotion understanding combines continuously arriving audio with historical state, often by feeding the model’s previous prediction back as context.This design is formalized as direct history conditioning.
  • Failure diagnosis: Changing only the injected previous emotion label alters perception of unchanged current audio, defining previous-belief contamination (PBC).The intervention holds audio, instruction, and decoding fixed while varying the previous label.
  • Proposed repair: EMOUPDATE is a training-free framework that forms a prior-blind acoustic observation before introducing historical state for revision.Its components include an acoustic firewall, an evidence-shrunk causal transition rule, and a closed-form decontamination repair when firewalling is unavailable.
  • Validation: 38.41 and 69.71 percentage points: EMOUPDATE improves step accuracy and state-balanced accuracy by up to these amounts over the strongest controlled baselines.It achieves the best step accuracy and state-balanced accuracy in all eight model–benchmark settings.

2 RELATED WORK

The related work connects streaming speech emotion modeling to audio-language models and temporal emotion trajectories, while distinguishing PBC from established self-conditioning and prompt-bias failures. The paper’s distinction is that previous self-generated labels can change perception of unchanged current audio.

  • Speech emotion understanding and streaming emotion modeling: Audio-language models increasingly support direct speech input, while recent benchmarks study multi-turn and streaming emotional trajectories.Related work also covers prosodic prompting, lexical–paralinguistic conflict, and temporal dependence in conversational emotion recognition.
  • Self-conditioning, prompt bias, and error accumulation: Prior research links self-conditioning to error accumulation and prompted language models to asserted beliefs, prompt position, and conflicting context.The paper situates its problem within this broader literature but identifies a distinct closed-loop setting.
  • Self-conditioning, prompt bias, and error accumulation: PBC differs from ordinary exposure bias because the previous self-generated label changes the model’s observation of unchanged current audio, not merely a later prediction.The paper studies this distinction through controlled counterfactual intervention and separates current-audio grounding from historical revision.

3 METHOD

The method diagnoses how injected previous labels contaminate current-audio perception, then separates prior-blind observation from causal historical revision. EMOUPDATE also provides a closed-form fallback for systems that cannot firewall the perception prompt.

  • 3.1 STREAMING EMOTION BELIEF REVISION: Streaming emotion belief revision models current speech chunks causally and uses KEEP or REVISE actions to update the emitted state.The system outputs a belief distribution and label after each chunk without observing future audio.
  • 3.1 STREAMING EMOTION BELIEF REVISION: Direct history conditioning feeds the previous emitted label into the same prompt that extracts current acoustic evidence.This entangles temporal evidence with perception and motivates separating the two roles.
  • 3.2 COUNTERFACTUAL PREVIOUS-BELIEF INTERVENTION: Counterfactual PBC measurement varies only the injected previous label while holding the current audio and other inputs fixed, using prior flip rate, wrong-prior pull, and posterior drift.The diagnostic calls the model with candidate previous labels, but those calls are not used during deployment.
  • 3.3 ACOUSTIC HISTORY FIREWALL: EMOUPDATE firewalling forms a prior-blind acoustic observation from the current audio, then fuses history through a causal Bayesian belief filter.The perception call excludes previous labels, filtered state, episode history, future audio, speaker identity, and filename cues.
  • 3.5 EVIDENCE-SHRUNK TRANSITION PRIOR: Evidence-shrunk transitions retain label-asymmetric destinations only when development evidence exceeds the uniform-null expectation.Otherwise, the positive-part shrinkage sets α to zero and reduces the filter to the symmetric base case; the rule is closed-form and adds no tuned scalar or training.
  • 3.6 PROMPT-FREE DECONTAMINATION OPERATOR: The prompt-free decontamination fallback estimates per-label log offsets from clean and contaminated posteriors and inverts them when firewalling is unavailable.It requires no prompt access, gradient training, or extra SpeechLM calls, and is evaluated by fitting on one stratified grid half and applying to the held-out half.

4 EXPERIMENTS

Experiments diagnose previous-belief contamination, evaluate EMOUPDATE across streaming benchmarks and frozen SpeechLMs, and test its components and robustness. EMOUPDATE consistently improves trajectory-level performance by separating acoustic grounding from causal history revision.

  • PBC diagnostic: 72.50% accuracy without previous-label exposure fell to 30.42% under injected-label exposure, with 65.69% of predictions flipping.The diagnostic used 720 injection conditions, including 600 pull-eligible conditions.
  • Benchmark comparison: Direct history conditioning loses up to 49 step-accuracy points, while EMOUPDATE ranks highest in all eight model–benchmark settings.Instruction-only mitigation leaves wrong-prior pull above 59.7% across ten grounding candidates.
  • Difficulty analysis: History helps ambiguous chunks by +17.2 to +40.9 accuracy points when correct, but wrong labels are adopted in 26.4–86.9% of eligible cases.Easy chunks are numerically indistinguishable because the audio already settles the label.
  • Benchmark comparison: On HumDial-En, EMOUPDATE gains +40.57 S-BAcc and +21.59 step accuracy for Qwen2-Audio, while preventing a 69.71-point S-BAcc collapse for Phi-4MM.On CREMA-D-Stream, it wins all four reported metrics on every backbone; across both benchmarks, it is best on S-BAcc and step accuracy in 8/8 settings.
  • Component analysis: Replacing contaminated observations with prior-blind acoustic posteriors is the larger effect in all eight settings, while causal filtering further improves S-BAcc in seven of eight.For Qwen2-Audio on HumDial-En, filtering contributes +8.29 S-BAcc before grounding and +16.86 after grounding.
  • Robustness and repair: EMOUPDATE’s measurement-parameterized repairs lift exposed accuracy from 30.42 to 53.89 and reduce wrong-prior pull from 71.00 to 12.17 without extra calls or training.The operator is cross-validated on the balanced intervention grid and does not require prompt access.

5 CONCLUSION

The paper identifies previous-belief contamination as a failure in which a SpeechLM’s previous prediction alters its perception of identical current audio. EmoUpdate addresses it by keeping perception prior-blind and introducing history afterward through evidence-shrunk causal revision, with a training-free fallback when firewalling is unavailable.

  • Previous-belief contamination occurs when a streaming SpeechLM’s previous prediction alters its perception of identical current audio.
  • EMOUPDATE keeps current perception prior-blind and introduces historical state afterward through an evidence-shrunk causal filter.
  • EMOUPDATE includes a training-free decontamination fallback for systems that cannot firewall historical state from perception.

ETHICS STATEMENT

The authors emphasize that emotion recognition is culturally and contextually uncertain. They therefore reject clinical diagnosis, hiring decisions, and consequential surveillance as appropriate uses, and recommend uncertainty exposure and correction mechanisms.

  • Emotion recognition is culturally and contextually uncertain, limiting its suitability for consequential applications.
  • The paper states that emotion-recognition systems should not be used for clinical diagnosis, hiring decisions, or consequential surveillance.
  • Deployments should expose uncertainty and permit correction rather than treating filtered beliefs as facts.

REPRODUCIBILITY STATEMENT

The paper supports reproducibility through released artifacts, locked evaluation procedures, and detailed controls for causal intervention and streaming-state metrics. Its methodological framing distinguishes previous-belief contamination from conventional exposure bias and prompt-belief effects.

  • Released manifests, hashes, caches, selection locks, predictions, bootstrap summaries, plot-source tables, and deterministic scripts map claims to reproducible artifacts.
  • Unlike exposure-bias remedies that change next-token production, PBC rewrites the input representation, motivating a perception firewall and counterfactual diagnosis.
  • The causal diagnostic fixes audio, instruction, and decoding while varying only the injected previous label to measure its effect on the posterior.
  • The evaluation defines transition F1, change-state accuracy, stable-state accuracy, and S-BAcc for streaming trajectory assessment.
  • Inertia and volatility are conditioned on previously correct predictions, preventing already-wrong states from being counted as desirable stability or destructive revision.

C DATASET CONSTRUCTION AND LEAKAGE CONTROLS

The evaluation constructs causal streaming episodes from controlled source data and separates development from evaluation to limit leakage. Baselines and prior-blind prompts are selected under locked, development-only procedures with explicit input restrictions.

  • CREMA-D-Stream reorganizes unique acted utterances into four-chunk, same-speaker trajectories for causal evaluation.
  • HumDial-En preserves dialogue turns in temporal order and uses non-overlapping source groups for development and evaluation.
  • Each audio file appears in at most one trajectory, according to the construction and leakage audit.
  • Controlled baseline thresholds are selected on development episodes and locked before evaluation, with DHC deliberately testing direct history exposure.
  • The seven causal systems use at most one frozen-SpeechLM call per chunk, while most controls share the same ten-candidate scalar budget.
  • Grounding prompts are selected using development-only accuracy screening and lexicographic ranking by S-BAcc, Transition F1, step accuracy, and final accuracy.
  • Selected prompts instruct the model to use current speech audio while excluding previous labels, history, filenames, and speaker identifiers from perception.

F DIFFICULTY-STRATIFIED ANALYSIS

Difficulty and trajectory analyses show that EMOUPDATE helps most when acoustic evidence supports revision, while improving both stable and changing-state performance where trajectories carry information.

  • Difficulty-stratified analysis: On lower-information CREMA-D-Stream posteriors, policy differences are correspondingly small, limiting margin-based difficulty separation.The reported chunkwise accuracies are 19.6% and 24.6%, with p ≥0.42.
  • Stable versus shift steps: Stable accuracy rises by up to 14.29 points, shift accuracy by up to 19.43 points, and false-switch rate falls by 2.86–11.43 points on HumDial-En.EMOUPDATE improves shift accuracy on all three backbones, improves stable accuracy on two, and preserves it on the third.
  • Cross-setting comparison: Direct history conditioning loses up to 49 step-accuracy points, whereas EMOUPDATE is highest in all eight model–benchmark settings.The comparison uses the same step-accuracy quantities as Table 2.

G COMPUTE AND FAIRNESS AUDIT

The audit matches deployment-time comparison conditions across policies, but distinguishes matched deployment resources from additional development calls used for prompt selection.

  • Matched conditions: Deployment calls, trainable parameters, chunk boundaries, decoding, label extraction, and policy-candidate budgets are matched across comparisons.Within a fixed perception source, every policy comparison replays exactly the same posterior cache.
  • Development-cost boundary: Total development SpeechLM calls are not identical because EMOUPDATE includes ten prompt candidates for prompt selection.The unoptimized seed prompt and selected grounding prompt are not claimed to be the same.

H ADDITIONAL INTERVENTION RESULTS

Additional intervention analyses characterize label-asymmetric contamination, its dependence on available evidence, and measurement-based repair when perception firewalling is unavailable.

  • Cross-backbone diagnostics: The Phi-4MM HumDial-En accuracy increase under exposure does not indicate harmless history conditioning because its flip rate remains 43.67%.The intervention summary also reports posterior JSD of 0.30 and attributes the accuracy increase to regularization of a weak control prompt.
  • Per-prior intervention: Wrong-prior pull is semantically asymmetric: happy and angry have comparable base rates but pull rates of 4.76% and 92.05%, respectively.Pull is uncorrelated with label surface length (r = +0.020) and prior-blind base rate (r = +0.022).
  • Per-prior intervention: A wrong happy prior is adopted 4.8% of the time, with most remaining probability mass landing on neutral.Eligibility counts only conditions where the injected prior differs from the prior-blind control prediction.
  • Instructional mitigation: No grounding instruction falls below 59.7% wrong-prior pull, and pull rises with prior-blind accuracy (Pearson r = 0.81).Seven of the ten candidates explicitly forbid using the previous label.
  • Measurement-based decontamination: The closed-form operator recovers 56% of accuracy damage, raising exposed accuracy from 30.42 to 53.89 against a 72.50 control.Wrong-prior pull falls from 71.00 to 12.17, using two-fold cross-validation.
  • Measurement-based decontamination: The operator is intended as a fallback for legacy or third-party stacks where firewalling is unavailable and the deployment-time prior is observable.It adds no SpeechLM calls and uses the system’s own previous label.
  • Measurement-grid size: The decontamination operator nearly saturates with 36 audited conditions, recovering exposed accuracy from 28.61 to 53.33 on held-out folds.This lies within the seed-to-seed range of the 360-condition fit, whose recovery reaches 53.89.
  • Trajectory metrics: EMOUPDATE improves step accuracy, TF1, and S-BAcc by 21.59, 31.12, and 41.43 points against the same baseline in one setting.Trajectory metrics distinguish beneficial or harmful incidental accuracy changes from sensitivity to injected history.

P REPRODUCIBILITY AND RESPONSIBLE USE

The paper documents locked artifacts and a staged reproduction procedure while limiting responsible use because emotion recognition is uncertain and prior beliefs can amplify erroneous labels.

  • Reproducibility artifacts: Reproduction stores manifests, posterior caches, candidate locks, configuration and data hashes, predictions, metric summaries, and plot-source tables.Selection locks prevent post-evaluation candidate changes.
  • Reproduction protocol: The six-stage reproduction order verifies hashes, runs frozen models, selects prompts and policies on development data, and replays locked policies on evaluation caches.No evaluation label is read during prompt or policy selection.
  • Responsible use: Emotion recognition should not be used for clinical diagnosis, hiring, or consequential surveillance because it is culturally and contextually uncertain.Applications should expose uncertainty and permit correction rather than treating filtered beliefs as facts.
Loading 2608.20769v1…