PaperlayerView exact revision on arXiv

Source-linked AI summary

VoxReason: Auditing Source-Grounded Speech Plans Before Synthesis

Mengzhe Geng

arXiv:2609.03203v3cs.SDcs.CLcs.LGeess.AS

TL;DR

VoxReason addresses the gap between plausible speech plans and plans grounded in source evidence by testing a planner before waveform synthesis. Its controlled source-label benchmark and verifier expose a contrast: high plan-slot accuracy can coexist with no consistent cue-driven changes. The resulting measurement layer is limited to derived records, not audio inputs.

  • Problem

    Existing audio evaluations do not isolate whether a planner uses the correct source evidence when the utterance stays fixed and one contextual cue changes.

    Source: §1 · p. 1

  • Method

    VoxReason provides a controlled source-label benchmark and verifier that check evidence citations and whether a cue edit changes only linked delivery fields before synthesis.

    Source: §1 · p. 2, §4.3 · p. 4

  • Results

    The prior-only control reaches 0.958 original plan-slot accuracy, yet counterfactual consistency and required-change accuracy remain zero.

    Source: §6.2 · p. 6

  • Takeaways & Limitations

    Plan-slot accuracy alone should not establish context awareness; evidence recall and counterfactual consistency should be reported alongside it.

    Source: §7 · p. 8

    The benchmark uses derived RAVDESS source labels and contains no public context-audio records, so it measures source-label planning rather than audio-conditioned planning.

    Source: §4.1 · p. 3, §8 · p. 8

Abstract

from arXiv · show

Speech systems increasingly infer how an utterance should be delivered from context, but a plausible delivery plan may not be supported by the input. VoxReason is a small public benchmark and verifier for testing this failure before waveform synthesis. Each of its 100 cases fixes the utterance, provides derived records that name the source emotion and intensity, and changes one licensed cue. A system must cite the record for its delivery decision and update only the plan fields associated with the edit. We use deterministic verifier references, not a model leaderboard. This holdout excludes every emotion and intensity combination observed in training. A prior-only predictor achieves 0.958 accuracy across plan fields but never changes its plan consistently after a cue edit. This contrast shows that plan agreement does not demonstrate source grounding. The released suite provides an auditable measurement layer for screening structured speech plans before synthesis. It is limited to derived records, not audio inputs.

1 Introduction

Context-aware speech planners must ground delivery decisions in the input, a question existing audio benchmarks do not isolate. VoxReason tests source attribution and plan changes before synthesis, complementing broader audio-understanding and generation evaluations.

1 Introduction

  • Existing audio benchmarks assess clip understanding or waveform quality, but do not isolate whether a planner uses the correct source evidence when one cue changes.Holding the utterance constant also prevents stable text, speaker, or emotion priors from explaining a plan.Source: §1 · p. 1
  • VoxReason evaluates source attribution and plan consistency without listener judgments, checking evidence identity, citation coverage, plan slots, and whether an edit changes only linked fields.It does not score speech quality or listener preference.Source: §1 · p. 1
  • The paper contributes a controlled source-grounded planning task, a compact source-label suite and verifier, and construct-validity and anti-shortcut analyses.The package is a narrow measurement layer; listener studies, waveform synthesis, and large-scale audio benchmarking are left to future work.Source: §1 · p. 2

2 Related Work

  • SpeechGPT and AudioGPT motivate evaluating intermediate decisions as multimodal capabilities expand, while VoxReason tests grounding in a supplied source record without requiring waveform input.SpeechGPT demonstrated cross-modal speech interaction Zhang et al. (2023); AudioGPT organized audio understanding and generation through foundation-model tools Huang et al. (2023).Source: §2 · p. 2
  • VoxReason targets unsupported or uncited evidence in structured plans, a narrower and more auditable question than broad reasoning and context-aware speech-generation settings address by themselves.Relevant settings include the benchmark in Ma et al. (2025), UniCATS Du et al. (2024), ThinkSound Liu et al. (2025), and the ISCSLP 2026 CoT-TTS challenge Xue et al. (2026).Source: §2 · p. 2
  • VoxReason uses derived RAVDESS emotion and intensity labels rather than audio, testing a planning decision before codec or synthesis stages.The package stores labels, prompts, source cues, and expected plans; the acoustic preflight is a repository consistency check, not a perceptual result. RAVDESS is described by Livingstone and Russo (2018), and semantic audio codecs by Ye et al. (2024).Source: §2 · p. 2

3 Task Formulation

The task represents each case as a target utterance, context, source evidence, expected delivery plan, and one-cue counterfactual edit. A planner returns cited records and a structured plan, which the verifier checks for evidence validity, slot agreement, and localized changes.

3 Task Formulation

  • Each case input contains a target utterance, role profile, scene label, source-label evidence records, expected delivery plan, and one-cue counterfactual edit.The planner returns cited evidence records, a structured plan, and an optional rationale; the verifier scores evidence and plan fields rather than free-form rationale.Source: §3 · p. 2
  • The benchmark profile table summarizes counts from the released files and deterministic construct-validity checks.The repository contains derived labels, prompts, plans, and verification code, not the original RAVDESS audio files.Source: §3 · p. 2, Table 1 · p. 3
  • The plan has seven scalar fields and a set-valued emphasis field; an edit changes one source cue while the target text and other case fields stay fixed.The expected plan delta specifies which fields may change.Source: §3 · p. 3
  • The verifier checks that cited records are present and match gold evidence, plan values agree across eight scored slots, and only linked fields change after an edit.These checks assess evidence identity, plan agreement, and counterfactual locality.Source: §3 · p. 3
  • A planner that repeats the same plan across cues can agree on a biased split yet fail the counterfactual locality test when the licensed cue changes.This separates source grounding from surface plausibility.Source: §3 · p. 3

4 VoxReason Benchmark and Verifier

VoxReason packages derived source-label cases with a verifier that checks cited evidence and delivery-plan changes before synthesis. Its metrics assess evidence grounding, plan agreement, and whether counterfactual edits change only the permitted fields.

4.1 Released source-label suite

  • The 100-case suite uses 15 emotion-intensity keys, each mapped deterministically to one expected plan, creating a shortcut risk.It is organized into 67 training, 17 development, and 16 test cases.Source: §4.1 · p. 3
  • Derived RAVDESS labels, rather than packaged raw audio, define the main measurement; acoustic checks do not assess listener perception or rendered speech quality.The repository’s preflight and matched-case acoustic-anchor checks serve data auditing.Source: §4.1 · p. 3

4.2 Verification path

  • A typed plan cites source evidence, then a one-cue edit tests whether licensed delivery slots change while other slots remain fixed.The verifier checks citation legality, slot agreement, evidence coverage, citation-required grounding, and counterfactual locality before the waveform branch.Source: §4.2 · p. 3, Figure 1 · p. 4

4.3 Metrics

  • Evidence precision, recall, and F1 match predicted and gold records by stable cue identity; decisive-cue recall focuses on the cue changed by the edit.Hallucinated evidence rate is 1 − P when evidence is predicted, while uncited evidence rate is 1 − R when gold evidence exists.Source: §4.3 · p. 4
  • Plan-slot accuracy averages agreement across seven scalar fields and exact emphasis-set matching; the released gold plan is a task specification, not a perceptual target.The citation-required grounded score applies an evidence-recall gate to the grounded score.Source: §4.3 · p. 4
  • Counterfactual consistency combines required-change accuracy with preservation of stable fields, while unexpected-change rate is one minus preservation rate.A required field earns credit only when it changes to the declared value; unchanged stable fields earn preservation credit.Source: §4.3 · p. 4

5 Evaluation Protocol

The evaluation compares text-only input with a deterministic source-label oracle across paired cases, then probes whether prior-only predictions react to cue edits. The checks are designed to expose shortcut behavior, not to establish learned-model performance.

5.1 Controlled information comparison

  • The primary comparison scores the text-only control and source-label oracle on all 100 paired cases using the same verifier.The oracle receives released gold evidence and plans as a deterministic reference, not as a learned model.Source: §5.1 · p. 5
  • The text-only control lacks source-label evidence for identifying the decisive cue, so the comparison tests the information exposed by the source-label task.The package’s 67/17/16 train/development/test split remains distinct from this 100-case paired measurement.Source: §5.1 · p. 5

5.2 Anti-shortcut checks

  • Anti-shortcut checks remove each test key from lookup training and repartition the pool into source-key-disjoint sets of 60 training, 16 development, and 24 test cases.The prior-only predictor sees source emotion alone; its scores are calibration diagnostics, intended to expose stable emotion-conditioned plans that ignore intensity-triggered edits.Source: §5.2 · p. 5

6 Results

The verifier separates source-grounded, cue-responsive planning from high plan-slot agreement alone. Across controlled interventions, prior-only predictors can match plans while failing to respond consistently to edited cues; the package supports only planning-stage conclusions.

6.1 The verifier exposes the source-grounding gap

  • The text-only control scores 0.857 evidence F1 but 0.000 decisive-cue recall, 0.185 plan-slot accuracy, and 0.427 citation-required grounded score; the source-label oracle reaches 1.000 on positive metrics by construction.Its zero hallucinated-evidence rate accompanies a 0.250 uncited-evidence rate; Gcite multiplies grounded quality by evidence recall.Source: §6.1 · p. 5

6.3 Source-key shortcuts fail under intervention

  • 0.958 source-key-disjoint plan-slot accuracy coexists with zero counterfactual consistency and unrelated-field changes on every edit; leave-key-out exact and slot accuracy fall to 0.000 and 0.242.On the source-emotion-disjoint split, plan-slot accuracy falls to 0.219 on 32 cases, with no citations; these are prior-only calibration diagnostics, not leaderboard results.Source: §6.3 · p. 6, Table 4 · p. 7

6.2 Evidence sensitivity under a source-key holdout

  • The prior-only control pairs 0.958 original plan-slot accuracy with zero counterfactual consistency and required-change accuracy, while the oracle and declared-delta controls satisfy locality checks.Text evidence-only and citation-echo controls preserve unrelated slots but have zero decisive-cue recall and required-change accuracy; shuffled delta is not an independent comparison because fixture deltas are identical.Source: §6.2 · p. 6, Figure 2 · p. 7

6.4 What the current package does and does not establish

  • The results support deterministic construction checks, verifier discrimination of legal versus missing evidence, and the conclusion that a stable emotion prior is insufficient for intervention testing.These conclusions concern a planning-stage protocol, not naturalness, listener preference, speaker similarity, intelligibility, or generalization to unconstrained conversational audio.Source: §6.4 · p. 7

7 Discussion

VoxReason makes cue-to-plan grounding inspectable by holding the utterance fixed during edits and recording source attribution. Its narrow diagnostic separates planning from rendering errors, while its anti-shortcut results motivate reporting evidence and counterfactual consistency alongside plan accuracy.

7 Discussion

  • Holding the utterance fixed makes a plan change attributable to the edited cue, and the evidence ledger keeps that attribution inspectable.Grounding links a cue, a plan field, and a cited source record.Source: §7 · pp. 7–8
  • The small source-label suite separates planning errors from rendering errors within its setting, but cannot characterize context-aware speech planning in the wild.It can be checked line by line and regenerated without private audio.Source: §7 · p. 8
  • Reporting evidence recall and counterfactual consistency beside plan-slot accuracy distinguishes unsupported agreement from cue-sensitive reasoning.Include source-key-disjoint tests when a label-to-plan mapping could be memorized.Source: §7 · p. 8

8 Limitations and Responsible Use

The benchmark measures derived source-label planning within a deliberately narrow construct and exact-match metric. It does not measure audio-conditioned planning, perceptual quality, or listener judgments.

Data scope

  • The released benchmark measures source-label planning from derived RAVDESS labels, not audio-conditioned planning or a representative sample of natural conversation.The source corpus contains acted emotional speech and no public context-audio records.Source: §8 · p. 8

Construct scope

  • With two utterances, one scene, two roles, and 15 source keys, the suite remains narrow; its deterministic mapping creates memorization risk.The source-key holdout reduces one shortcut but is only a calibration check, not a complete remedy.Source: §8 · p. 8

Metric scope

  • Exact structured-field agreement and evidence matching do not evaluate alternative reasonable plans, waveform prosody, or human-validated perceptual quality.The weighted grounded score is an engineering metric defined by the public implementation.Source: §8 · p. 8

Human and audio evaluation

  • No human listeners or evaluators are required; future work should use a separately governed human-evidence lane for questions about perceived naturalness, appropriateness, or preference.Listener judgments should remain distinct from the verifier score.Source: §8 · p. 8

9 Conclusion

VoxReason tests whether a speech planner grounds delivery decisions in source records and changes only linked fields when a contextual cue changes. The controlled comparison and anti-shortcut checks show why plan accuracy alone is insufficient, while the package remains a pre-synthesis audit layer.

9 Conclusion

  • VoxReason tests whether planners cite the source record licensing a decision and update only linked delivery fields after a contextual cue changes.The public source-label suite and verifier measure this before synthesis.Source: §9 · p. 8
  • The controlled comparison exposes an information gap between a text-only control and a source-label oracle, while anti-shortcut checks show plan accuracy alone can mislead.Evidence and counterfactual locality help distinguish unsupported agreement from cue-sensitive reasoning.Source: §9 · p. 8
  • The package does not address listener experience or waveform quality; it is a pre-synthesis audit layer that could later be extended with audio-context inputs and governed perceptual evaluation.Source: §9 · p. 8

Reproducibility Statement

The release supports reproducing the paper's deterministic measurements from derived source-label data and scripts, without raw audio or audio generation.

Reproducibility Statement

AI Use Statement

The author used generative AI for language and consistency assistance and manuscript review, but not to generate scientific statements, synthetic data, or experimental results.

AI Use Statement

  • Generative AI assisted with language editing, consistency checks, manuscript review, and comparison against the repository implementation and reported summaries.The author verified the comparisons and made the final scientific decisions.Source: AI Use Statement · p. 9
  • No generative AI tool generated synthetic data, experimental results, or the paper’s scientific statements.Source: AI Use Statement · p. 9
  • The author reviewed all AI-assisted text and takes responsibility for the final manuscript, results, and artifacts.The same disclosure will be provided in the submission form.Source: AI Use Statement · p. 9

Appendix

The appendix documents the scope of the planning-stage measurements, metric corner cases, and split and matrix construction for the shortcut analysis.

A Appendix Roadmap

  • The appendix supports Sections 3–5’s planning-stage measurements; it adds no audio-conditioned, waveform, or listener evidence.Section B specifies metric corner cases, while Section C records split and matrix construction for shortcut analysis.Source: A Appendix Roadmap · p. 10

B Metric Details

  • Emphasis receives credit only for an exact normalized-set match; missing evidence has zero recall, while illegal evidence lowers precision through hallucinated-evidence rate.Undefined precision or recall terms and F1 when P + R = 0 use zero; hallucinated and uncited rates are zero when their reference sets are absent.Source: B Metric Details · p. 10

C Split and Matrix Details

  • The public split has 67/17/16 training/development/test cases; shortcut analyses use 60/16/24 source-key-disjoint and 56/12/32 source-emotion-disjoint splits.The evidence-sensitivity matrix crosses six conditions with 24 held-out pairs and three seeds, totaling 432 paired records.Source: C Split and Matrix Details · p. 10
  • Acoustic preflight and anchor checks cover 300 feature records and 100 matched cases, but do not establish audio-conditioned planning, waveform quality, or listener preference.These checks audit derived source labels against lightweight acoustic statistics and are excluded from the main result table because they do not measure listener judgments or alter the source-grounding finding.Source: C Split and Matrix Details · p. 10
Loading 2609.03203v3…