PaperlayerView exact revision on arXiv

Source-linked AI summary

DecisionTraceBench: Tracking Decision-Critical Information in Multi-Party Spoken Conversations

Wenchao Du, Fangxin Shang, Long Wu, Runze Cui, Qing Yang, Yehui Yang

arXiv:2610.34746v1cs.CL

TL;DR

Existing speech benchmarks leave joint recovery of decision evolution across speakers insufficiently evaluated, despite the need to trace how proposals change and who is responsible. DecisionTraceBench evaluates complete decision-event chains from audio, jointly checking attribution, scope, relations, and temporal evidence. On human recordings, the highest DCIST score is 21.35%, and transcription error rates do not adequately reflect complete trajectory recovery.

  • Problem

    Existing benchmarks cover audio understanding, speaker-attributed reasoning, or turn-level state tracking, but do not jointly establish complete decision trajectories, their sources, and evidence.

    Source: §1 · p. 1

  • Method

    DecisionTraceBench makes the complete decision-critical event chain the prediction target and evaluates required events, attributes, relations, scope, and temporal evidence from audio.

    Source: §1 · p. 1, §3.3 · p. 3, §5.1 · p. 5

  • Results

    21.35% was the highest DCIST score on human recordings, and overall transcription error rates did not adequately reflect complete trajectory recovery.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Complete decision-process recovery must be evaluated directly rather than inferred from transcription accuracy or terminal-value correctness.

    Source: §6 · p. 8

    Both subsets are fixed research evaluation cohorts, and validation in other record-sensitive settings requires independent data and experiments.

    Source: Limitations · p. 9

Abstract

from arXiv · show

In high-stakes domains such as finance, critical information evolves through multi-party interaction. Recovering these changes and their sources is fundamental to reliable speech understanding. Existing speech benchmarks focus on transcription, local understanding, and turn-level state prediction, leaving joint decision evolution across speakers insufficiently evaluated. We introduce DecisionTraceBench, a speech state-tracking benchmark for Decision-Critical Information State Tracking (DCIST). It jointly evaluates decision event sequences, speaker attribution, scope, and temporal evidence, using trajectory-level F1 to measure complete event-chain recovery. Our dataset comprises Chinese financial dialogues involving two to four speakers, with 100.39 hours of synthetic audio and 8.61 hours of human-performed recordings. On a fixed cohort of human-performed recordings, five cascaded systems and two direct-audio models achieve DCIST scores of 5.73% to 21.35%. Ablations with speaker-labeled reference scripts and no temporal localization requirements yield a maximum score of only 38.72%. Audio perturbation experiments further show that overall transcription error rates do not adequately reflect complete trajectory recovery. These findings reveal limitations in jointly understanding information provenance and state evolution, providing a benchmark for evaluating decision-process understanding and decision traceability in high-stakes speech applications.

1 Introduction

Financial decisions require recovering how proposals change and who is responsible, not merely what was said or the terminal value. DecisionTraceBench makes the complete decision-critical event chain the target and evaluates its recovery across speakers, with provenance and temporal evidence.

1 Introduction

  • Local correctness does not guarantee trajectory correctness: a correct terminal amount can conceal omitted changes or misattribution of an authorized decision.The benchmark gap is joint recovery of effective events, their relations, sources, and evidence.Source: §1 · p. 1
  • DecisionTraceBench defines DCIST as prediction of the complete decision-critical event chain, with completeness based on required events and relations rather than every possible speech segmentation.Evidence-supported refinements cannot compensate for missing required events.Source: §1 · p. 1
  • The framework evaluates required-event coverage, attributes and relations, and temporal evidence across full trajectories.Its event actions are Start, Pause, Progress, and Finalize.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The benchmark contributes Chinese financial dialogues with two to four speakers, synthetic and human-recorded speech, event provenance, temporal evidence, and paired variants.The paired variants change speaker assignments or confirmation relations.Source: §1 · pp. 1–2
  • Its evaluation includes cascaded and direct-audio systems, reference-script controls, and targeted acoustic perturbations to examine recovery failures not captured by transcription quality alone.The experiments ask whether systems recover full trajectories, whether the task remains difficult with reference text and speaker labels, and whether aggregate transcription errors reflect trajectory quality.Source: §1 · pp. 1–2

2 Related Work

Prior resources cover audio understanding, speaker-attributed reasoning, spoken events, dialogue states, and state evolution. DecisionTraceBench instead evaluates the joint correctness of a complete decision-event chain, including speaker, scope, and time.

2 Related Work

  • Audio and financial-speech benchmarks evaluate content understanding, while DecisionTraceBench also tracks when information changes effectively, who causes the change, and its scope.The cited resources include AudioBench, MMAU, SpeakerSleuth, MultiFinBen, and FinAudio.Source: §2 · p. 2
  • Speaker-attributed reasoning, spoken-event extraction, and turn-level slot-state benchmarks address components that DecisionTraceBench connects over an entire required chain.The comparison includes M3-SLU, HEAR, SpeechEE, and SpokenWOZ.Source: §2 · p. 2
  • State-evolution resources represent belief states, procedural entity changes, or incremental propositions, while DecisionTraceBench targets complete decision-record recovery from raw audio.VoxMem instead evaluates question-conditioned retrieval; DecisionTraceBench predicts the complete event chain itself.Source: §2 · p. 2

3 Full-Trajectory State Tracking

The task represents decisions as temporally ordered event chains for entity–field pairs, then scores complete required-chain recovery across structure, attribution, scope, and time. Its matching rules distinguish required events from supported refinements and invalid additions.

3.1 Representation and Task

  • A trajectory is an entity–field pair containing ordered decision cycles whose events record action, value, speaker, scope, predecessor, and evidence interval.The public schema supplies 193 entity–field combinations, 64 scope labels, and permitted value structures, but not sample-specific fields, roles, or answers.Source: §3.1 · pp. 2–3
  • Turns and events are not one-to-one: an event requires an observable substantive change, and its interpretation may depend on earlier turns.An unapproved numeric candidate is not necessarily an approved result.Source: §3.1 · p. 3

3.2 Events, Cycles, and Evidence

  • A cycle’s actions encode state changes: a revision that leaves processing suspended is hold, while settlement closes the cycle and explicit reopening starts another.A first observed event may be hold or settle; no unobserved start is invented.Source: §3.2 · p. 3, Table 1 · p. 3

3.3 Matching and DCIST

  • Required-chain matching aligns every required event one-to-one and in order, checking action, value, speaker, scope, cycle boundaries, predecessor relations, and time.The terminal state is derived from the last event rather than scored independently; evidence intervals locate the principal triggering utterance.Source: §3.3 · p. 3, §3.2 · p. 3
  • A predicted interval must overlap at least 80% with the coarse gold interval; supported extra nodes can be exempt, but omissions and unsupported insertions remain errors.Exemptions are removed only in the scoring view, with predecessor edges contracted; they cannot repair invalid edges or remove an extra cycle.Source: §3.3 · pp. 3–4, Figure 2 · p. 4
  • Legitimate new trajectories are excluded from the predicted denominator but do not add true positives, and reference-script experiments retain structural requirements while disabling time checks.The scoring protocol also reports the proportion of unique exempt nodes among parseable predicted nodes.Source: §3.3 · p. 4

4 Dataset

The dataset is built by designing gold decision trajectories, generating and checking colloquial scripts, and producing audio aligned to event evidence. It combines controlled synthetic speech with human-recorded scripted conversations and paired variants.

4 Dataset

  • The construction pipeline designs gold trajectories, generates colloquial scripts conditioned on them, verifies script–trajectory consistency, and produces audio with aligned event evidence.Synthetic speech supports controlled variation, while human recordings test the same task with natural voices.Source: §4 · p. 4
  • Synthetic dialogues cover 14 financial scenarios and human recordings cover 10, including due diligence, supply-chain finance, drawdown checks, audit adjustments, and insurance claims.The human recordings involve 22 participants.Source: §4 · p. 4
  • The synthetic subset contains 790 A/B pairs and 666 standalone scripts; the human subset contains 10 pairs and six standalone scripts.Paired variants change speakers, role assignments, or confirmation relations, with coherence checks on references and subsequent responses.Source: §4 · p. 5
  • Both subsets are fixed research evaluation cohorts without a training/development partition, and their scoring uses monetary equivalence, expression mappings, and minimum-chain matching.Shared scripts and A/B variants remain grouped within dialogue families.Source: §4 · p. 5

5 Experiments

The experiments test complete-trajectory recovery on human and synthetic speech, reference-script controls, and targeted acoustic perturbations. Across these evaluations, complete trajectories remain difficult to recover, and terminal-value or transcription measures do not by themselves establish process recovery.

5.1 Setup

  • The human-data comparison fixes 25 recordings, 293 gold trajectories, and 16 dialogue families, with DCIST as the primary metric.The evaluated systems include cascaded speech-recognition pipelines and two direct-audio models; confidence intervals and pairwise tests use family-level procedures.Source: §5.1 · p. 5

5.2 Full-Trajectory Recovery

  • The seven systems score 5.73–21.35% DCIST; FunASR+Astra has the highest numerical score, while only selected pairwise differences pass Holm correction.FunASR+Astra recalls 42.75%, 17.48%, and 13.56% of one-node, two-node, and three-or-more-node trajectories, respectively; speaker-count and duration strata are not monotonic.Source: Table 3 · p. 6, §5.2 · p. 6

5.3 Synthetic Speech

  • On synthetic speech, timed DCIST is 32.71% for FunASR+DeepSeek, while untimed DCIST reaches 74.59% for VibeVoice+DeepSeek.The four systems share 2,224 recordings and 5,269 gold trajectories; untimed scoring disables temporal checks.Source: §5.3 · p. 6, Table 4 · p. 6

5.4 Reference Text and Speaker Labels

  • With labeled scripts and no time requirement, the highest DCIST is 38.72%; supplying labels benefits Astra and DeepSeek significantly, but not Qwen or Kimi.Removing labels lowers Astra, DeepSeek, and Qwen by 9.73, 6.73, and 5.10 percentage points, respectively, while Kimi increases by 0.86 points.Source: §5.4 · pp. 6–7, Table 5 · p. 6

5.5 Terminal Values and Paired Dialogues

  • Terminal-value recall exceeds full-chain recall, and correct terminal values often belong to incomplete or incorrect trajectories; paired full-trajectory recovery is also limited.FunASR+Astra has 51.20% incomplete or incorrect trajectories among its recovered terminal values; it jointly recovers 21/87 paired audio fields, versus 41/102 with labeled scripts.Source: §5.5 · p. 7

5.6 Scoring Sensitivity

  • Across 41 conditions, adopting second-round scoring decisions changes DCIST by at most 1.81 percentage points, indicating relative stability under the observed review differences.For labeled-script Astra, DCIST rises from 38.72% to 40.50%, with ten trajectories becoming matches and two ceasing to match.Source: §5.6 · p. 7

5.7 Targeted Acoustic Perturbations

  • Under B2 keyword noise, VibeVoice DCIST falls from 9.66% to 6.34%, while FunASR DCIST falls from 9.68% to 7.64%.Only the VibeVoice difference is statistically detectable in this test; automatic word boundaries and retained redundant evidence limit attribution to individual words.Source: §5.7 · pp. 7–8, Table 6 · p. 7

5.8 Failure Analysis

  • Among five cascaded systems, required-chain mismatch accounts for 807 of 1,273 failures, while missing field candidates account for 340.These are mutually exclusive matching barriers, not independent judgments of underlying model errors; exempt output content measures granularity, not hallucination rate.Source: §5.8 · p. 8

6 Conclusion

DecisionTraceBench evaluates speech systems on complete decision-event histories, including who changed information, its scope, and temporal evidence. Experiments find complete-history recovery challenging and show that terminal-value correctness and transcription error rates do not adequately reflect it.

6 Conclusion

  • DecisionTraceBench jointly evaluates decision evolution across speakers, information attribution, scope, and temporal evidence.It is a speech-based benchmark for event-chain tracking.Source: §6 · p. 8
  • The benchmark provides process-oriented evaluation for audio settings requiring accurate event reconstruction, including financial operations, accident investigations, and critical meetings.Source: §6 · p. 8
  • The highest DCIST score on human recordings is 21.35%, and 51.20% of FunASR+Astra’s correctly recovered terminal values belong to incomplete or incorrect trajectories.The paper also reports that higher overall transcription error does not necessarily imply lower DCIST.Source: §6 · p. 8

Limitations

The benchmark’s conclusions are bounded by the human subset, scoring target, temporal annotations, and evaluation design. These limits constrain claims about representativeness, fine-grained capability, and generalization.

Representativeness

  • The human subset comprises 26 scripted Chinese recordings, limiting coverage of spontaneous meetings and multilingual practice.Because A/B variants share content, the number of independent examples is smaller than the recording count.Source: Limitations · p. 8

Representation and scoring

  • Once the required chain is recovered, additional accurate fine-grained events do not improve the primary score.The metric therefore cannot distinguish those incremental capabilities.Source: Limitations · p. 8

Temporal localization

  • Coarse-window overlap tests evidence placement more directly than word-boundary precision, and small score changes across overlap thresholds do not establish accurate boundaries.Requiring half-window coverage sharply reduces scores; fine-grained temporal evaluation requires finer annotation.Source: Limitations · pp. 8–9

Comparability and generalization

  • The reported family-clustered intervals characterize variation within the collected dialogue set, not repeated-generation or participant-independent population variation.Both subsets are fixed research cohorts, and validation in other record-sensitive settings requires independent data and experiments.Source: Limitations · p. 9

Ethical Considerations

The section describes participant protections and uses constructed examples to illustrate how the benchmark represents and scores decision trajectories. The examples distinguish complete event histories from correct terminal values, refinements, and unrelated trajectories.

Ethical Considerations

  • Participants consented to recording, research, model evaluation, and public release; dialogues are fictional and do not request real financial information.Public-release permissions govern distribution, commercial use requires separate permission, and withdrawal affects future official versions but cannot guarantee recall of third-party copies.Source: Ethical Considerations · p. 9

A.1 A Four-Event Required Chain

  • The four-event example is constructed, and its event value records the current candidate or result rather than necessarily a settled value.It distinguishes S1’s materials role from S2’s technical verification, which does not approve a loan.Source: §A.1 · p. 12

A.2 Correct Terminal Value, Incorrect Chain

  • A prediction with the correct terminal amount still fails when it omits required suspension and resumption events.The example scores zero for one incomplete trajectory; valid interval overlap cannot compensate for incorrect event attributes or relations.Source: §A.2 · p. 12

A.3 Additional Nodes and Trajectories

  • A supported, materially changed hold can be treated as a legitimate refinement, while a repeated unchanged hold is an invalid insertion.A separate supported field can count as a new trajectory without increasing F1 beyond 1.Source: §A.3 · p. 13

B Relation to Existing Tasks

DecisionTraceBench evaluates complete decision records rather than states alone, and its corpus combines synthetic and human-recorded financial dialogues. The section details the corpus, evidence alignment, construction workflow, and direct-audio evaluation conditions.

B Relation to Existing Tasks

  • The target is full decision-record recovery: states are evaluated together with the effective events that produce them.Financial fields instantiate the representation, but the structural target is not intrinsically financial.Source: B Relation to Existing Tasks · p. 13

C.1 Distribution and Scenarios

  • Synthetic dialogues cover 14 financial scenarios, while human recordings cover 10 scenarios across 26 recordings.The human subset includes credit due diligence, supply-chain finance, drawdown checks, audit adjustments, and insurance claims.Source: §C.1 · p. 13

C.2 Voice Sources, Evidence, and Speaking Rate

  • The 100.39-hour synthetic batch uses 193 source-speaker entries, and all 3,538 human-script turns have aligned start and end times.Human event evidence relies on supervised collection and coarse event-level windows, not exhaustive word-level annotation; the automatic transcript-character rate is 229.53 characters per minute.Source: §C.2 · pp. 13–14

C.3 Generation and Inference

  • Construction moves from gold-trajectory design through colloquial script generation, cross-review, rule checks, recording, and time alignment.All 1,456 admitted synthetic script groups have GLM and Qwen review records, while human gold review covers all 308 trajectories and 3,538 scripted turns.Source: §C.3 · p. 14

C.4 Direct-Audio Gemini Evaluation

  • Direct-audio requests provide a complete recording and frozen query without transcripts, cast lists, speaker counts, or gold labels.The fixed comparison cohort has 25 recordings; Argon returned 25 complete answers, while Flash returned 24, including one format-invalid answer, and had one absent answer.Source: §C.4 · pp. 14–15
  • Direct-audio frontends, chunk lengths, and auxiliary inputs define distinct evaluation conditions rather than matched-compute ablations against cascades.The same semantic, speaker, structure, and 80% temporal-overlap rules apply; sequential review left the main DCIST scores unchanged.Source: Table 10 · p. 15, §C.4 · p. 15

C.5 Pairs, Evaluation Cohorts, and Gold Revisions

The experiments examine paired decisions, fixed evaluation cohorts, scoring changes, audio perturbations, speaker controls, and matching rules. Their results show non-monotonic transcription and trajectory patterns, while the evaluation design preserves key structural constraints.

C.5 Pairs, Evaluation Cohorts, and Gold Revisions

  • Evaluation covers fixed research cohorts of 2,246 synthetic and 26 human recordings, with no training or development partition, and assesses existing systems rather than train-to-test generalization.Family identifiers preserve paired dependence in analyses.Source: §C.5 · p. 15
  • The human gold revision affected three files, four trajectories, and six events, yet left DCIST unchanged for all 939 predictions.Other required constraints still prevented full matches.Source: §C.5 · p. 16

C.6 Noise Verification

  • B2 changed 8.20% of samples, while FunASR completed 26 transcriptions and VibeVoice completed 22; four missing outputs are not measured zeros.The noise-verification passage reports samplewise checks and protected identity, authority, premise, and gold regions.Source: §C.6 · p. 16

D Acoustic Perturbations

  • The acoustic perturbation design expands 3,153 anchors covering 562 gold events by 160 ms on each side, merging overlapping intervals.Boundaries are automatic rather than manually verified words, and redundant evidence outside masks remains audible.Source: D Acoustic Perturbations · p. 16

D.1 Historical Evidence-External Noise

  • FunASR DCIST is 12.41% on clean audio, 7.91% on A1, and 10.19% on A2, although script-referenced WER rises from 21.69% to 24.17% and 29.51%.On 23 shared VibeVoice recordings, WER rises from 11.13% to 12.34% while DCIST rises from 10.10% to 11.22%; these are not a controlled causal contrast with B2.Source: §D.1 · p. 16

D.2 Automatic Speaker Control

  • Automatic attribution controls changed Kimi’s score by +1.82 pp for FunASR A1, +1.24 pp for VibeVoice A1, and −1.25 pp for FunASR A2.None of the three corrected tests is significant, and the controls use automatic identities rather than human gold speakers.Source: §D.2 · pp. 16–17

C.7 Synthetic Evaluation

  • With temporal constraints, expression mappings add 11.21–21.35 pp on fixed outputs, compared with 0.02–0.28 pp for monetary normalization and 0.05–0.81 pp for extra-node adjudication.These increments depend on scoring order, not improved model predictions; the table decomposes successive scoring changes on the same 2,224 recordings.Source: §C.7 · p. 16, Table 12 · p. 17

E.1 Required Attributes and Relations

  • Required-event matching checks cycles and ordered event attributes, including action, speaker, settlement result, and predecessor structure.Consistent identifier renaming is allowed, but speaker permutation and content editing cannot repair a prediction; an incorrectly expressed required event is a mislabeled substitution, not an extra.Source: §E.1 · p. 17

E.2 Sequential Semantic Review

Sequential semantic review resolves expression equivalence before deterministic trajectory scoring, with human review of disagreements. Across sensitivity conditions, adopting second-model proposals changes DCIST by at most 1.81 percentage points.

E.2 Sequential Semantic Review

  • Across 41 conditions, maximum absolute DCIST change is 1.81 pp; Astra rises from 28.99% to 30.80% unlabeled and 38.72% to 40.50% labeled.These changes indicate relative stability within sequential review while leaving the full-recovery challenge intact.Source: §E.2 · p. 18

E.3 Additional-Content Adjudication

  • Astra reviews unmatched nodes and absent-gold fields against actual model input and frozen rules; unresolved evidence boundaries and permitted refinements can be exempt.Invalid insertions can invalidate a trajectory, and a coarse script cannot resolve discrepancies that require listening or repair a transcript.Source: §E.3 · pp. 18–19

E.4 Aggregation and Verification

  • Every required event must pass, and maximum one-to-one trajectory matching supplies TPi; global micro-F1 is supplementary.Reproduction tests cover equivalence, timing, identities, actions, edges, cycles, exemptions, and matching; the scoring snapshot passed 40 regression tests.Source: §E.4 · p. 19

F.1 Clustered Intervals and Paired Tests

  • Three contrasts involving direct-audio models pass 0.05, while the two original significant cascade contrasts remain significant under the 21-comparison Holm correction.Tests use a fixed 25-recording cohort in 16 families, exact family-level sign permutations, and jointly corrected comparisons.Source: §F.1 · p. 20

F.2 Terminal-Value Projection

  • Terminal-value projection checks the final required event’s non-null value without requiring intermediate events, action, speaker, scope, or time.It projects the same answer rather than evaluating a separately prompted task; field assignment is one-to-one and prioritizes existing full-chain matches.Source: §F.2 · p. 20

F.3 A/B Joint Recovery

  • A/B joint recovery requires both variants to match their own full gold trajectories, covering nine audio pairs and 87 shared fields and ten script pairs and 102 fields.The changed subset contains 35 audio and 40 script field pairs; fields unique to one variant are excluded.Source: §F.3 · p. 20

G.1 Speakers, Duration, and Chain Length

  • One-node full-chain recall ranges from 7.63–42.75%, versus 1.69–13.56% for trajectories with at least three required events across systems.Speaker-count and duration strata are not independently controlled, and four long recordings cannot establish a stable length-degradation law.Source: §G.1 · p. 21

G.2 Extra Nodes and Micro-Averaging

  • The extra-node proportion describes output granularity, not hallucination, and cannot cancel omissions; micro-F1 weights recordings with more trajectories more heavily than recording-macro DCIST.Its node denominator includes parseable outputs, including outputs that fail a schema check.Source: §G.2 · p. 21

G.3 Time Windows and Speaker Identifiers

Frozen diagnostics show that stricter evidence-window coverage sharply lowers scores, while identifier remapping addresses numbering conventions rather than diarization accuracy. Local anchor-absence signals accompany many lost trajectories but are not verified ASR errors.

G.3 Time Windows and Speaker Identifiers

  • Requiring at least half of the gold window in addition to the official 80% predicted-window criterion reduces scores to 0.00–3.99%.Valid predicted windows have median lengths of 5.06–10.20 seconds, and predicted/gold window-length ratios within fully matched trajectories range from 15.54% to 48.56%.Source: §G.3 · pp. 21–22
  • Identifier permutation diagnoses numbering convention, not diarization accuracy; one FunASR+Astra and four FunASR+Kimi recordings improve, while three other systems show no improvement.The diagnostic uses one globally consistent, one-to-one speaker mapping per recording.Source: §G.3 · p. 22

G.4 Local Evidence and Trajectory Changes

  • 26/26 FunASR and 13/16 VibeVoice lost trajectories contain an anchor-absence signal, but it also appears in gained trajectories and is not a verified ASR error rate.The signal marks a normalized anchor form present in original local context but absent in B2 context.Source: §G.4 · p. 22

H Astra Error Distributions on Reference Scripts

Astra’s reference-script error analysis finds content and scope errors most often among originally unmatched trajectories. Speaker-error rates differ between labeled and unlabeled scripts, but categories can co-occur and do not provide a paired causal decomposition.

H Astra Error Distributions on Reference Scripts

  • 38 unmatched trajectories (21.71%) with speaker labels and 72 (34.62%) without labels involve speaker errors; content and scope errors are the two most frequent categories in both conditions.The table counts trajectories involving each error type, and categories may co-occur.Source: H Astra Error Distributions on… · p. 23, Table 30 · p. 26
  • The analysis covers originally unmatched gold trajectories—175 with speaker labels and 208 without—among 308 gold trajectories per condition, using ordered alignment of unmatched nodes between preserved matches.The failure sets differ, so percentages describe their composition rather than mutually exclusive causes or a paired causal decomposition of score differences.Source: H Astra Error Distributions on… · pp. 23, 26
Loading 2610.34746v1…