PaperlayerView exact revision on arXiv

Source-linked AI summary

AVERT: Audio-Verified Adjudication for Spoken Dialogue State Tracking

Chunggi Lee, Hanspeter Pfister

arXiv:2609.01828v1cs.CL

TL;DR

Spoken dialogue state tracking must recover noisy slot values while correcting errors that persist across turns, but transcript-only editing cannot use audio to adjudicate candidates. AVERT combines cross-turn agreement with a trained audio verifier and slot-restricted vote, add, and swap operators. On SpokenWOZ, it reaches 40.13 JGA versus 38.34 for the text editor, without retraining either model.

  • Problem

    Spoken dialogue state tracking is challenged by ASR errors concentrated in entity values and by errors that accumulate across turns.

  • Method

    AVERT combines cross-turn agreement with a trained audio-conditioned verifier to apply vote, add, and swap corrections to candidate values.

  • Results

    40.13 JGA is achieved on SpokenWOZ, compared with 38.34 for the text editor and 33.04 for the base model, without retraining R or E.

  • Takeaways & Limitations

    Slot-restricted value-level audio adjudication improves a strong text editor while keeping audio use per turn rather than consuming the full spoken history.

  • Takeaways & Limitations

    Only 21.0% of the editor’s errors have reference values attested in the cumulative ASR transcript, limiting the current add and swap operators.

Abstract

from arXiv · show

Spoken dialogue state tracking recovers slot-value pairs from speech, where ASR errors concentrate in entity values and persist across turns, making it both a generation and an editing problem. A strong per-turn text editor corrects much of this but, operating on the transcript alone, leaves three recoverable errors: a value predicted inconsistently across turns, an omitted slot, and a value the audio does not support. We present AVERT, which scores each candidate value by combining cross-turn agreement with a trained audio-conditioned verifier and resolves the three error types with three operators, vote, add, and swap, each restricted to the slots where its error is common. On SpokenWOZ, a base speech-LLM reaches 33.04 JGA, a text editor 38.34, and AVERT 40.13, without retraining either. This is in the range of a 1B end-to-end system that consumes the full spoken history (39.32), though AVERT uses two 1B decoders rather than one. The audio verifier contributes a statistically significant gain, and restricting each operator to a selected slot subset matters: removing it lets unrestricted voting overwrite correct categorical values and fall below the editor.

1 Introduction

Spoken dialogue state tracking must recover slot values from noisy speech while correcting errors that accumulate across turns. AVERT combines cross-turn agreement with audio-conditioned value verification to repair errors that transcript-only editing leaves unresolved.

  • Motivation: Spoken DST is difficult because ASR frequently mistranscribes or drops entity values, and early errors persist across later turns.These errors can directly produce wrong actions in deployed voice assistants.
  • Motivation: Transcript-only editing leaves inconsistent values, omitted slots, and values unsupported by the audio.Cross-turn agreement and audio evidence provide complementary signals for these recoverable errors.
  • Method: AVERT scores candidate values using cross-turn agreement and a trained audio verifier, then applies vote, add, and swap operators.A lexical attestation gate controls when values may be inserted or substituted.
  • Method: AVERT restricts each operator to slots where its associated error is common.This ties each correction to a specific slot and audio condition.
  • Results: 40.13 JGA is achieved by AVERT on SpokenWOZ, compared with 33.04 for the base model and 38.34 for the text editor.The gain is 1.79 points over the editor without retraining either model.
  • Results: Unrestricted voting overwrites correct categorical and boolean values, causing JGA to fall below the editor.The reported controls show that slot restriction is essential rather than a cosmetic design choice.

2 Related Work

Prior spoken-DST systems either rely on ASR output or process audio end to end, while correction methods generally operate on text alone. AVERT instead combines cross-turn candidate aggregation with value-level audio verification after the state has been formed.

  • Spoken dialogue state tracking: Early spoken-DST systems consumed ASR output or confidence scores, leaving audio unavailable as evidence for a specific slot value.SpokenWOZ showed that text DST methods transfer poorly to spoken data.
  • Speech-LLM systems: Speech-LLM systems address spoken DST by taking audio as input, enlarging spoken data, or converting textual data into speech-style training examples.These approaches include full or compressed spoken-history processing and joint spoken-text training.
  • Self-correction and re-ranking: Text correction methods revise an initial state or select among alternatives, but operate on transcript-level information.AVERT aggregates across past turns and prediction sources while adding audio-conditioned candidate verification.
  • Self-correction and re-ranking: AVERT conditions on a queried slot and re-ranks values already placed in the structured state rather than correcting words before state construction.This distinguishes its value-level audio conditioning from transcript correction such as ClozeGER.

3 Method

AVERT keeps the existing spoken-DST model and text editor, then adjudicates candidate slot values with cross-turn agreement, audio-conditioned verification, and slot-restricted operators. Its verifier checks individual values against audio while the operators address inconsistent, omitted, and unattested values.

  • Pipeline: AVERT’s pipeline uses the base model R, text editor E, and adjudication over candidate values across turns.R produces a transcript and first-pass state; E revises it before AVERT adjudicates candidates.
  • Candidate collection and scoring: AVERT scores candidates using agreement across sources and prior turns, weighted by σ’s audio-conditioned confidence.Repeated predictions contribute separately to the score even though each distinct value appears once in the candidate set.
  • Candidate collection and scoring: The attestation indicator α is a cumulative-transcript substring check used only to gate Add and Swap, not to rank candidates.First-token attestation is used because ASR may corrupt only part of a multi-token entity value.
  • Audio-conditioned verifier: The audio verifier σ estimates whether turn audio supports a candidate value for a queried slot.It combines pooled audio, slot, and candidate-value representations and is the only newly trained component beyond R and E.
  • Verifier training: AVERT uses a frozen speech encoder and trains a verifier with focal loss to score candidates from both editor and base-model sources.The verifier reuses R’s speech features and transfers across the shared SpokenWOZ value space without source-specific training.
  • Operators: Vote resolves inconsistent present values, Add inserts an attested confident value for an omitted slot, and Swap replaces an unattested value with an attested alternative.Each operator is enabled only for development-selected slots where it corrects more errors than it introduces.

4 Experiments

On SpokenWOZ, AVERT improves causal spoken-DST performance by combining a fixed base model and editor with audio-conditioned value verification, while ablations test its signals and slot-restricted operators.

  • Main Results: 40.13 JGA is AVERT’s result on SpokenWOZ, improving over the editor’s 38.34 and the base model’s 33.04 without retraining either.The gain over the editor is 1.79 points.
  • Main Results: AVERT reaches 40.13 JGA with two 1B decoders, compared with 39.32 for a comparable 1B system that consumes the full spoken conversation.AVERT keeps dialogue history as text and uses audio in a per-turn verifier, whereas the comparison system feeds the entire spoken conversation to the LLM.
  • Main Results: The gain over the editor holds under exact and fuzzy matching, reaching +1.54 under exact matching and +1.79 under fuzzy matching.Fuzzy matching is the metric used for comparison with Table 1 baselines.
  • Ablation: Removing the audio verifier drops JGA from 40.13 to 39.74, a significant +0.40-point contribution with 95% CI [+0.08, +0.71].Zeroing the pooled audio representation reduces JGA to 39.80, indicating that the verifier’s gain depends largely on its audio input.
  • Ablation: Vote, add, and swap reach 39.42, 39.97, and 40.13 cumulatively, with swap contributing a significant additional +0.16 over vote and add.Vote is the principal operator, while swap provides a smaller but significant marginal gain.
  • Effect of Slot Restriction: Removing slot restrictions lowers JGA to 38.14 for B1 and 36.53 for B2, as unrestricted voting overwrites correct categorical and boolean values.B2 raises such changes from 114 to 675 and falls 3.60 points relative to AVERT.

5 Analysis

AVERT improves the editor by correcting omissions, inconsistencies, and unsupported values, with gains concentrated in numeric slots and longer dialogues. Its audio verifier contributes to voting, while slot restriction prevents harmful overwrites.

  • 1.79 JGA points are gained over the editor, despite increases in hallucinations and other wrong values from ADD insertions.Omission and inconsistency each fall by about 16%, while hallucinations rise 6.2% and other wrong values rise 23.6%.
  • 57.5% of corrected slot errors are wrong values and 42.5% are missing or spurious slots.Slot-presence F1 rises from 92.02 to 92.50, as recall increases and precision decreases.
  • 37.93 JGA from consistency-filtered merging remains below the editor’s 38.34, showing that merging predictions alone is insufficient.A naive union reaches 37.34, while the filter cannot correct values that are consistently wrong.
  • +2.51 is AVERT’s largest absolute gain by slot type, exceeding categorical (+1.61) and proper-noun (+1.54) gains.Within vote, uniform agreement contributes +0.73 and verifier weighting adds +0.34.
  • 4.3% of vote decisions with multiple candidates tie on raw frequency, and audio-weighted scoring separates every such tie.The verifier also accounts for about a third of vote’s gain and a fifth of the total improvement.
  • +19.4% is AVERT’s relative improvement on turns 31+, compared with +0.6% on turns 1–5.Long dialogues benefit most because cross-turn evidence accumulates while the editor’s accuracy falls to 11.92% at turn 31+.
  • Vote, add, and swap respectively restore a consistent value, recover an omitted slot, and replace an unattested misspelling.The qualitative traces use cambridge, anatolia, and james thornton as examples.

6 Conclusion

AVERT combines cross-turn agreement, audio-conditioned verification, and slot-restricted operators to repair three recoverable editor errors. On SpokenWOZ, it raises a 1B text editor from 38.34 to 40.13 JGA without retraining the existing models.

  • AVERT combines cross-turn agreement with a trained audio-conditioned verifier and three slot-restricted operators for editor error correction.The operators repair three recoverable error types: inconsistent values, omitted slots, and unsupported values.
  • 40.13 JGA is achieved versus 38.34 for the 1B text editor, without retraining R or E.Audio enters only through a per-turn verifier, so the language-model context does not grow with dialogue length.

Limitations

The evaluation is limited to a 1B backbone and SpokenWOZ-dependent design choices, while lexical attestation cannot reach consistently mistranscribed values. Reproducibility is also constrained by Fisher’s LDC licensing requirement.

  • AVERT is evaluated only with a 1B backbone, so transfer of the editor delta to larger models remains a hypothesis.The paper does not run 9B or larger backbones.
  • Literal first-token attestation can reject values that are phonetically close to the audio when ASR consistently mistranscribes them.Soft phonetic attestation is left for future work.
  • 79.0% of the editor’s errors are unattested in the cumulative transcript and therefore unreachable by the current literal-attestation design.These errors may require stronger phonetic or generative audio grounding, bounding current add and swap headroom.
  • The ASR pre-training stage uses Fisher data that requires an LDC license to reproduce.SpokenWOZ and Loquacious are open, but Fisher is not freely reproducible under the stated setup.
  • Operator slot subsets and the add threshold are tuned on SpokenWOZ development data, and zero-shot schema transfer is not evaluated.The authors leave selection without target-domain labels to future work.
  • AVERT does not measure how the unattested error share is distributed across speakers.Consistently mistranscribed values are more likely for uncommon names and underrepresented accents.

A Operator Slot Subsets

Each operator is enabled only for development-selected slots where it corrects more errors than it introduces. The subsets are fixed before testing and cover 27 of 35 candidate slots.

  • Operator Slot Subsets: Each operator’s slot subset includes a slot only when development results show more corrections than introduced errors.The subsets are fixed before test evaluation.
  • Operator Slot Subsets: 27 of 35 candidate slots are covered by the fixed operator subsets.A slot may belong to multiple operators, but only one fires per turn according to slot presence or attestation.
Loading 2609.01828v1…