PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents

Yanjie Zhang, Nanchen Hu, Yushi Sun

arXiv:2609.32536v1cs.SDcs.AIcs.CLcs.MMeess.AS

TL;DR

Voice agents must decide not only what a command requests, but whether its acoustic and conversational context warrants action. VGBENCH tests that decision, and its results show that supervised training can improve action gating while cross-scene proximity gating remains unreliable.

  • Problem

    Command recognition does not ensure that an agent uses acoustic and conversational context to decide whether an action is addressed to it.

    Source: §1 · p. 1, §7 · p. 8

  • Method

    VGBENCH tests action-level addressedness through MUTE, TOOL, and ANSWER decisions, including speaker-switch pairs that hold words fixed while changing source, distance rendering, and timing.

    Source: §3.1 · p. 3, §3.2 · p. 4

  • Results

    Raw models’ strongest switch mute rate is 14%, while supervised training mutes 91.3% of switched commands and selects the correct tool for all nearby wearer commands and text-only controls.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VGBENCH measures multi-cue acoustic-context gating, and factorized controls identify an independent source-change effect while far-field sensitivity varies across acoustic scenes.

    Source: §8 · p. 9, §7 · p. 9

    Results use one split and one run per setting, and the balanced test does not vary room, template, natural bystander speech, or open-set speakers.

    Source: §3.3 · p. 4, §7 · p. 9

Abstract

from arXiv · show

Audio language models can recognize spoken commands and invoke tools, but an agent must first decide whether the acoustic and conversational context warrants action. We introduce VGBench, a 1,018-item diagnostic benchmark for action-level addressedness across side-talk, self-talk, and speaker-switch scenarios. Each item uses a shared action space comprising silence, a tool call, and a natural-language answer. Speaker-switch pairs hold the specified words fixed while source, distance rendering, and a temporal boundary define a controlled wearer-to-bystander shift. Six raw Audio LLMs and three training-free adaptations often identify the target tool yet rarely withhold action under this shift; the highest raw switch mute rate is 14%. We then use VoxGate as a post-training case study. Supervised training mutes 91.3% of switched commands while choosing the correct tool for all nearby wearer commands and text-only controls. An exploratory GRPO stage has similar switch performance; side-talk accuracy rises from 68.4% to 70.9%, and self-talk muting from 52.0% to 60.0%. Factorized controls identify an independent source-change effect, while sensitivity to the far-field manipulation varies across acoustic renderings. The benchmark therefore measures multi-cue acoustic-context gating rather than isolated speaker identity.

1 INTRODUCTION

The paper frames voice-agent action as a joint decision about command meaning and whether acoustic and conversational context warrants action. VGBENCH exposes that distinction, and a VOXGATE case study examines whether the action boundary can be learned.

1 INTRODUCTION

  • Voice agents must distinguish a command's textual content from acoustic and conversational evidence about whether it is addressed to the assistant.A bystander may utter the same command as the user, or a user may mention one while thinking aloud.Source: §1 · p. 1
  • VGBENCH contains 1,018 items across side-talk, self-talk, and speaker-switch scenarios, using a shared action space of mute, tool call, and natural-language answer.Speaker-switch pairs preserve specified words while jointly changing trigger source, far-field rendering, and a 600 ms boundary.Source: §1 · pp. 1–2
  • The strongest raw switch mute rate is 14%, while Step-Audio-R1.1 selects the target tool on 96% and 97% of same-speaker and text-only controls but mutes only 1% of switched commands.This contrast exposes a gap between recognizing command content and using acoustic-pragmatic evidence to control action.Source: §1 · p. 2
  • Supervised training mutes 91.3% of switched commands while selecting the correct tool for all nearby wearer commands and text-only controls.An exploratory GRPO stage yields similar switch muting; one run raises side-talk accuracy from 68.4% to 70.9% and self-talk muting from 52.0% to 60.0%.Source: §1 · p. 2
  • The paper presents VGBENCH as an action-level benchmark and VOXGATE as a post-training case study of whether the measured gate is learnable without collapsing positive controls.The case study also uses exploratory GRPO and factorized controls to characterize remaining gains and cue dependence.Source: §1 · p. 2

2 RELATED WORK

Related work covers addressedness in voice-assistant pipelines, broader Audio LLM and paralinguistic benchmarks, and inference-time or post-training adaptations. VGBENCH is positioned around text-matched contrasts that test whether context changes an executable action.

Addressedness and voice-agent evaluation

  • VGBENCH complements related voice-agent benchmarks with text-matched action contrasts, linking acoustic-context use to action selection rather than a separate front-end addressedness score.Related resources include WearVox, Audio2Tool, and ProVoice-Bench.Source: §2 · p. 2

Acoustic and paralinguistic evidence in Audio LLMs

  • Audio and paralinguistic benchmarks cover broad reasoning, prosody, speaker attributes, and acoustic-linguistic conflicts, while VGBENCH asks whether such evidence changes silence, answer, or tool choice.VoxParadox tests perceptual answers to linguistic-acoustic conflicts; VGBENCH targets action selection.Source: §2 · p. 2

Post-training and inference-time adaptation

  • Prior work includes GRPO and other audio post-training methods, as well as training-free adaptations using orchestration, acoustic DSP, or chunked listening.The paper evaluates representative inference-time adaptations as behavioral probes and uses post-training to test whether the VGBENCH action boundary is learnable.Source: §2 · p. 3

3 VGBENCH: ACTION-LEVEL ADDRESSEDNESS DIAGNOSTICS

VGBENCH treats addressedness as an executable choice among mute, tool, and answer, and separates complementary attribution and action-gating challenges. Its construction and evaluation controls specify both what the benchmark measures and where its generalization claims stop.

3.1 TASK FORMULATION

  • An agentic voice assistant chooses MUTE, TOOL, or ANSWER based on textual semantics, intended recipient, and acoustic source.The task tests whether contextual evidence changes an executable decision, rather than perceptual classification alone.Source: §3.1 · p. 3

3.2 DIAGNOSTIC DESIGN

  • Side-talk tests attribution within an exchange, self-talk tests command-like monologues framed as non-instructions, and speaker-switch tests action reversal under a controlled source-and-scene shift.Speaker-switch preserves specified words: a same-speaker near-field trigger targets a tool, while a bystander trigger under far-field rendering after 600 ms targets Mute; text-only retains the tool target.Source: §3.2 · pp. 3–4

3.3 CONSTRUCTION AND SPLITS

  • Speaker-switch cases combine 100 seed and 300 generated cases across ten tools, with separately synthesized context and trigger segments and a fixed source, boundary, and room intervention.Post-training uses an approximately 80/20 item split, including 320/80 speaker-switch pairs; item-level separation does not establish held-out voice, template-family, or tool-family generalization.Source: §3.3 · p. 4

3.4 EVALUATION CONTRACT

  • Scenario-specific metrics jointly assess switch muting and positive-control tool selection, self-talk muting, and acting only on the assistant-directed side-talk utterance.Tool scoring checks parseable tool names rather than complete executable-call accuracy; the side-talk free-response judge has no reported human-agreement estimate.Source: §3.4 · pp. 4–5

4 OFF-THE-SHELF ACTION POLICIES

Off-the-shelf Audio LLMs and training-free adaptations show distinct action errors under the standardized VGBENCH interface. High tool selection can coexist with little or no muting when the acoustic source and scene change.

4.1 SETUP

  • Step-Audio selects the target tool on 96% and 97% of same-speaker and text-only controls, yet mutes only 1% of switched commands; Kimi has the highest raw switch mute rate at 14%.Kimi selects the target tool on 53% and 44% of the two controls, respectively.Source: §4.1 · p. 5
  • Systems exhibit different shortcuts, including first-utterance preference, later-utterance preference, and responding to both side-talk utterances.Audio Flamingo 3 receives zero canonical tool-selection credit because it emits a different bracketed action language, a format mismatch that does not explain its low mute rates.Source: §4.1 · p. 5
  • Inference-time adaptations shift the error profile: AURA improves side-talk and self-talk, while muting no switched commands; TwS raises self-talk muting but lowers side-talk accuracy.SHANKS also raises self-talk muting, but chunked output often violates the action grammar, leaving 1% same-speaker tool selection.Source: §4.1 · pp. 5–6

5 CAN ACOUSTIC-CONTEXT GATING BE LEARNED?

VoxGate is used as a post-training intervention to test whether acoustic-context gating can be learned without treating it as a new speaker-identification mechanism. The section describes supervised joint training and exploratory counterfactual-pair GRPO.

5.1 SUPERVISED JOINT TRAINING

  • Supervised training mixes VGBENCH addressedness examples with WearVox answer, abstention, tool-use, and translation examples.Switch pairs use matched words with tool targets for same-source near-field speech and mute targets for bystander speech; text-only examples retain tool targets.Source: §5.1 · p. 6
  • The cue-balanced SFT control replaces 1,212 of 5,748 training slots, balancing near/far rendering and 0/600 ms separation within both action labels.Other task proportions, optimization steps, and the LoRA configuration remain fixed; far-field authorized-source/tool examples make this a cue-control experiment, not proximity-gate training.Source: §5.1 · p. 6

5.2 EXPLORATORY COUNTERFACTUAL-PAIR GRPO

  • Counterfactual-pair GRPO groups samples from matched same-speaker/tool and switched/mute conditions so reward normalization reflects separation of the pair.Groups with zero reward variance are excluded because they provide no within-group policy-gradient signal.Source: §5.2 · p. 6
  • The reward scheme scores correct mute or tool actions at +1 and incorrect actions at −1, with additional task-specific rewards for answers and WearVox tasks.Malformed tool syntax incurs an additional −0.2; scenario-specific reward tracking is used because aggregate curves can hide saturated controls or failed mute boundaries.Source: §5.2 · p. 6
  • The GRPO experiment has no unpaired-GRPO or repeated-seed control, so it cannot establish that pair-aware grouping caused an SFT-to-GRPO difference.The stage tests whether paired optimization preserves or improves the learned action boundary.Source: §5.2 · p. 6

6 POST-TRAINING RESULTS

Supervised training substantially mutes held-out speaker switches while retaining target-tool selection on positive controls; GRPO has similar switch performance. Factorized and cue-balanced tests show that the measured gating depends on multiple cues and acoustic rendering, while downstream WearVox performance is retained.

6 POST-TRAINING RESULTS

  • 73 of 80 switched commands are muted by supervised training, versus 74 of 80 after GRPO; side-talk accuracy rises from 68.4% to 70.9% and self-talk muting from 52.0% to 60.0%.Both models select the target tool for every near-field same-speaker and text-only control; these are descriptive differences from one training run and do not establish that pair-aware grouping caused the improvement.Source: §6 · p. 7
  • With near-field rendering and a fixed gap, changing only the trigger source raises mute rate from 1.25% to 50.0% for SFT and to 53.75% for GRPO.Same-speaker near-to-far rendering raises muting to 60.0% for both models, while the gap alone changes same-speaker near-field muting by only 1.25 points; the main switch result combines source and distance dependence.Source: §6 · p. 7, Table 3 · p. 7

6.1 CUE-BALANCED SFT AS A CUE CONTROL

  • 39.22% → 66.88% equal-weight target-action accuracy is reported for original versus cue-balanced SFT across the scene-consistent eight-condition test.Each condition has 80 examples; same-speaker triggers target a tool when near and mute when far, while different-speaker triggers target mute.Source: Table 4 · p. 8
  • Distance rejection varies across renderings: direct retesting raises same/far/600 muting from 60.00% to 90.00%, while scene-consistent far-field audio yields only 3.75%/5.00% muting of same/far triggers.The different WAVs make distance rejection scene dependent; the direct-retest different-speaker rates rise from 35.00%/50.00% to 66.25%/82.50% at 0/600 ms.Source: §6.1 · p. 8

6.2 DOWNSTREAM TASK RETENTION

  • Joint supervised training and joint GRPO reach 72.14% and 76.30% overall on WearVox, versus 63.28% and 67.71% for task-only SFT and GRPO.Joint models improve tool use and translation relative to task-only controls; WearVox retention does not measure transfer to natural addressedness interactions.Source: §6.2 · p. 8, Table 5 · p. 8

7 DISCUSSION

The discussion argues that voice agents must use acoustic and conversational context to govern execution, not merely recognize commands. Post-training improves benchmark action gating, but cue-balancing results remain scene-dependent and current evidence has limited generalization scope.

7 DISCUSSION

  • Paired reporting separates false execution from false rejection, because mute scores alone reward never acting and tool accuracy alone rewards executing every recognized command.Near-field same-speaker and text-only controls expose false rejection, while different-speaker conditions expose false execution.Source: §7 · p. 9
  • Voice-agent design needs both proximity and source evidence, since distance alone cannot reject a nearby bystander.Evaluation should report false execution and false rejection separately.Source: §7 · p. 9
  • Side-talk tests recipient attribution, self-talk tests whether discourse cancels command force, and speaker-switch tests a source-and-scene shift with words fixed.Performance on one scenario family does not imply performance on the others, so the paper reports them separately.Source: §7 · p. 9
  • Supervised training learns the benchmark’s conditional action mapping while preserving near-field and text-only tool controls and WearVox tasks.The cue study shows that source and distance can each change the decision, so an aggregate action reversal cannot identify the rule.Source: §7 · p. 9
  • Cue-balanced SFT raises different/near muting to 66.25%/82.50% on original validity WAVs, but same/far muting reaches only 3.75%/5.00% on scene-consistent audio.Proximity-rule accuracy rises from 39.22% to 66.88%, while WearVox tool-call accuracy falls from 90.18% to 83.04%; the matched-WAV gains do not establish a reliable cross-scene proximity gate.Source: §7 · p. 9
  • Current evidence uses one split and one run per setting, and the balanced test varies wearer voice but not room, template, natural bystander speech, or open-set speakers.Side-talk uses an LLM judge without reported human agreement; VGBENCH tool accuracy checks tool names rather than arguments.Source: §7 · p. 9

8 CONCLUSION

VGBENCH tests whether Audio LLMs choose the intended action across controlled addressedness scenarios, rather than merely recognize spoken commands. Post-training improves benchmark action gating, while factorized controls identify a source-change effect and show that far-field sensitivity varies across acoustic scenes.

8 CONCLUSION

  • Across 1,018 controlled items, Audio LLMs often recognize spoken commands without reliably choosing the intended action.The benchmark covers side-talk, self-talk, and speaker-switch scenarios.Source: §8 · p. 9
  • Supervised training learns much of the benchmark action boundary, while exploratory GRPO also adds small gains in one run.Factorized controls identify an independent source-change effect, but far-field sensitivity varies across acoustic scenes.Source: §8 · p. 9
  • Cue-balanced SFT improves near-bystander suppression without establishing a reliable cross-scene proximity gate.The paper concludes that voice-agent evaluation should measure executable action rather than command recognition alone.Source: §8 · p. 9

A EVALUATION CONTRACT AND REPRODUCIBILITY RECORD

The evaluation protocol fixes benchmark records, action formats, and scoring rules to make raw, training-free, and post-trained results interpretable. Its action grammar distinguishes muting, tool calls, and natural-language answers, with controls guarding against an always-mute policy.

A EVALUATION CONTRACT AND REPRODUCIBILITY RECORD

  • The corpus contains 395 side-talk recordings, 223 self-talk recordings, and 400 speaker-switch pairs; raw and training-free systems use retained items, while post-training is reported on a disjoint held-out partition.Table 6 distinguishes the complete retained corpus (“Full”) from the fixed post-training test partition (“held-out”).Source: A EVALUATION CONTRACT AND… · p. 12, Table 6 · p. 12

B ACTION GRAMMAR AND SCORING

  • The action grammar maps outputs to [Mute], one structured tool call, or a natural-language answer; tool-selection accuracy checks the target tool name, not its arguments.Side-talk uses a fixed four-label judge, and switch muting is interpreted alongside same-speaker and text-only tool controls so an always-mute policy cannot score well.Source: B ACTION GRAMMAR AND SCORING · p. 12

C COMPLETE RESULTS FOR RAW AND TRAINING-FREE SYSTEMS

The paper documents full-corpus raw and training-free results, implementation details, downstream WearVox evaluation, and benchmark construction controls. These sections specify the result scopes and protocols without supplying most underlying table values in the passages here.

C COMPLETE RESULTS FOR RAW AND TRAINING-FREE SYSTEMS

  • Table 7 reports full-corpus percentage results for raw and training-free systems across 395 side-talk recordings, 223 self-talk recordings, and 400 speaker-switch pairs.Side-talk combines tool items scored by the canonical parser with response items scored by the fixed judge; AURA, TwS, and SHANKS use distinct training-free inference procedures.Source: C COMPLETE RESULTS FOR RAW AND… · p. 12, Table 7 · p. 13
  • Audio Flamingo 3’s native bracketed action language fails the canonical tool contract, explaining strict tool-selection zeros but not its 1% self-talk muting or zero switched-command muting.Equation 1 defines paired normalized advantage, with action rewards specified in Section 5 and rollout reward distributions recorded by family.Source: C COMPLETE RESULTS FOR RAW AND… · p. 13, D VOXGATE IMPLEMENTATION DETAILS · p. 13

E DOWNSTREAM WEARVOX EVALUATION

  • Table 5 evaluates all WearVox breakdown rows on the same 384 items with identical task prompts, judge, and scoring code; Table 10’s controlled conditions each contain the same 80 held-out commands and follow the proximity rule.Table 9 is identified as covering speaker-switch construction controls.Source: E DOWNSTREAM WEARVOX EVALUATION · p. 13, Table 10 · p. 14, Table 9 · p. 14

F CONSTRUCTION AND QUALITY CONTROLS

  • Self-talk scripts place command cores in discourse frames such as planning, regret, quotation, sarcasm, and rhetorical questions, while speaker-switch cases balance ten tools and synthesize source changes across eight voices.The switch condition inserts 600 ms of silence and renders source B’s trigger with fixed far-field room rendering; recordings are retained after consistency checks.Source: F CONSTRUCTION AND QUALITY CONTROLS · p. 14

G SPEAKER-SWITCH CUE VALIDITY STUDY

The cue-validity study finds that post-trained models respond to source changes under near-field rendering, while distance also strongly gates muting. A cue-balanced SFT control rebalances selected speaker-switch audio slots while leaving other task rows unchanged.

G SPEAKER-SWITCH CUE VALIDITY STUDY

  • The validity set reuses synthesized segments across six conditions while holding text, tool target, TTS, sample rate, loudness, and scoring fixed.Far-field segments are normalized after room rendering, and all 480 files pass format, hash, peak, gap, and loudness checks.Source: G SPEAKER-SWITCH CUE VALIDITY STUDY · p. 14
  • 48.75 points (SFT) and 52.50 points (GRPO) are the mute-rate increases from changing trigger speaker at fixed near-field rendering and 600 ms separation.Table 11 reports mute and matching-tool selection rates for each condition.Source: G SPEAKER-SWITCH CUE VALIDITY STUDY · p. 14, Table 11 · p. 15
  • Far-field rendering raises mute rate by 58.75 points for both models; removing the gap from a different source lowers it, while a same-speaker gap barely changes muting.The same-speaker gap condition preserves 98.75% target-tool selection; the 600 ms boundary amplifies muting when the trigger source changes.Source: G SPEAKER-SWITCH CUE VALIDITY STUDY · pp. 14–15, Table 11 · p. 15

H CUE-BALANCED SFT CONTROL

  • Only 1,212 of 5,748 SFT rows are rebalanced across four near/far-by-gap conditions; other task rows and 286 text-only tool rows remain unchanged.The 320 training commands and 80 test commands are disjoint, with all variants of each command confined to its split; the run is a cue-control experiment, not proximity-policy training.Source: H CUE-BALANCED SFT CONTROL · p. 15
  • The earlier trigger-only far-field intervention and later scene-consistent rendering produce different WAVs, so their condition cells cannot be pooled as matched measurements.Table 13 covers eight scene-consistent conditions, with 80 held-out commands per row.Source: H CUE-BALANCED SFT CONTROL · p. 15, Table 12 · p. 15, Table 13 · p. 16
Loading 2609.32536v1…