PaperlayerView exact revision on arXiv

Source-linked AI summary

Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Change

Bhavik Mangla

arXiv:2609.35922v1cs.CLcs.AIcs.LGcs.SD

TL;DR

Voice agents can miss rules that depend on how a call sounds, despite handling its words. VoxParity tests executed actions across paired audio variants against a words-only null; only 11 of 23 systems with transcript paths pass, and errors on protective calls tend toward the routine request.

  • Problem

    Whether voice agents act on audible cues in their executed typed actions is rarely measured against a baseline that cannot hear the call.

    Source: §1 · p. 1

  • Method

    VoxParity holds transcripts fixed across audio variants requiring different tool calls and compares each system’s action shift with a words-only cascade’s shift.

    Source: §1 · p. 1, §3.1 · p. 5

  • Results

    Only 11 of 23 systems with transcript paths pass the words-only null test, while all 28 carry out routine requests on protective calls more often than they over-trigger on clean calls.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In exploratory analyses, stating the rule and describing the caller’s delivery each recover part of the leading systems’ shortfall, but neither closes the gap on emotion.

    Source: §1 · p. 3

    The 28-system roster omits Amazon’s Nova 2 Sonic and does not cover every model size or generation, so roster-level fractions may shift with a different mix.

    Source: §11 · p. 13

Abstract

from arXiv · show

A voice agent can handle almost every call on the words alone and still fail the few its sector's rules were written for. Emergency-call standards, fraud guidance, radio phraseology and vulnerability rules recognise that how a caller sounds, or what else is audible, can change the right action. VoxParity tests whether agents act on it. In 183 scenarios from 14 sectors, one transcript stays fixed while the audio changes (a coaching voice, a medical monitor beeping, a mayday under a radio check, noise over a drug name, a child's voice placing a bet, a frightened whisper), and with it the correct typed tool call. A words-only null test credits a system only if hearing the call moves its actions more than it moves a pipeline that only reads the words. Only 11 of the 23 systems that can also be run on the transcript pass. Descriptively, errors run toward the words: when the audio calls for protection, all 28 systems carry out the routine request more often than they over-react on clean calls (41% against 12% pooled; the words-only pipeline, 58% against 15%). Exploratory analyses place most of the leading systems' misses on cues they heard; systems beat the null almost entirely on items that state the rule; the leading systems overrule heard resignation or confusion far more often than acute alarm; and, in the models tested, describing the voice and stating the rule each recover part of the shortfall, leaving a gap on emotion.

1 Introduction

VoxParity asks whether voice agents change their executed actions when audio changes the right response, testing them against a pipeline that reads only the words. Across the reported findings, agents often recognize audio cues without acting on them, and the benchmark’s exploratory analyses locate parts of that gap in applying the rule.

Contributions

  • VoxParity measures executed typed calls against a words-only null across 183 scenarios from 14 sectors, with identical transcripts and audio variants requiring different tool calls.A deterministic matcher scores calls without a judge.Source: §1 · p. 1

What we find

  • Leading systems recognize most cues about as often as volunteer players do, yet often do not act on cues they recognize.Guess-corrected recognition is 0.77 for leading systems against 0.79 for players.Source: §1 · p. 2
  • 41% against 12% pooled: all 28 systems carry out the routine request on protective calls more often than they over-trigger on clean calls.The words-only cascade carries out the routine request on 58% of protective calls.Source: §1 · p. 2
  • On emotional cells, the four leading systems beat the words-only null by +0.33 when the rule is stated, but show no detectable gain when it is unstated.They take the words’ action on 52% of heard quiet states versus 6% of heard acute alarm.Source: §1 · p. 2
  • Only 11 of 23 systems with a transcript path pass the words-only null test; accuracy and cue recognition disagree with its verdicts.Passes come almost entirely on items that state the rule.Source: §1 · pp. 2–3
  • In exploratory tests, describing the caller’s delivery and stating the rule each recover part of the shortfall, but neither closes the gap on emotion.For emotional calls with unstated rules, gemini-3.7-flash reaches 0.83 with both, against 0.40 with neither.Source: §1 · p. 3

2 VoxParity: 183 scenarios in which only the sound changes the right action

VoxParity constructs paired calls with fixed words but different audio and required tool calls, drawing cues and actions from sector rules and practice. It checks that people hear the intended cues and scores executed calls deterministically.

2.1 One transcript, two sounds, two correct tool calls

  • Each item fixes a transcript across audio variants whose gold tool calls differ, making it a contrast-set test of whether sound changes the correct action.The gold flips in 130 of 132 items with two runnable variants.Source: §2.1 · p. 3

2.2 Seven cues, fourteen sectors, golds drawn from each sector’s rules and practice

  • The benchmark spans seven audible cue types across 14 sectors, with 182 counterfactual items grouped into five harm classes and one invariant control.Of the counterfactual items, 163 rest on a written mandate, permission, or documented sector practice.Source: §2.2 · p. 4

2.3 People hear the cues

  • Listeners chose the intended cue on 0.78 of 143 judgments, compared with a chance rate of 0.26.Recognition was 0.50 on sarcastic readings.Source: §2.3 · p. 4

2.4 A judge-free score on the executed call

  • The deterministic score gives full credit for a matching tool and typed arguments, partial credit for specified acceptable alternatives, and zero otherwise.Selection credit scores tool choice alone when comparing players.Source: §2.4 · pp. 4–5

3 A test for acting on audio, 28 systems and a human reference

The study compares each system’s audio actions with its transcript actions and a cascade that cannot hear the call. It evaluates a 28-system roster, uses volunteer players as a reference, and applies item-clustered statistical comparisons.

3.1 The words-only null test

  • The words-only null test subtracts the cascade’s audio-minus-transcript change from the system’s change on identical cue-bearing cells; passing requires a positive difference after Holm correction.It assumes that without using a cue, switching from transcript to audio moves the system as it moves the cascade.Source: §3.1 · p. 5

3.2 Twenty-eight systems, nine of them production realtime agents

  • The roster includes 28 audio-native systems from 11 vendors in three serving modes; 23 have transcript paths and five are tested separately against the cascade on accuracy.Nine are production realtime agents.Source: §3.2 · p. 5

3.3 Volunteer players as a reference

  • Volunteer players chose actions from the same clips and menus without seeing the stated policy, so they serve as a reference rather than a baseline to beat.The study reports 638 answers from about 20 unpaid volunteers, with one player supplying 23% of answers.Source: §3.3 · p. 5

3.4 Statistics

  • System comparisons use item-clustered 95% bootstrap intervals, pair identical cells, and apply Holm correction within test families.Comparisons involving players resample items and players together.Source: §3.4 · pp. 5–6

4 When the audio calls for protection, every system errs toward the words

Across protective calls, systems more often follow the routine wording than take the protective action, including in life-safety cases. Audio reduces unsafe execution on average but does not reverse this asymmetry.

4 When the audio calls for protection, every system errs toward the words

  • The error audit distinguishes unsafe execution, missed duty, over-triggering, and deferral, but reports rates without severity weights.An audit of 60 sampled classifications agreed with 57 classifications read against item rationales.Source: §4 · p. 6

4.1 The asymmetry holds for every system, vendor and serving mode

  • 41% of protective calls led to unsafe execution, versus 12% over-triggering on clean calls, pooled across systems; the four leading systems were at 34% versus 18%.The asymmetry held for every system, vendor, and serving mode; the median ratio was 4 to 1.Source: §4.1 · p. 6

4.2 The errors reach life-safety calls, and the stakes do not detectably raise systems’ action on a heard cue

  • On 17 of 53 life-safety protective cells, most systems followed the words, and on five cells none chose the prescribed action.Across 27 systems with perception probes, a heard protective cue was acted on in 40% of life-safety calls and 40% of other calls; the difference was not detectable.Source: §4.2 · p. 6
  • Healthcare and telecom had the lowest sector-level median score, 0.24, while emotional delivery accounted for 24 of 29 cells where no system acted correctly.These figures cover sectors with at least five scenarios.Source: §4.2 · p. 7

4.3 Hearing lowers unsafe execution on average but never reverses its direction

  • Against each system’s own transcript, audio lowered unsafe execution by 0.12 and left over-triggering unchanged at +0.00, pooled across 23 systems with transcript paths.Eleven systems showed a significant reduction after Holm correction; hearing changed the size, not the direction, of the asymmetry.Source: §4.3 · p. 7
  • Volunteer players had 18% unsafe execution and 21% over-triggering, while the four leading systems executed unsafely more often on the same protective calls.The player and system contexts differed in both directions.Source: §4.3 · p. 7

5 Audio that changes the facts moves agents; the caller’s state moves them mainly under a stated rule

The paper distinguishes cues that change request-relevant facts from cues that change what the caller’s state asks of the agent. Systems act on emotional delivery mainly when an item states the rule, and describing the voice helps but does not close the gap.

5 Audio that changes the facts moves agents; the caller’s state moves them mainly under a stated rule

  • The analysis defines facts as cues changing request-relevant facts and feelings as cues changing what the caller’s state asks of the agent.Feelings cells left the rule implicit more often than facts cells: 59% versus 20%.Source: §5 · p. 7

5.1 The caller’s state is acted on mainly where the item states its rule

  • Where the rule is stated, the four leading systems’ gain over the null on emotional cells was +0.33; where unstated, it was +0.00.Across 23 systems with transcript paths, cue cells without a stated rule sat on the null (+0.01), while those with a rule cleared it (+0.18).Source: §5.1 · p. 7
  • Among 27 probed systems, the emotional-delivery credit gap versus other cues shrank from −0.10 pooled to −0.04 after stratifying by whether the rule was stated.For gemini-3.7-flash, the gap remained −0.20 after stratification.Source: §5.1 · p. 7

5.2 Even the leading systems overrule heard quiet states far more often than heard acute alarm

  • The four leading systems took the words’ action on 52% of heard quiet states, versus 6% of heard acute alarm and 20% of heard facts.Quiet states exceeded facts by 32 points; the grouping was exploratory.Source: §5.2 · p. 7

5.3 In its own words, the highest-credit system understates the state it hears

  • In describe-then-act runs, gemini-3.7-flash usually understated emotional states it had heard, and its own description added only +0.03 to correct action.In 22 of 30 understated, denied, or mislabelled notes, the same model’s forced-choice probe identified the cue correctly; the authors treat the note as an intermediate description, not a perception test.Source: §5.3 · p. 8

5.4 Described in the prompt, the caller’s state is acted on more; facts are already at ceiling

  • Adding a description note raised gemini-3.7-flash’s emotional-call credit on its own audio to 0.66, while emotion remained the lowest-performing cue in the figure’s conditions.With both a description and stated rule, implicit-rule emotional calls rose from 0.40 to 0.83, still 0.17 below other cues; the notes are specification-derived upper bounds.Source: §5.4 · pp. 8–9, Figure 2 · p. 9

6 Acting on audio beyond the words is far from universal, comes almost entirely from items that state the rule, and accuracy and recognition disagree on which

The words-only null compares audio-driven changes in actions with changes in a pipeline that reads transcripts. Eleven of 23 systems with transcript paths pass, but the median system does not, and common accuracy or recognition measures can give different verdicts.

6.1 The words-only null holds

  • The words-only cascade’s audio-minus-twin change was +0.01, while alternative text models and a filler-preserving transcriber stayed near the floor.A transcriber adding emotion and sound-event tags rose only slightly, to +0.06.Source: §6.1 · p. 9

6.2 Eleven of 23 systems pass; one of seven realtime agents with a transcript path does

  • 11 of 23 systems with transcript paths passed the words-only null; the median of all 28 systems matched the cascade’s 0.34, and only one of seven production realtime agents with a transcript path passed.Passes came almost entirely from items stating the rule; no system without a transcript path beat the cascade.Source: Figure 3 · p. 10, §6.2 · pp. 10–11
  • Under a stricter both-deliveries criterion, eight systems handled more than the cascade on 130 scenarios, and all eight also passed the null test.The best system, gemini-3.7-flash, scored 0.35 against the cascade’s 0.05.Source: §6.2 · p. 10

6.3 Accuracy and cue recognition reach different verdicts

  • Accuracy and cue recognition disagreed with null-test verdicts for some systems because recognizing a cue and acting on it can come apart.Accuracy disagreed on four of 23 systems; recognition disagreed on eight at a 0.5 threshold and at least four at any scored threshold.Source: Table 5 · p. 9, §6.3 · p. 11

7 At the frontier, the headroom lies in the bridge from hearing to deciding

At the leading systems’ frontier, the paper estimates more headroom in deciding than in hearing. Description notes and stated rules recover part of that shortfall, but the comparisons do not establish parity with human players.

7.1 At the frontier, better hearing alone would add little: perfect hearing would add 0.04, perfect deciding 0.28

  • Perfect hearing would raise leading systems’ cue-bearing credit by 0.04, while perfect deciding would raise it by 0.28.The estimated intervals were [+0.01, +0.08] for hearing and [+0.23, +0.34] for deciding; description and stated rules each recover part of the headroom, leaving an emotional-cue residual.Source: §7.1 · p. 11

7.2 An upper bound: with the delivery described, even a text-only model outscores every unaided audio system

  • With a description note, gemini-3.7-flash reached 0.79 on its own audio, while the cascade reached 0.76 on the exact transcript with the same note.The note is an upper bound derived from the rendering specification, not a deployable fix.Source: §7.2 · p. 11

7.3 The players are a reference, not a parity line

  • On 171 cue-bearing cells answered by players, gemini-3.7-flash’s estimate was −0.02 relative to them, but equivalence within ±0.10 was not established.Players saw no stated rule and were told the voice decides the move, so the authors draw no parity claim.Source: §7.3 · p. 11

8 Related work

Prior work identifies a gap between hearing vocal cues and acting on them, while existing action evaluations often do not test whether delivery changes tool calls. VoxParity compares executed actions against a words-only null and adds a human action-selection reference.

Related work and measurement

  • VoxParity scales delivery-sensitive action testing to 183 scenarios and 28 systems, using typed calls, a words-only null, clustered intervals, and volunteer choices on the same audio.The earlier direct study tested four realtime agents in three scenarios, without statistical analysis or a human decision reference.Source: §8 · p. 11
  • The earlier four-agent study found word-led actions in three scenarios, while VoxParity adds a controlled comparison of voice descriptions and stated rules on identical cells.The earlier study is “Hears but Does Not Listen” [5].Source: §8 · pp. 11–12

Perception without action

  • Earlier studies report that models can recognise distress or other cues yet fail to act on them, including audio models whose optimal-solution rate barely changed when audio was added to transcripts [43].Other probing work describes a gap between encoded and used information [34].Source: §8 · pp. 11–12

Scoring what voice agents do

  • Most action benchmarks hold task logic invariant to delivery or use transcripts alone; ProVoice-Bench conditions intervention on an acoustic event, whereas VoxParity changes the correct typed action for a fixed transcript.The passage reports that none of the neighboring benchmarks has more than one of VoxParity’s four properties in full.Source: §8 · p. 12

Human reference points

  • Prior delivery-sensitive action benchmarks lack a human action-selection baseline, while the nearest cited case measures background-fact retrieval rather than an action whose gold depends on delivery.MSI-Bench reports whether a background fact was captured under SNR stress.Source: §8 · p. 12

9 Implications

The implications propose using the words-only null to audit deployed agents, exploring counterfactual-pair training, and treating a grounded description plus an explicit policy as a hypothesis for bridging hearing and action.

Audits and procurement

  • Deployers can screen with 81 development-split cells and use the full words-only null test to check whether paired audio-versus-transcript action changes exceed the pipeline’s.The paper recommends asking for this test alongside accuracy or recognition measures because it measures action changes beyond the words.Source: §9 · p. 12

Training

  • Delivery-counterfactual pairs could provide a contrastive training signal for tool-calling policies, but whether that training works is untested.The paper says such pairs should be built with the harness rather than taken from the held-out bank.Source: §9 · p. 12

Architecture (a hypothesis)

  • The authors hypothesize that a calibrated, decision-usable description of caller state under a policy mapping that state to action may address a missing bridge rather than missing hearing.The proposed architecture is not a deployable estimate, and the reported association between naming the right cue and acting correctly does not establish that the label is used.Source: §9 · p. 12

10 Robustness of the comparisons

Robustness checks preserve the principal null-test verdicts and error direction, but comparisons of context and probe format show some sensitivity in individual measurements.

Robustness of the comparisons

  • Context shifts model credit in both directions: qwen3.5-omni-plus loses 0.24 on stated-rule items, while gemini-3.7-flash gains +0.15 on items without a stated rule.The passage describes player comparisons as tilted both ways.Source: §10 · p. 12
  • Leniently reading malformed MiMo-V2.6-Flash probe answers raises cue recognition from 0.40 to about 0.77, while reshuffling probe options or action menus changes no result.Counting realtime no-option answers as missing shrinks the apparent hearing drop from −0.18 to −0.10.Source: §10 · p. 13
  • No robustness check reverses the null-test verdicts, the words-directed error pattern, or the finding that systems sometimes hear a cue yet take the words’ action.The error direction holds with and without a stated rule, and emotion and stated-rule patterns retain their sign and size on 158 admitted cue-bearing cells.Source: §10 · p. 13

11 Limitations

The words-only null test establishes audio-following only on this benchmark, not deployment safety. The authors also qualify conclusions by exploratory analyses, constrained stimuli and participants, and limited model and roster coverage.

Limitations

  • The null test is necessary but not sufficient for safe deployment, and its verdict is limited to the calls in this bank.It tests whether action follows audio beyond what the words explain.Source: §11 · p. 13
  • The emotion gap relies almost entirely on Gemini-TTS affect, with 298 of 309 scored cells using one Gemini voice; variation across synthetic voices remains untested.Stereotyped synthetic affect may be easier to recognise and discount than natural affect, but the study does not test that possibility.Source: §11 · p. 13
  • Human recordings come from one author who knew the hypothesis, so they cannot establish effect size across speakers, genders, or accents.Independent listeners confirmed intended cues, and the effect did not differ detectably from the synthetic one on identical cells.Source: §11 · p. 13
  • Self-selected volunteers without headphone screening or catch trials are not a population estimate, and their post-action perception question may inflate recognition.One player supplied almost a quarter of the answers, and players had different context from the systems.Source: §11 · p. 13
  • Only the null-test verdicts are confirmatory; the error, emotion, hearing-versus-deciding, and stated-rule analyses are post hoc or observational.The emotion split was named after looking at the emotion types.Source: §11 · p. 13
  • The feelings-versus-facts pattern may partly reflect rule grounding: within mandate-grounded cells, the difference in P(right | heard) was not detectable.Feelings more often rested on practice or permission than written mandate.Source: §11 · p. 13
  • Context tests covered two models, and their effects differed; rules were generated from grounding notes that describe the intended response and were not blind to the gold.The stated-rule effect agreed across the two models, while the players’ context effect did not.Source: §11 · p. 13
  • The roster omits Amazon’s Nova 2 Sonic and not every size or generation; its 11-of-23 pass fraction could shift with roster mix, while protocol grounding is the authors’ reading rather than a legal finding.Nineteen legacy items without written grounding are set aside where relevant, and harm weights are judgment-based; headline rates are weight-free.Source: §11 · pp. 13–14

12 Ethics and availability

The paper describes its human-reference ethics and the materials and procedures made available for evaluating VoxParity.

12 Ethics and availability

  • Volunteer players consented in-app, were unpaid, and provided no names, contact details, IP addresses, or audio.The authors contributed no reference answers; every child voice was synthetic.Source: §12 · p. 14
  • VoxParity tests protection of vulnerable callers; its over-trigger controls reward protective reactions, not reflexive escalation.It trains and releases no emotion classifier.Source: §12 · p. 14
  • The released harness, scorer, and regeneration scripts include a development split with audio, while 143 items remain held out.The paper says submitted agents can be evaluated on the held-out bank and reviewers can request access.Source: §12 · p. 14

13 Conclusion

VoxParity tests whether agents change tool actions appropriately when audio changes but words stay fixed. Across the benchmark, systems often act toward the words rather than the audio, and the authors identify limits in the emotion results and human reference.

13 Conclusion

  • Across 183 scenarios from 14 sectors, all 28 tested systems err toward the words when audio calls for protection, while only 11 of 23 systems with a transcript path pass the words-only null test.Source: §13 · p. 14
  • Systems beat the null almost entirely on items that state the rule, and accuracy and cue recognition disagree with the null test about which agents act on audio.For leading systems, perfect hearing would add little compared with perfect deciding; exploratory tests find that voice descriptions and stated rules each recover part of the shortfall, while emotion remains a gap.Source: §13 · p. 14
  • The emotion findings rest almost entirely on one synthetic voice, and the human reference draws on about 20 unpaid volunteers.The authors identify recordings from more speakers and a larger reference panel as natural next steps.Source: §13 · p. 14
  • The authors recommend evaluating tool-calling policies on same-word pairs with different required actions and auditing against a words-only null.They also suggest training on such pairs.Source: §13 · p. 14

A.1 Item schema and scoring rules

Items hold the transcript fixed while varying delivery or acoustic scene, then score typed tool calls against audio-specific requirements. The section also specifies first-turn scoring and reports how alternative scoring choices affect results.

A.1 Item schema and scoring rules

  • Each item pairs one fixed transcript with at least two audio variants, each carrying a gold typed tool call and a perception probe.Items define the agent role and tool surface; 104 of 183 also state a policy, and menus are shuffled to block a structural menu-order exploit.Source: §A.1 · p. 21
  • Acceptable credit rewards only actions warranted by that variant’s audio; when policy is stated, non-exercise scores zero, with two interpreter-item exceptions.Three implicit-policy variants also score non-exercise at zero; it occurred on 27 of 8,652 system cells.Source: §A.1 · p. 21
  • Scoring the first turn gives 11 of 23 clears; adding a scripted follow-up turn changes no system’s difference-in-differences by more than 0.02 and gives 12 clears.Stacking all conservative choices—first turn, no legacy items, and all three floors—leaves 11 of 23.Source: §A.1 · p. 21
  • First-call scoring counts eight cells as correct even though the system made both the protective call and the routine one, an unsafe execution pattern.The authors describe these as a hold and a wire.Source: §A.1 · p. 21

A.2 The text twin, the null distribution and the ladder

These appendices examine how transcript-based null tests, robustness checks, and scoring characterize systems’ ability to act on decision-changing audio cues, including where protective actions are missed and how explicit descriptions or rules affect the shortfall. They also assess the development split and 2PL model as measures of system ability and ranking.

A.2 The text twin, the null distribution and the ladder

  • The three transcript-reading floors move by +0.01 [−0.02, +0.05], +0.02 [−0.01, +0.05], and +0.03 [−0.00, +0.07]; 10 of 11 clears survive alternative floors and 8 pass the stricter shift test.The verbatim cascade remains at −0.02 [−0.06, +0.02], while adding local emotion and acoustic-event tags moves it to +0.06 [+0.00, +0.11]; ladder rungs are references, not pooled nulls or audio-native systems.Source: §A.2 · p. 21

A.3 Stimuli, gates, recordings and the game

  • Across 46 author-recorded variants, listeners identified intended cues in 0.78 [0.72, 0.83] of judgments (chance 0.26), while gemini-3.7-flash beat the words-only cascade by +0.30 [+0.16, +0.47] on 34 cue-bearing cells.On 158 cue-bearing cells admitted by either non-Google judge or a person, the pooled P(right | heard) gap between emotional and other cues was −0.13 [−0.23, −0.04] (−0.10 on all cells).Source: §A.3 · p. 22

A.4 Systems and provenance

  • The roster excludes several systems for access, cost, audio-delivery, platform, tool-schema, or benchmarking-term constraints, and some vendor voice models remain unverified.The authors also note that sibling sizes and generations within covered vendors were not all run.Source: §A.4 · p. 22

A.5 Full leaderboard and robustness

  • 7 of the 11 floor-clearing systems still clear after excess over-reaction is penalized, and every clear remains above zero when any item family or domain is dropped.On scenarios whose correct action differs between deliveries, all text twins score 0.00 and eight systems exceed the cascade after correction; this stricter check corroborates rather than replaces the main comparison. Alternative recognition and credit measures disagree with the null test, and realtime serving has pooled probe accuracy −0.10 [−0.15, −0.05] versus file mode across tested pairs.Source: §A.5 · pp. 23–25
  • 11 of 23 transcript-path systems clear the words-only floor; Qwen3.8-Omni and gemini-3.7-flash lead with gains of +0.25 [+0.17, +0.32] and +0.23 [+0.16, +0.31].No transcript-path system sits significantly below the floor after correction. Table A2 reports the full leaderboard on 206 cue-bearing cells.Source: §A.5 · pp. 22–23, Table A2 · p. 24
  • Across seven available model-generation steps, the gain over the null rises significantly in two, falls in one, and is unchanged detectably in four.Across both Gemini Live steps together it falls by −0.08 [−0.15, −0.01]; the authors caution that these few, unadjusted comparisons describe this roster, not a trend.Source: §A.5 · p. 26

A.6 Scenario atlas and case studies

  • 37% of responses on protective cells take the words’ default action, versus 30% correct, and no system acts correctly on 29 cue-bearing cells.Examples include a silent 911 caller with a threat behind: 3 of 28 dispatch, while 21 take the clean variant’s TTY path; in a breathless inhaler-refill call, none escalate clinically and 25 process the refill.Source: §A.6 · p. 27

A.7 Harm rubric, weights and rule-level departures

  • The rubric defines protective, proceeding, deferring, and trap actions, while sensitivity checks show that charging deferral and inaction—not tier weights—can change the risk ranking.A seeded review found the assigned class right in 57 of 60 outcomes, with three disagreements reflecting rule ambiguities. A linear 5-4-3-2-1 tier scheme ranked systems identically (ρ = 1.00), while weight-free rates and player contrasts survived all nine tested weightings.Source: §A.7 · pp. 28–29
  • All 28 systems erred toward the words: unsafe execution was 41% [36%, 46%] on protective calls versus 12% [9%, 15%] over-triggering on clean calls; the words-only cascade was 58% [50%, 66%] versus 15% [10%, 22%].Adding audio lowered unsafe execution for 11 of 23 systems after Holm correction. Mandate and guidance cells had similar departure rates (58% and 54%), but equivalence within ±0.10 was not established.Source: §A.7 · pp. 29–30, Figure A4 · p. 30

A.8.1 Four definitions of “heard”; A.8.2 Decompositions

  • After 1/k correction, pooled systems, the four leading systems, and players reached P(right | heard) of 0.53, 0.64, and 0.74; neither hearing nor deciding dominates for the field.For the frontier, perfect deciding would add +0.28 [+0.23, +0.34], compared with +0.04 [+0.01, +0.08] for perfect hearing.Source: §A.8.1 · p. 30, §A.8.2 · p. 31

A.8.3 Converging analyses (exploratory); A.8.6 Failure taxonomy and over- and under-reaction

  • Hearing the cue raised the odds of a correct action 3.2 [2.6, 4.0] times, yet only 44% of heard cues ended in the strict right action; salience improved perception, not action.Across 13 model pairs, hearing contributed at most 0.05 to credit changes while the using component ranged from −0.20 to +0.24. Naming a different cue was associated with 0.11 [0.06, 0.16] fewer correct actions; systems also left 15% to 50% of calls with a heard cue unacted on.Source: §A.8.3 · p. 32, Figure A6 · p. 33

A.8.4 The human-relative contrast and its robustness; A.8.5 Same-reply notes

  • The player–frontier contrast on heard emotion was suggestive, not robust across measures; in same-reply notes, Gemini often understated or mislabelled emotional states even when its probe identified the cue.On 30 emotional calls where it took the words’ action, the note understated, denied, or mislabelled the state; on 22 of these, its forced-choice probe had identified the cue. The authors caution that notes may rationalise choices already made.Source: §A.8.4 · p. 32, §A.8.5 · p. 33

A.9 Cue notes and other levers

  • For Gemini-3.7-flash, a description note improved emotional-call credit by +0.07 to +0.18 over a label note; adding a stated rule to the description raised it to 0.83 [0.75, 0.90].The description-plus-rule result remained below the 1.00 credit on other cues, and a stated rule alone did not close the emotion gap.Source: §A.9 · pp. 33–35, Figure A7 · p. 35
  • The players’ context changed credit differently by model, while attend-to-voice instructions and describe-first prompting yielded small or nonsignificant gains; Gemini’s three-rollout pass3 was 0.49 [0.43, 0.55].The context cost qwen3.5-omni-plus 0.24 [0.15, 0.34] on stated-rule cue cells but changed its implicit-rule items by +0.01 [−0.06, +0.08].Source: §A.9 · pp. 35–36

A.10 Further related work: nulls, levers and axes beyond emotion

  • Unlike prior work on perception, safety, or delivery-conditioned responses, VoxParity tests typed actions that flip on a fixed transcript against a words-only null.The related-work review distinguishes this setup from fixed-transcript benchmarks that do not change the correct action with delivery.Source: §A.10 · pp. 36–37

A.11 Psychometrics and the development split

  • The 2PL model had marginal reliability 0.985 over the 28 systems, and random 100-item subsets reproduced system rankings at mean Kendall τ=0.94.The model’s information peaked at θ=−0.2 for all cells, +0.3 for cue-bearing cells, and −1.6 for neutral cells.Source: §A.11 · p. 37, Figure A8 · p. 37
  • Cue-bearing cells carry the bank’s information where systems and players are.Figure A8 compares test information for all, cue-bearing, and neutral cells over ability; the players are marked by a dotted line.Source: Figure A8 · p. 37, §A.11 · p. 37
  • Ability estimates from the split correlated with full-bank estimates at ρ=0.991, and headline difference-in-differences rankings matched at Kendall τ=0.86 (Spearman 0.96).Random whole-item subsets had mean correlation 0.969 and 5th-percentile correlation 0.942; selecting on half the systems and testing on the other half gave ρ=0.989 (minimum 0.969 over 20 splits). The split comprised 40 items and 81 cells, 47 cue-bearing, with every cue axis represented.Source: §A.11 · p. 38

Related papers

Loading 2609.35922v1…