When Does a Spoken Agent Have Enough Evidence to Act? The PACT-SLM Contract Test

1 INTRODUCTION

• Early action matters when an exposed action changes external state or interrupts the user, even if the final action is correct.Final-response scores can miss whether the speech available at the time supported acting.Source: §1 · p. 1
• A valid timing test must track the visible prefix and first valid action time, distinguishing action-bearing evidence from duration or endpoint shortcuts.Without that contract, high final accuracy can reward systems that guess late, act early, or recognize only an endpoint.Source: §1 · p. 1
• PACT-SLM pairs utterances that match through the last WAIT prefix, then diverge before their distinct actions become valid, with held-out semantic families and 15 dB noise.Matched controls test duration, scalar-acoustic, text, and representation shortcuts.Source: §1 · p. 1
• WavLM Base Plus exceeds matched controls in post-onset accuracy but falls below the permutation reference, exposes pre-onset actions, and predicts few complete trajectories exactly.Elapsed time is more onset-exact but less accurate about action identity; branch IDs are not semantic targets.Source: §1 · p. 2

2 RELATED WORK AND EVALUATION GAP

• Speech encoders and modality adapters provide input interfaces for SLMs, while simultaneous and full-duplex models address listening, reasoning, speaking, and interruption during ongoing turns.Broader duplex benchmarks evaluate conversation quality and turn management.Source: §2 · p. 2
• PACT-SLM complements these systems by testing whether the current user-speech prefix supports an exposed action.The paper uses frozen WavLM representations as a reproducible probe, not as a new SLM architecture.Source: §2 · p. 2
• Prosody and contextual speech benchmarks test paralinguistic or semantic reasoning, but do not by themselves establish when an action becomes valid.An action-timing evaluation also needs a known onset, strictly nested observations, and controls that separate action evidence from time.Source: §2 · p. 2

3.1 WHAT THE TEST MUST IDENTIFY

• The contract scores whether a model identifies the target action after its first-valid index and waits before that point, which final-action accuracy alone cannot establish.Each trajectory has ordered speech prefixes, a target action, and a first-valid action index.Source: §3.1 · p. 2

3.2 PAIRED BRANCH CONSTRUCTION

• Paired branches are byte-identical before onset, then acquire branch-specific samples at the first action-labeled prefix; the assigned onset is not a listener judgment.The five nested crops use one branch waveform; noisy pairs receive the same pair-level 15 dB noise realization before cropping.Source: §3.2 · pp. 2–3, Figure 1 · p. 3

3.3 DATA AND SPLIT

• The diagnostic uses 340 eligible action-contrast groups, with 80 test groups from four held-out semantic families and 1,600 clean and noisy prefix predictions.The test families are excluded from training and development; clean test speech provides a matched reference for held-out noise.Source: §3.3 · p. 3, Table 2 · p. 4

4 MODELS AND MEASUREMENTS

• The audio-only probe mean-pools standardized final hidden states from frozen WavLM Base Plus representations and fits balanced multinomial logistic regression.The audio-plus-text probe adds TF–IDF word and bigram features; no model or threshold is selected on the test set.Source: §4 · p. 4
• Matched controls test endpoint, elapsed-time, authored-text, scalar-acoustic, and shuffled-representation shortcuts using the same classifier and training rows.Prefix-shuffled embeddings preserve position-specific action frequencies while breaking speech–action correspondence.Source: §4 · p. 4
• The measurements separate complete-trajectory exactness, pre-onset exposed actions, pooled post-onset semantic-label accuracy, and exact onset timing.Onset exactness checks whether the first non-WAIT prediction occurs at the assigned first-valid prefix.Source: §4 · pp. 4–5
• Group-bootstrap intervals condition on fitted probes and held-out predictions, excluding variability from feature extraction and probe fitting.The generated speech and embeddings are not included in the accompanying materials.Source: §4 · p. 5

5.1 ACTION IDENTITY AND ONSET ARE DIFFERENT CAPABILITIES

• 26.03% pooled post-onset semantic-label accuracy coexists with 18.99% pre-onset exposed actions and 5.94% exact complete trajectories for WavLM Base Plus.The post-onset score includes exposed-action and internal-state labels.Source: §5.1 · p. 5
• WavLM exceeds shuffled, scalar-acoustic, and text-only controls, yet its 26.03% score sits at permutation percentile 96, below the 97.5th-percentile reference of 26.73%.The permutation check does not support a robust speech-specific effect at the stated reference.Source: §5.1 · p. 5
• 36.25% onset exactness for elapsed-only versus 23.13% for WavLM contrasts with 9.92% versus 26.03% post-onset accuracy.Duration predicts the assigned onset more often, while WavLM identifies the target action more often.Source: §5.1 · p. 5

5.2 HELD-OUT NOISE BREAKS COMPLETE TRAJECTORIES

• 18.99% to 15.19% is the change in pre-onset exposed-action rate with authored text, which does not improve action identification or overall performance.The pooled post-onset label difference is -0.93% [-2.06%, 0.10%].Source: §5.2 · p. 5
• 38.43% clean to 13.64% noisy pooled post-onset label accuracy accompanies a fall in complete-trajectory exactness from 11.25% to 0.62%.Table 3 reports results by held-out rendering with contrast-group-bootstrap intervals.Source: §5.2 · p. 5, Table 3 · p. 6

5.3 WHY THE CONTRACT CHANGES THE CONCLUSION

• Single-chunk and fixed-endpoint diagnostics do not test intermediate-prefix support; the variable-onset results show imperfect behavior, not a reliable timing policy.A single-chunk design has no WAIT target, while a fixed-endpoint design marks every action valid only at the final crop.Source: §5.3 · p. 6

• PACT-SLM is a controlled diagnostic, not a natural-speech benchmark or deployed policy evaluation; no listener study establishes naturalness, intelligibility, or human agreement on onset.The four test families were used during development, so held-out results are exploratory rather than independent confirmation on untouched families.Source: §6 · p. 6
• Six families lacked complete audio pairs and 20 audio-backed pairs lacked action contrast; all four test families also appeared in development.Consequently, held-out results are exploratory rather than independent confirmation on untouched families.Source: §6 · p. 6
• Results use frozen WavLM Base Plus with a shallow classifier and token-fraction authored text, while native streaming SLMs, time-aligned transcripts, natural speech, and human judgments require separate evidence.The authored text control is not a timestamped automatic transcript.Source: §6 · pp. 6–7

7 CONCLUSION

• 26.03% pooled post-onset semantic-label accuracy contrasts with 5.94% exact trajectories, while 18.99% of eligible pre-onset prefixes expose an action.The evidence supports separate action-identity and timing measures in this controlled test, not a robust speech-specific effect, reliable policy, or natural-conversation performance.Source: §7 · p. 7

A SAMPLE UNITS AND SPLIT

• The test contains 340 groups, with 208 training, 52 development, and 80 test groups; the 4,200 prefix records are repeated observations, not independent examples.Four test families are held out from both training and development, and groups—not records—are resampled for reported intervals.Source: A SAMPLE UNITS AND SPLIT · p. 8

B LIMITS OF ENDPOINT-ONLY DIAGNOSTICS

• Endpoint-only designs cannot establish whether an intermediate speech prefix supports an action, and the single-chunk design cannot measure timing because it has no WAIT target.In the fixed-endpoint design, endpoint plurality reaches 37.50% final-action accuracy with no pre-final non-WAIT prediction, while WavLM reaches 39.06% but predicts non-WAIT on 25.78% of pre-final records.Source: B LIMITS OF ENDPOINT-ONLY DIAGNOSTICS · p. 8

C SEMANTIC ACTION TARGETS

• Each source trajectory takes its semantic action target from its utterance, because counterbalanced branch codes do not name fixed actions.Equation 1 assigns WAIT before valid onset and the source utterance’s semantic action at and after onset.Source: C SEMANTIC ACTION TARGETS · p. 8

D FULL HELD-OUT RESULTS AND UNCERTAINTY

• The 95% group-bootstrap intervals quantify variation across observed test groups conditional on the fixed fitted probes and predictions.They exclude uncertainty from fitting, new generated speech, a new synthesis system, and natural conversations.Source: D FULL HELD-OUT RESULTS AND UNCERTAINTY · p. 9
• 26.03% pooled post-onset accuracy [22.14%, 29.68%] contrasts with 5.94% trajectory exactness [3.44%, 9.06%] for WavLM Base Plus.It exposes an action on 18.99% of eligible pre-onset prefixes [14.19%, 24.85%].Source: D FULL HELD-OUT RESULTS AND UNCERTAINTY · p. 9, Table 4 · p. 9
• Elapsed time is less accurate about action identity at 9.92% [6.96%, 12.86%], but more often identifies onset at 36.25% [25.00%, 47.50%].The best action-identity and onset scores belong to different controls, and neither approximates a reliable joint policy.Source: D FULL HELD-OUT RESULTS AND UNCERTAINTY · p. 9

E PAIRED DIFFERENCES AND COMPETING METRICS

• On the same 80 resampled groups, WavLM Base Plus had higher pooled post-onset label accuracy than prefix-shuffled WavLM by 10.74% [6.73%, 14.56%] and scalar acoustics by 11.16% [5.66%, 15.95%].Source: E PAIRED DIFFERENCES AND COMPETING… · p. 9
• WavLM minus elapsed time is -13.12% [-23.76%, -1.88%] for onset exactness, reversing the ranking on that metric.Audio plus authored text lowers pre-onset exposure by 3.80% [-6.00%, -1.94%], without a supported improvement in pooled post-onset label accuracy or trajectory exactness.Source: E PAIRED DIFFERENCES AND COMPETING… · p. 9, Table 5 · p. 10
• The observed 26.03% score is at the 96th percentile of 100 within-prefix permutations, below the 97.5th-percentile reference of 26.73%.This finite permutation diagnostic uses one fixed split and probe pipeline, and does not support a robust speech-specific claim.Source: E PAIRED DIFFERENCES AND COMPETING… · p. 10

F RESULTS BY HELD-OUT FAMILY

• Audio-only pooled post-onset label accuracy ranges from 47.08% for message barge-in to 8.75% for calendar noise.Table 6 reports descriptive percentages for four families with 20 contrast groups per family.Source: F RESULTS BY HELD-OUT FAMILY · p. 10, Table 6 · p. 10
• Audio-only exact full trajectories range from 0.00% to 15.00% across the four held-out families.The family cell-wise intervals are not a correction for inspecting multiple families.Source: F RESULTS BY HELD-OUT FAMILY · p. 10
• The pooled estimate should not be read as uniform transfer across semantic families.The table’s point estimates do not establish a between-family effect.Source: F RESULTS BY HELD-OUT FAMILY · p. 10, Table 6 · p. 10

G CONSTRUCTION AND IMPLEMENTATION DETAILS

• Pairs share transcripts and pre-divergence samples, then branch between the preceding observed prefix and the first action-labeled prefix; paired 15 dB noise is added before nested cropping.The construction tests an information contract, not perceived naturalness or listener agreement.Source: G CONSTRUCTION AND IMPLEMENTATION… · p. 11
• The WavLM probe uses standardized, mean-pooled 768-dimensional final hidden states and balanced logistic regression, alongside text, scalar-acoustic, shuffled-representation, endpoint-plurality, and always-wait controls.Training and development use clean rendering only, and test scores do not guide threshold or model choice.Source: G CONSTRUCTION AND IMPLEMENTATION… · p. 11
• Targets are WAIT before assigned onset and source-utterance actions at and after onset; trajectory exactness requires all five predictions to match, while exposed-action rate excludes THINK and ABSTAIN.Onset exactness counts trajectories whose first non-WAIT prediction occurs at assigned onset; never-acting trajectories receive predicted onset 1.2 for timing-error calculations.Source: G CONSTRUCTION AND IMPLEMENTATION… · p. 11
• WavLM Base Plus gets 252 of 968 post-onset predictions correct: 195/606 exposed-action labels (32.18%) and 57/362 internal-state labels (15.75%).SPEAK is not a label in this six-class diagnostic.Source: G CONSTRUCTION AND IMPLEMENTATION… · p. 11
• Bootstrap intervals resample 80 contrast groups with replacement, retain paired utterances and renderings, and condition on fitted probes and predictions without refitting.Paired model differences reuse sampled group IDs; the 100-permutation analysis is separate from bootstrap uncertainty.Source: G CONSTRUCTION AND IMPLEMENTATION… · p. 11

H INTERPRETATION BOUNDARIES

• The results are limited to a machine-only probe diagnostic on synthesized speech with assigned onset labels, not human judgments, naturalness, native streaming conversation, or general claims about speech representations or agent timing.The observed score did not exceed the 97.5th percentile of the 100-permutation reference distribution.Source: H INTERPRETATION BOUNDARIES · p. 11