Source-linked AI summary
Repairing Speech Access After Text Updates with Frozen Affine Fields
Ming-Hao Hsu, Yuxiang Wang, Yingda Shen, Zhizheng Wu
TL;DR
After a text update, a model may answer an updated fact when asked in text but fail on the same spoken query. AFFIRM repairs speech access with a detachable operator around a frozen checkpoint; it has no clear advantage over SALSA or ReFT on balanced synthetic-speech PopQA, while recorded-SLUE transfer differs across methods. The authors conclude that text acquisition and speech access can diverge after an update.
Problem
A text update can make a fact available to typed queries while the same query remains difficult to answer through speech.
Source: §1 · p. 1Method
AFFIRM freezes a shared base correction and adds a zero-mean query-specific correction conditioned by paired speech and text hidden-state coordinates.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
Text acquisition and speech access can diverge after an update and should be verified and repaired separately.
Source: §8 · p. 8PopQA and TOFU use synthetic English speech from one fixed voice, while recorded SLUE changes task, prompt, speakers, and acoustics and includes text passages, so it tests no-refit transfer rather than closed-book recall from human recordings.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
Speech-capable language models inherit most of their knowledge from text-rich language backbones, and continued model maintenance is likewise dominated by text updates. Knowledge acquired through a text update can nevertheless remain difficult to access when the same query is spoken. We formulate this mismatch as post-update speech access repair, where the knowledge-bearing checkpoint stays frozen and a detachable operator repairs speech-only access in one model pass. We introduce Affine Fields for Interface Repair across Modalities (AFFIRM), which first learns and freezes a base correction for speech hidden states, then adds query-specific corrections with zero mean over the operator-training queries. Paired speech and text states provide the coordinates that condition these corrections on each query. On balanced synthetic-speech PopQA, paired comparisons do not establish a clear advantage for AFFIRM over global SALSA steering or low-rank ReFT interventions under the same frozen single-state constraint. On recorded SLUE-SQA-5 questions with text passages, a full-scale PopQA-fitted AFFIRM operator transfers without refitting and improves ROUGE-L by 0.0941 over its paired No Repair baseline. Low-rank ReFT interventions also transfer positively, whereas the matched SALSA variant falls 0.0707 below its paired baseline. These results show that text acquisition and speech access can diverge after an update and should be verified and repaired separately.
1 Introduction
A successful text update can leave the same fact inaccessible through speech, motivating repair that keeps the acquired checkpoint frozen. AFFIRM combines a shared base correction with a centered query-specific correction; experiments find no clear synthetic-speech advantage but differing recorded-speech transfer.
1 Introduction
- A model may answer an updated fact when the question is typed yet fail on the same spoken question, defining a post-update speech-access gap.Text success indicates the acquired checkpoint can express the updated fact; the paired speech failure motivates repair.Source: §1 · p. 1
- The repair contract keeps the acquired checkpoint immutable and uses a detachable operator on one speech decoder state in one pass, without a transcript.The deployment question is supplied only as speech.Source: §1 · p. 1
- AFFIRM first learns and freezes a shared access parent, then adds a query-specific centered route field whose mean over training queries is zero.Paired spoken and typed queries provide hidden-state coordinates for conditioning the correction.Source: §1 · p. 2
- On balanced PopQA, paired comparisons show no clear AFFIRM-DIRECT advantage, while recorded SLUE transfer improves over No Repair for AFFIRM and ReFT but regresses for the tested SALSA variant.The operators transfer without refitting; the paired text–speech gap follows successful text acquisition.Source: §1 · p. 2
- The paper formulates post-update speech access repair as separating text knowledge acquisition from detachable speech access through an immutable acquired checkpoint..Source: §1 · p. 2
2 Related Work
Prior work addresses speech–text gaps through multimodal adaptation, knowledge acquisition or editing, and hidden-representation interventions. AFFIRM instead targets speech access after a text update, using a detachable operator around a frozen acquired checkpoint.
Text-centered speech language models and modality gaps
- Multimodal speech-language models use alignment and cross-modal distillation, yet speech can remain weaker on semantic and reasoning tasks; this work addresses later maintenance after text updates.Prior studies include Tang et al. (2024), Hu et al. (2024), Xu et al. (2025), and Hsu et al. (2026a), among others.Source: §2 · p. 2
Cross-modal knowledge acquisition and access
- Cross-modal acquisition can leave information unequally available across routes, while knowledge-editing work changes stored associations; AFFIRM repairs access after acquisition succeeds.The passage distinguishes access repair from acquisition or editing.Source: §2 · pp. 2–3
Frozen representation interventions
- Frozen representation interventions modify behavior without changing model weights, but do not separate text acquisition from speech access to acquired knowledge.The cited intervention family includes ReFT, DiReFT, and LoReFT.Source: §2 · p. 3
Global and conditional steering
- Global and input-conditioned steering provide complementary mechanisms for reducing speech–text discrepancies, whereas AFFIRM targets post-update factual access with a detachable one-state decoder operator.The passage names global speech steering, input-dependent steering, Steer-MoE, and angle and length interventions.Source: §2 · p. 3
3 Preliminaries and Problem Formulation
The formulation measures whether a text-acquired fact is less accessible through a matched speech query, then evaluates a detachable repair while the acquired checkpoint remains frozen. Evaluation uses disjoint groups and separately tests no-refit transfer to recorded speech with text passages.
3.1 Post-update access
- For each acquired fact, the setup pairs semantically matched text and speech queries and considers the corresponding reference answer.The checkpoint is obtained through text-only answer-likelihood training.Source: §3.1 · p. 3
- At decoder block ℓ, the compared states are taken from the final attended prompt position before prediction and come from the same checkpoint.This makes the text and speech states directly comparable for matched queries.Source: §3.1 · p. 3
- For PopQA and TOFU, operator-training, development, and evaluation facts are disjoint subsets of the text-acquisition set, and evaluation facts provide no operator gradients or selection signals.The setup verifies the acquired checkpoint through text and then freezes it.Source: §3.1 · p. 3
- A positive text-to-speech access difference operationally means the acquired checkpoint answers more reliably through text than through the matched speech interface.The route-specific access scores are defined over a scored set.Source: §3.1 · p. 3
3.2 Frozen repair contract
- A detachable repair operator is fitted on the operator-training set and applied to the speech state while the acquired checkpoint stays frozen.The intervention strength is fixed before test evaluation.Source: §3.2 · p. 3
- At deployment, the query is speech-only: there is no transcript, matched text-question state, retrieved example, or second model pass.Held-out speech improvement measures repair after text-query success establishes acquisition and the paired route gap diagnoses inaccessible behavior.Source: §3.2 · p. 3
- The separate SLUE transfer test uses recorded questions with text passages and changes the task, prompt, speakers, and acoustics together without refitting.It evaluates transfer behavior rather than closed-book recall from human recordings.Source: §3.2 · p. 3
4 AFFIRM: Shared Access Parent with Centered Route Field
AFFIRM repairs speech access with a shared parent correction plus query-specific corrections conditioned on paired speech–text states. The acquired checkpoint stays frozen while the operator is trained and deployed.
4.1 Method overview
- AFFIRM predicts paired-route differences with a low-dimensional code, then trains a separate readout to turn fixed coordinates into query-specific corrections.The coordinate fit uses paired prompt states without answer labels; the acquired checkpoint remains frozen, and deployment stores only the compact operator.Source: §4.1 · pp. 3–4, §4.3 · p. 4
4.2 Affine repair field
- Each affine repair adds a shared parent and a centered route correction, whose zero mean makes it represent variation around the fitted parent.The route field is affine in its coordinate, which itself depends nonlinearly on the speech state.Source: §4.2 · p. 4
4.3 Answer-free paired-route coordinates
- The coordinate map is fitted from paired speech and text prompt states, frozen, and used without answer labels to condition later corrections.A temporary readout predicts state differences during coordinate fitting, then is discarded before Stage B learns a separate answer-improving readout.Source: §4.3 · p. 4
4.4 Stage A: decoder-aware access parents
- Stage A trains the parent through the frozen decoder with teacher-forced answer cross-entropy, then freezes its parameters before Stage B.The direct and expert parent forms are alternative parameterizations; in the balanced PopQA diagnostic, the K = 64 router concentrated on nearly the same expert across states.Source: §4.4 · p. 4
4.5 Stage B: selective correction around a frozen parent
- Stage B freezes the checkpoint, coordinate map, and selected parent, then trains a zero-initialized readout using text guidance when it outperforms the parent and parent guidance otherwise.Zero initialization makes the first Stage-B pass reproduce parent-only predictions; backpropagation through the frozen decoder updates only the readout.Source: §4.5 · p. 4
4.6 Fitting and deployment
- At inference, AFFIRM edits the final prompt state during prefill and continues ordinary greedy generation without editing answer-token states.The deployment uses one model pass and requires no question transcript, paired text-question state, pre-update checkpoint, or external ASR system.Source: §4.6 · p. 5
5 Experimental Setup
The experiments freeze text-acquired checkpoints and compare speech-only representation repairs under a one-pass deployment constraint, with recorded SLUE transfer tested using PopQA-fitted operators without refitting.
Experimental models and acquisition
- Full-scale Qwen2.5-Omni-7B checkpoints are acquired from all 14,267 PopQA facts or 4,000 TOFU questions, then frozen before repair operators are fitted.The study also evaluates compact operators on separately acquired Kimi-Audio-7B and MiMo-Audio-7B checkpoints.Source: §5 · p. 5
Operator fitting and splits
- PopQA operator splits keep shared-subject or duplicate-question groups together, while TOFU splits separate authors across training, development, and test.The balanced PopQA comparison uses a separate 1,000-fact checkpoint and a 600/199/201 split; full-scale PopQA uses 8,572/2,848/2,847 examples.Source: §5 · p. 5
Methods and evaluation
- The compared repairs include No Repair, low-rank DiReFT and LoReFT, input-dependent L2S, global SALSA, and AFFIRM parent variants.AFFIRM-DIRECT is primary, AFFIRM-EXPERT uses an expert parent, and AFFIRM-K1 uses a compact scale–direction parent; reported metrics include ROUGE-L recall, normalized exact match, and SLUE answer-span accuracy.Source: §5 · p. 5
Deployment constraints and controls
- Speech-only methods receive audio without transcripts or paired text-question states and use one pass through the frozen acquired checkpoint.Speech LoRA, distillation, ASR cascade, and full speech supervision are reported separately because they relax the frozen single-state contract; Angle and Length use a paired text-state donor.Source: §5 · p. 5
Recorded-speech transfer
- The recorded-speech transfer test applies frozen PopQA-20 operators to all 256 SLUE-SQA-5 questions without SLUE examples, refitting, or target-domain strength retuning.SLUE questions are spoken and accompanied by text passages, testing transfer under changes in task, prompt, speakers, and acoustics.Source: §5 · p. 5
6 Results
Text acquisition leaves a measurable speech-access gap, and repair methods that are comparable on synthetic PopQA diverge on recorded SLUE transfer. Results also vary with parent parameterization, while a compact variant shows cross-backbone gains.
6.1 The text update leaves a measurable speech-access gap
- Text-trained PopQA acquisition reaches 0.9213 RL and 0.9187 EM through text but 0.1840 RL and 0.1519 EM through speech; speech-state retrieval is 50.3% versus 0.17% chance.On balanced synthetic-speech PopQA, paired bootstrap comparisons do not establish an AFFIRM-DIRECT advantage over DiReFT, LoReFT, or SALSA, although AFFIRM-DIRECT has the strongest TOFU RL and EM point estimates among speech-only rows.Source: §6.1 · p. 6, §6.2 · p. 6
6.3 Parent parameterization affects source and transfer performance
- AFFIRM-EXPERT has higher TOFU scores, while AFFIRM-DIRECT has higher SLUE point estimates, exceeding AFFIRM-EXPERT by +0.0430 ± 0.0185 EM.The difference is reported as a bootstrap standard error; the point-estimate ranking therefore changes between TOFU and SLUE.Source: §6.3 · p. 6
6.4 Scale, stability, and cross-backbone scope
- On a separately acquired Kimi-Audio checkpoint, compact AFFIRM-K1 improves held-out RL/EM from 0.6321/0.0850 to 0.6451/0.1188.Three Stage-B seeds reproduce the PopQA-20 AFFIRM-DIRECT and TOFU AFFIRM-EXPERT development results.Source: §6.4 · p. 6
6.2 Methods comparable on synthetic speech diverge on recorded speech
- 0.0941 RL gain for AFFIRM-DIRECT and 0.0774 for LoReFT contrast with SALSA’s 0.0707 RL deficit against paired No Repair on SLUE, without refitting.The full PopQA-20 operator transfers to 256 recorded questions with text passages; the comparison reflects joint shifts in task, prompt, speakers, and acoustics.Source: §6.2 · p. 6, Table 1 · p. 7
7 Component and Depth Analysis
AFFIRM-DIRECT combines a shared access parent with a centered query-specific route field, and this composition gives the strongest RL across the reported PopQA and TOFU settings. The analysis also finds a parent-driven early-block advantage in a compact TOFU sweep.
Shared parent and route field
- AFFIRM-DIRECT gives the strongest RL across PopQA-20, PopQA-60, and TOFU; adding the route field raises RL and EM on both PopQA budgets.Adding the shared parent to the route field raises RL on both PopQA budgets, while TOFU differences are not resolved by paired bootstrap analysis.Source: §7 · p. 7, Table 4 · p. 8
- On TOFU, paired-bootstrap analysis resolves none of the four corresponding RL and EM differences between component configurations.Either component recovers substantial access under the same frozen one-state contract.Source: §7 · p. 7, Table 4 · p. 8
Field geometry
- The shared parent has norm 31.48, versus mean per-query route corrections of 8.69 with mean cosine 0.0104 to the parent.The centered route code's mean norm over operator-training queries is below 2 × 10^-6, consistent with a dominant shared translation and smaller, nearly orthogonal refinement.Source: §7 · p. 7
Compact-parent depth
- In the two-seed compact TOFU sweep, both seeds select block 4, while full-field RL declines from 0.5550 at block 4 to 0.4788 at block 28.The parent alone also peaks at block 4 in both seeds; the route field changes RL by at most 0.0068 and its contribution varies in sign.Source: §7 · p. 7
Parent parameterization
- Table 3 compares direct and expert parents at block 4 on TOFU and on no-refit transfer to recorded SLUE questions.TOFU reports RL and EM on 800 author-disjoint questions; SLUE reports Span, RL, and EM on 256 recorded questions.Source: Table 3 · p. 8
8 Discussion and Conclusion
The paper frames AFFIRM as a detachable speech-access repair that leaves the acquired checkpoint unchanged. Its results differ across evaluation settings: synthetic-speech comparisons show no clear advantage, while recorded-speech transfer favors AFFIRM and ReFT over the matched SALSA variant.
Discussion and conclusion
- AFFIRM combines a shared access parent with a smaller centered route field for query-specific variation as a detachable operator.The structure is intended to complement speech-pathway retraining when checkpoint immutability and one-pass speech inference matter.Source: §8 · p. 8
- Text acquisition and speech access can diverge after an update, so the paper concludes they should be verified and repaired separately.The acquired checkpoint can remain unchanged during speech-access repair.Source: §8 · p. 8
- On balanced synthetic speech, paired comparisons do not establish a clear AFFIRM advantage over SALSA or ReFT; on recorded SLUE transfer, AFFIRM and ReFT improve over No Repair while SALSA regresses.The SLUE operators transfer from full-scale PopQA without refitting.Source: §8 · p. 8
- With the centered route field included, the expert parent gives higher block-4 TOFU scores, while the direct parent gives higher SLUE point estimates.The compact TOFU sweep favors early decoder blocks.Source: §8 · p. 8
Limitations
The evaluations focus on short-answer factual access after text-only updates, primarily with Qwen2.5-Omni-7B. Their scope is limited to transfer across acquired facts and specific speech settings, rather than unseen-knowledge acquisition or broad speech performance.
Evaluation scope
- The study focuses on short-answer factual access after text-only updates, primarily using Qwen2.5-Omni-7B.PopQA and TOFU test access to facts included in text acquisition, with disjoint operator-training, development, and test groups.Source: Limitations · p. 8
- The paired text–speech generation gap measures access but does not localize every acoustic or internal cause.Source: Limitations · p. 8
Recorded-speech transfer
- Recorded SLUE changes the task, prompt, speakers, and acoustics, and its text passage means it tests no-refit transfer rather than closed-book recall from human recordings.The transfer result covers the complete fitted operator and does not isolate the centered route field's contribution.Source: Limitations · p. 8
Scope and future directions
- The single-state intervention restricts adaptation capacity; long-form generation, multilingual access, repeated updates, and architecture-specific scaling remain future directions.Broader speech fine-tuning or an ASR cascade can attain higher accuracy when the deployment contract is relaxed.Source: Limitations · pp. 8–9
Ethical Considerations
The paper flags incomplete characterization of language, accent, and user-group performance, alongside potential answer changes and licensing conditions. Its appendix documents dataset splits, checkpoints, operator fitting, calibration, evaluation, and computing settings.
Ethical Considerations
- English synthetic- and recorded-speech tests leave performance across languages, accents, and user groups incompletely characterized.The paper also notes that interventions can change correct answers or expose incorrect stored information, and that benchmark gains do not establish reliability for consequential applications.Source: Ethical Considerations · p. 9
A.1 Datasets and splits
- PopQA questions are grouped by shared subjects or normalized duplicates before splitting, while TOFU questions are grouped by fictional author.The acquired checkpoint saw every PopQA fact through text, including operator development and test facts; SLUE evaluation uses 256 recordings from 50 speakers without fitting on SLUE.Source: §A.1 · p. 11
A.2 Acquired checkpoints and prompts
- Text acquisition updates all Qwen2.5-Omni-7B parameters for ten epochs, after which the acquired checkpoint remains frozen during operator fitting and evaluation.Speech prompts use the native audio wrapper, and SLUE prompts pair a text passage with a spoken question without inserting a transcript for speech-only methods.Source: §A.2 · p. 11
A.3 State extraction and operator optimization
- The primary operator state is the final attended prompt position after decoder block 20, and the route field is trained with a 0.1-nat teacher-selection threshold.Every realization edits one prompt-final state during the first generation step; learned-value counts range from 3,584 for DIRECT PARENT to 917,505 for AFFIRM-EXPERT.Source: §A.3 · p. 11
A.4 Frozen inference calibration
- Intervention strengths are fixed before test evaluation, and matched Qwen operators are calibrated to the same mean edit norm on development states.AFFIRM-DIRECT’s PopQA-20 strength is reused unchanged for SLUE.Source: §A.4 · p. 11
A.5 Decoding, metrics, uncertainty, and failures
- Evaluation uses greedy decoding, defines ROUGE-L, exact match, answer-span accuracy, and retention, and reports paired-bootstrap uncertainty and no failed generations.PopQA and TOFU allow at most 160 new tokens, while SLUE allows at most 64.Source: §A.5 · p. 12
A.6 Software and hardware
- Qwen runs use BF16 on one NVIDIA A800-SXM4-80GB accelerator, with learned operator tensors and calibration statistics retained in FP32.The timing study reports standalone end-to-end wall time and peak allocated CUDA memory.Source: §A.6 · p. 12
B Baseline Implementations and Inference Inputs
On recorded SLUE speech, AFFIRM-DIRECT and ReFT improve over their paired No Repair baselines, while SALSA has a resolved negative ROUGE-L difference. Balanced PopQA comparisons do not resolve AFFIRM-DIRECT’s ROUGE-L differences against its comparators.
C.1 Balanced source-domain comparisons
- The paired-bootstrap analysis does not resolve the ROUGE-L differences between AFFIRM-DIRECT and its comparators on balanced PopQA.The comparison uses paired per-item score differences and group-bootstrap estimates.Source: §C.1 · p. 13, Table 8 · p. 14
C.4 Block-4 repair comparisons
- AFFIRM-DIRECT has a resolved exact-match advantage over the block-4 repair comparators on SLUE, with positive point estimates in answer span and ROUGE-L.The comparison includes block-4 AFFIRM-EXPERT and block-20 expert parent; the expert-parent row uses a different intervention block.Source: §C.4 · p. 14
D.1 Synthesized-question content retention
- Speech-to-text exact-pair retrieval reaches 0.5033 top-1 across 600 synthesized-question candidates, showing substantial item-specific question identity in speech states.Within-relation top-1 is also 0.5033, against item-averaged chance of 0.0267; the results do not isolate the cause of the remaining generation gap.Source: §D.1 · p. 14
D.2 Full-scale PopQA matrix
- The block-4 AFFIRM-EXPERT test row covers 2,847 PopQA questions, but no same-layer parent-only result is available for an ablation.Source: §D.2 · p. 14
C.2 No-refit recorded-speech transfer
- AFFIRM-DIRECT and DiReFT/LoReFT improve over paired No Repair on all three SLUE metrics, while SALSA has a resolved negative ROUGE-L difference and L2S intervals include zero.The transfer evaluations apply fitted operators without refitting on SLUE; each method’s improvement uses its own paired No Repair outputs.Source: §C.2 · p. 13, B Baseline Implementations and… · p. 13, Table 9 · p. 15
C.3 Full-scale component contrasts
- +0.06250 ± 0.02163 is AFFIRM-DIRECT’s SLUE answer-span accuracy difference versus ROUTE FIELD; +0.03516 ± 0.01835 is its difference versus EXPERT PARENT.Positive values favor AFFIRM-DIRECT.Source: §C.3 · p. 13, Table 10 · p. 15
D.3 Stage-B seed stability; D.4 Compact-parent depth characterization
- PopQA-20 development scores are stable across three Stage-B seeds; on TOFU, full-field ROUGE-L falls from 0.5550 at block 4 to 0.4788 at block 28.The route field changes ROUGE-L by at most 0.0068 across blocks and seeds, and its two-seed average never exceeds 0.0025.Source: §D.4 · p. 15, §D.3 · p. 15
D.5 Cross-backbone characterization
- On Kimi-Audio, the selected operator raises test ROUGE-L/exact match from 0.6321/0.0850 to 0.6451/0.1188; MiMo shows mixed changes versus No Repair.MiMo’s exact match increases by 0.0125 while ROUGE-L decreases by 0.0044.Source: §D.5 · p. 15
D.6 Geometric and capacity controls
The paper contrasts representation controls and broader adaptation options with the detachable one-state operator. Its development comparison finds Speech LoRA at least as accurate at each tested unique-pair budget, while the route map stores fewer learned values.
D.6 Geometric and capacity controls
- The route map beats the mean shift on balanced PopQA, while residual KRR stores 9.4 times more learned values without improving test scores.The remaining AFFIRM-DIRECT gap is consistent with decoder-facing training contributing beyond representation-only mapping.Source: §D.6 · pp. 15–16, Table 15 · p. 17
E.1 Matched-data direct speech fine-tuning
- At every unique-pair budget, Speech LoRA is at least as accurate on development data; at 150 pairs, route-map and first-epoch Speech LoRA ROUGE-L/exact-match scores are 0.5720/0.5628 and 0.5695/0.5578.Their approximate training times are 202.6 and 197.8 seconds, respectively.Source: §E.1 · p. 16, Table 16 · p. 17
E.2 Test accuracy and deployment requirements
- Broader adaptation and extra inference resources can reach higher accuracy than the detachable one-state operator.The test comparison covers balanced PopQA, PopQA-60, and TOFU under frozen and relaxed deployment contracts.Source: §E.2 · p. 16, Table 17 · p. 18