Source-linked AI summary
REVE: Efficient Hallucination Correction for Large Audio-Language Models via Reused Encoder States
Hongjin Song, Jiasheng Kuang, Xinyu Yang, Qiuyu Fang, Ziyu Wu, Guowu Tan, Xiang Xie
TL;DR
Large audio-language models may mention events absent from the audio, while separate detectors require another encoding pass. REVE reuses the target model’s encoder states to verify caption events; on AudioSet it removes 92.9% of false mentions at 0.756 recall, comparable to CED-Tiny and CED-Base.
Problem
LALM captions can contain plausible but unsupported event mentions that text or token probabilities cannot reliably verify at the event-span level, while separate detectors require another audio-encoder pass.
Source: §1 · p. 1Method
REVE reuses target-model encoder states for frame-score and four-segment readouts, fuses them class-wise, and verifies ontology-mapped caption events.
Source: §1 · p. 1, §2 · p. 1, §2.4 · p. 3Results
REVE removes 92.9% of false mentions at 0.756 recall, comparable to CED-Tiny’s 92.7% and CED-Base’s 93.8%, without a separate audio-encoder pass.
Source: §4.1 · p. 3Takeaways & Limitations
REVE adds 3.38M parameters and has 0.57 ms complete verification latency, about 1/18 of the CED-Base path.
Source: §4.1 · p. 3Hallucination reduction must be reported with faithful-mention recall; thresholds are selected to maximize reduction under a development-set recall constraint.
Source: §3.3 · p. 3
Abstract
from arXiv · showhide
Large audio-language models may mention acoustic events that are absent from the input. A separate audio event detector can verify these mentions, but doing so requires a second audio encoder and a separate forward pass. We propose Reused Encoder States for Verifying Events (REVE), a lightweight method that uses states already computed by the target model. One readout summarizes class scores across audio frames, while another uses pooled states from four consecutive frame intervals. Class-aware score fusion combines their outputs to verify generated event mentions without encoding the audio again. On AudioSet, REVE removes 92.9% of label-unsupported mentions under a faithful-mention recall constraint. With fewer added parameters and no second audio-encoding pass, REVE achieves a reduction comparable to those of CED-Tiny and CED-Base. Its complete verification latency is about 1/18 of the CED-Base path. Results on controlled DESED mixtures and different target-model architectures further confirm the effectiveness of encoder-state reuse.
1. INTRODUCTION
Large audio-language models can produce plausible event mentions unsupported by the audio, which text or token probabilities alone cannot reliably verify at the event-span level. REVE reuses the target model’s encoder states to verify caption events, addressing the added encoding cost of separate detectors.
Motivation
- Plausible but absent acoustic events can appear in LALM captions, and caption-level scores cannot locate the exact unsupported span.Training or decoding changes may also be difficult to apply to deployed models.Source: §1 · p. 1
- A separate detector can map caption event mentions to ontology classes and link acoustic evidence to spans, but CED adds a second audio-encoder pass.CED-Tiny reduces detector size but still re-encodes the waveform, increasing verification latency.Source: §1 · p. 1
Contribution
- REVE uses frame-score and four-segment readouts from existing encoder states, then class-aware fusion to verify ontology-mapped caption events.It formulates post-hoc correction as ontology-aligned event verification.Source: §1 · p. 1
- The evaluation measures hallucination reduction under a faithful-mention recall constraint and reports verification computation and storage costs.Source-disjoint DESED mixtures and cross-model experiments test generalization across acoustic domains and LALM architectures.Source: §1 · p. 1
2. METHOD
REVE reuses the target LALM’s encoder states to verify ontology-mapped caption events without re-encoding the audio. Its two-scale readouts combine frame-level score statistics with temporal segment representations before class-aware fusion and selective editing.
2. METHOD
- REVE reuses pre-projector encoder states as acoustic evidence and maps caption event spans to ontology classes for verification.Only event mentions supported by the corresponding class scores are retained.Source: §2 · p. 1, Figure 1 · p. 2
2.1. Encoder State Reuse
- The target LALM’s audio encoder produces frame-level states that are also available to REVE, avoiding a second audio encoding pass.The states are extracted before the modality projector converts them into language-model-compatible audio tokens.Source: §2.1 · pp. 1–2
2.2. Ontology Mapping
- REVE normalizes official AudioSet class names to match whole-word event spans, mapping matched nodes to readout classes or their nearest represented ancestors.Unmapped spans are not edited, and reference labels use the same mapping.Source: §2.2 · p. 2
2.3. Two-Scale Event Scoring
- To preserve short-event evidence that global mean pooling may weaken, REVE combines frame-score statistics with means from four contiguous segments and class-aware calibration.The calibrator fuses the statistical and segment readouts; its shared and class-specific parameters add 1,057 coefficients.Source: §2.3 · p. 2
2.4. Event Mention Correction
- After ontology mapping, REVE scores caption-mentioned classes and removes event spans whose scores fall below the event-presence threshold.All other caption text remains unchanged.Source: §2.4 · p. 3
3. EXPERIMENTAL SETUP
Experiments evaluate REVE on AudioSet and controlled DESED mixtures, with Qwen2-Audio as the main target and two additional LALM architectures. The protocol compares internal, confidence-based, and external signals while selecting thresholds under a development-set recall constraint.
3.1. Datasets and Models
- AudioSet evaluation uses 1,000 clips split into 500-clip development and test sets; DESED evaluation uses 1,000 ten-second mixtures with source-disjoint 500-mixture splits.DESED mixtures contain one to three events from eight families without background noise; the main target is Qwen2-Audio-7B-Instruct, with Qwen2.5-Omni-7B and SALMONN-13B also tested.Source: §3.1 · p. 3
3.2. Baselines
- Baselines include token confidence, decoder states, a mean-pooled encoder readout, and external audio models that reprocess the audio.External models include CLAP, AST, PANN Cnn14, CED-Tiny, and CED-Base.Source: §3.2 · p. 3
3.3. Evaluation Protocol
- The protocol maximizes hallucination reduction subject to a development-set faithful-mention recall constraint, since deleting every mention could otherwise yield 100% reduction.Thresholds and REVE calibration are selected using development data.Source: §3.3 · p. 3
4. RESULTS
REVE removes 92.9% of false mentions on AudioSet at 0.756 recall, with reductions comparable to CED-Tiny and CED-Base. Ablations, controlled DESED mixtures, and cross-model evaluation examine its components and scope.
4.1. AudioSet Results
- 92.9% of false mentions are removed by REVE at 0.756 recall, comparable to 92.7% for CED-Tiny and 93.8% for CED-Base.Table 1 reports results at a development-set recall target of 0.75.Source: Table 1 · p. 3, §4.1 · p. 3
- 0.57 ms is REVE’s complete added latency, about 1/18 of the CED-Base path, with 3.38M added parameters.CED-Tiny and CED-Base still re-encode the waveform; Fig. 2(a) compares reduction against added latency and parameters.Source: §4.1 · pp. 3–4, Table 2 · p. 4, Figure 2 · p. 4
4.2. Ablation Study
- Class-aware calibration lowers residual density to 0.060 while keeping AUROC nearly unchanged, improving decisions with one threshold across classes.The two readouts provide complementary evidence.Source: §4.2 · p. 4, Table 3 · p. 4
4.3. Controlled DESED Results
- 0.959 AUROC and 0.915 AP are reported for REVE on controlled DESED mixtures; CED-Base reaches 0.901/0.791.The mixtures provide exact presence labels for all eight event families, and REVE remains above both CED variants from one to three sources.Source: Table 4 · p. 4, §4.3 · p. 4, Figure 2 · p. 4
4.4. Cross-Model Results
- REVE is evaluated on Qwen2.5-Omni-7B and SALMONN-13B to test performance across target-model architectures.Table 5 reports complete REVE at a development recall target of 0.75.Source: §4.4 · p. 4
Target LALM
REVE reduces residual density to 0.158 on Qwen2.5-Omni and 0.207 on SALMONN.
Target LALM
- 80.1% and 74.3% reductions in residual density are reported for Qwen2.5-Omni and SALMONN, respectively.Residual density falls to 0.158 on Qwen2.5-Omni and 0.207 on SALMONN.Source: Target LALM · p. 4
5. CONCLUSION
REVE uses encoder states already available after caption generation to correct LALM captions, removing unsupported mentions with lower verification overhead than a separate detector path.
5. CONCLUSION
- 92.9% of false mentions are removed under the recall constraint, matching CED-Tiny and CED-Base.The result uses REVE’s statistical and segment readouts.Source: §5 · p. 4
- REVE uses fewer parameters, avoids another audio-encoding pass, and adds about 1/18 of CED-Base’s complete verification latency.Source: §5 · p. 4
- DESED and cross-model results confirm the effectiveness of encoder-state reuse.Source: §5 · p. 4