PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-Woodpecker: Hallucination Correction via Evidence-Prior Rebalancing for Omni LLMs

Zijian Liu, Chaoyou Fu, Yan Yang, Caifeng Shan, Ran He

arXiv:2610.27103v1cs.AIcs.CLcs.CVcs.MM

TL;DR

Omni-modal hallucinations can arise when modality cues or learned priors override query-relevant evidence in fused representations, while existing methods mainly intervene at the output level. OWP diagnoses cross-modal conflicts and rebalances representations before decoding. It reports gains on hallucination benchmarks, general reasoning benchmarks, and another OLLM architecture.

  • Problem

    In early-fusion OLLMs, output-level mitigation does not control grounding evidence and learned priors that become entangled in fused representations, a challenge when modalities provide incomplete or conflicting cues.

    Source: §I · p. 1

  • Method

    OWP uses complementary views of a frozen model to diagnose cross-modal conflicts, separate target evidence from competing priors, and adaptively rebalance fused representations without modifying model parameters.

    Source: §I · p. 1, C. Evidence-Preserving Conflict… · p. 3

  • Results

    12.9 points on CMM and 8.6 points on AVHBench: OWP improved accuracy with Qwen2.5-Omni-7B, and it also improved both benchmarks with VideoLLaMA2-AV.

    Source: §VI · p. 10

  • Takeaways & Limitations

    OWP’s reported gains extend beyond hallucination benchmarks to general omni-modal reasoning and a second OLLM architecture.

    Source: §I · p. 2, §VI · p. 10

    OWP targets cross-modal hallucinations and can restore only evidence the frozen backbone encodes; it also requires access to model internals and extra forward passes.

    Source: A. Limitations · p. 9

Abstract

from arXiv · show

Recent Omni LLMs (OLLMs) have demonstrated remarkable capabilities in joint audio-visual reasoning, yet cross-modal hallucinations remain a major challenge. Existing mitigation methods mainly operate at the output level, overlooking the entanglement of grounding evidence and learned priors within fused representations. To address this issue, we propose Omni-Woodpecker (OWP), a training-free framework that shifts hallucination mitigation from output-level manipulation to conflict-conditioned representation rebalancing. Specifically, OWP first identifies cross-modal conflicts from complementary modality-specific views. It then employs orthogonal decomposition to separate grounding evidence from competing priors within the hidden states. Finally, OWP adaptively rebalances fused representations by suppressing misleading priors and restoring ignored evidence before decoding. Extensive experiments demonstrate the effectiveness and strong cross-architecture generalization of OWP. Using Qwen2.5-Omni-7B as the backbone, OWP improves performance by 12.9 and 8.6 points on CMM and AVHBench, respectively, while consistently achieving gains of 0.67-2.25 points across four general OLLM reasoning benchmarks. Furthermore, OWP generalizes effectively to VideoLLaMA2-AV, improving performance by 4.6 and 3.9 points on the same benchmarks. Code is publicly available at https://github.com/ZiiJianLiu/Omni-Woodpecker

I. INTRODUCTION

Omni-modal hallucinations arise when competing modality cues or learned priors outweigh query-relevant evidence in fused representations, even when the model encodes that evidence. OWP addresses this imbalance by diagnosing conflicts and rebalancing representations before decoding; experiments report gains on hallucination benchmarks and broader reasoning tasks.

I. INTRODUCTION

  • Existing mitigation methods calibrate final decoding distributions or optimize response preferences, without controlling evidence-prior interactions already entangled in early-fusion representations.The paper argues that output-level methods therefore struggle to repair corrupted hidden states.Source: §I · p. 1, Figure 1 · p. 1
  • OWP reframes omni-modal hallucination as evidence-prior imbalance in fused representations, rather than failure to perceive the inputs.The dog example shows that the model encodes audio evidence of silence, but a visual prior overrides it during fusion.Source: §I · pp. 1–2
  • OWP is a training-free framework that diagnoses query-conditioned cross-modal conflicts and rebalances fused representations without modifying model parameters.It uses complementary modality-specific views to identify target evidence and competing sources, then restores suppressed evidence and attenuates misleading priors before decoding.Source: §I · pp. 1–2
  • On CMM and AVHBench, OWP outperforms the strongest baseline by 4.9 and 7.7 points, respectively, and improves original OLLMs by up to 12.9 points on hallucination-centric metrics.The introduction also reports gains across architectures and diverse omni-modal reasoning benchmarks.Source: §I · p. 2

II. RELATED WORK

The related work covers cross-modal hallucinations in omni-modal models, output-level mitigation, and representation-level intervention. OWP is distinguished by identifying query-specific competing streams and adapting its representation edits to each input.

A. Omni-Modal Models and Cross-Modal Hallucination

  • Early fusion supports joint reasoning but can let irrelevant sounds, images, or language priors override query-relevant evidence; CMM and AVHBench test several such cross-modal failure types.The section describes CMM’s language, visual, and audio dominance and spurious-correlation cases, and AVHBench’s audio-driven visual, visual-driven audio, and matching tasks.Source: A. Omni-Modal Models and Cross-Modal… · p. 2

B. Output-Level Hallucination Mitigation

  • Output-level methods adjust decoding distributions or model weights, while early-fusion OLLMs mix evidence and priors before decoding begins.The section notes that these approaches do not identify which stream competes with a given query.Source: B. Output-Level Hallucination Mitigation · p. 2

C. Representation-Level Intervention

  • Representation-level methods edit internal activations, but prior approaches described here use fixed splits or directions; OWP instead builds query-specific edits and allocates them across attention paths and residual states.The contrast includes HulluEdit’s fixed visual-evidence versus anti-prior split and MAD’s use of modality relevance to weight output branches.Source: C. Representation-Level Intervention · p. 2

III. METHOD

OWP intervenes in a frozen OLLM by diagnosing query-conditioned conflicts, decomposing evidence and competing information, and adapting representation edits. Its decomposition protects target evidence while isolating competing content and measuring evidence lost in fusion.

A. Overview

  • OWP edits a frozen OLLM through three stages: conflict diagnosis, evidence-preserving decomposition, and conflict-adaptive rebalancing.The final stage divides one correction budget between attention-path and residual-state edits according to the diagnosed conflict.Source: A. Overview · pp. 2–3, Figure 2 · p. 3

B. Query-Conditioned Conflict Diagnosis

  • The four-view diagnosis compares full, audio-only, video-only, and text-only inputs to identify the query’s target modality and a competing source.A conflict occurs when the target modality supports a different answer from the full view while another modality agrees with the full view; source selection also uses source-normalized attention, and no labels or reference answers are used.Source: B. Query-Conditioned Conflict Diagnosis · p. 3

C. Evidence-Preserving Conflict Decomposition

  • The evidence direction anchors on target evidence and keeps only cross-view components that support it; text-only state subtraction removes the query and shared language component.The fusion shift is decomposed into an orthogonal competing direction and an evidence deficit: the competing direction can be suppressed without changing the evidence component, while the clipped projection measures missing target evidence and is zero when none is missing.Source: C. Evidence-Preserving Conflict… · p. 4
  • OWP localizes competing and supporting heads per input using source attention and answer-aligned value contributions, then ranks them under a fixed per-layer cap.An update is skipped when contributions are numerically unstable.Source: C. Evidence-Preserving Conflict… · p. 4

D. Conflict-Adaptive Representation Rebalancing

OWP splits conflict correction between token–head paths and residual states according to the full-view answer preference. Its edits suppress competing priors and restore evidence before decoding.

D. Conflict-Adaptive Representation Rebalancing

  • OWP uses full-view answer preference ΔF(x) to allocate correction between token–head paths and residual states, with λ(x) = σ((τ−ΔF(x))/T).The remaining share, 1−λ(x), goes to the residual edit.Source: D. Conflict-Adaptive Representation… · p. 4
  • A weak answer preference assigns more correction to what each source sends, while a strong preference shifts it to the residual state.The allocation matches the edit site to how far the conflict has progressed.Source: D. Conflict-Adaptive Representation… · p. 4
  • Path and state coefficients control evidence enhancement, prior suppression, evidence restoration, and path strength.The coefficients are αP, βP, αR, βR, and ηP, respectively.Source: D. Conflict-Adaptive Representation… · p. 4
  • At selected layers, the path edit reweights routes before aggregation, while the residual edit removes an orthogonal prior component and restores the measured evidence deficit.The two shares divide one correction budget; edit sites and coefficients are global, while directions, head sets, and allocation weights are input-specific.Source: D. Conflict-Adaptive Representation… · p. 4
  • OWP compares full and target views first, returns the backbone answer when they agree, and otherwise uses an edited full-view pass without changing the language head or decoding rule.Non-target and text-only views identify the competing source, edit directions, and allocation weight without selecting an output token.Source: D. Conflict-Adaptive Representation… · p. 4

E. Implementation Details

Implementation settings are fixed before evaluation, with selected thresholds, localization limits, and Qwen2.5-Omni-7B edit parameters. Transfer to another backbone adapts only the edited layers and two residual gains.

E. Implementation Details

  • All implementation constants are global and fixed before evaluation, including margins for target typing and competing-source verification.The passage also specifies evidence-calibration settings and limits on selected tokens and heads.Source: E. Implementation Details · p. 4
  • On Qwen2.5-Omni-7B, OWP edits evidence layers 12, 16, and 20 and prior layers 24, 26, and 27.The passage gives layer weights and the αP, αR, βP, βR, ηP, τ, and T settings.Source: E. Implementation Details · p. 4
  • For transfer to another backbone, only the edited layers and two residual gains are adapted.Source: E. Implementation Details · p. 4

IV. EXPERIMENTS

The experiments assess whether OWP corrects cross-modal conflicts while retaining joint audio-visual evidence and general omni-modal reasoning. They also examine transfer, design ablations, and where edits change predictions.

IV. EXPERIMENTS

  • The experiments assess whether OWP corrects cross-modal conflicts and preserves joint audio-visual evidence.Source: §IV · p. 5
  • The evaluation also tests transfer to another backbone and preservation of general omni-modal reasoning.Source: §IV · p. 5
  • The study ablates OWP’s design and analyzes where and on which inputs edits change the backbone’s predictions.Source: §IV · p. 5

A. Experimental Setup

OWP is evaluated on frozen Qwen2.5-Omni-7B and VideoLLaMA2-AV backbones across conflict benchmarks and general omni-modal reasoning tasks. It improves conflict correction in both directions, preserves joint audio-visual evidence, transfers across backbones, and adds inference cost under a conflict-triggered schedule.

A. Experimental Setup

  • The evaluation covered all 1,200 CMM conflict cases, 5,302 AVHBench examples, and official splits of four general reasoning benchmarks using frozen backbones and shared examples.OWP used no reference answers or per-example supervision; coefficients and margins were fixed before evaluation.Source: A. Experimental Setup · p. 5

B. Main Results

  • OWP identified conflicts in 25.8% of inputs and changed final answers in 16.9%; target typing agreed with benchmark-defined target modality on 99.48% of rows.The other 74.2% ran the unmodified backbone; only 8 of 24 target-typing disagreements reached an edit, with no net accuracy effect.Source: B. Main Results · p. 5
  • OWP corrected both conflict directions and all three dominance types, raising CMM hallucination resistance from 57.0% to 85.5% while perception accuracy was 86.7%.The CMM perception-accuracy and hallucination-resistance gap fell from 32.5 points to 1.2 points.Source: B. Main Results · p. 5
  • On AV Matching, OWP reached 81.1%, exceeding the backbone, MAD, and AVCD by 6.6, 13.1, and 17.9 points, respectively.OWP treats joint audio-visual input as evidence and the text-only view as the competing prior.Source: B. Main Results · p. 5
  • On VideoLLaMA2-AV, OWP raised CMM accuracy from 76.2% to 80.8% and AVHBench accuracy from 70.2% to 74.1%.These were the best overall results in that group; the decomposition transferred to a backbone with a different audio-visual fusion design.Source: B. Main Results · p. 5
  • 86.1% on CMM and 83.7% on AVHBench: OWP had the highest overall accuracy among controlled methods, exceeding MAD by 4.9 and 7.7 points, respectively.The comparisons use Qwen2.5-Omni-7B; Table I reports the main results.Source: B. Main Results · p. 5, Table I · p. 6
  • OWP improved general reasoning accuracy on all four benchmarks: OmniBench by 1.93 points, WorldSense by 2.25, MUSIC-AVQA by 1.00, and AVQA by 0.67.Table II reports Qwen2.5-Omni-7B results with and without OWP.Source: B. Main Results · p. 5, Table II · p. 6
  • Under the conflict-triggered schedule, average latency was 1.71× and 1.77× the backbone on Qwen2.5-Omni-7B and VideoLLaMA2-AV, respectively.OWP used at most five forward calls per input; Table III summarizes latency and calls.Source: B. Main Results · p. 6, Table III · p. 6

C. Ablation Studies

The ablations show that evidence enhancement and prior suppression work best together, with adaptive allocation and carefully chosen edit sites improving results. Shared-executor tests further examine cross-backbone gains, answer-prior control, and the role of input views.

Evidence enhancement and prior suppression

  • 85.17% AVHBench accuracy and 85.50% HR came from combining evidence enhancement with prior suppression, while reducing PA cost to 2.80 points.Each component improved the backbone alone, but together they gave the best result and offset each other’s perception cost.Source: C. Ablation Studies · p. 6

Residual and path interventions

  • Residual-only editing reached 83.23% AVHBench accuracy and 84.33% HR, while path-only editing reached 81.68% and 69.67%, respectively.Using both sites with a fixed split raised accuracy to 83.93% but lowered HR to 82.33%, motivating input-dependent allocation.Source: C. Ablation Studies · p. 7

Adaptive allocation and layer assignment

  • Adaptive allocation raised AVHBench accuracy from 83.93% to 85.17%, PA from 84.67% to 86.70%, and HR from 82.33% to 85.50%.Its total budget is unchanged; moving the budget toward the residual branch when the full view favors the competing answer improved all three metrics. Swapping component layers lowered accuracy by 6.79 points (Holm-adjusted p < 0.001).Source: C. Ablation Studies · p. 7

One shared configuration across backbones

  • A shared configuration improved Qwen2.5-Omni-7B and VideoLLaMA2-AV by +2.85 and +2.14 points, respectively, while editing only 11.33% and 20.90% of rows.Every unedited row matched the backbone output exactly; gains concentrated on negative-ground-truth rows (+5.53 and +5.97 points).Source: C. Ablation Studies · p. 7

Answer-prior control and component necessity

  • A constant shift matching OWP’s yes rate gained only +0.60 and +0.67 points, versus larger OWP gains; prior suppression alone contributed more than evidence enhancement alone.Under the shared executor, full-model gains were +2.85 and +2.14 points, compared with +0.04 and −0.02 for evidence enhancement alone and +1.34 and +1.51 for prior suppression alone.Source: C. Ablation Studies · p. 7

View necessity

  • Keeping only the target stream raised accuracy by +2.23 and +0.71 points, but OWP exceeded oracle routing at 84.91% versus 84.28% and 78.99% versus 77.56%.No single stream matched the full input, and the competing stream alone sharply lowered accuracy.Source: C. Ablation Studies · p. 7, Table V · p. 7

D. Mechanism Analysis

Mechanism analyses find that OWP edits evidence and priors at different depths, shifts hidden representations toward evidence, and makes answers more selective to the queried stream. Direction controls indicate that both semantic edit directions matter.

Layer-wise intervention analysis

  • Evidence enhancement worked across middle layers, whereas prior suppression became effective only in later layers and peaked late.The profiles support recovering evidence while it remains separable, then suppressing the competing prior after it forms.Source: D. Mechanism Analysis · p. 8, Figure 3 · p. 8

Changes in hidden representations

  • OWP shifted the evidence-minus-prior balance by 14.25 points, compared with 2.19 for HulluEdit, and raised evidence-dominant states from 23.0% to 50.8%.Across prior-suppression layers, the prior coordinate fell steadily while the evidence coordinate changed by at most 0.14 points.Source: D. Mechanism Analysis · p. 8, Figure 3 · p. 8

Behavioral selectivity under modality corruption

  • OWP’s answer-change selectivity gap was 41.0 points, versus 31.3 for the backbone, 32.6 for MAD, and 29.6 for HulluEdit.Under strong corruption, answers changed for 45.3% of relevant-stream corruptions but only 4.3% of irrelevant-stream corruptions.Source: D. Mechanism Analysis · p. 8, Figure 3 · p. 8
  • In 1,000 matched inputs, corruption of the queried stream changed OWP’s answer more selectively than corruption of the other stream.Mild corruption muted or blacked out alternating one-second segments; strong corruption removed the stream.Source: D. Mechanism Analysis · p. 8

Semantic direction controls

  • OWP reached high accuracy and HR only with both semantic directions intact; randomizing the evidence direction yielded 72.5% AVHBench accuracy and 83.9% CMM HR, versus OWP’s 75.8% and 87.4%.Reversing or randomizing the prior direction and removing orthogonal protection also lowered performance.Source: D. Mechanism Analysis · p. 8, Figure 3 · p. 8

E. Qualitative Results

Qualitative examples show OWP answering reciprocal audio- and visual-grounded queries from the queried modality while retaining compatible context.

Qualitative results

  • OWP described the beehive without inventing a visible rooster and reported both spraying and concurrent speech, unlike MAD and HulluEdit.The baselines followed the salient audio or visual cue rather than the modality queried.Source: E. Qualitative Results · p. 9

V. LIMITATIONS AND FUTURE WORK

OWP’s scope is cross-modal hallucination and depends on evidence the frozen backbone already encodes, while its interventions require model internals and additional forward passes. The paper discusses extensions to other hallucination types and concludes that internal rebalancing is a practical route toward more reliable omni-modal reasoning.

A. Limitations

  • OWP addresses cross-modal hallucinations where another modality or language prior overrides evidence, but cannot restore evidence the backbone fails to perceive in a single-stream view.Its diagnosis is designed for CMM and AVHBench’s cross-modal failure setting.Source: A. Limitations · p. 9
  • OWP requires access to hidden states and attention outputs, limiting it to open-weight OLLMs, and its extra passes cost 1.71–1.77× backbone latency.The reported latency remains lower than MAD on both backbones.Source: A. Limitations · p. 9

B. Future Work

  • The authors propose extending OWP’s diagnosis to temporal and acoustic or paralinguistic hallucinations, and note that unchanged weights allow combination with training-based methods.Temporal diagnosis would compare event timing; acoustic diagnosis would separate spoken content from acoustic attributes.Source: B. Future Work · p. 9

VI. CONCLUSION

  • OWP improved Qwen2.5-Omni-7B by 12.9 points on CMM and 8.6 on AVHBench, and improved all four general omni-modal reasoning benchmarks by up to 2.25 points on VideoLLaMA2-AV.Analyses identify complementary prior suppression and evidence restoration, with residual-state correction and different preferred layer ranges.Source: §VI · p. 10

Related papers

Loading 2610.27103v1…