PaperlayerView exact revision on arXiv

Source-linked AI summary

Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models

Aditi Sarker, Nazreen Shah, Rafi Ibn Sultan, Rhongho Jang, Dongxiao Zhu, Prashant Khanduri

arXiv:2608.29996v1cs.CVcs.AIcs.LG

TL;DR

LVLMs may rely on spurious object-background correlations, limiting controlled diagnosis of whether predictions are grounded in objects or contextual shortcuts. PURGE constructs structured partitions and applies partition-aware unlearning, consistently reducing hallucination and spurious-correlation errors while maintaining or improving overall performance in most evaluated settings.

  • Problem

    Existing LVLM evaluations provide limited control for distinguishing object-grounded predictions from predictions induced by correlated background cues.

  • Method

    PURGE constructs structured retain/forget partitions through complementary data, behavioral, and semantic strategies, then uses partition-aware unlearning to suppress spurious associations while preserving object-grounded reasoning.

  • Results

    PURGE consistently reduces hallucinations and spurious-correlation-driven errors while maintaining or improving overall performance across multiple models and benchmarks.

  • Takeaways & Limitations

    PURGE provides a reusable evaluation protocol and mitigation framework for more reliable LVLMs.

  • Takeaways & Limitations

    PURGE depends on the quality of masks, counterfactual images, cue proposals, and detector outputs, and adds data-processing and fine-tuning costs.

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) achieve strong performance across many multimodal tasks; however, they often exploit spurious object-background correlations, resulting in predictions driven by contextual shortcuts rather than object-relevant visual evidence. Despite growing interest in hallucination and robustness evaluation, existing benchmarks provide limited control over whether model predictions are grounded in the target object or induced by correlated background cues. In this work, we introduce PURGE (\underline{P}artition-aware \underline{U}nlearning for \underline{R}emoving spurious-correlation \underline{G}enerated \underline{E}rrors), a framework for constructing, benchmarking, and mitigating spurious-correlation-induced failures in LVLMs. The framework consists of: -- (1) Structured dataset construction wherein we develop three complementary structured data construction strategies that partition examples by object-relevant evidence and spurious background cues, enabling controlled diagnosis of shortcut reliance; and -- (2) Partition-aware unlearning, which uses these partitions to selectively remove spurious object-background associations while preserving object-based reasoning. We evaluate the \algo~framework across multiple LVLMs, including LLaVA-1.6-7B, Qwen3-VL-8B-Instruct, and Qwen3.5-9B, together with CLIP as a vision-language encoder, on a diverse suite of benchmarks, including CHAIR, POPE, Causal-HalBench, MM-SpuBench, AMBER, MMHal, and Waterbirds. Our results show that PURGE consistently reduces hallucinations and spurious-correlation-driven errors while maintaining or improving overall performance in most evaluated settings, providing both a reusable evaluation protocol and an effective mitigation framework for more reliable LVLMs.

1 Introduction and motivation

LVLMs can hallucinate by relying on object-background correlations instead of object-relevant evidence, while existing evaluations provide limited control over diagnosing this shortcut reliance. PURGE addresses this gap through structured partitions and partition-aware unlearning, reducing hallucinations while preserving competitive utility.

  • Motivation: Spurious object-background correlations can drive LVLM hallucinations when contextual cues replace object-relevant visual evidence.These failures threaten reliability in safety-critical applications and can persist when target-object evidence is absent, ambiguous, or contradicted.
  • Motivation: Existing evaluations often measure hallucination or robustness in aggregate rather than separating object-grounded predictions from background-induced predictions.Natural datasets such as MSCOCO contain strong object-background co-occurrence patterns that can encourage shortcut reliance.
  • PURGE framework: PURGE uses structured retain/forget partitions to diagnose background reliance and target spurious associations for removal while preserving object-based reasoning.The framework combines controlled evaluation with mitigation through partition-aware unlearning.
  • PURGE framework: Three construction strategies expose correlations at pixel, behavioral, and semantic levels: PURGE-D, PURGE-H, and PURGE-M.They use masking and inpainting, annotations with model predictions and counterfactual inputs, or object-presence and contextual-cue partitions, respectively.
  • PURGE framework: Partition-aware unlearning suppresses spurious object-background associations while preserving object-grounded reasoning using GA, KL, and NPO objectives.The framework is evaluated across multiple LVLMs and CLIP on hallucination and spurious-correlation benchmarks.

2 Related work

Prior work mitigates multimodal shortcuts through training-time changes, inference-time corrections, or unlearning, but offers limited control over object-grounded versus shortcut-driven predictions. PURGE combines structured partitioning with targeted unlearning to address these gaps.

  • Training-time debiasing: Multimodal shortcut mitigation includes contrastive debiasing, prompt-based methods, and region-aware training-time approaches.These methods modify parameters, representations, prompts, or supervision signals to reduce spurious associations.
  • Inference-time debiasing: Inference-time methods suppress shortcuts through embedding transformations, visual or instruction-based contrastive decoding, and output-distribution adjustments.They mitigate behavior without updating model parameters.
  • Unlearning-based approaches: Multimodal unlearning removes harmful, biased, or unwanted associations through targeted updates, attention guidance, and cross-modal interventions.Existing approaches extend from broad removal to finer-grained concept- and domain-level forgetting.
  • Research gaps: Existing methods leave limited mechanisms for constructing controlled partitions that distinguish object-grounded predictions from shortcut-driven ones.They also do not explicitly model object-background entanglement across mitigation and unlearning settings.
  • PURGE response: PURGE combines structured data partitioning with partition-aware unlearning to measure shortcut reliance and define targeted retain/forget sets.This design aims to remove spurious object-background associations while preserving object-based reasoning.

3 PURGE: Structured data partitioning for spurious correlation analysis

PURGE constructs structured retain and forget partitions that separate object-relevant evidence from contextual cues across pixel, behavioral, and semantic levels. These partitions support controlled diagnosis of shortcut reliance and identify examples for targeted analysis.

  • 3 PURGE: Structured data partitioning for spurious correlation analysis: PURGE models images as object-specific evidence and contextual features, then constructs retain and forget sets to separate grounded reasoning from shortcut-inducing cues.The framework instantiates this design on MSCOCO and Waterbirds.
  • 3.1 PURGE-D (Data-level disentanglement): PURGE-D uses masking or inpainting to create object-focused retain views and context-only forget views, including object-absent hard negatives.The retain view preserves object-centric information, while the forget view removes the object and retains contextual cues.
  • 3 PURGE: Structured data partitioning for spurious correlation analysis: The three strategies provide complementary views of spurious correlations through data-level disentanglement, behavior-driven partitioning, and model-informed semantic partitioning.Their coverage spans pixel-level evidence, model behavior, and contextual semantics.
  • 3.2 PURGE-H (Hybrid behavior-driven partitioning): PURGE-H combines ground-truth object presence, original predictions, and counterfactual object removal to partition reliable predictions from missed detections and hallucinations.Its hallucination cases are object-absent images predicted as containing the target object, directly capturing contextual reliance; imperfect counterfactuals may add noise.
  • 3.3 PURGE-M (Model-informed semantic partitioning): PURGE-M scores candidate contextual cues and crosses cue presence with object presence to form four semantic partitions.Object-present, low-cue samples provide object-centric supervision, while object-absent, cue-present samples capture spurious dependencies; the remaining groups are evaluated but excluded from training.

4 PURGE: Partition-aware unlearning for spurious correlation mitigation

PURGE applies partition-aware unlearning to suppress context-driven predictions while preserving object-grounded reasoning. It combines retain and forget sets with GA, KL-regularized, or NPO objectives, which trade off forgetting strength, stability, and relative suppression.

  • 4 PURGE: Partition-aware unlearning for spurious correlation mitigation: PURGE formulates unlearning over structured retain sets supporting object-grounded reasoning and forget sets exposing spurious or failure-inducing context dependence.The target behavior is Pθ′(c | z) →0 while preserving Pθ′(c | o) ≈Pθ(c | o).
  • 4 PURGE: Partition-aware unlearning for spurious correlation mitigation: The unlearning objective preserves useful object-based predictions on retain data while suppressing spurious predictions on forget data.The framework uses the same retain/forget principle across its three partitioning strategies.
  • Unlearning objectives: GA preserves retain-set predictions while reducing forget-set label likelihood, but its aggressive updates can cause instability or unintended degradation.Because the objective is minimized, its forget term decreases probability assigned to forget labels.
  • Unlearning objectives: KL-regularized unlearning suppresses forget-set predictions while keeping the updated model close to the original model on retain samples.Strong regularization can weaken forgetting when retain and forget objectives conflict.
  • Unlearning objectives: NPO penalizes forget-set predictions relative to a reference model, discouraging high-confidence forget labels while maintaining retain-set performance.Its relative penalty is controlled by β and provides a more targeted forgetting signal than absolute likelihood reduction.
  • Partition-aware unlearning: PURGE-D, PURGE-H, and PURGE-M instantiate different retain/forget labels and supervision choices for object-only, behavior-defined, and semantic partitions.PURGE-D adds explicit no supervision for background-only forget samples, whereas PURGE-M makes it redundant because all forget samples are object-absent.

5 Experimental evaluation

PURGE is evaluated across multiple multimodal models and benchmarks measuring hallucination, grounding, causal consistency, and spurious-correlation robustness. Across these settings, partition-aware variants generally improve robustness and grounding while preserving overall performance.

  • Evaluation setup: The evaluation covers LLaVA-1.6-7B, Qwen3-VL-8B-Instruct, Qwen3.5-9B, and CLIP across hallucination, grounding, causal-consistency, and spurious-correlation benchmarks.Benchmarks include CHAIR, AMBER, POPE, MMHal, Causal-HalBench, MM-SpuBench, and Waterbirds.
  • Waterbirds: 79.6 WGA: PURGE-M-NPO achieves the highest Waterbirds worst-group accuracy while maintaining competitive average accuracy on CLIP-RN50.Waterbirds evaluates binary landbird-versus-waterbird classification, with background used to characterize spurious-correlation groups.
  • Robustness across models: PURGE-M improves both Overall and worst-group accuracy over corresponding baselines across evaluated LVLMs, with especially large gains in worst-group accuracy.The evaluation uses four object-context groups on a held-out test split.
  • Causal consistency: PURGE improves Qc and Qa while reducing CAC, AAC, and CHR on Causal-HalBench, indicating more stable predictions under counterfactual changes.Lower CAC, AAC, and CHR indicate better causal robustness.
  • Category-wise hallucination analysis: PURGE variants improve MMHal scores across categories, with the largest gains from PURGE-M, especially for counting and adversarial examples involving misleading context.MMHal scores are higher when performance is better.
  • Object existence: All PURGE variants improve POPE object-existence performance, with PURGE-M-NPO generally strongest across models and other NPO variants best in several combinations.The Random split reports accuracy and F1 for object-presence prediction.
  • Ablations: Joint LoRA-plus-projector training further improves performance beyond LoRA-only updates, particularly on CHAIR.LoRA-only updates already improve MM-SpuBench accuracy and reduce CHAIR hallucination.

6 Conclusion and limitations

The paper identifies spurious object-background correlations as an important source of hallucination and proposes PURGE to diagnose and unlearn these dependencies. Across models and benchmarks, PURGE reduces hallucination and improves robustness while preserving competitive overall utility, but depends on data and detector quality and adds processing costs.

  • Conclusion: PURGE uses structured retain/forget partitions for controlled evaluation and partition-aware unlearning of spurious object-background dependencies.The framework targets shortcut reliance in vision-language models.
  • Conclusion: Across multiple LVLMs, vision-language encoders, and benchmarks, PURGE reduces hallucination and improves robustness while preserving competitive overall utility.This is the paper’s reported cross-setting conclusion.
  • Limitations: PURGE depends on mask, counterfactual-image, cue-proposal, and detector quality; effectiveness may decrease when object and background cues are highly entangled.The method also adds data-processing and fine-tuning costs.

A Training details

Training constructs structured retain and forget datasets, then applies parameter-efficient unlearning objectives to object- and context-dependent samples. The protocol preserves target labels, evaluates held-out robustness, and uses fixed configurations to compare partitioning strategies fairly.

  • Dataset construction: COCO-based retain and forget samples come exclusively from MSCOCO train2017, while COCO val2017 is held out from partition-aware training.PURGE-M uses C ∩ S_c as retain and C_c ∩ S as forget, with the same sampling procedure and budget across variants.
  • Training configuration: Training uses five epochs, bf16 precision, gradient checkpointing, AdamW, LoRA learning rate 2 × 10^-5, projector learning rate 1 × 10^-5, and LoRA rank 8.These settings provide the reported common training configuration.
  • Objective controls: The unlearning strength λ scales the forget-set contribution, β regularizes KL and NPO deviation, and γ adds negative supervision for PURGE-D object-absent samples.Hyperparameters are kept consistent across methods and models unless otherwise specified.
  • Backbone-specific adaptation: All PURGE variants freeze the visual encoder and adapt the language model with LoRA; LLaVA and Qwen3-VL additionally optimize the multimodal projector.Qwen3.5-9B uses LoRA only because its multimodal fusion is integrated.
  • Waterbirds protocol: Waterbirds retains bird class as the binary target while using background only to define four groups for partitions and worst-group evaluation.WGA is the minimum accuracy across those four bird-class-background groups.
  • Sampling: Using 100 samples per class balances coverage and feasible training time, while smaller subsets of 30–60 samples per class show consistent trends.The observation comes from preliminary experiments on dataset-size variation.
  • PURGE-D partitioning: PURGE-D-IP masks the target object to create an object-centric retain view and an inpainted context-only forget view, suppressing confidence in the latter’s yes response.Object-absent samples provide explicit no supervision.

C.2 Experimental result

PURGE-D improves hallucination and multimodal reasoning by using structured partitions and unlearning objectives that suppress spurious object-context dependencies. Performance improves with stronger forget supervision, with PURGE-M generally strongest across partition strategies.

  • CHAIR evaluation: All PURGE-D variants significantly reduce CHAIR hallucination scores across models, with PURGE-D-NPO achieving the lowest scores.Performance improves progressively from GA to KL to NPO, indicating stronger suppression of spurious object-context dependencies.
  • MMHal evaluation: PURGE-D variants improve MMHal scores across categories, with PURGE-D-NPO showing the largest gains in adversarial reasoning and counting.The improvements indicate stronger visual grounding and reduced reliance on context-only signals.
  • Forget-set ratio: Increasing the forget-set ratio consistently improves MM-SpuBench robustness, although gains vary across partition strategies.The evaluation uses forget-set ratios of 25%, 50%, 75%, and 100% while keeping the retain set fixed.
  • Forget-set ratio: PURGE-M performs best across forget-set ratios, followed by PURGE-H and PURGE-D, while gains plateau at higher ratios.PURGE-D saturates early, whereas PURGE-H improves steadily; plateauing is more pronounced for LLaVA-1.6-7B than Qwen3-VL-8B-Instruct.
  • Partition construction: Automatic cue discovery ranks candidate contextual cues by OWLv2 confidence and selects the cue with the strongest spurious association without manual selection.GPT-4 proposes 12 candidate cues per target object before normalization and filtering.
  • Partition construction: PURGE-M retains images containing the target object without its cue and forgets images containing the cue without the target object.This partition separates object evidence from contextual evidence for selective unlearning.

F Group-wise analysis and visualization of PURGE-M

Group-wise analysis shows that PURGE-M-NPO reduces context-driven false positives while improving recognition when contextual support is absent. The framework still struggles when object-level visual evidence is weak or partition quality is imperfect.

  • Group-wise analysis: Any directly diagnoses contextual shortcut reliance because the target object is absent while its associated cue remains present.A false-positive yes prediction in Any indicates reliance on contextual rather than object-specific evidence.
  • Group-wise analysis: PURGE-M-NPO raises Any accuracy from 32% to 59%, 46% to 63%, and 47% to 58% across the three evaluated backbones.These results indicate fewer context-driven false-positive predictions when object-specific evidence is absent.
  • Group-wise analysis: PURGE-M-NPO raises Ayn accuracy from 52% to 77%, 65% to 80%, and 62% to 75% across the three backbones.Ayn requires recognizing the target object without its associated contextual cue.
  • Group-wise analysis: PURGE-M-NPO outperforms PURGE-M-KL across Ayy, Ayn, Any, and Ann, indicating a stronger forgetting signal while maintaining partition-wide performance.The diagnostic uses 6,332 samples from the PURGE-M training split, separate from held-out test evaluation.
  • Visualization: The PURGE-M partitions separate joint object-cue presence, object-only presence, cue-only presence, and joint absence.For the boat example, water is the contextual cue; Ayn is retained and Any is forgotten for unlearning.
  • Failure analysis: Both baseline and PURGE-trained models fail mainly on small, distant, occluded, or visually ambiguous objects, which PURGE does not directly address.These cases reflect insufficient object-level visual evidence rather than context-driven shortcut reliance.
  • Partition verification: Independent Grounding DINO checks show that automatically constructed partitions largely satisfy their intended semantic configurations.The lower verification rate for the PURGE-H edited partition makes partition quality an explicit limitation.

I Prediction transition and regression analysis

PURGE corrects substantially more prediction errors than it introduces across transition analyses and reduces hallucinations on object-absent examples. Its gains extend across POPE splits and CHAIR evaluation, while remaining regressions cluster in visually difficult cases.

  • Transition analysis: The transition analysis compares identical evaluation samples before and after PURGE using Correct-to-Correct, Incorrect-to-Correct, Correct-to-Incorrect, and Incorrect-to-Incorrect categories.POPE and AMBER analyses use fixed 4,000-sample subsets.
  • Transition analysis: Correct-to-incorrect regressions remain below 1% across all three benchmarks, while incorrect-to-correct transitions are substantially more frequent.Remaining regressions primarily involve small, distant, heavily occluded, or ambiguous objects.
  • Object-absent evaluation: PURGE reduces context-driven false-positive predictions from 56-65% to approximately 12-23% when objects are absent but associated contextual cues remain.This controlled evaluation provides behavioral evidence of reduced contextual shortcut reliance without establishing a unique causal origin.
  • POPE evaluation: PURGE-M-NPO achieves the best overall POPE results across Random, Popular, and Adversarial splits, with larger gains under stronger bias conditions.PURGE-H-NPO often ranks second, and POPE evaluation uses official parsing across all variants.
  • CHAIR evaluation: PURGE-M-NPO reduces LLaVA-1.6-7B CHAIRs from 41.2 to 11.3 on COCO validation.The result demonstrates a substantial reduction in generated non-existent objects.

O Fine-grained evaluation on MM-SpuBench

PURGE improves MM-SpuBench accuracy across models and spurious-correlation categories, with PURGE-M and especially PURGE-M-NPO producing the strongest results. Gains are largest on bias-sensitive categories and under stronger contextual bias.

  • Overall accuracy: PURGE improves MM-SpuBench accuracy from 59.7 to 78.3 for LLaVA-1.6-7B and from 73.8 to 88.7 for Qwen3-VL-8B-Instruct.PURGE-M achieves the largest gains across evaluated models.
  • Category-wise evaluation: PURGE consistently improves performance across MM-SpuBench categories, with the largest gains in bias-sensitive metrics.PURGE-M-NPO achieves the best results across models, categories, and overall accuracy.
  • Category-wise evaluation: The category-wise results identify strong baseline bias particularly in BG and RS, where contextual over-reliance is most evident.Improvements across these categories support enhanced generalization under spurious-correlation conditions.

P Robustness and hallucination evaluation

On AMBER, PURGE reduces hallucination and improves object grounding across evaluated models. Ablations indicate that partition-aware unlearning and projector-based visual alignment contribute complementary benefits.

  • AMBER benchmark: PURGE consistently reduces hallucination while improving coverage and discriminative scores across AMBER evaluations.The framework improves both generative hallucination metrics and discriminative classification performance.
  • AMBER benchmark: On Qwen3-VL-8B-Instruct, CH decreases from 5.9 to 2.8 and Hal from 26.5 to 8.3.These changes demonstrate improved factual consistency and object grounding.
  • Ablation study: Partition-aware unlearning alone reduces hallucination relative to the baseline.The ablation attributes this improvement to removing spurious dependencies.
  • Ablation study: Adding the projector further lowers hallucination and increases coverage by improving visual-language alignment.The ablation identifies unlearning and alignment as complementary components for robust predictions.
Loading 2608.29996v1…