Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMs

Jiali Wei; Ming Fan; Mingkun Zhang; Haoyu Wang; Jun Sun; Guoheng Sun; Xiaoning Ren; Haijun Wang; Ting Liu

Source-linked AI summary

Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMs

Jiali Wei, Ming Fan, Mingkun Zhang, Haoyu Wang, Jun Sun, Guoheng Sun, Xiaoning Ren, Haijun Wang, Ting Liu

arXiv:2608.24354v1Aug 25, 2026cs.CRcs.AIcs.CL

TL;DR

MLLM backdoor defenses either transfer poorly from conventional classifiers or operate mainly at inference time without repairing poisoned weights. RACER repairs models by exploiting modality-dependent layer-wise inconsistency through region-aware normalization and min-max adversarial fine-tuning, reducing mean ASR to 1.1% and reaching 0% in 32 of 36 settings while preserving clean-task utility.

Problem
Existing model-level backdoor removal methods have limited effectiveness on MLLMs, while MLLM-specific defenses mainly filter inputs at inference time and leave poisoned weights intact.
Method
RACER separately normalizes visual and textual layer-wise inconsistency over a deep-layer window and uses min-max adversarial consistency repair with clean autoregressive training.
Results
1.1% mean ASR after repair, down from 98.6% without defense, with 0% ASR in 32 of 36 settings and clean-task utility remaining close to pre-repair levels.
Takeaways & Limitations
RACER removes diverse image, text, and multimodal backdoors while preserving clean-task utility on both backdoor and clean models.
Takeaways & Limitations
RACER requires manual backbone-specific configuration of the window start and textual weight, and evaluation focuses on 7B-scale open-source MLLMs.

Abstract

from arXiv · show

MLLMs are increasingly deployed in user-facing applications, yet they inherit backdoor risks from the pipelines used to construct them: triggers may reside in images, texts, or both. Existing model-level backdoor removal methods, largely designed for conventional classifiers, show limited effectiveness on MLLMs, while MLLM-specific defenses mainly operate at inference time, filtering suspicious inputs without removing the backdoor embedded in the model. To address this gap and eliminate latent backdoors from MLLMs at their source, we present RACER, a model-level repair framework motivated by a key observation: backdoors induce abnormal layer-to-layer evolution in internal representations, which we term the layer-wise inconsistency anomaly. Importantly, this anomaly is modality-dependent, concentrating primarily in the token region encoding the trigger features that the backdoor model actually relies on. RACER therefore decomposes the fused representation into visual and textual token regions, normalizes their layer-wise inconsistency separately, and recomposes them using modality-aware weights over a deep-layer window, yielding a region-aware inconsistency objective that better captures localized backdoor-induced anomalies. Through a min-max optimization, this objective drives worst-case perturbation synthesis and adversarial fine-tuning against the resulting perturbation to repair the model, suppressing the deep representational directional shifts on which backdoor behaviors rely. RACER requires only 100 clean samples and no knowledge of the trigger, attack objective, or even whether the input model contains a backdoor. Evaluations on three open-source MLLMs across 36 backdoor settings spanning image, text, and multimodal triggers show that RACER reduces the average ASR to 1.1%, reaching 0% in 32 settings, while preserving clean-task utility on both backdoor and clean models.

I. INTRODUCTION

RACER addresses MLLM backdoors by targeting modality-dependent layer-wise inconsistency anomalies in trigger-relevant token regions. Across diverse settings, it repairs models with strong backdoor removal while preserving clean-task utility.

Motivation: MLLM backdoors can use image, text, or multimodal triggers, making trigger type and modality difficult for defenders to anticipate.Attacks include localized image patches, blended watermarks, periodic signals, rare words, inserted sentences, and trigger combinations.
Motivation: Existing model-level defenses rely on assumptions from conventional classifiers, while MLLM-specific defenses mainly filter inputs at inference time and leave poisoned weights intact.This limits model-level removal and allows the backdoor to persist in redistributed checkpoints.
Evaluation: RACER reduces mean ASR from 98.6% without defense to 1.1% after repair and reaches 0% ASR in 32 of 36 settings.The evaluation covers three open-source MLLMs, six attacks, three trigger modalities, and two attack objectives, while clean-task utility remains close to its pre-repair level.
Key observation: Backdoors induce a layer-wise inconsistency anomaly that concentrates in the trigger-relevant token region and becomes more pronounced in deeper layers.Image triggers predominantly affect visual tokens, whereas text triggers predominantly affect textual tokens.
RACER: RACER separately normalizes visual and textual inconsistency, recombines the regions with modality-aware weights, and constrains repair over a deep-layer window.Its adversarial consistency repair uses min-max optimization without backdoor-specific information.

B. Backdoor Attacks on MLLMs

MLLM backdoors can use image, text, or multimodal triggers, while conventional model-level defenses do not readily transfer to autoregressive MLLMs. RACER instead uses layer-wise inconsistency anomalies, which localize by modality and depth, to guide repair.

MLLM backdoor triggers may reside in images, text, or both, and can manipulate open-ended generation or induce forced refusals.
Backdoors disrupt the otherwise smooth layer-to-layer evolution of hidden representations when activated, producing a layer-wise inconsistency anomaly.
The anomaly separates mainly in deeper layers, while shallow and middle layers show largely similar inconsistency patterns for clean and backdoor models.
Image-trigger anomalies concentrate in visual tokens, whereas text-trigger anomalies concentrate in textual tokens, making sequence-level aggregation potentially obscure localization.

IV. DESIGN OF RACER

RACER preserves the regional structure of backdoor-induced inconsistency and uses it in a min-max repair objective. The procedure synthesizes perturbations and fine-tunes the model against them while retaining clean-task loss.

RACER decomposes layer-wise inconsistency into visual and textual terms, normalizes each region, and recombines them before adversarial repair.
The inner maximization constructs a worst-case perturbation, while the outer minimization fine-tunes model parameters against that perturbation.
The clean sample set D, perturbation budget ε, and consistency weight α balance adversarial consistency repair against clean-task utility.

B. Region-Aware Inconsistency Objective

Sequence-level averaging can hide modality-specific inconsistency because token counts determine regional influence. RACER instead decomposes and normalizes regions separately, weights them explicitly, and focuses regularization on deeper layer pairs.

Sequence-level aggregation cannot preserve where inconsistency occurs and assigns influence according to region size rather than anomaly relevance.
RACER measures visual and textual inconsistency separately, normalizes each by its region size, and recombines them with modality-aware weights.
Within-region normalization keeps each modality term stable across different visual resolutions and instruction lengths.
The sequence-level objective is recovered when wv = |V|/|S| and wt = |T|/|S|, whereas RACER can break cross-region symmetry.
The deep-layer window concentrates regularization where backdoor-specific inconsistency is more pronounced instead of spreading it across less informative layer pairs.
The same region-aware consistency objective is maximized over perturbations and minimized over model parameters in the two optimization stages.

C. Region Identification

RACER identifies visual and textual token regions from model input structure without trigger knowledge. It then synthesizes bounded fused-embedding perturbations with fixed model parameters before adversarial fine-tuning.

For each sample, RACER derives visual and textual region masks from the model input structure without knowing the backdoor or trigger.
The worst-case perturbation is constructed in fused embedding space, allowing one optimization to explore deviations from either modality.
Projected gradient ascent generates perturbations within an ℓ∞ budget ε while model parameters remain fixed during the inner maximization.
RACER detaches the final perturbation, computes clean and adversarial losses, and updates model parameters during outer minimization.
The standard loss anchors clean-task utility, while the consistency term suppresses sensitivity to the worst-case representational deviation.

E. Mechanism of Region-Aware Consistency Repair

RACER uses regional inconsistency control to limit deep layer-to-layer directional shifts associated with backdoor activation. Its design separately constrains modalities, focuses on deep layers, and searches worst-case perturbations during repair.

The inconsistency measure quantifies directional changes between consecutive hidden states, whose accumulation across layers is bounded by adjacent-layer inconsistency.Angular distance satisfies the triangle inequality, linking local consistency control to multi-layer directional stability.
RACER evaluates accumulated directional change over a designated deep-layer window spanning layers l0 through l1 − 1.The window defines where consistency is constrained between the first and last selected layers.
Regional mean inconsistency limits the prevalence of large adjacent-layer directional changes within each modality region.Markov’s inequality bounds the fraction of positions exceeding a threshold γ, while the layer window bounds accumulated change by |W| arccos(1 −γ).
The min-max objective suppresses deep representational directional shifts associated with backdoor activation under worst-case perturbations in the ε-neighborhood.The inner maximization exposes inconsistency, while the outer repair minimizes it through model updates.
Region-aware aggregation prevents anomalies in one modality from being obscured by sequence-level aggregation over the other.The deep-layer window targets depths where backdoor-specific inconsistency is most pronounced.

V. EXPERIMENTAL EVALUATION

The evaluation tests RACER across backdoor removal, clean-model safety, component contributions, configuration sensitivity, and suppression of the layer-wise inconsistency anomaly.

RQ1 tests whether RACER removes backdoors across MLLMs, attacks, and objectives while preserving clean-task utility.
RQ2 tests whether RACER preserves clean-task utility when applied without knowing whether the received model contains a backdoor.
RQ3 evaluates each RACER component and sensitivity to key configurations using one configuration per backbone across attacks and objectives.
RQ4 tests whether RACER suppresses the layer-wise inconsistency anomaly induced by backdoor activation.

A. Experimental Setup

The experiments construct poisoned MLLMs from LLaVA-Instruct-150K and measure security and utility across clean and backdoor evaluations. Utility uses VQAv2 and COCO Captions, while security uses attack success rate.

The study randomly selects 2,000 clean samples for fine-tuning and creates poisoned training sets with a 15% poisoning rate.Evaluation uses 250 clean and 250 backdoor test samples.
Table I compares defense methods using ASR, VQA accuracy, and CIDEr across model–attack–objective settings, with lower ASR preferred.
The setup includes fine-tuning, quantization, pruning, Fine-Pruning, and LC-Uniform as comparison methods.Quantization uses training-free INT4 weight quantization, while pruning uses 50% magnitude pruning.
Security is measured by ASR, the fraction of 250 backdoor samples meeting the objective-specific malicious-content or refusal criterion.
Clean-task utility is measured with VQA accuracy and CIDEr on COCO image-question and image-caption benchmarks.

B. RQ1: Backdoor Removal and Utility Preservation

Across 36 settings, RACER achieves broad backdoor removal while retaining clean-task utility and outperforming baselines in the security–utility trade-off. It also preserves utility when repairing clean models, though some baselines trade security for benign-performance degradation.

Backdoor Removal and Utility Preservation: 1.1% mean ASR after repair, versus 98.6% undefended, with 0% ASR in 32 of 36 settings.Only four settings retain nonzero ASR, and RACER achieves the largest ASR reduction across all six attacks and both objectives.
Backdoor Removal and Utility Preservation: Utility-preserving baselines leave most text and multimodal backdoors intact, whereas pruning reduces ASR more broadly but substantially damages clean-task performance.
Backdoor Removal and Utility Preservation: RACER changes mean VQA accuracy from 70.9% to 70.7% on LLaVA-1.5-7B across its 12 evaluated settings.
Backdoor Removal and Utility Preservation: RACER remains the only method in the high-removal, high-utility region, combining a 97.5-percentage-point mean-ASR reduction with utility close to undefended models.
Backdoor Removal and Utility Preservation: The overall conclusion is consistent removal across trigger types and objectives while largely preserving clean-task utility with one configuration per MLLM.
Backdoor Removal and Utility Preservation: On clean models, RACER causes only marginal utility changes, while quantization and pruning produce much larger degradations in selected model metrics.RACER is evaluated on the clean model itself without a separate clean reference model.

D. RQ3a: Contribution of Individual Design Components

The ablations show that RACER’s main structural gains come from combining a deep-layer window with region-wise aggregation, while modality weighting further suppresses residual ASR.

Multi-Step Inner Maximization: Multi-step inner maximization alone provides limited benefit, with ASR changes of at most 5.1 percentage points and values remaining between 52.8% and 96.7%.The perturbation budget is fixed across configurations, so the later gains are not explained by stronger inner maximization alone.
Deep-Layer Window and Region-Wise Aggregation: Combining the deep-layer window and region-wise aggregation reduces malicious-injection ASR to 9.9% on Qwen2-VL-7B and 5.1% on Qwen2.5-VL-7B.Their combination consistently outperforms either component alone, reflecting complementary structural roles.
Modality Weighting: Modality weighting lowers targeted-refusal ASR from 83.5%, 48.7%, and 81.3% to 0.0%, 0.0%, and 0.1%, respectively.Malicious-injection ASR on LLaVA-1.5-7B also falls from 59.3% to 6.4%.
Sensitivity to Key Configurations: Increasing the textual weight consistently improves backdoor removal across all three MLLMs, with adopted configurations remaining within effective operating regions.At wt=3, Qwen2-VL-7B reaches 0.0% ASR under both objectives, while Qwen2.5-VL-7B reaches 0.0% and 0.1%.

F. RQ4: Effect on the Layer-Wise Inconsistency Anomaly

RACER repairs the internal anomaly associated with backdoors by narrowing deep-layer inconsistency profiles toward clean-control models, with residual separations localized to trigger-relevant regions.

Overall Anomaly Profiles: RACER substantially narrows deep-layer separation between repaired and clean-control inconsistency-gap profiles for both image-trigger and text-trigger cases.The comparison uses the same illustrative cases analyzed earlier.
Regional Decomposition: Residual separations from clean-control profiles concentrate in visual regions for image triggers and textual regions for text triggers.This regional pattern is consistent with modality-dependent anomaly localization.
Mechanistic Interpretation: The layer-wise inconsistency anomaly is modality-dependent, concentrating in trigger-relevant token regions and becoming more pronounced in deeper layers.RACER separately normalizes visual and textual regions and recomposes them with modality-aware weights over a deep-layer window.

APPENDIX A IMPLEMENTATION DETAILS

The appendix documents trigger construction, evaluation views, and qualitative and mechanistic evidence across image, text, and multimodal attacks.

Region Masks: LLaVA-1.5-7B identifies the visual token region by instrumenting input preparation to return the span of 576 visual embeddings.The span has a constant offset and length under the fixed chat template.
Attack Construction: The evaluation covers six attacks spanning image, text, and multimodal triggers, with two attack objectives.BadNets-MM combines image and text trigger components during poisoning, although either component may suffice at inference.
Trigger-Component Contributions: Under malicious injection, the BadNets-MM text component alone yields 100.0% ASR on all three MLLMs, whereas the image component alone yields 0.0%.Under targeted refusal, the dominant component differs across MLLMs.
Qualitative Example: On one Qwen2-VL-7B sample, all 12 repaired outputs return to the clean model’s correct answer across six attacks and two objectives.The repaired outputs contain neither attacker-specified injected content nor refusal responses.
Mechanistic Evidence: For a SIG image-trigger case, LC-Uniform reduces ASR from 93.6% to 8.0% while narrowing visual-region separation from the clean-control profile.This links successful suppression with reduced trigger-relevant inconsistency in that case.