Source-linked AI summary
ARISE: Authentic Reference-Informed Stability Enhancement for Continual Multimodal Forgery Detection
Yuchen Zhang, Yaxiong Wang, Yujiao Wu, Lianwei Wu, Qi Zhang, Li Zhu, Chaofeng Dong, Le Wu, Zhedong Zheng
TL;DR
Multimodal forgery detectors face evolving attacks that make static generalization and repeated retraining inadequate. ARISE uses authentic replay geometry to stabilize a detector as it learns new attacks, with bounded replay emphasis for vulnerable stages. On Fable-Bench, it achieves the highest average accuracy across all three protocols among the continual-learning methods reported.
Problem
Static generalization is fragile as new forgery mechanisms emerge, while retraining from scratch repeatedly requires reconstructing historical data and does not explicitly preserve accumulated forensic knowledge.
Source: §1 · p. 2Method
ARISE uses replayed authentic representations as a historical geometric reference, with GRASP stabilizing representation directions and VAR assigning bounded replay emphasis to vulnerable stages.
Source: §1 · p. 2Results
85.37% average accuracy on P1, with ARISE outperforming top competitor LPR by 1.50 percentage points; ARISE has the highest average accuracy on all three protocols.
Source: §4.1 · p. 7Takeaways & Limitations
Across Fable-Bench, two public benchmarks, and multiple backbones, experiments show consistent improvements in final performance, retention, and stability–plasticity balance, with no inference-time computation added.
Source: §5 · p. 8The setting assumes a bounded replay memory and unavailable complete historical training data.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Multimodal forgery detection is commonly studied under a static assumption: training attacks are expected to approximate those encountered after deployment. This assumption becomes increasingly fragile as generators, editing systems, and multimodal large language models continually introduce new forgery mechanisms and forensic evidence. We formulate this problem as attack-incremental continual multimodal forgery detection, where the binary real/fake interface remains fixed while emerging attacks must be learned without forgetting previous ones. We introduce Fable-Bench, which organizes attack evolution through complementary joint-modality, text-side, and image-side protocols. We further propose ARISE (Authentic Reference-Informed Stability Enhancement), motivated by the open-ended forged class and the comparatively reusable geometry of authentic content. ARISE fits a real-only Gaussian mixture model to projected authentic replay representations as a compact historical reference. Meanwhile, the proposed Gaussian-Responsibility-Aware Stability Preservation (GRASP) derives a sample-adaptive anisotropic metric that strongly constrains historically stable representation directions while retaining plasticity elsewhere. Our Vulnerability-Aware Replay (VAR) further estimates stage-wise vulnerability from authentic-density degradation, forged-to-authentic boundary confusion, and authentic-mode occupancy drift, then assigns bounded replay emphasis to fragile stages. All adaptive signals are obtained from training-time replay statistics, while inference uses only the original multimodal backbone and binary classification head. Experiments across three attack-evolution protocols and multiple detector architectures demonstrate improved historical retention and final average performance over representative continual-learning baselines.
1 Introduction
As forgery mechanisms evolve, static generalization and repeated retraining are inadequate deployment strategies. The paper frames continual learning with a fixed real/fake interface and introduces Fable-Bench and ARISE to acquire new attack evidence while retaining prior forensic knowledge.
1 Introduction
- Persuasive fabricated image–text news can combine manipulated images, misleading captions, and machine-written narratives, with plausible pairs spreading faster than manual verification.The passage situates this risk in news and social media.Source: §1 · p. 1
- Continual adaptation must learn emerging attacks while retaining prior capabilities, because a frozen detector degrades as unseen mechanisms emerge and retraining from scratch is operationally unattractive.The label space stays fixed while forged distributions and the evidence separating forged from authentic content evolve.Source: §1 · p. 2
- Fable-Bench organizes attack evolution into joint-modality, text-side, and image-side protocols under a fixed binary label space.A single detector sequentially learns emerging forensic evidence while retaining competence on previously observed attacks.Source: §1 · p. 2
- ARISE uses replayed authentic samples as a compact geometric reference, rather than modeling all historical forgeries, to preserve stable forensic knowledge across attack stages.GRASP uses a real-only diagonal Gaussian mixture over projected authentic replay representations to constrain stable directions while preserving plasticity along naturally variable ones.Source: §1 · p. 2
- VAR estimates stage-wise vulnerability from authentic-geometry drift, forged-to-authentic boundary confusion, and authentic-mode occupancy drift, then assigns bounded weights for class-balanced replay.Adaptive signals come from training-time replay statistics; auxiliary components are not used at inference.Source: §1 · p. 2
2 Related Work
Prior multimodal misinformation work progresses from event-level and out-of-context detection toward fine-grained manipulation reasoning, while continual-learning research addresses forgetting through several method families. The paper distinguishes attack-incremental multimodal forensics by its fixed binary labels and evolving forged distributions.
2.1 Multimodal Misinformation Detection
- Earlier multimodal misinformation methods use event-invariant representations, user comments, or out-of-context image–text detection; later work targets grounded, aligned, coordinated, and forensic reasoning.These methods generally assume joint access to attack distributions during training.Source: §2.1 · p. 2
2.2 Continual Learning
- Figure 2 depicts Fable-Bench’s 821K image–text pairs across 30+ attack mechanisms and its joint-modality, text-side, and image-side evolution protocols.The protocols provide complementary forms of attack evolution.Source: Figure 2 · p. 3
- Continual learning includes architecture-based, replay, and regularization methods, but most benchmarks use class-, task-, or generic domain-incremental classification rather than fixed-label multimodal forensics.Here, forged distributions and discriminative evidence evolve while the real/fake label space remains unchanged; ARISE combines bounded replay with authentic-reference-informed stabilization.Source: §2.2 · pp. 2–3
3 Methodology
The methodology frames forgery detection as continual learning with fixed real/fake labels and evolving attacks, then introduces Fable-Bench and ARISE to evaluate and address that setting. ARISE combines authentic-reference-based representation stabilization with vulnerability-weighted replay, using auxiliary components only during training.
3.1 Problem Setting
- The attack-incremental setting keeps binary labels fixed while new attack distributions arrive sequentially, with bounded replay memory and evaluation across observed stages.The complete historical training data are unavailable, and evaluation does not use stage identity.Source: §3.1 · p. 3
3.2 Fable-Bench
- Fable-Bench evaluates evolving attacks through joint-modality, text-side, and image-side protocols, spanning 821,387 image–text pairs and over 30 manipulation models and mechanisms.The detector retains the same real/fake semantics and is evaluated on previously observed attacks after each update.Source: §3.2 · p. 3
3.3 Arise: Authentic Reference-Informed Stability Enhancement
- ARISE freezes the previous detector and fits a stage-fixed Gaussian reference to projected historical authentic replay, which supplies shared historical information to GRASP and VAR.The reference uses mixture weights, means, and diagonal variances and remains fixed throughout the stage.Source: §3.3 · p. 4
3.4 GRASP: Gaussian-Responsibility-Aware Stability Preservation
- GRASP uses sample-specific Gaussian responsibilities to combine local authentic-distribution precisions, constraining stable representation directions while preserving flexibility along variable ones.Its preservation metric is estimated solely from authentic geometry, a comparatively reusable reference as forgery patterns evolve.Source: §3.4 · pp. 4–5
3.5 VAR: Vulnerability-Aware Replay
- ARISE uses authentic-density degradation, forged-boundary confusion, and authentic-mode drift to estimate stage vulnerability, then gives fragile stages bounded replay emphasis.Above-average-vulnerability stages receive stronger replay supervision, while bounded weights prevent one stage from dominating; VAR reweights replay classification loss without changing GRASP or memory replacement.Source: §3.5 · pp. 5–6
3.6 Training and Inference
- During incremental training, replay passes through current and frozen detectors for VAR and GRASP signals, but inference uses only the multimodal backbone and binary classification head.At the initial stage, only the current-stage classification loss is optimized; auxiliary components and replay memory are excluded at inference.Source: §3.6 · p. 6
4 Experiments
Experiments evaluate ARISE on Fable-Bench, public misinformation benchmarks, and alternative backbones, alongside analyses of its stability–plasticity balance and component ablations.
4.1 Main Results on Fable-Bench
- ARISE achieves the highest average accuracy among continual-learning methods on all three Fable-Bench protocols: 85.37% on P1, 81.22% on P2, and 90.47% on P3.These results exceed top competitor LPR by 1.50, 1.87, and 1.45 percentage points, respectively.Source: §4.1 · p. 7
4.2 Generalization Evaluation
- On Weibo, ARISE obtains 96.54% Accuracy and 96.46% Macro-F1; on FineFake, it obtains 77.21% Accuracy and 71.63% Macro-F1.It also improves over standard experience replay by 2.73–6.78 Accuracy points and 3.57–6.62 Macro-F1 points across protocols with CLIP and ViLT backbones.Source: §4.2 · p. 7, Table 2 · p. 7, Figure 4 · p. 8
4.3 Stability–Plasticity Analysis
- The stability–plasticity analysis examines whether ARISE can acquire emerging attacks while preserving performance on previously observed attacks.Figure 4a presents the trade-off across continual-learning methods, with bubble size representing parameter count.Source: §4.3 · p. 7, Figure 4 · p. 8
4.4 Ablation Studies
- Adding GRASP to experience replay raises average Accuracy across P1–P3 by 3.26 points, VAR by 2.20 points, and their combination by 4.71 points.The combined gains support complementary, rather than interchangeable, contributions from representation anchoring and vulnerability-aware replay.Source: §4.4 · p. 8, Table 3 · p. 8
- Full GRASP attains 91.11% Plasticity, 92.47% Stability, and 84.24% average Accuracy, the highest values for these metrics among the tested designs.The comparison evaluates component-specific anisotropic precision and responsibility-aware anchoring against alternatives including uniform precision, global precision, and hard assignment.Source: §4.4 · p. 8, Table 3 · p. 8
- Combining authentic-density degradation, boundary confusion, and authentic-mode occupancy drift lowers Average Forgetting from 11.48% to 8.67% and raises average Accuracy to 83.18%.Plasticity decreases by only 0.28 points; geometry signals reduce forgetting more effectively than boundary confusion alone, while the boundary signal better preserves Plasticity.Source: §4.4 · p. 8, Table 3 · p. 8
5 Conclusion
The conclusion presents attack-incremental continual multimodal forgery detection as a fixed-label setting in which detectors learn emerging attacks while retaining prior forensic knowledge. Fable-Bench and ARISE support this approach through complementary attack-evolution protocols, authentic-reference stabilization, and vulnerable-stage replay emphasis.
5 Conclusion
- The paper introduces fixed-label attack-incremental detection, Fable-Bench’s joint-modality, text-side, and image-side protocols, and ARISE with GRASP stabilization and VAR replay emphasis.Experiments across Fable-Bench, two public benchmarks, and multiple backbones show consistent improvements in final performance, retention, and the stability–plasticity balance; auxiliary components add no inference-time computation.Source: §5 · p. 8