Source-linked AI summary
FUSED: Forensic-Semantic Mixture-of-Experts for AI Inpainting Detection and Localization
Anton Nuzhdin, Marcel Worring, Ivona Najdenkoska
TL;DR
AI inpainting detectors must localize edits while avoiding global generator artifacts that limit transfer. FUSED combines forensic and semantic streams with sparse per-token expert routing and joint segmentation supervision, achieving the strongest reported transfer performance across the evaluated benchmarks while retaining residual sensitivity to global VAE cues.
Problem
AI inpainting modifies localized regions, but detectors often rely on global artifacts that vary across generators and fail to provide reliable localization under distribution shifts.
Method
FUSED combines a trainable forensic branch and frozen semantic branch through a sparsely-gated Mixture-of-Experts with token-wise routing, jointly predicting manipulation scores and tampering masks.
Results
FUSED achieves state-of-the-art detection and localization transfer from a single training source, remaining strongest on OpenSDID, AutoSplice, CocoGlide, and under both global VAE-artifact conditions.
Takeaways & Limitations
The results support token-wise forensic-semantic fusion as a strong approach for cross-generator AI-inpainting detection and localization.
Takeaways & Limitations
Experiments focus on generation-based inpainting, and FUSED’s accuracy drop on INP-X indicates residual sensitivity to the global VAE artifact.
Abstract
from arXiv · showhide
Diffusion-based inpainting models modify only a localized part of an image, while many AI-image detectors rely on global artifacts and do not localize. These artifacts vary across generators, limiting detector transfer under distribution shifts. Recent work shows that restoring the authentic pixels outside the inpainted region removes these cues and can degrade pretrained detectors. To address this, we present FUSED, a unified framework for the joint detection and localization of AI-generated inpainting. FUSED combines low-level forensic cues with high-level semantic features using a sparsely-gated Mixture-of-Experts architecture, enabling the model to adaptively prioritize the most relevant signal for each token. For each input, FUSED predicts both an image-level manipulation score and a pixel-level mask of the inpainted area. On the OpenSDID cross-generator benchmark, FUSED achieves the best average detection and localization, with the largest gains on unseen generators. The same model transfers directly to the held-out AutoSplice and CocoGlide benchmarks, more than doubling localization performance. Evaluating each held-out benchmark with and without the global generator artifact further shows that all evaluated methods, ours included, partly read the artifact as evidence of manipulation, and FUSED remains the strongest under both conditions. Code and pretrained models are available at https://github.com/AntonNuzhdin/FUSED.
Introduction
AI inpainting detection must determine both whether an image was manipulated and where, while avoiding global generator artifacts that fail under transfer. FUSED addresses this by combining complementary forensic and semantic evidence through token-level expert routing.
- AI inpainting changes localized regions while preserving realistic surroundings, requiring both image-level detection and pixel-level localization.
- Global cues such as VAE traces can span authentic background pixels, and restoring those pixels can reduce strong detectors to chance.
- Low-level forensic cues localize accurately but generalize poorly, whereas semantic features transfer across generators but cannot identify synthesized pixels.
- FUSED routes forensic and semantic tokens through selected sparse experts, replacing whole-image fusion with token-level adaptation.
- The model jointly predicts an image-level manipulation score and tampering mask, with localization supervision extending into fusion and routing modules.
- Evaluation uses transfer from one training generator to unseen generators and directly held-out AutoSplice and CocoGlide benchmarks.
Related Work
Prior work largely separates global AI-image detection from pixel-level manipulation localization and commonly relies on low-level forensic traces or fixed feature fusion. FUSED instead applies sparse, cross-stream, per-token routing to heterogeneous forensic and semantic representations.
- AI-image detectors commonly classify images using generator fingerprints, including spatial, spectral, frequency, and reconstruction-based cues.
- Image manipulation methods produce both image-level forgery decisions and per-pixel tampering maps, often relying on low-level forensic traces.
- Existing forensic MoE methods target fully synthetic or face-manipulated imagery and typically route within a single representation or predefined partition.
- FUSED represents forensic and semantic streams as token sets, routes each token through sparse experts, and models cross-stream interactions with self-attention.
- FUSED differs from referenced MoE approaches by organizing experts end-to-end across two heterogeneous encoders using a single undivided training set.
- Shortcut-learning research documents detector reliance on dataset biases and studies interventions that remove or align such global cues.
Methodology
FUSED combines a trainable non-semantic forensic encoder with a frozen semantic backbone through sparse token routing, then decodes classification and localization outputs from fused representations. Joint losses supervise detection, boundaries, localization, routing balance, and expert diversity.
- Overview: The trainable forensic branch learns manipulation-sensitive representations, while a frozen semantic backbone supplies object-level context.
- Forensic branch: SparseViT suppresses semantic content through grouped sparse attention and aggregates multi-scale features into a fused forensic map.
- Forensic branch: Attention pooling summarizes the forensic map with m=8 learnable query latents into a compact forensic token set, with diversity regularization.
- Semantic branch: The frozen semantic branch projects its final 16 × 16 feature grid into semantic tokens, keeping trainable parameters modest and regularizing against dataset-specific artifacts.
- Mixture-of-experts fusion: FUSED concatenates forensic and semantic tokens, then routes each token to k=2 of E=8 experts before cross-branch self-attention.
- Prediction heads and segmentation bridge: A segmentation bridge concatenates the forensic map with upsampled post-fusion semantic tokens and decodes the tampering mask through a convolutional decoder.
- Training details: End-to-end training combines localization, boundary, classification, load-balancing, and diversity terms, applying localization losses only to manipulated images.
Experiments and Results
FUSED is evaluated for cross-generator detection and pixel-level localization, including unseen generators, held-out benchmarks, artifact-controlled conditions, and robustness tests. It achieves strongest overall transfer while showing that localization depends on the segmentation bridge, forensic branch, adaptive routing, and semantic-backbone capacity.
- Cross-generator evaluation: FUSED achieves the highest average OpenSDID localization, with IoU of 44.5 and pixel F1 of 55.3 versus 42.7 and 49.4 for MaskCLIP.The largest gains occur on the three most distant generators.
- Cross-generator evaluation: 11.3, 10.9, and 11.5 pixel-F1 points separate FUSED from the comparison on unseen SDXL, SD3, and Flux.1, respectively.MaskCLIP and CAT-Net remain stronger on in-domain SD1.5, while MaskCLIP has the best SD2.1 IoU at 55.5 versus 52.4.
- Cross-generator evaluation: FUSED reaches average detection F1 of 88.0 and accuracy of 89.5, compared with 77.8 and 82.0 for MaskCLIP.Its F1 margins reach 16.1 points on SD3 and 14.7 on Flux.1.
- Held-out transfer: 51.4 and 62.8 pixel F1 on AutoSplice and CocoGlide improve over CAT-Net by 18.3 and 37.2 points, respectively.FUSED also obtains the highest ROC-AUC on both held-out benchmarks, while fixed-threshold accuracy is affected by score-distribution shifts.
- Artifact reliance: FUSED remains strongest with and without the global artifact, though removing it lowers accuracy from 80.8 to 68.3.Every method above chance is more accurate when the artifact is present, so none decides fully within the manipulated region.
- Large-scale OOD transfer: +14.5 pixel-F1 points and +13.1 IoU points are FUSED's largest gains on So-Fake-OOD after heterogeneous multi-generator training.Detection gains are modest, within 1.5 points, while localization gains are substantially larger.
- Ablations: Removing the segmentation bridge reduces average localization pixel F1 from 55.3 to 36.2, while removing the forensic branch lowers it to 49.5.On in-domain SD1.5, removing the forensic branch reduces pixel F1 from 70.5 to 61.6.
- Ablations: Routing, rather than matched dense capacity, accounts for the transfer benefit: replacing sparse routing reduces unseen-generator detection F1 by 4.4 to 8.8 points.Per-token expert utilization differs between forensic and semantic branches and varies across generators.
Conclusion
FUSED unifies AI-inpainting detection and localization through token-wise sparse expert fusion and achieves strong transfer across datasets, generator families, and artifact conditions. Its remaining limitations include residual VAE-artifact sensitivity and a focus on generation-based inpainting.
- FUSED integrates forensic and semantic representations through a token-wise sparse Mixture-of-Experts, cross-stream attention, and a segmentation bridge.
- FUSED achieves state-of-the-art transfer from a single training source, with the largest gains on unseen generators and strong performance with and without the global VAE artifact.
- After multi-generator retraining, FUSED remains ahead on So-Fake-OOD, including held-out commercial generators.
- FUSED remains sensitive to the global VAE artifact, localization is harder for the smallest edits, and experiments focus on generation-based inpainting.
(Supplementary Material)
Supplementary analyses show that FUSED localizes manipulated regions more consistently across representative, fully synthetic, and small-edit examples, while retaining some spurious or fragmented components.
- A Additional Localization Examples: FUSED responds across representative OpenSDID examples and usually preserves the manipulated region’s coarse extent, though masks sometimes include neighboring areas.
- B Fully Synthetic Images: FUSED produces full-frame predictions for all five fully synthetic images, matching the correct mask when every pixel is generated.
- C Small Manipulated Regions: FUSED produces non-empty, localized predictions in all four small-edit cases, although some masks contain spurious or fragmented components.
D Performance by Manipulated-Area Fraction
Across manipulated-area bins, FUSED outperforms MaskCLIP on localization and detection. Edit size affects localization more strongly than detection, with the smallest edits localized less precisely.
- FUSED exceeds MaskCLIP in every manipulated-area bin on both localization IoU and detection recall.
- Localization IoU increases from below 0.25 for edits under 2% of the image to above 0.60 in medium and large partial-edit bins.
- Detection recall remains above approximately 0.90 and is considerably less sensitive to edit size than localization.
- Both methods decline on fully generated images, while small edits may remain detectable but be localized imprecisely.
E Qualitative Ablation Results
Qualitative ablations show that removing key FUSED components visibly degrades mask quality, especially when the segmentation bridge, SparseViT, or ConvNeXt branch is removed.
- Removing the segmentation bridge, SparseViT, or ConvNeXt branch damages predictions most, producing fragmented or displaced regions.
- The qualitative mask degradation tracks the quantitative drops observed for the ablated variants.
F.1 Detection Errors
FUSED’s remaining detection errors span varied image content rather than one dominant semantic shortcut, while the figure organizes errors by actual and predicted labels.
- F.1 Detection Errors: Qualitative ablations show the largest prediction changes after removing the segmentation bridge, SparseViT, or ConvNeXt branch.These variants compare the full model with four component-ablation configurations on OpenSDID test examples.
- F.1 Detection Errors: Figure 8 groups images by actual and predicted real/fake labels, with off-diagonal cells representing false positives and false negatives.The examples span people, sporting events, animals, isolated objects, natural scenes, and graphics.
- F.1 Detection Errors: No single content category or visual characteristic explains all detection errors in either direction.The observed error patterns are heterogeneous rather than dominated by one obvious semantic shortcut.
F.2 Localization Failures
FUSED’s zero-IoU failures occur mainly on small manipulated regions and usually reflect incorrect spatial attribution rather than empty masks.
- F.2 Localization Failures: Zero-IoU examples generally have ground-truth regions that are small relative to the image, corresponding to the lowest-area regime.Figure 9 shows ten examples, with two examples per generator.
- F.2 Localization Failures: Most failed predictions are non-empty but respond to regions elsewhere, sometimes covering an area comparable to or larger than the target.Thus, the failure is spatial misattribution rather than mask collapse.
- F.2 Localization Failures: Improving these cases requires more precise spatial grounding rather than simply stronger mask activation.This contrasts with baselines’ frequent empty-mask behavior in Figure 11.
G So-Fake-OOD Qualitative Results
Qualitative results show FUSED generally localizing manipulated regions across unseen generators and challenging image regimes, while retaining fragmentation and over-segmentation errors.
- G So-Fake-OOD Qualitative Results: On So-Fake-OOD, FUSED more often localizes manipulated regions than MaskCLIP, whose predictions are frequently empty or near-full.The comparison covers GPT-4o and Ideogram 3.0 examples.
- G So-Fake-OOD Qualitative Results: Several FUSED predictions remain fragmented or extend beyond the ground-truth region despite generally producing localized responses.These residual errors keep the task challenging under the qualitative comparison.
- G So-Fake-OOD Qualitative Results: Across ten OpenSDID examples, FUSED responds consistently and generally recovers the coarse manipulated region, although some masks are over-segmented.Each row contains the input, ground-truth mask, and predictions from all compared methods.
- G So-Fake-OOD Qualitative Results: For fully synthetic images, FUSED predicts a full-frame mask in all five examples, matching ground truth that covers the entire image.Compared baselines are less consistent in this setting.
- G So-Fake-OOD Qualitative Results: For small manipulated regions, FUSED generally localizes the target but may include additional disconnected areas.The edited region occupies only a small fraction of each image.