Source-linked AI summary
DINO-FIL: Multi-Layer Transformer Fusion for Generative Inpainting Forgery Localization
Simon Freitter, David Fischinger, Silvia Poletti, Marcel Hasenbalg, Martin Boyer
TL;DR
Generative inpainting can make edited regions difficult to localize because synthesized content may blend into image context and obscure local traces. DINO-FIL fuses multiple DINOv3 feature depths for pixel-level localization, achieving strong cross-dataset results against existing baselines.
Problem
Generative inpainting can blend synthesized regions into their context, and fully regenerated images may lack clear local traces for pixel-level localization.
Source: §1 · pp. 1–2Method
DINO-FIL combines intermediate representations from a pretrained DINOv3 ViT-H+/16 encoder with a lightweight multi-layer fusion decoder for dense inpainting localization.
Source: §1 · p. 2, §6 · p. 13Results
Across five benchmarks, DINO-FIL has the highest average IoU and F1 versus DinoLizer and TruFor, leads on four datasets, and maintains a high true-negative rate on pristine SAGI-D images.
Source: §6 · p. 14Takeaways & Limitations
Multi-depth representations let DINO-FIL combine local appearance, texture-sensitive cues, and deeper semantic information for pixel-level inpainting localization.
Source: §6 · p. 13In the evaluated fixed-resolution setup, 1024×1024 training added no benefit, so the interaction of training resolution, test resolution, and dataset image statistics needs further investigation.
Source: §5.4 · p. 13, §6.1 · p. 14
Abstract
from arXiv · showhide
We propose DINO-FIL, a transformer-based framework for pixel-level localization of generative image forgeries. The method leverages multi-layer representations from a pretrained DINOv3 ViT-H+/16 encoder combined with a lightweight fusion decoder to capture both semantic structure and fine-grained texture inconsistencies introduced by image editing. By aggregating intermediate transformer features across depths, the model improves sensitivity to subtle, spatially localized artifacts that are often missed by standard single-layer approaches. We evaluate DINO-FIL in a cross-dataset setting, training on B-Free and testing on Beyond the Brush, CocoGlide, TGIF, and SAGI-D. The model achieves strong generalization, outperforming existing baselines on average and on four of five evaluated datasets, while maintaining a high true negative rate on pristine images.
1 Introduction
Generative inpainting can blend synthesized regions into an image, making pixel-level localization difficult when clear forensic traces are absent. DINO-FIL addresses this with multi-depth DINOv3 features and reports strong cross-dataset results.
1 Introduction
- Fully regenerated edits can alter pixels across an image while preserving background semantics, potentially removing local forensic traces needed to locate inpainting.Generative editing can synthesize content within a mask and blend it into the surrounding context, making visual inspection alone difficult.Source: §1 · pp. 1–2
- Existing DINO-based localization methods typically use final-layer features, although intermediate transformer states offer local, structural, and semantic information for dense prediction.The introduction motivates multi-level features because forensic methods often rely on low-level artifacts while vision foundation models encode higher-level structure.Source: §1 · p. 2
- DINO-FIL fuses hidden states from multiple DINOv3 depths through a lightweight decoder to combine local appearance, texture-sensitive representations, and semantic context.Its fusion hierarchy is transformer depth rather than spatial scale because the selected hidden states share a patch-grid resolution.Source: §1 · p. 2
- Partial fine-tuning of later DINOv3 layers improves localization, while full encoder fine-tuning provides limited additional benefit.The contribution also reports that high-resolution inference substantially improves localization while maintaining low false-positive rates on pristine images.Source: §1 · p. 3
- DINO-FIL outperforms strong forensic baselines in cross-dataset evaluation, while high-resolution inference improves localization and pristine images retain low false-positive rates.The evaluation is described as cross-dataset; the supplied passages do not provide the individual dataset scores.Source: §1 · p. 3
2 Related Work
Prior forensic models and DINO-based approaches provide relevant backbones and localized representations, but differ in task focus and feature use. DINO-FIL distinguishes itself through multi-depth features and targeted encoder adaptation.
2.1 Pre-trained Vision Transformers in Image Forensics
- TruFor targets broad image manipulation localization, whereas DinoLizer uses DINOv2 patch embeddings to segment localized forgery artifacts for inpainting.DinoLizer processes overlapping crops sequentially and aggregates their logit maps into a heatmap.Source: §2.1 · p. 3
- Yu et al. apply LoRA to a frozen DINOv3 backbone with a lightweight decoder, targeting diverse forensic manipulations rather than generative inpainting alone.The passage describes their approach as a baseline for broad image forensics.Source: §2.1 · p. 3
2.2 Architectural Distinctions in DINO-FIL
- Unlike DinoLizer and Yu et al., DINO-FIL extracts intermediate features at layers 8, 16, 24, and 32 to capture texture boundaries and semantic inconsistencies.The compared methods rely exclusively on final-layer DINO features.Source: §2.2 · p. 3
- DINO-FIL investigates full and later-layer-only fine-tuning and uses separate learning rates for the pretrained encoder and newly initialized decoder.This strategy is described as preserving foundational backbone features while allowing decoder specialization.Source: §2.2 · p. 4
3 Methodology
DINO-FIL localizes inpainted regions by combining features from four depths of a pretrained DINOv3 encoder. A context-enhanced, depth-wise top-down decoder produces patch logits that are upsampled into a pixel-level probability mask.
3.1 Model Architecture
- DINO-FIL extracts hidden states from layers 8, 16, 24, and 32 to provide the decoder representations from different transformer depths.The earlier states can retain local appearance and spatial detail, intermediate states can provide texture- and pattern-sensitive cues, and deeper states can contribute semantic context.Source: §3.1 · p. 4
- For each selected DINO layer (8, 16, 24, 32), the model discards one CLS token and four register tokens, then reshapes the remaining spatial patch tokens into a patch-grid feature map.Each feature map is independently projected from 1280 backbone channels to 1024 decoder channels.Source: §3.1 · pp. 4–5
- An ASPP-inspired context block enriches each projected feature map with three parallel depthwise-separable 3 × 3 convolution branches at dilation rates 1, 2, and 4.The branch outputs are summed and fused with a 1 × 1 convolution to capture cues at multiple spatial scales.Source: §3.1 · pp. 5–6
- A depth-wise top-down decoder merges the context-enhanced features from deepest to shallowest, even though all selected layers share the same patch-grid resolution.The decoder progressively adds shallower features and refines each merged representation; spatial resizing between fusion stages is unnecessary.Source: §3.1 · p. 6
- A 1 × 1 prediction head produces patch-level logits that are bilinearly upsampled and passed through a sigmoid to form the inpainting probability mask.The predicted mask is supervised with the binary inpainting ground-truth mask using the segmentation loss described in Section 4.2.Source: §3.1 · p. 6
4 Experimental Setup
The experiments train DINO-FIL on B-Free and evaluate it on B-Free and four other inpainting datasets, with additional tests of encoder tuning and image resolution. The setup also specifies training, pixel-level metrics, pristine-image evaluation, and the protocol for comparisons with existing detectors.
4.1 Datasets
- DINO-FIL is trained on B-Free and evaluated on B-Free, Beyond the Brush, CocoGlide, TGIF, and SAGI-D to measure in-domain performance and cross-dataset generalization.The benchmark evaluation uses 10,000 B-Free test images and 28,835 test images overall.Source: §4.1 · pp. 6–7
4.2 Training Protocol
- Training uses binary cross-entropy with logits plus Dice loss, with 512 × 512 inputs unless otherwise stated and augmentations including flips, color jitter, and B-Free-style perturbations.The decoder learning rate is 1 × 10−4, the unfrozen-encoder learning rate is 5 × 10−6, batch size is 12, and decoder dropout rate is 0.05.Source: §4.2 · p. 7
4.3 Encoder Fine-Tuning Ablation
- The encoder fine-tuning ablation compares 0, 6, 12, 16, 24, and 32 unfrozen final transformer layers, using an encoder learning rate of 5 × 10−6.All variants are trained for two epochs on B-Free.Source: §4.3 · p. 7
4.4 Test Resolution Evaluation
- Test-resolution evaluation compares models trained at 512 × 512 when tested at 512 × 512 and 1024 × 1024.The comparison is motivated by diverse native resolutions in test datasets and whether finer evidence is available beyond 512 × 512.Source: §4.4 · p. 8
4.5 Training Resolution Evaluation
- The training-resolution experiment compares models trained at 512 × 512 and 1024 × 1024 using SAGI-D, which spans a broad range of image resolutions.B-Free and CocoGlide are excluded because their fixed low resolutions offer no additional image information when upsampled; the models are trained for two epochs.Source: §4.5 · p. 8
4.6 Evaluation Metrics
- Manipulated-image localization is evaluated with per-image IoU and F1 scores after binarizing predicted maps at a fixed threshold of 0.5, then macro-averaging across images.Each image contributes equally regardless of manipulated-region size, and the separately averaged mean IoU and mean F1 need not rank models identically.Source: §4.6 · pp. 8–9
4.7 Comparison with Existing Detectors
- Pristine-image evaluation reports pixel-level false-positive rate and image-level true-negative rate, while detector comparisons use reported DinoLizer and TruFor results.The baselines are compared without externally downsampling test images; the reported SAGI-SP and SAGI-FR scores are combined into SAGI-D with an image-count-weighted average.Source: §4.6 · p. 9, §4.7 · p. 9
5 Results
DINO-FIL achieves the strongest average cross-dataset localization performance, while 1024×1024 inference improves results and pristine-image false positives remain low. Partial fine-tuning of later DINOv3 layers is sufficient, and higher-resolution training adds no benefit in the tested setup.
5.1 Encoder Fine-Tuning and Test Resolution
- 0.61/0.67 to 0.71/0.77: average IoU/F1 rises when test resolution increases from 512×512 to 1024×1024.Beyond the Brush, TGIF, and SAGI-D improve; CocoGlide’s small increase is cautious to interpret because its 256×256 images are only upsampled, while B-Free performs slightly better at 512×512.Source: §5.1 · pp. 10–11
- 0.71/0.77 average IoU/F1 is reached with 24 unfrozen layers at 1024×1024; unfreezing all 32 layers provides no clear additional benefit.At 1024×1024, unfreezing later layers improves performance up to 24 layers; the average F1 slightly decreases from 0.774 to 0.771 with all 32 unfrozen.Source: §5.1 · pp. 10–11, Table 1 · p. 10
5.2 Comparison with Existing Detectors
- 0.71/0.77 average IoU/F1: DINO-FIL outperforms DinoLizer (0.49/0.61) and TruFor (0.36/0.43), leading on four of five datasets but not TGIF.On TGIF, TruFor scores 0.70/0.80 and DINO-FIL scores 0.57/0.63.Source: §5.2 · p. 11, Table 2 · p. 11
5.3 False Positives on Pristine Images
- 0.0032 pixel-level false-positive rate and 0.9719 image-level true-negative rate are reported across 9,081 pristine images.True-negative rates are 97.67% on COCO, 96.84% on Open Images, and 98.10% on RAISE; this does not establish spatial accuracy on manipulated images.Source: §5.3 · p. 12, Table 3 · p. 12
5.4 Effect of Training Resolution
- 0.51/0.56 to 0.71/0.78 average IoU/F1: a SAGI-D-trained model improves with 1024×1024 inference, but 1024×1024 training and testing scores 0.68/0.74.Under this fixed-resolution setup, high-resolution training does not add benefit; the authors note that the relationship between training resolution, test resolution, and dataset image statistics needs further investigation.Source: §5.4 · p. 13, Table 4 · p. 13
6 Conclusion
DINO-FIL combines intermediate DINOv3 representations in a lightweight fusion decoder for pixel-level inpainting localization, with strong cross-dataset results. The conclusion highlights high-resolution inference and identifies resolution strategy and parameter-efficient adaptation as future directions.
6 Conclusion
- Multi-layer DINOv3 features are fused to combine local appearance cues, texture-sensitive intermediate representations, and deeper semantic information for pixel-level localization.The decoder uses projection, context aggregation, and top-down refinement rather than relying only on the final transformer output.Source: §6 · p. 13
- DINO-FIL achieves the highest average IoU and F1 against DinoLizer and TruFor, with best performance on four of five datasets and a high image-level true-negative rate on pristine SAGI-D.The model is trained on B-Free and evaluated across five inpainting localization benchmarks.Source: §6 · p. 14
- High-resolution inference improves localization, but training at 1024 × 1024 gives slightly lower average performance than training at 512 × 512 and evaluating at 1024 × 1024.The reported benefit is especially apparent on datasets with larger or more diverse native image resolutions.Source: §6 · p. 14
6.1 Future Work
- The interaction between training resolution, test resolution, and dataset image statistics remains unresolved, motivating investigation of adaptive-resolution training and evaluation.One proposed approach groups images by native resolution and uses higher target resolutions when sufficient high-resolution evidence is available; targets should be multiples of 16.Source: §6.1 · p. 14
- LoRA is proposed as a future backbone-adaptation direction that keeps pretrained weights frozen while adding trainable low-rank updates.The paper suggests integrating LoRA with multi-layer feature fusion; prior work is described as suggesting possible performance improvement over full fine-tuning [14].Source: §6.1 · p. 14