PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Before You Judge: Training-Free Region Mining for Grounded and Explainable Deepfake Detection

Chia-Ling Chen, Yu-Ting Ta, Jian-Yu Jiang-Lin, Tai-Ming Huang, Ling Lo, Po-Ching Chen, Yan-Tsung Wang, Pei-Heng Li, Ling Zou, Hong-Han Shuai, Wen-Huang Cheng

arXiv:2609.35536v1cs.CV

TL;DR

MLLM explanations may not reflect visual evidence, and forensic cues are subtle and localized. Look Before You Judge mines candidate regions through blur-contrastive attention, inspects them locally, and combines observations with global context. Across five open-source MLLMs and two benchmarks, it consistently improves detection accuracy and explanation grounding.

  • Problem

    MLLMs can produce plausible explanations that do not faithfully reflect visual evidence, while forensic cues are weak and spatially localized.

    Source: Introduction · p. 1

  • Method

    Look Before You Judge contrasts decoder-to-visual attention for original and blurred images to identify candidate regions, inspects them individually, and integrates local evidence with global context before judgment.

    Source: Introduction · p. 2

  • Results

    Across five open-source MLLMs and two benchmarks, the framework demonstrates consistent improvements in detection accuracy and explanation grounding.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    The results support identifying potential evidence locations and examining them locally before reaching a global forensic conclusion.

    Source: Conclusion · p. 7

    The framework requires whitebox access to decoder attention, limiting its applicability to open-weight MLLMs.

    Source: A. Discussion and Limitations · p. 10

Abstract

from arXiv · show

Multimodal large language models (MLLMs) can explain deepfake verdicts in natural language, but such explanations are not necessarily visually grounded in the visual evidence underlying the prediction. A model may describe plausible artifacts inferred from language priors rather than from image evidence. Existing grounding methods improve visual reliance through decoding or attention interventions, but they generally strengthen grounding over the entire image, making them ill-suited for forensic artifacts that are subtle, spatially localized, and image-dependent. We propose Look Before You Judge, a training-free framework that formulates explainable deepfake detection as a sequential evidence acquisition process. Instead of directly predicting image authenticity from holistic visual reasoning, our framework first identifies image-specific candidate evidence regions by contrasting the MLLM's decoder-to-visual attention between an original image and its Gaussian-blurred counterpart. The identified regions are then inspected individually, and the resulting local evidence is integrated with the global image context before reaching a final verdict. The framework operates without manipulation masks, external forensic models, or parameter updates, making it directly applicable to off-the-shelf MLLMs. Across five open-source MLLMs on TriDF and MMTD-Set, our framework improves detection accuracy by up to 12.8%, reduces CHAIR by up to 33.4% and hallucination rate by up to 21.3%, and outperforms representative training-free decoding and attention methods.

Introduction

MLLM forensic explanations can be plausible yet unfaithful to image evidence, while subtle, localized artifacts are difficult to acquire through holistic reasoning. Look Before You Judge makes evidence discovery and local inspection precede global judgment.

Introduction

  • A general MLLM may produce a plausible forensic explanation while overlooking manipulation artifacts and making an incorrect verdict.The paper identifies acquiring image-specific evidence before judgment as the central challenge.Source: Introduction · p. 1
  • Forensic cues are weak and spatially localized, so semantically dominant image content can overwhelm them during holistic reasoning.Useful cues may appear around facial structures in DeepFakes or around edited objects and regenerated regions in broader AIGC images.Source: Introduction · pp. 1–2
  • The proposed look-before-you-judge formulation separates region discovery, local evidence inspection, and final image-level judgment.It treats pre-decision evidence acquisition as an explicit stage rather than a by-product of explanation generation.Source: Introduction · p. 2
  • The framework contrasts decoder-to-visual attention for original and blurred images to mine candidate regions, then inspects them individually and combines local observations with global context.It requires no manipulation masks, predefined facial priors, external forensic encoders, learned localization modules, or task-specific training.Source: Introduction · p. 2
  • Across five open-source MLLM backbones and two benchmarks, experiments show consistent gains in detection accuracy and explanation grounding.The introduction reports improvements across both evaluation dimensions without specifying their magnitude here.Source: Introduction · p. 2

Related Work

Prior detection and explanation systems, whole-image grounding methods, and supervised localization methods address different parts of forensic reasoning. The paper positions its approach as bridging these gaps through automatic, image-specific region discovery without localization supervision or auxiliary modules.

From Detection to Local Evidence Inspection

  • Earlier detectors provide image-level verdicts, while MLLM forensic systems add explanations or localization, commonly learning them jointly with decisions from annotated masks, artifact labels, or curated explanations.The cited examples include FaceForensics++ and DFDC (Dolhansky et al. 2020), and FakeShield, SIDA, and LEGION.Source: From Detection to Local Evidence… · p. 2

Visual Grounding and Forensic Localization

  • Training-free grounding interventions generally act over the whole image, while ControlMLLM assumes regions of interest are specified in advance.The cited grounding methods are Lu et al. 2026; Basile et al. 2026; Tu et al. 2026; Huang et al. 2024; Leng et al. 2024; ControlMLLM is Wu et al. 2024.Source: Visual Grounding and Forensic… · p. 2
  • Localized forensic methods use frequency priors, saliency, or dedicated networks; LAA-Net remains training-dependent.Its localization is governed by task-specific objectives and the artifact distributions represented in training data.Source: Visual Grounding and Forensic… · pp. 2–3
  • The proposed method bridges training-free grounding and supervised localization by deriving image-specific forensic regions from inference-time attention, without localization supervision, auxiliary modules, or parameter updates.The overview describes blur-sensitive attention contrast, candidate-region construction, and region-wise inspection with evidence aggregation.Source: Visual Grounding and Forensic… · p. 3, Figure 2 · p. 3

Method

The framework mines image-specific, blur-sensitive visual evidence, groups it into regions for separate inspection, then filters and verifies those observations against the full image before producing a verdict.

Method

Stage 1: Blur-Sensitive Visual Prior Mining

  • Stage 1 contrasts output-to-visual attention for an image and its Gaussian-blurred counterpart, using fixed response tokens to identify detail-sensitive evidence and score spatially reliable heads.The selected heads’ positive raw–blur attention changes form a token-level prior; head scoring favors directional change, spatial redistribution, and responses not dominated by borders or a single token.Source: Stage 1: Blur-Sensitive Visual Prior… · pp. 3–4

Stage 2: Local Forensic Region Construction

  • Stage 2 maps the token prior onto the backbone’s visual grid, groups adjacent high-response tokens into components, and ranks retained regions by their prior mass.The method expands surviving components slightly for local context and retains their normalized steering distributions for inspection.Source: Stage 2: Local Forensic Region… · p. 4

Stage 3: Region-Wise Evidence Inspection and Global Aggregation

Experiments

The experiments assess the framework across MLLM backbones and benchmarks, compare it with training-free grounding methods, and examine its components and qualitative grounding. Results report consistent improvements in detection and explanation grounding, with qualitative examples showing locally inspected evidence contributing to a corrected verdict.

Experimental Setup

  • Evaluation uses TriDF Type-B and MMTD-Set with five off-the-shelf MLLMs in a fully training-free setting and the same three-stage inference procedure across backbones.TriDF evaluation includes ACC, Cover, CHAIR, Hal, and F 0.5; MMTD-Set reports ACC and F1 on DeepFake and AIGC-Editing subsets.Source: Experimental Setup · p. 5

Overall Quantitative Comparison

  • The framework improves image-level accuracy and reduces CHAIR and hallucination rates across five MLLM backbones, while improving ACC and F1 across nearly all MMTD-Set model–subset combinations.For InternVL-3.5-8B, accuracy rises from 0.418 to 0.546 and CHAIR falls from 0.975 to 0.641.Source: Overall Quantitative Comparison · p. 6

Comparison with Training-Free Visual Grounding

  • On TriDF with InternVL-3.5-8B, the framework improves all five metrics and reduces CHAIR from 0.9745 to 0.6407, unlike VCD and AttnReal’s marginal gains and nearly unchanged hallucination metrics.The reported results indicate that whole-image visual-reliance interventions are insufficient, while spatially selective discovery and inspection precede the final judgment.Source: Comparison with Training-Free Visual… · p. 6, Table 2 · p. 6

Ablation Study

  • The full framework performs best overall; prompt-only prompting leaves hallucination metrics largely unchanged, raw attention underperforms prompt-only, and removing component-wise steering weakens focused inspection.The ablation evaluates the procedure, blur-contrastive region mining, and connected-component steering on TriDF using InternVL-3.5-8B.Source: Ablation Study · pp. 6–7, Table 3 · p. 6

Qualitative Spatial Grounding

  • On fake MMTD-Set images, mined candidate regions frequently cover manipulated facial boundaries and locally edited structures despite no localization training or ground-truth masks at inference.Figure 3 depicts candidate regions aligning with annotated tampered areas without localization supervision.Source: Figure 3 · p. 6, Qualitative Spatial Grounding · p. 7

Qualitative Analysis

  • In the illustrated case, region-wise inspection finds blending, abnormal smoothness, and texture inconsistencies around the eyes, facial boundary, and jawline, supporting a correct manipulated verdict.Vanilla inference instead overlooks the annotated artifacts and predicts the image as authentic.Source: Qualitative Analysis · p. 7, Figure 4 · p. 7

Conclusion

Look Before You Judge frames explainable deepfake detection as sequential evidence acquisition rather than single-pass prediction. Its training-free framework mines detail-sensitive regions, inspects them individually, and integrates local evidence into a final image-level judgment; experiments report consistent gains in detection accuracy and explanation grounding.

Conclusion

  • Look Before You Judge performs inference-time region discovery, individual inspection, and local-evidence integration on frozen MLLMs without localization supervision, auxiliary modules, or parameter updates.The framework identifies detail-sensitive candidate regions from blur-contrastive attention responses.Source: Conclusion · p. 7
  • Across five open-source MLLMs and two benchmarks, experiments show consistent improvements in detection accuracy and explanation grounding.The paper presents these results, ablations, and qualitative analyses as support for identifying and examining potential evidence before reaching a global conclusion.Source: Conclusion · p. 7

Supplementary Material

The supplementary material covers limitations, evaluation protocols, alternative region sources, qualitative results, implementation, runtime, and prompt templates.

Supplementary Material

  • The supplement includes a discussion of the approach’s boundary and definitions of the TriDF and MMTD-Set evaluation metrics.The metric protocols cover artifact-level Cover, CHAIR, Hal, and F 0.5 on TriDF and ACC/F1 on MMTD-Set.Source: Supplementary Material · p. 10
  • Additional experiments examine language-level guidance and fixed landmark-defined facial components as alternatives to the proposed region source.The landmark-defined components correspond to the eyes, nose, and mouth, with Stage 3 unchanged.Source: Supplementary Material · p. 10
  • The supplement provides additional side-by-side qualitative comparisons, implementation and hyperparameter details, runtime measurements, and Stage-3 prompt templates.Runtime is quantified over 100 images using model invocations, peak GPU memory, and end-to-end inference time per image.Source: Supplementary Material · p. 10

A. Discussion and Limitations

The discussion treats mined regions as evidence probes rather than causal attributions and notes design boundaries; the evaluation defines artifact-grounding metrics for TriDF and classification metrics for MMTD-Set.

A. Discussion and Limitations

  • The selected regions are probes for evidence-sensitive attention, not causal attributions or necessarily the sole causes of the final prediction.The authors describe the blur-contrastive prior as a proposal mechanism for acquiring evidence.Source: A. Discussion and Limitations · p. 10
  • Blur-contrastive proposals favor high-frequency-sensitive artifacts, require white-box decoder attention, and may degrade with low-frequency inconsistencies or lost fine detail.The authors identify severe compression, downsampling, and coarse visual tokens as conditions that may reduce performance.Source: A. Discussion and Limitations · p. 10
  • The formulation is presented as a flexible foundation for future proposal mechanisms, perturbation strategies, and visual representations.The authors frame these extensions as adaptable to future MLLMs.Source: A. Discussion and Limitations · p. 10

B. Evaluation Metrics

  • The evaluation follows TriDF's protocol without modification and uses standard binary-classification metrics on MMTD-Set.Artifact-level metrics are restated to support interpretation and reproduction.Source: B. Evaluation Metrics · p. 10

Artifact-Level Metrics on TriDF

  • On TriDF, an external LLM maps free-form responses to a predefined artifact taxonomy, and claims it cannot align are discarded before scoring.Cover measures recall of annotated artifacts; CHAIR measures the proportion of asserted artifacts unsupported by annotation.Source: Artifact-Level Metrics on TriDF · p. 10
  • Hal records whether a response contains any unsupported artifact claim, while CHAIR reflects their proportion; Hal does not measure claim severity.The TriDF F 0.5 score weights precision twice as much as recall; artifact-level metrics apply to manipulated samples, while accuracy uses the full set.Source: Artifact-Level Metrics on TriDF · pp. 10–11

Detection Metrics on MMTD-Set

  • On MMTD-Set, ACC summarizes overall classification and F1 emphasizes detection of manipulated samples on the DeepFake and AIGC-Editing subsets.The study treats manipulated as the positive class and reports FakeShield scores at the original publication's two-decimal precision, zero-padded for alignment.Source: Detection Metrics on MMTD-Set · p. 11

C. Self-Proposed Regions

Self-proposed region inspection gives a useful local-inspection structure but does not match the full framework, which combines image-specific region discovery with attention steering.

C. Self-Proposed Regions

  • Self-proposed regions do not reliably identify manipulation-relevant evidence and may follow semantic saliency or language priors instead.The selected regions may therefore fail to capture image-specific forensic cues.Source: C. Self-Proposed Regions · p. 11
  • The authors identify detail-sensitive, image-specific discovery and region-specific attention steering as necessary beyond textual region decomposition.The self-proposed variant improves over vanilla inference, indicating that local inspection and final aggregation can still provide useful reasoning structure.Source: C. Self-Proposed Regions · p. 11
  • The full framework outperforms Self-Proposed Region Inspection, which improves over vanilla inference but trails Prompt-only on several metrics.Table S1 compares these methods on TriDF using InternVL-3.5-8B; the cited results also place the full framework strongest across detection, grounding, and hallucination reduction.Source: Table S1 · p. 11, C. Self-Proposed Regions · p. 11

D. Alternative Region Source

A facial-component prior improves on vanilla inference, but the full method performs best; the authors attribute the remaining gap to the region targets' adaptivity.

D. Alternative Region Source

  • The facial-component result indicates that localized inspection and attention reallocation can help even with a simple face-centric prior.The authors state that explicitly directing inspection to localized visual evidence accounts for part of the improvement.Source: D. Alternative Region Source · p. 11
  • The full method consistently performs best across all metrics, whereas the facial-region variant improves over vanilla on detection and explanation-grounding metrics.The facial variant uses landmark-based eye, nose, and mouth regions, while keeping the downstream inspection and aggregation procedures unchanged.Source: D. Alternative Region Source · p. 11
  • Image-derived regions adapt across facial artifacts, edited objects, and regenerated regions, unlike a fixed facial prior that can miss non-facial evidence.This adaptability matters in image editing when the face occupies little of the image and useful evidence appears elsewhere.Source: D. Alternative Region Source · p. 11

E. Qualitative Results

Qualitative examples show the framework correcting both missed manipulations and false positives by calibrating local evidence, while a representative failure exposes cases where localized cues remain ambiguous. It also grounds explanations in specific inspected artifacts when vanilla inference gives generic claims.

Correcting erroneous global judgments

  • The framework corrects vanilla inference in complementary cases: it finds localized blur and texture inconsistencies in a manipulated image and rejects hallucinated cues in an authentic face.For the authentic face, ambiguous local regions are assigned low evidential strength, and global aggregation retains a negative observation.Source: E. Qualitative Results · p. 12

Improving explanation grounding

  • When both methods predict manipulation, the framework ties its explanation to blending and texture irregularities around the hairline and forehead, unlike vanilla’s generic claims.The inspected findings more closely match the annotated artifacts, making the explanation more localized and visually traceable.Source: E. Qualitative Results · p. 12

Failure case

  • In a manipulated outdoor scene, both methods predict authentic: the framework identifies background blur, but natural focus variation and lack of complementary evidence leave the cue inconclusive.The selected regions do not reveal sufficiently strong blending, color inconsistency, or object-integrity flaws.Source: E. Qualitative Results · p. 12
  • The failure suggests lower effectiveness when manipulation evidence is spatially diffuse, background-dominated, or naturally explainable in isolation, although the local cue remains better calibrated than vanilla’s.The framework identifies a relevant cue without treating it as conclusive evidence.Source: E. Qualitative Results · p. 12

F. Implementation Details

The three-stage framework is instantiated consistently across backbones, with architecture-dependent steering layers and visual-token grids and a specific adjustment for Qwen3.5-9B. Implementation examples illustrate both corrected predictions and an unresolved ambiguous cue.

F. Implementation Details

  • Across backbones, the three-stage setup changes only architecture-dependent quantities, while other hyperparameters are shared apart from stated Qwen3.5-9B adjustments.The architecture-dependent quantities are decoder layer/head counts, visual-token grid, and steering-layer range; InternVL-3.5-8B is the detailed ablation instantiation.Source: F. Implementation Details · p. 12
  • For InternVL-3.5-8B, Stage 1 uses a 3.0px Gaussian blur and retains up to 20 heads; Stage 2 selects up to three regions from a 16×16 token grid.Stage 3 steers layers 12–31 with retention coefficient 0.5; region inspection and final aggregation are capped at 96 and 384 tokens.Source: F. Implementation Details · p. 12
  • On a manipulated image, the framework identifies blur and texture/color inconsistencies in selected regions and produces the correct manipulated verdict.On an authentic face, it assigns weak, uncertain, or negative evidence to inspected regions and recovers the correct authentic verdict; a second example grounds findings in hairline and forehead irregularities.Source: Figure S1 · p. 13, Figure S2 · p. 13, Figure S3 · p. 14
  • The inference-efficiency comparison evaluates a balanced 100-image TriDF subset with the same InternVL-3.5-8B backbone under identical single-GPU settings; an invocation is autoregressive generation or explicit attention forward evaluation.Source: Table S3 · p. 15
  • Qwen3.5-9B steers only its eight full-attention layers; lowering the peak-distance gate from 2 to 1 restores selection to 15–17 heads with reduced spatial diversity.Its region-inspection and aggregation budgets are raised to 256 and 512 tokens, respectively, while other hyperparameters remain unchanged.Source: F. Implementation Details · p. 15

G. Computational Cost & Runtime

On the balanced TriDF runtime subset, the framework adds 48.0% latency and approximately 3.1% peak memory over vanilla inference. Its prompt protocol inspects local regions first, then verifies retained cues against the full image before deciding.

G. Computational Cost & Runtime

  • Our method averages 6.970 invocations and takes 19.951 seconds per image (1.480× vanilla inference’s 13.480 seconds), while peak memory rises 0.495 GB, from 16.129 GB to 16.624 GB.Source: G. Computational Cost & Runtime · p. 15
  • The extra calls are cheaper than a full vanilla generation because short intermediate outputs are distributed across stages, the Stage 1 continuation is reused, and Stage 2 uses lightweight processing.Regional inspections yield short structured observations, while final aggregation uses the overall generation budget rather than repeating a full-length response at each stage.Source: G. Computational Cost & Runtime · p. 15
  • Despite nearly seven high-level invocations, the method adds 48.0% latency and approximately 3.1% peak GPU memory over vanilla inference.The average is 6.970 invocations; the total is 4 + K, where K ≤ 3, with input-dependent retained regions.Source: G. Computational Cost & Runtime · p. 15

H. Prompt Templates

  • Region-wise prompts request concrete local comparisons and classify each observation as yes, uncertain, or no, with confidence and artifact type recorded.Generic blur, smoothness, compression, focus variation, lighting variation, or image quality alone is not sufficient manipulation evidence.Source: H. Prompt Templates · p. 16
  • Global aggregation verifies retained tentative cues against the full image, discounts ambiguous or duplicate observations, and bases the final decision on verified evidence strength.Only minor wording adjustments accommodate backbone differences; they do not change the evidence-before-judgment workflow.Source: H. Prompt Templates · pp. 15–16

Related papers

Loading 2609.35536v1…