PaperlayerView exact revision on arXiv

Source-linked AI summary

From Sharp Eyes to Expert Mind: Internalizing Expert Knowledge in MLLMs for Tampered Text Detection

Kaiqing Lin, Songze Li, Shen Chen, Yunfei Guo, Xiaoye Qiu, Haodong Li, Taiping Yao, Bo Wang, Youchang Xiao, Bin Li, Shouhong Ding

arXiv:2609.36145v1cs.CV

TL;DR

Expert models capture subtle manipulation traces but may generalize poorly, while MLLMs transfer across domains yet miss fine-grained forensic artifacts. EKI uses a two-stage framework to internalize expert perception in the MLLM. The paper reports state-of-the-art benchmark performance and identifies limits in explanation and exact-shape localization.

  • Problem

    Expert models capture subtle forensic traces but may generalize poorly, while MLLMs transfer across domains yet remain insensitive to fine-grained artifacts.

    Source: §I · pp. 1–2

  • Method

    EKI first focuses the MLLM on small text regions, then uses FGRA to transfer forensic perception into its shallow representations.

    Source: §I · p. 2

  • Results

    EKI achieves state-of-the-art performance across diverse document benchmarks and consistently improves different base MLLMs.

    Source: §VII · p. 12

  • Takeaways & Limitations

    The forensic expert is used only during training and removed at inference, so deployment requires no external expert.

    Source: §I · p. 2, §VII · p. 12

    The method does not advance human-readable explanations of tampering evidence, and bounding boxes do not delineate exact shapes of irregular regions.

    Source: §VI · p. 11

Abstract

from arXiv · show

Tampered Text Detection (TTD) is essential for safeguarding document authenticity in security-critical workflows. Existing expert models are effective at capturing subtle manipulation traces but often generalize poorly across diverse document domains, while Multimodal Large Language Models (MLLMs) offer stronger semantic understanding and transferability yet remain insensitive to fine-grained forensic artifacts. This complementarity motivates us to investigate how expert forensic perception can be internalized into an MLLM rather than merely accessed through an external module. We identify a fundamental Double Mismatch that hinders this goal: a Spatial Precision Mismatch between coarse visual tokens and tiny tampered regions, and a Perceptual Granularity Mismatch between semantics-oriented pre-training and low-level forensic perception. To address these challenges, we propose Expert Knowledge Internalization (EKI), a progressive two-stage framework that transfers forensic expertise into the MLLM itself. In Stage 1, Text-Focused and Image-Focused strategies establish precise spatial focus on small text regions. In Stage 2, the proposed Forensic-General Representation Alignment (FGRA) loss aligns shallow LLM representations with those of a pre-trained forensic expert, enabling the model to acquire fine-grained artifact perception before such cues are diluted by deeper semantic abstraction. Extensive experiments on multiple in-domain and cross-domain benchmarks demonstrate that EKI achieves state-of-the-art performance and stronger generalization than existing expert-model-based and MLLM-based methods. Moreover, the expert is required only during training, allowing the resulting MLLM to maintain inference efficiency nearly identical to the vanilla model without relying on any external expert at inference.

I. INTRODUCTION

Expert forensic models capture subtle manipulation traces but may generalize poorly, while MLLMs transfer across domains yet miss fine-grained artifacts. EKI aims to internalize expert perception in the MLLM rather than add an external module.

I. INTRODUCTION

  • Expert models capture subtle traces but may generalize poorly, whereas MLLMs transfer across domains yet remain insensitive to fine-grained tampering.This complementarity motivates equipping one MLLM with specialist forensic perception.Source: §I · p. 1
  • External-reliance systems insert expert predictions or features but do not give the MLLM a sufficient opportunity to learn forensic representations itself.The externally coupled system fails to surpass the standalone expert despite a more complex and costly inference pipeline.Source: §I · p. 2
  • The Double Mismatch combines tiny regions under-resolved by coarse visual tokens with semantics-oriented representations insensitive to low-level forensic signals.Together, these mismatches hinder intrinsic forensic perception through external predictions or standard fine-tuning alone.Source: §I · p. 2, Figure 2 · p. 2
  • EKI outperforms the standalone expert and external-reliance paradigm while showing stronger cross-domain generalization.The comparison is shown in Fig. 1(b).Source: §I · p. 2, Figure 1 · p. 2
  • EKI first focuses spatially on small text regions, then uses FGRA to align shallow LLM representations with a pre-trained forensic expert.The expert is discarded at inference, and the resulting model requires no external expert module.Source: §I · p. 2

II. RELATED WORK

Document text tampering can be tiny, visually subtle, and varied across domains. Related work spans image-forensics methods, specialized document-TTD experts, and the limits of adapting natural-image localization methods to dense text.

A. Document Text Tampering

  • Document text tampering includes five types, and tiny manipulated regions can become visually imperceptible after post-processing while forgery styles vary across domains.The types are copy-move, splicing, removal, insertion, and replacement.Source: A. Document Text Tampering · p. 3

B. General Image Manipulation Localization

  • Natural-image manipulation localizers use varied architectures and forensic cues but may perform suboptimally on documents because they lack text priors and face dense textual interference.Examples include multi-scale feature aggregation, high-resolution modeling, noise-sensitive fingerprints, and frequency components.Source: B. General Image Manipulation… · p. 3

1) Expert Models:

  • Document-TTD expert models exploit low-level cues, including frequency, residual, and contrastive features, but remain task-specific and limited in adapting to unseen document distributions.DITL2 [4] uses diverse-image pre-training for transferable domain-invariant representations, yet expert models have limited ability to exploit textual and structural information.Source: 1) Expert Models · p. 3

2) MLLM-based Approaches:

Existing MLLM-based TTD approaches combine semantic capabilities with external forensic expertise, but do not internalize that expertise. This work proposes targeted alignment and bounding-box outputs compatible with autoregressive decoding.

2) MLLM-based Approaches:

  • TextSleuth [9] and TVSIP use external experts, and current MLLMs remain insensitive to subtle forensic artifacts.TVSIP fuses independently modeled branches at the prediction level without feature-level interaction or joint representation learning.Source: 2) MLLM-based Approaches · p. 3

1) External Reliance Strategies:

A. Task Reformulation

  • EKI reframes localization from dense pixel masks to bounding-box regression, using rectangles derived from existing masks and serialized as short coordinate-token sequences.This output format is compatible with autoregressive decoding.Source: A. Task Reformulation · p. 4

B. Problem Definition

The task asks an MLLM to judge authenticity and return coordinates for manipulated regions. EKI addresses spatial focus first, then transfers forensic perception into the model’s representations.

B. Problem Definition

  • The model receives a document image and instruction, then generates authenticity judgments and bounding-box coordinates for manipulated regions.The output coordinates are [x1, y1, x2, y2].Source: B. Problem Definition · p. 4

A. Overview

  • EKI uses a pre-trained forensic expert as a teacher in a two-stage framework: first teaching where to look, then how to discern artifacts.The stages target spatial precision and perceptual granularity mismatches, respectively.Source: A. Overview · p. 4

B. Stage 1: Precise Spatial Focus

  • Stage 1 fine-tunes the vision encoder and projector while freezing the LLM, using Text-Focused and Image-Focused strategies to improve grounding of tiny regions.The approach addresses noise from background semantics under coarse-grained visual tokenization.Source: B. Stage 1: Precise Spatial Focus · p. 4, A. Overview · p. 4

1) Text-Focused:

  • Text-Focused prompts use OCR-detected regions as coarse anchors and supervise precise boxes, directing attention toward text-associated spatial tokens.This coarse-to-fine refinement establishes a spatial foundation for later forensic analysis.Source: 1) Text-Focused · p. 4

2) Image-Focused:

Image-Focused training addresses the difficulty of localizing tiny tampered regions by pairing magnified-patch classification with full-image localization. Hard samples are selected by localization failure and trained with both tasks to strengthen detection in cluttered documents.

2) Image-Focused:

  • The strategy adds an easier classification task on magnified local patches because direct localization of regions smaller than 1% of an image is difficult for general MLLMs.The local classification task asks whether a patch is real or tampered.Source: 2) Image-Focused · pp. 5–6
  • Samples with predicted-box IoU below 0.7 after preliminary baseline training are designated hard samples for decomposition training.These cases often involve tiny tampering regions or subtle traces overwhelmed by background noise.Source: 2) Image-Focused · p. 5
  • The figure describes OCR-based text-focused instructions alongside Local Classification and Global Localization for mining hard samples.The local task distinguishes manipulation in isolated magnified views, while the global task retains full-image localization.Source: Figure 4 · p. 6, 2) Image-Focused · p. 6
  • Magnified-patch classification is jointly trained with full-image localization so local discrimination can support finding difficult regions in global context.Tampered and authentic patches are resized to the original image resolution before classification.Source: 2) Image-Focused · p. 6

3) Training Objectives (Stage 1):

Stage 1 unifies local classification and global localization with an autoregressive language-modeling objective. The target response format depends on the task, and optimization updates the vision encoder and projector for fine-grained spatial features.

3) Training Objectives (Stage 1):

  • Given instruction T and image I, the model maximizes the likelihood of target response sequence Y using the standard autoregressive language-modeling objective.Y has length L, and y<t denotes preceding tokens.Source: 3) Training Objectives (Stage 1) · p. 6
  • The Stage 1 objective jointly optimizes standard language-modeling losses on hard-mined local patches and global localization samples.The local target is a binary label, whereas the global target includes a judgment and bounding-box coordinates.Source: 3) Training Objectives (Stage 1) · p. 6
  • Minimizing the Stage 1 objective updates the vision encoder and projector to capture fine-grained spatial features for precise spatial focus.The loss terms are standard language-modeling losses computed on the local and global datasets.Source: 3) Training Objectives (Stage 1) · p. 6

C. Stage 2: Forensic Artifact Perception

Stage 2 preserves Stage 1 spatial features while addressing whether localized regions contain genuine tampering traces. It uses a frozen forensic expert to guide shallow LLM representations toward forensic artifacts.

C. Stage 2: Forensic Artifact Perception

  • Stage 2 freezes the vision encoder and fine-tunes the LLM and projector to align forensic features in the language model’s latent space.This follows Stage 1 localization of suspected regions from the full image to character-level precision.Source: C. Stage 2: Forensic Artifact Perception · p. 6

1) Pre-trained Forensic Expert:

  • A DINOv2 backbone fine-tuned on the forensic training set is frozen as the expert, supplying generalized forensic representations as supervision.Its parameters remain frozen during internalization.Source: 1) Pre-trained Forensic Expert · p. 6

2) Forensic-General Representation Alignment (FGRA):

  • FGRA addresses sparse, indirect text-generation supervision by aligning shallow LLM tokens with expert embeddings before deeper semantic abstraction dilutes forensic cues.Shallow layers retain structural and textural details, including local patterns and high-frequency cues correlated with forensic artifacts.Source: 2) Forensic-General Representation… · pp. 6–7
  • FGRA uses a learnable projection and patch-wise pairing to align LLM representations with expert features, encouraging the LLM to encode discernible artifacts.The loss is negative cosine similarity between projected LLM tokens and expert features.Source: 2) Forensic-General Representation… · p. 7

3) Training Objectives (Stage 2):

Stage 2 combines text-generation and FGRA losses, using the former to preserve instruction-following and the latter to inject forensic discrimination. Evaluation covers two training datasets, in-domain tests, and five cross-domain benchmarks; the expert setup retains only the frozen backbone for internalization.

3) Training Objectives (Stage 2):

  • The Stage 2 objective combines text-generation loss with FGRA loss to preserve instruction-following while injecting forensic discriminative power into the LLM’s latent space.The expert feature alignment is applied during internalization.Source: 3) Training Objectives (Stage 2) · p. 7

1) Datasets:

  • Evaluation trains on DocTamper-TrainingSet or FSTS-T, tests in-domain on DocTamper-TestingSet or FSTS-S, and uses five other datasets as cross-domain benchmarks.The cross-domain sets are DocTamper-FCD, DocTamper-SCD, FSTS-1.5K, SACP, and RIFLC.Source: 1) Datasets · p. 7
  • Table I’s abbreviations denote copy-move, splicing, removal, insertion, and replacement manipulation types.The table summarizes the datasets used in the experiments.Source: Table I · p. 7

2) Setting of the Expert Model:

  • The forensic expert uses DINOv2-ViT-L with a Mask2Former segmentation head, trained on the same forensic data as its corresponding MLLM without additional training data.For internalization, the Mask2Former head is discarded and only the frozen DINOv2 backbone is retained; no decoder is involved.Source: 2) Setting of the Expert Model · p. 7

3) Setting of the EKI:

The evaluation uses a shared document-image setup and localization protocol to compare EKI with general image, expert TTD, and MLLM competitors. EKI ranks best on average across the reported test sets, with stronger robustness on cross-domain benchmarks despite a substantial domain gap.

3) Setting of the EKI:

  • At inference, EKI produces an authenticity judgment and tampered-region boxes by greedy decoding, without OCR preprocessing or an external expert.LoRA fine-tuning uses rank r = 32 and α = 64.Source: 3) Setting of the EKI · p. 7

4) Optimization Details:

  • The expert and MLLM are trained for 10 epochs with AdamW at a 1e−4 learning rate and batch size 8; comparison models use batch size 32.All inputs are resized to 512×512 pixels for training and testing.Source: 4) Optimization Details · p. 7

5) Evaluation Metrics:

  • IoU and F1 are averaged per image at the pixel level to measure overlap between predicted and ground-truth regions.For segmentation-mask methods, each connected component is converted to its minimum enclosing rectangle before evaluation; all methods use the same protocol.Source: 5) Evaluation Metrics · pp. 7–8

1) Competitors:

  • The evaluation compares general image manipulation methods, expert TTD models, and TVSIP after retraining all competitors on DocTamper-TrainingSet and FSTS-T.The base MLLM is Qwen2.5-VL-7B; inference uses the document image and a plain detection instruction, without OCR preprocessing or an external expert.Source: 3) Setting of the EKI · p. 7, 1) Competitors · p. 8

2) Quantitative Analysis:

  • EKI achieves the best average performance across all test sets, surpassing competitors by significant margins in both IoU and F1.It attains the best IoU on Doc-Test and FSTS-S and ranks at or near the top on cross-domain SACP and RIFLC, where absolute performance remains relatively low because the training and test forgeries differ substantially.Source: 2) Quantitative Analysis · p. 8

3) Qualitative Analysis:

EKI produces compact, precise localization, while component ablations identify FGRA as the strongest performance driver and show that shallow-layer alignment is preferable. The external expert is removed at inference, leaving speed nearly identical to the vanilla Qwen2.5-VL baseline.

3) Qualitative Analysis:

  • EKI generates compact, precise bounding boxes, contrasting with general methods’ blind spots, expert models’ fragmented masks or false positives, and baseline MLLM’s coarse boxes.The comparison uses models trained on DocTamper-TrainingSet.Source: 3) Qualitative Analysis · p. 8

4) Inference Efficiency Analysis:

  • EKI’s inference speed is nearly identical to vanilla Qwen2.5-VL because the external expert is discarded after training.Its latency is higher than that of compact expert models, which have substantially fewer parameters.Source: 4) Inference Efficiency Analysis · p. 8

1) Component Analysis:

  • FGRA produces the largest ablation gain: FGRA alone surpasses both Stage 1 focus strategies combined, and the full model performs best.Text-Focused and Image-Focused strategies each improve localization, while their combination improves it further; the full result supports their complementary roles with internalized forensic knowledge.Source: 1) Component Analysis · p. 8

2) Analysis of the Alignment Layer Depth:

  • Aligning FGRA with the first LLM layer gives optimal performance, while detection performance declines as alignment moves deeper.Shallow layers retain structural and textural details; deeper semantic-dominant representations dilute subtle tampering traces.Source: 2) Analysis of the Alignment Layer Depth · pp. 8–9
  • EKI’s first-layer visual responses are distinctive on manipulated regions, unlike the noisy or blind activation patterns of the original model and standard SFT.Layerwise visualizations show forensic signals distinct in shallow feature maps but fading in deeper layers.Source: 2) Analysis of the Alignment Layer Depth · p. 9

3) Analysis of Different Fusion Strategies:

The comparison tests external expert coupling against internalizing forensic perception in the MLLM. EKI performs best overall, with average IoU/F1 of 0.505/0.566, and needs no external expert at inference.

3) Analysis of Different Fusion Strategies:

  • The four compared paradigms are a fine-tuned MLLM baseline, a standalone DINOv2-ViT-L expert, external reliance with late fusion, and EKI internalization.The same fine-tuned expert is used in the expert, external-reliance, and EKI paradigms.Source: 3) Analysis of Different Fusion… · pp. 9–10
  • In the qualitative comparison, EKI produces precise, compact localization, unlike general methods’ blind spots, expert TTD models’ fragmented masks or false positives, and the baseline’s coarse boxes.The figure compares methods trained on DocTamper-TrainingSet.Source: Figure 5 · p. 10
  • FGRA performs best when aligned to the shallowest LLM layer, with deeper layers showing a declining trend.The layer-selection ablation is shown in Fig. 6.Source: Figure 6 · p. 10
  • External reliance does not outperform the expert, whereas EKI achieves the best overall performance.The comparison supports transferring expert perception into the MLLM rather than relying on externally coupled predictions.Source: 3) Analysis of Different Fusion… · pp. 10–11
  • EKI reaches an average IoU/F1 of 0.505/0.566, achieves the best overall performance, and requires no external expert at inference.The comparison averages metrics over the test sets.Source: 3) Analysis of Different Fusion… · p. 10, Table VI · p. 11

4) Analysis of the Base MLLM:

Across tested base MLLMs, EKI consistently improves performance, while the paper identifies limits of bounding-box localization and describes a two-stage approach to internalizing forensic perception. The expert is removed at inference, and the conclusion reports state-of-the-art performance across diverse document benchmarks.

VI. LIMITATIONS

  • Bounding boxes identify manipulated text regions but do not delineate irregular tampered shapes, so the formulation is not intended for pixel-accurate boundary localization.The paper focuses on detecting and localizing tampered text rather than advancing human-readable explanations of tampering evidence.Source: §VI · p. 11

4) Analysis of the Base MLLM:

  • Across LLaVA-1.5-7B, Qwen2.5-VL 7B, and Qwen3-VL 8B, EKI consistently improves performance, with a stronger base model reaching a new state-of-the-art performance ceiling.All models are trained on DocTamper-TrainingSet.Source: 4) Analysis of the Base MLLM · p. 11, Table VII · p. 12

VII. CONCLUSION

  • EKI addresses spatial precision and perceptual granularity mismatches through spatial-focus training followed by FGRA-based transfer of fine-grained forensic perception.Stage 1 uses Text-Focused and Image-Focused training; Stage 2 uses the FGRA loss.Source: §VII · pp. 11–12
  • The expert guides training but is removed at inference, leaving no external expert required during deployment.The conclusion also reports stronger cross-domain generalization than external reliance.Source: §VII · p. 12
Loading 2609.36145v1…