PaperlayerView exact revision on arXiv

Source-linked AI summary

MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answering

Dang Quang Thien Tran, Quang V. Dang, Vinamra Tyagi, Sai Soorya Rao Veeravalli, Trang Nguyen, Ryan A. Rossi, Franck Dernoncourt, Nedim Lipka, Koustava Goswami, Samyadeep Basu

arXiv:2607.01420v3cs.CLcs.AIcs.CV

TL;DR

Fine-grained attribution in multimodal long documents remains underexplored, despite its importance for verifying grounded answers. MultAttnAttrib uses a training-free, single-pass approach to locate modality-aware evidence and consistently outperforms strong baselines while operating at roughly 14% of their inference latency.

  • Problem

    Fine-grained attribution in multimodal long documents remains comparatively nascent, especially for jointly locating supporting text and images within a full-length document.

  • Method

    MultAttnAttrib is a training-free multimodal attribution method that uses specialized cross-modal retrieval heads to produce modality-aware citations in a single inference pass.

  • Results

    Roughly 14% latency: MultAttnAttrib consistently outperforms most strong baselines on Qwen3-VL-30B and a frontier model.

  • Takeaways & Limitations

    MultAttrEval provides a test-bed for evaluating fine-grained multimodal attribution over long-context documents.

  • Takeaways & Limitations

    MultAttnAttrib requires a small labeled probe set for attention-head identification and threshold calibration.

Abstract

from arXiv · show

As grounded QA systems are increasingly deployed in AI assistants, accurately attributing generated answers to evidence is critical for user trust and model safety. While unimodal attributions have been explored in depth, the multimodal setting remains relatively under-researched. As a result, we introduce MultAttnAttrib, a training-free attribution-generation method that leverages a model's prefill pass, selected attention heads, and calibrated thresholds to locate source evidence within a document. To establish baseline results for the method, we introduce MultAttrEval, a complementary benchmark dataset annotated with fine-grained, ground-truth attributions for answer components grounded in multimodal source documents. To our knowledge, this is the first evaluation dataset designed specifically for multimodal attribution in long-form documents. Experimental results show that MultAttnAttrib consistently outperforms a variety of attribution-generation methods, including several strong prompting-based approaches and matches the latest frontier models such as GPT 5.4. Our method not only substantially improves attribution accuracy for both unimodal and multimodal attribution types, but also produces attributions at up to one-seventh of the direct inference latency compared to prompting on the same base model.

1 Introduction

Multimodal attribution in long documents remains underdeveloped and uniquely challenging because systems must identify both the relevant modality and fine-grained source evidence. The paper introduces a training-free attention-based method and benchmark, reporting stronger attribution with substantially lower inference cost.

  • Motivation: Accurate attributions are fundamental for trustworthy deployment because users often must verify generated answers, and modern systems accurately cite fewer than 52% of statements.This requirement is especially important in high-impact domains such as medicine, where ungrounded answers can cause real negative impacts.
  • Problem: Multimodal long-document attribution remains nascent, with prior work emphasizing citation selection from retrieved passages or images rather than fine-grained localization within a full document.Unlike unimodal attribution, multimodal systems must determine both the correct modality or modalities and the correct source.
  • Method: MULTATTNATTRIB uses retrieval-head attention from a model’s prefill pass to jointly score text spans and image regions, then applies lightweight calibration for modality-aware citations in one inference pass.The method is training-free and avoids iterative generation and additional reasoning overhead.
  • Benchmark: MULTATTREVAL is a complementary benchmark spanning five domains and evaluating fine-grained attribution in both unimodal and multimodal settings.The evaluation covers prompting-based, captioning-based, and retrieval-augmented baselines across open-source and frontier MLLMs.
  • Results: 14% of the inference latency is required by MULTATTNATTRIB while it consistently outperforms most strong baselines on Qwen3-VL-30B and a frontier model.The method also reduces peak memory usage by approximately 15GB (non-vLLM) per QAA instance, while results confirm a substantial multimodal–unimodal attribution performance gap.

2 Related Work

Prior attribution research comprises training-based citation generation, post-processing with external evaluators, and methods that recover attribution from model computations. Existing multimodal benchmarks often reduce evidence attribution to selecting from pre-curated candidates rather than localizing evidence in full documents.

  • Attribution methods: Attribution methods span fine-tuned models that interleave citations with generated text, including work built on Attributed QA and ALCE.This family includes training-based citation-generation methods.
  • Attribution methods: A second family separates attribution from generation by post-processing outputs with retrievers, natural-language-inference verifiers, or LLM judges.The cited approaches include Gao et al. (2023a) and Qian et al. (2025).
  • Multimodal attribution benchmarks: Existing multimodal benchmarks evaluate citation selection from small, pre-curated pools of passages, figures, or tables rather than evidence localization over full multimodal documents.Examples include MCiteBench, MM-DocRAG, and MAVIS; SciClaimEval similarly pre-identifies relevant figures.

3 MULTATTNATTRIB: A Training-Free Approach for Multimodal Attribution

MultAttnAttrib frames multimodal attribution as mapping a question, document, and answer to supporting text spans, images, or joint text–image evidence. It is a training-free method that identifies cross-modal retrieval heads, calibrates modality-specific thresholds, and generates citations in one forward pass.

  • Head identification: MultAttnAttrib identifies cross-modal retrieval heads from a small labeled probe set and uses their attention signals to score image slots and text passages.Retrieval heads are modality-specific at the highest ranks but largely shared across the broader population, motivating the cross-modal selection strategy.
  • Problem formulation: The task attributes an answer to a text span, an image set, or a joint text–image evidence pair in a multimodal document.The predicted attribution maps (q, D, a) to an attribution space containing these evidence types.
  • Head identification: Causal Mediation Analysis uses two forward passes per probe to compare clean and corrupted inputs, while Mean Attention Scoring uses one but lacks causal validity.CMA corrupts multimodal evidence by replacing text spans or resizing and replacing images, then measures each head’s indirect effect while suppressing uniformly attending heads.
  • Threshold calibration: F1-maximizing sweeps over probe attribution scores derive separate thresholds Timg and Ttxt for image and text citations.The thresholds are selected from modality-labeled probe distributions and reused during inference.
  • Online inference: Attribution requires a single forward pass: selected-head attention scores are averaged, image and text evidence are min-max normalized, and thresholded with a highest-scoring fallback.Images are scored over patch tokens and text over sliding windows.

4 MULTATTREVAL: A Dataset for Multimodal Attribution in Long Document Understanding

MultAttrEval is a dataset of fine-grained question-answer-attribution triplets from long-form PDFs, spanning five domains and text-only, image-only, and multimodal settings. Its construction combines modality-specific generation with strict grounding and answerability checks.

  • Dataset construction: The pipeline obtains PDFs from MINT-1T, filters documents by image count and valid URLs, then extracts, preprocesses, and embeds text and images.Embedding-based similarity pairings support subsequent attribution generation.
  • Attribution generation: For unimodal items, an MLLM generates questions and answers using only selected image or text-chunk spans to create image-only or text-only attributions.The provided construction description distinguishes unimodal attribution generation from the combined text-image case.
  • Attribution generation: For text-image items, the pipeline reranks similar text-image pairs, identifies text entities, and verifies that entities belong to the corresponding image.Both modalities must be relevant and mutually support the answer.
  • Quality control: Generated QAA candidates undergo strict acceptance criteria, including verifier quality, attribution support, layout-artifact exclusion, answerability, and multimodal grounding checks.Multimodal-only items additionally require image and text grounding of distinct answer entities plus answer circularity prevention.
  • Dataset scope: MultAttrEval contains question-answer-attribution triplets for long-form PDF documents spanning five domains and three attribution settings: text-only, image-only, and multimodal.The corpus distributions and modality-level statistics are reported in Appendix A.

5 Experiments

The experiments use a 90-triplet Probe set for attention-head analysis, head identification, and threshold calibration, while 608 remaining items form the Test set for evaluating all methods. Evaluation covers QWEN3-VL-30B-A3B-INSTRUCT, GPT-5.4, multimodal and text-only baselines, and retrieval-augmented variants using macro-averaged attribution metrics.

  • Dataset splits: 90 probe QAA triplets support attention-head analyses, head identification, and threshold calibration in MULTATTNATTRIB.The remaining 608 items constitute the Test set for evaluating all methods.
  • Dataset splits: 608 Test-set items are used to evaluate all methods.
  • Compared methods: Baselines include QWEN3-VL-30B-A3B-INSTRUCT, GPT-5.4, image-and-OCR VLM attribution, caption-based text-only LLM attribution, and k = 5 RAG variants.Cohere retrieves five text chunks and five images, while ColQwen retrieves five full PDF pages.
  • Evaluation metrics: Attribution quality is measured with macro-averaged precision, recall, and F1, using exact match for image citations and fuzzy substring scoring for text citations.Text scores use three tiers and penalize under- or over-quoting according to length ratio.

6 Results and Analysis

MultAttnAttrib outperforms prompting and VLM baselines in attribution quality while reducing inference cost, and its gains arise from sparse, modality-structured retrieval heads. Its attention-based attribution also exhibits distinct strengths across visual and textual grounding settings.

  • Attribution Performance: Over 20%, MultAttnAttrib outperforms all prompting baselines across all three modality splits.The comparison uses the same QWEN3-VL-30B-A3B-INSTRUCT backbone.
  • Attribution Performance: Universal F1 gains improve image precision and text recall over direct and RAG-augmented VLM baselines.Attention-based attribution is especially effective for localization-heavy long-document settings.
  • Attribution Performance: Thresholding suppresses spurious citations and mitigates baseline text overprediction, improving unimodal attribution quality.Baseline text over-attribution particularly reduces image-regime precision and text-regime recall.
  • Retrieval Ablations: Cohere RAG substantially improves text performance when combined with MultAttnAttrib, whereas ColQwen degrades text and image metrics.Fine-grained retrieval is therefore more effective for attribution than page-level retrieval.
  • Efficiency: 7.3× better latency and nearly 15 GB lower peak VRAM usage distinguish MultAttnAttrib from non-OOM VLM inference on singular QA inputs.A single forward pass bypasses KV-cache growth and token-level decoding overhead; direct non-vLLM inference frequently encounters OOM errors.
  • Retrieval Head Analysis: At k = 4, CMA yields ρ4 = −0.107 and MAS yields ρ4 = −0.657, while CMA recovers to ρ20 = 0.042 as more heads are included.CMA produces higher cross-modal overlap than MAS at k = 20: IoU(20) = 0.379 versus 0.212.
  • Retrieval Head Analysis: About 80% of heads score below 0.1 and fewer than 2% score above 0.6, confirming a sparse retrieval circuit sufficient for efficient single-pass attribution.Image heads concentrate in layers 22–36, text heads span early and late layers, and crossmodal heads cluster in the mid-to-late transition zone.

7 Conclusion

The paper introduces MULTATTNATTRIB, a training-free multimodal attribution method that outperforms strong prompting and inference-time strategies at a fraction of their latency while remaining competitive with GPT-5.4. It also introduces MULTATTREVAL, a test bed for multimodal attribution in long-context documents.

  • Contributions: MULTATTNATTRIB uses cross-modal and specialized retrieval heads in a training-free attribution method.The method is designed for multimodal attribution over long-context documents.
  • Results: MULTATTNATTRIB outperforms strong prompting and inference-time strategies on the same backbone at a fraction of the latency.The comparison is made against a range of strong strategies using the same backbone.
  • Results: MULTATTNATTRIB remains competitive with frontier-scale models such as GPT-5.4.The conclusion positions the method as competitive with GPT-5.4 despite its training-free design.
  • Contributions: MULTATTREVAL provides a test bed for evaluating multimodal attribution over long-context documents.The benchmark complements the proposed attribution method.

8 Limitations

The work identifies limitations in both its benchmark and method, with MultAttrEval’s image-dense documents and single-image ground truth creating challenges for attribution evaluation. Visually similar alternatives can depress baseline performance, motivating stricter image filtering and support for multiple image attributions.

  • MultAttrEval uses long, image-dense documents containing near-duplicate or decorative images with limited semantic value.These document characteristics complicate multimodal attribution evaluation.
  • Single-source QAA triplets assign one ground-truth image, while baselines often retrieve visually similar alternatives, depressing performance.The benchmark’s image regime therefore penalizes retrievals that are visually similar but not the annotated image.
  • Future curation should apply stricter image-relevance filtering and support multiple image attributions through embedding-cluster or entity-based grouping.

A MultAttrEval Dataset Statistics and Analysis · B Baseline Design

MultAttrEval is characterized through domain, document, and attribution-regime statistics, while baselines compare multimodal and text-only attribution under full-context and retrieval-augmented settings. Experiments use Qwen3-VL-30B-A3B-Instruct and GPT-5.4 to assess diverse attribution-generation approaches.

  • A MultAttrEval Dataset Statistics and Analysis: MultAttrEval source documents are distributed across multiple domains.The domain distribution is presented in Figure 6.
  • A MultAttrEval Dataset Statistics and Analysis: Document statistics and QAA counts are reported across domains and attribution regimes.Table 3 separates document statistics from QAA counts across domains and regimes.
  • A MultAttrEval Dataset Statistics and Analysis: QAA statistics are analyzed across attribution regimes.Table 4 provides the regime-level statistical analysis.
  • A MultAttrEval Dataset Statistics and Analysis: The dataset’s QAA items are distributed by attribution regime and by the combination of regime and document domain.Figures 7 and 8 provide these two distribution views.
  • B Baseline Design: Baseline experiments use Qwen3-VL-30B-A3B-Instruct and GPT-5.4 for long documents containing interleaved text and images.The models support long multimodal documents, and the comparison targets diverse attribution-generation methods.
  • B Baseline Design: Baselines test whether full document context helps attribution and whether image captions make text-only attribution competitive, with and without retrieval augmentation.These settings isolate the contribution of context compression.
  • B Baseline Design: The VLM baseline provides raster images, document text, and batched QAAs, then prompts a vision-language model to locate answer sources.The LLM variant instead provides image captions and document text, while RAG retrieves sources using k = 5.

C Evaluation Metrics … F Comparing GPT-5.4 to MULTATTNATTRIB

The paper evaluates attribution with citation-level precision, recall, and F1, and describes MultAttnAttrib’s attention-head identification and calibration procedures. It also reports separate text, image, and multimodal regime metrics for GPT-5.4 against Qwen3-VL and MultAttnAttrib.

  • C Evaluation Metrics: Image citations use exact-match scoring, whereas text citations use fuzzy substring matching with a partial_ratio score in [0, 1].The match score is discretized, and length ratios penalize over-quoting for precision and under-quoting for recall.
  • C Evaluation Metrics: Precision, recall, and F1 measure attribution quality, with macro-averaged P, R, and F1 reported across the dataset.Per-item precision averages scores over predicted citations, while recall averages scores over ground-truth citations.
  • D MultAttnAttrib: MeanAttn identifies attention heads by aggregating head-specific attention ratios across items, queries, and document regions, weighted by attention entropy.The selected head set H is the arg max over layer-head scores.
  • D MultAttnAttrib: CMA identifies heads by comparing clean and corrupted inputs, then aggregating an intervention-effect score with the same entropy-based weighting.The algorithm returns the layer-head set maximizing the accumulated score.
  • D MultAttnAttrib: Calibration separately processes image and text modalities to return modality-specific thresholds from probe attribution scores and ground-truth modality labels.The calibration procedure returns Timg and Ttxt.
  • E Comparing GPT-5.4 to QWEN3-VL-30B: GPT-5.4 and Qwen3-VL are compared using text-regime metrics.The supplied passage identifies these results as Table 5, without providing metric values.
  • E Comparing GPT-5.4 to QWEN3-VL-30B: GPT-5.4 and Qwen3-VL are also compared using image- and multimodal-regime metrics.The supplied passages identify these results as Tables 6 and 7, without providing metric values.
  • F Comparing GPT-5.4 to MULTATTNATTRIB: GPT-5.4 and MultAttnAttrib are compared using text-, image-, and multimodal-regime metrics.The supplied passages identify these results as Tables 8, 9, and 10, without providing metric values.

G Domain Difficulty Analysis

Document domain affects overall attribution difficulty but not the relative ordering among baseline methods. Academic documents are hardest, while legal documents are comparatively easiest, likely because their references are densely structured.

  • Cross-domain method relationships: MULTATTNATTRIB and its Cohere variant generally outperform other methods across domains, while VLM and LLM baselines perform worst.Cohere + VLM and Cohere + LLM are similarly better than the VLM and LLM baselines.
  • Cross-domain method relationships: Changing document type does not affect relationships among baselines, indicating robust and impartial baseline implementations across tested domains.The domain comparison therefore changes absolute difficulty without materially changing intra-baseline relationships.
  • Domain difficulty: Academic documents are hardest, with a regime-wide unified F1 score of 0.54.Marketing is the second-worst domain, with an approximate 8% increase in F1 scores—the largest domain-to-domain jump for unified F1.
  • Domain difficulty: 6% is the domain-to-domain jump in F1 metrics between Legal, with an F1 score of 0.74, and Business.Legal documents tend to receive more accurate attributions because references are densely structured within clauses, claims, laws, or cases.

H Baseline Findings … I.2 Judge Results

Baseline performance varies by model, regime, modality representation, and retrieval: GPT-5.4 benefits more consistently than Qwen, while combined attribution remains difficult. Attribution quality is also evaluated with a three-judge GPT-4o panel, with separate results reported for text, image, and multimodal regimes.

  • H Baseline Findings: Image captions improve GPT baseline performance, particularly text and multimodal precision, but produce mixed or negative effects for Qwen.Qwen sees a slight boost in text-only regimes but degradation in image-only and text + image QAAs.
  • H Baseline Findings: RAG generally improves attribution metrics, with GPT-5.4 nearly closing the text-only gap while Qwen gains modestly and can decline in image-only attribution.The passage attributes the asymmetry to stronger generators exploiting a smaller retrieval-preselected candidate pool more effectively.
  • H Baseline Findings: Multimodal attribution remains challenging despite frontier-model gains, especially when evidence must be combined across modalities.GPT-5.4 exceeds Qwen3-VL-30B by 35.7 F1 points on text-only, 14.6 on image-only, and 10.7 on combined attribution.
  • H Baseline Findings: 10.7 F1 points separate GPT-5.4 and Qwen3-VL-30B on combined attribution, compared with 35.7 on text-only and 14.6 on image-only.Under direct VLM comparisons, the gap narrows to 1.8 points for combined attribution, versus 11.1 for text-only and 11.5 for image-only.
  • I.1 Judge Setup: Attribution quality is additionally assessed with a panel of three GPT-4o judges using distinct evaluator, critic, and mediator personas.Judges share discussion history, deliberate for up to two rounds, terminate early on unanimous consensus, and determine the final decision by majority vote.
  • I.2 Judge Results: The judge-results tables report LLM evaluations for Qwen and MULTATTNATTRIB in the text regime.These results are presented in Table 11.
  • I.2 Judge Results: The judge-results tables separately report LLM evaluations for Qwen and MULTATTNATTRIB in the image and multimodal regimes.These results are presented in Tables 12 and 13, respectively.

J Head Analysis · K QAA Rubrics · L Prompts

The paper analyzes cross-modal attention-head agreement, defines graded QA and verification rubrics, and specifies modality-specific prompts that enforce grounded, standalone, nontrivial question-answer generation and evaluation.

  • J Head Analysis: The study scores 1,536 attention heads using Mean Attention Scoring and CMA Scoring, comparing image-text agreement with IoU and Spearman’s rank correlation.The head count is N = L × H = 48 × 32 = 1536, with score matrices Simg and Stxt.
  • K QAA Rubrics: Text-only QAA verification uses four quality levels, from poor unsupported grounding to excellent, precise, direct, and convincing evidence.The rubric labels the levels Poor, Acceptable, Good, and Excellent.
  • K QAA Rubrics: Multimodal verification separately rates entity visibility from absent to clear and modality synergy from redundant to strong.The entity rubric has five levels, while the multimodal QA rubric has four levels describing how essential and complementary each modality is.
  • L.1.1 Image-only: Image-only prompts require source-agnostic, domain-grounded questions whose factual answers rely on legible raster content rather than grounding text.They prohibit perceptual, co-occurrence, annotation-dependent, and hallucinated questions while favoring diverse reasoning-first coverage.
  • L.1.2 Text-only: Text-only prompts require comprehension-based QA pairs with answers fully supported by one paragraph, 12–25 words long, and independent of external knowledge.The required output is valid JSON with a top-level triplets list containing question and answer keys.
  • L.1.3 Multimodal: Multimodal prompts require standalone retrieval questions whose answers add new factual content and cannot be constructed from either image or text alone.They target intersections such as diagram-visible numbers explained in prose, while rejecting annotation-dependent and pure blank-filling questions.
Loading 2607.01420v3…