PaperlayerView exact revision on arXiv

Source-linked AI summary

What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering

Guanchen Wu, Jiayuan Ding, Subhabrata Mukherjee, Carl Yang

arXiv:2608.14841v1cs.AI

TL;DR

Long-document multimodal VQA needs better evidence selection because text-only rerankers miss information in tables, charts, figures, and layout. Trident uses structured visual-semantic page annotations for adaptive reranking and multi-view generation, substantially improving retrieval across candidate pools and downstream QA.

  • Problem

    Long-document VQA evidence selection remains difficult because text-only rerankers miss table, chart, figure, and layout evidence in candidate pages.

  • Method

    Trident annotates candidate pages with visual and structural semantics for adaptive LLM reranking, then prompts a VLM through topical, entity, and structural lenses for synthesis.

  • Results

    BGE-M3 retrieval F1 rises from 0.254 to 0.546, and annotation-plus-rerank exceeds PageIndex across all five candidate pools.

  • Takeaways & Limitations

    Structured page annotations improve evidence selection even for multimodal retrievers, with visual captions recovering most of the ranking lift.

  • Takeaways & Limitations

    Trident-R requires an offline per-document indexing pass costing $0.256 per 100 pages, though the cost is amortized across queries.

Abstract

from arXiv · show

Long-document visual question answering (VQA) over documents of tens to hundreds of pages mixing text, tables, charts, and figures typically follows retrieve-then-read pipelines. In our setting, the bottleneck shifts from retrieval recall to reranker-side evidence selection: on MMLongBench-Doc, BGE-M3 reaches Recall@20 = 0.86 but only F1@5 = 0.254, and even the visual retriever ColPali reaches only F1@5 = 0.332; a text-only rerank LLM seeing only raw snippets misses table, chart, and layout evidence even when the upstream retriever encoded images. We propose Trident, with two complementary components: Trident-R, a retriever-agnostic LLM reranker that converts each candidate into an LLM-readable semantic record, including a visual caption, section path, entity tags, multi-axis concept hits, and a text snippet, then performs a single adaptive-K rerank call; and Trident-S, a generation-side module that prompts the VLM under topical, entity, and structural lenses before synthesis. On two long-document datasets, the annotation+rerank protocol substantially improves retrieval F1 across five heterogeneous pools, with every reranked pool exceeding the strongest adaptive-K baseline PageIndex. An LLM rerank without the annotation barely changes first-hit ranking, indicating the lift comes from the structured annotation. Trident-S targets open-ended synthesis questions by design, adding up to 6.6 points in generation accuracy on these questions. The best Trident configuration is the strongest downstream QA pipeline in our evaluation, with rankings consistent across two LLM judges (kappa = 0.913).

1 Introduction

Long-document multimodal VQA remains bottlenecked by selecting a small evidence subset from retrieved candidates, because text-only rerankers miss visual and layout evidence. TRIDENT addresses reranking and generation with structured candidate annotation and multi-lens VLM prompting.

  • Motivation: Oracle evidence pages lift LVLM F1 by 10–30 points over full-document inputs on MMLongBench-Doc, highlighting evidence selection as a practical bottleneck.Retrieve-then-read pipelines select a small set of pages because processing entire documents is expensive and can suffer position-induced degradation.
  • Evidence-selection bottleneck: Recall@20=0.86 for BGE-M3 yields only F1@5=0.254, while ColPali reaches F1@5=0.332, showing the challenge is reranking within the top-20 pool.Evidence pages are typically already present among 20 surface-similar candidates; the difficulty is identifying the small evidence subset.
  • Evidence-selection bottleneck: Text-only LLM reranking leaves BGE-M3 MRR essentially unchanged at 0.524 → 0.523 and lowers ColPali MRR from 0.692 to 0.598.Raw text snippets do not expose the table, chart, figure, and layout evidence encoded by multimodal retrieval.
  • TRIDENT-R: TRIDENT-R annotates each top-20 candidate and uses one LLM call to select an adaptive set of K≤5 evidence pages.The approach addresses rerank-stage blindness without requiring a full multimodal LLM call for every rerank query.
  • TRIDENT-S: TRIDENT-S prompts the VLM under topical, entity, and structural lenses before synthesizing the final answer.It targets the generation-stage need to balance attention across text, tables, and figures within each retrieved page.

2 Related Work

Related work spans long-document multimodal VQA benchmarks and retrieval systems, structured or adaptive document retrieval, and multimodal reranking and prompting. TRIDENT differs through a modular, retriever-agnostic annotation-plus-rerank interface and a generation-side multi-view prompting extension.

  • Long-document VQA benchmarks and multimodal RAG systems: MMLongBench-Doc and LongDocURL evaluate VQA over hundred-plus-page documents containing text, tables, charts, figures, and layout evidence.Prior systems combine multimodal retrievers with multimodal LLMs, hierarchical chunks, chunk–query graphs, structural triage, or structured table and figure descriptions.
  • Long-document VQA benchmarks and multimodal RAG systems: TRIDENT separates a swappable Stage 1 retriever from a Stage 2 annotation-plus-rerank protocol that can be empirically isolated.This design distinguishes it from prior multimodal RAG systems using multimodal retrievers, hierarchical indexing, chunk–query graphs, document-structure triage, or structured visual descriptions.
  • Document retrievers and adaptive-K selection: Prior document retrievers include sparse BM25, dense BGE-M3, late-interaction ColBERT-style models, RRF fusion, and vision-aware ColPali and DSE, while none condition on document-level structure.Adaptive-K selection has also been studied alongside these page-level retrieval approaches.
  • LLM rerankers with structured features: TRIDENT uses a modular, prompting-based multi-aspect annotation layer that is pairable with any candidate pool and VLM without retraining.Its visual captions provide an LLM-readable interface for multimodal evidence, and cross-pool isolation tests transfer to the visual ColPali pool through caption ablations.
  • Caption-based vs MLLM-based multimodal reranking: Caption-based rerankers textualize page images for text-only LLMs, whereas MLLM-based rerankers let multimodal LLMs inspect page images directly.TRIDENT targets long-document multimodal QA with VLM-generated captions and extends the caption-and-rerank lineage from single-image QA to long-document retrieval and generation.
  • VLM-augmented LLM QA and multi-view prompting: TRIDENT-S extends multi-view prompting to long-document generation, building on approaches that combine complementary analytical descriptions or caption-plus-rerank pipelines.Its retrieval side identifies visual captions as the strongest universal annotation field, while its generation side uses three orthogonal lenses.

3 Method

TRIDENT decomposes long-document VQA into candidate retrieval, multi-aspect page annotation, adaptive-K LLM reranking, and VLM generation. TRIDENT-R exposes structured visual, structural, entity, concept, and text evidence, while TRIDENT-S uses topical, entity/value, and structural lenses before synthesis.

  • TRIDENT-R pipeline: TRIDENT-R retrieves up to 20 candidate pages, annotates each page, and uses one LLM call to return an ordered subset of at most 5 pages.The candidate pool is retriever-agnostic, while annotation and reranking are the method’s focus.
  • Multi-aspect annotation: The page index combines topic, entity-bundle, and structural concepts with per-page metadata for section path, visual caption, salient entity tags, and truncated raw text.Visual captions describe visible tables, figures, charts, and layout in at most 60 words; entity tags retain the top 15 salient entities or numbers.
  • Candidate records: Each reranker record exposes five signals: concept hits, section path, visual caption, entity tags, and a truncated text excerpt.Concept memberships are collapsed into a typed list of at most 6 tags, allowing the LLM to attend to structured evidence rather than re-derive it from raw text.
  • Candidate-pool sources: Candidate pools include BM25, BGE-M3, reciprocal-rank fusion, ColPali, and the multi-axis XKG pool, each supplying the top 20 pages by score.XKG combines topic, entity, and structural concept scores using a query-classifying router and class-conditional fusion.
  • TRIDENT-S generation: TRIDENT-S prompts the VLM in parallel under topical, entity/value, and structural lenses, then synthesizes their three candidate answers into a final response.The synthesis prompt prefers the entity/value view for numeric questions, the structural view for location questions, and consensus otherwise.

4 Experiments

Experiments on MMLongBench-Doc and LongDocURL evaluate Trident’s retriever-agnostic annotation-plus-rerank interface and generation-side synthesis module across retrieval and end-to-end QA metrics. Structured annotation transfers across heterogeneous candidate pools, improves downstream QA, and yields the largest synthesis gains on open-ended questions.

  • Experimental Setup: Experiments use MMLongBench-Doc and LongDocURL, reporting page-level retrieval metrics and end-to-end QA accuracy.Retrieval metrics use an N=830 answerable MMLongBench-Doc subset; end-to-end accuracy uses a 300-QA stratified subset.
  • Cross-Pool Retrieval: BGE-M3 F1 rises from 0.254 to 0.546 with annotation plus reranking, and all five candidate pools exceed PageIndex.The same protocol transfers across sparse, hybrid, graph, dense, and visual pools.
  • Cross-Pool Retrieval: ColPali F1 rises from 0.332 to 0.581 despite already encoding page images, showing the interface adds value beyond missing visual information.This result indicates that converting candidates into LLM-readable annotations improves visual-retriever pools as well as text-retriever pools.
  • Mechanism Ablations: Raw-snippet reranking leaves BGE-M3 MRR essentially unchanged at 0.524 →0.523 and reduces ColPali MRR from 0.692 →0.598.The ranking lift appears only when the reranker receives structured page annotation; the visual caption alone recovers most of the lift on both pools.
  • End-to-End QA: ColPali with annotation and reranking is the best observed end-to-end configuration, while adding TRIDENT-S produces the highest Gen Acc and Gen F1.Pipeline rankings are consistent across two judges, with κ=0.913.
  • Generation-Side Synthesis: TRIDENT-S adds +1.7 to +6.6 Gen Acc across all eight evaluated pipelines on open-ended MMLongBench-Doc questions, with PageIndex Plain gaining +6.6.It is bypassed by design on LongDocURL’s MCQ-style and short-extractive workloads, where the three lenses converge on the same extracted answer.

5 Limitations and Conclusion

The paper concludes that LLM-readable semantic annotations address a reranker-interface bottleneck in long-document multimodal QA, while noting practical and evaluation limitations. TRIDENT-R is retriever-agnostic and improves retrieval F1 across five candidate-pool families, with every reranked pool exceeding PageIndex.

  • Limitations: TRIDENT-R requires an offline per-document indexing pass costing $0.256/100 pages, amortized across queries.The reranker uses GPT-4.1 here, but smaller or open-source LLMs are drop-in substitutes.
  • Limitations: The evaluation isolates five candidate pools on MMLongBench-Doc and uses LongDocURL as an external transfer check.PageIndex is used only as a section-path parser, and public PDF parsers such as PyMuPDF or GROBID could replace it.
  • Conclusion: Evidence selection improves when candidate pages become LLM-readable semantic annotations, even with multimodal retrievers.TRIDENT-R implements this approach as a retriever-agnostic annotation-based LLM reranker.
  • Conclusion: Across five candidate-pool families on MMLongBench-Doc, the same interface substantially improves retrieval F1, and every reranked pool exceeds PageIndex.The conclusion identifies this improvement as a reranker-interface bottleneck in long-document multimodal QA.

A Additional End-to-End Results · B TRIDENT-R Rerank Prompt · C TRIDENT-S Lens Prompts

The appendix provides full end-to-end QA reporting and specifies the prompts for Trident-R reranking and Trident-S’s three-lens generation and synthesis. Trident-R uses structured candidate records for adaptive page selection, while Trident-S analyzes the same retrieved images topically, by entities and values, and structurally.

  • A Additional End-to-End Results: Appendix Tables 5 and 6 report full end-to-end QA results for 300-question subsets of MMLongBench-Doc and LongDocURL, including metrics omitted from the main paper.Table 5 includes answer accuracy and unanswerable-abstention columns; Table 6 covers an all-answerable LongDocURL subset.
  • B TRIDENT-R Rerank Prompt: The Trident-R prompt supplies GPT-4.1 with a runtime-filled candidate JSON list, the question, and a maximum page count of K=5.It requires JSON output containing reasoning and a page list, with at most K pages returned.
  • B TRIDENT-R Rerank Prompt: Trident-R prioritizes pages with multiple distinct concept hits and selects the strongest evidence pages from candidate records.Concept hits use topical, entity, and structural prefixes, while records include page identifiers, section information, visual descriptions, entities, and scores.
  • B TRIDENT-R Rerank Prompt: The uniform retriever_score field preserves pool-specific Stage-1 scores while making the rerank prompt retriever-agnostic.Its semantics cover BM25, cosine, reciprocal-rank fusion, ColPali MaxSim, and the paper’s multi-channel XKG fusion.
  • C TRIDENT-S Lens Prompts: Trident-S makes three parallel VLM calls over the same retrieved page images, using topical, entity/value, and structural/section lenses before synthesis.The topical lens connects page themes to the question; the entity/value lens extracts named entities and exact values; the structural lens uses section paths and captions.
  • C TRIDENT-S Lens Prompts: Each lens can return “Not answerable” when its retrieved pages lack sufficient topical, entity/value, or structural evidence.The synthesis stage also returns “Not answerable” when all three lenses agree that the question cannot be answered.
  • C TRIDENT-S Lens Prompts: The synthesis prompt prefers convergent lenses, resolves specificity and contradictions, and matches lens choice to question type and answer format.Numeric questions favor the entity/value view, section-location questions favor the structural view, and topic questions favor the topical view.

D Query-Type Router · E 300-QA Subset Construction

The query-type router classifies questions and extracts salient entities to adapt topical, entity, and structural score fusion. The 300-QA subset emphasizes long documents while largely matching the full dataset’s distributions, with released QA identifiers supporting reproducibility.

  • D Query-Type Router: GPT-4.1 at τ=0 classifies each question into entity-level, topical, structural, or mixed and extracts salient entities in JSON mode.The extracted entities include named entities, technical terms, monetary amounts, percentages, and figure or table references.
  • D Query-Type Router: The class label controls fusion weights αT, αE, and αS for topical, entity, and structural concept scores.These weights are applied when aggregating the three concept channels into a candidate-pool score.
  • E 300-QA Subset Construction: Most axes match the full MMLongBench-Doc distribution within ±3pp, including Evidence Source, Evidence Pages, and Unanswerable rate.The subset is therefore broadly distribution-matched despite its deliberate long-document emphasis.
  • E 300-QA Subset Construction: Answer format is biased toward Float by +5.9pp and away from Str by −6.1pp in the 300-QA subset.These are the principal reported answer-format deviations from the full dataset.
  • D Query-Type Router: −0.003 F1 results from replacing class-conditional routing with uniform weights (1/3, 1/3, 1/3) on the cross-pool isolation setup.The comparison uses the same N=830 aligned subset as Table 1, indicating a small but consistent router contribution.
  • E 300-QA Subset Construction: The 300-QA subset is sampled from MMLongBench-Doc with seed 42 and emphasizes the long-document setting.Its median document length is 72 pages, compared with 28 pages in the full dataset.
  • E 300-QA Subset Construction: The LongDocURL subset contains 200 QAs sampled analogously, stratified by document length, answer format, and evidence source.Released QAid lists for both subsets accompany the code release.

F Bootstrap CIs, Wilcoxon and McNemar tests

The paper uses bootstrap confidence intervals and Wilcoxon signed-rank tests for retrieval, while evaluating end-to-end generation accuracy with continuity-corrected McNemar tests. At N=300, no top-tier pipeline comparison reaches p < 0.05, although rankings remain consistent across the three LLM judges.

  • Retrieval: Retrieval evaluation uses non-parametric 95% bootstrap CIs on per-QA F1 with B=10,000 resamples from 830 aligned QAs.The reported CIs cluster tightly within ±0.024.
  • Retrieval: Wilcoxon signed-rank tests on per-QA F1 are preferred over McNemar tests on STRICT because methods use different effective K.Fixed-K baselines use K=5, whereas adaptive-K methods have mean K ∈[2.7, 3.0], which would structurally favor larger-K methods.
  • End-to-end: At N=300, none of the pairwise comparisons among the top-tier end-to-end pipelines reaches p < 0.05 under continuity-corrected McNemar testing.The comparisons involve ColPali+ann. +TRIDENT-S, TRIDENT-R+TRIDENT-S, BGE-M3+ann. +TRIDENT-S, and PageIndex+TRIDENT-S; rankings are consistent across all three LLM judges with inter-judge κ=0.913.

G Case Study Records · H Detailed Cost Breakdown

The case studies show TRIDENT-R answering chart, financial-ratio, and calibrated-refusal questions correctly while PageIndex and BGE-M3 raw fail. The cost analysis reports document-level and per-query costs, with entity extraction dominating offline indexing.

  • G Case Study Records: Scores use GPT-4o-extracted short answers and rulebased Generalized Accuracy, where 1.0 is correct and 0.0 is wrong.Answers follow the official MMLongBench-Doc evaluation pipeline.
  • G Case Study Records: TRIDENT-R retrieved pages {6, 5} and answered the chart-axis question “[Context Length, F1]” correctly with score 1.0.PageIndex and BGE-M3 raw both returned “Not answerable” with score 0.0.
  • H Detailed Cost Breakdown: Token accounting covers the 134-document MMLongBench-Doc corpus, totaling 6,492 pages with mean 103 text tokens and 40 caption tokens per page.The accounting uses stated public OpenAI 2025 rates and approximately 1,500 image tokens per full page at ZOOM=2.0.
  • H Detailed Cost Breakdown: $0.256 per 100-page document is reported, rising to approximately $0.36 when per-page text tokens double; per-query costs are $0.012 plain and $0.039 for TRIDENT-S.TRIDENT-S costs reflect 3 lens calls plus 1 synthesis, and the document estimate is conservative for dense documents.
  • H Detailed Cost Breakdown: 78% of offline indexing cost comes from entity extraction, while concept naming and visual captioning contribute approximately 10% each and page embedding 1%.The breakdown identifies entity extraction as the dominant offline indexing expense.

I Detailed Field Ablation on Two Pools · J Adaptive-K Distribution by Question Type · K Visual Caption QC: Substring and N-gram Overlap

Detailed ablations show that visual captions account for most annotation gains, while adaptive K remains stable across candidate pools but varies by question type. Caption audits further indicate that occasional answer overlap reflects visual evidence encoding rather than systematic leakage.

  • I Detailed Field Ablation on Two Pools: Each ablation row removes exactly one annotation field from the rerank candidate JSON on BGE-M3 and ColPali pools.The aligned evaluation contains N=830 examples with Kmax=5.
  • I Detailed Field Ablation on Two Pools: Removing captions causes a larger drop on BGE-M3 than ColPali: −0.130 versus −0.083.The drop is 36% smaller on the visual pool.
  • J Adaptive-K Distribution by Question Type: Mean adaptive K is stable across candidate pools, ranging from 2.55 to 3.11.This pattern suggests K selection is driven by evidence requirements rather than the retriever supplying candidates.
  • J Adaptive-K Distribution by Question Type: Pure-text questions request the most pages, with mean K approximately 3.0 and a K=5 cap rate near 30%.Figure or Chart questions request fewer pages, with mean K approximately 2.6 and K=1 exceeding 30% on Figure questions.
  • K Visual Caption QC: Substring and N-gram Overlap: Exact gold-answer substrings occur in 22/238 captions, or 9.2%, after auditing 200 randomly sampled answerable QA/candidate-page pairs.The audit retained N=238 caption-page records after dropping “Text-only page.” captions.
  • K Visual Caption QC: Substring and N-gram Overlap: A 0% bigram match rate indicates that captions do not near-verbatim paraphrase answer sentences.The 9.2% substring matches reflect visual content such as chart legend colours or diagram labels coinciding with gold answers.

L Inter-Judge Agreement

Two independent LLM judges scored identical outputs with the same rubric and gold answers, yielding high agreement and identical pipeline rankings. Human evaluation was omitted because the study covered 4,200 prediction-gold pairs, so inter-judge agreement served as a proxy.

  • Evaluation protocol: Two independent LLM judges scored the same 300-QA outputs from 14 pipelines using an identical three-level rubric and MMLongBench-Doc gold answers.The judges were GPT-4.1 and Claude Sonnet 4.5; the rubric labels were correct, partial, and wrong.
  • Agreement results: Mean κ=0.913 indicated almost perfect agreement between GPT-4.1 and Claude.Table 11 reports per-pipeline pairwise agreement and Cohen’s κ.
  • Ranking consistency: The pipeline-level ranking was identical under both judges, with ColPali + ann. + rk + TRIDENT-S best and TRIDENT-R + TRIDENT-S second under Claude and tied second under GPT.This agreement indicates that the observed end-to-end ranking was not dependent on a single judge’s quirks.
  • Evaluation limitation: Human evaluation was not performed because the study involved 14 × 300 = 4,200 prediction-gold pairs.The authors treated high inter-judge agreement and matching rankings between two independently developed LLMs as a reasonable proxy.

M Extended Limitations Discussion

The limitations concern dependence on GPT-4.1, incomplete replication of the full LongDocURL cross-pool study, negligible TRIDENT-S gains on extractive questions, and untested parser substitution. The authors leave smaller-reranker, broader transfer, and parser-robustness evaluations for future work.

  • Reranker LLM dependence: GPT-4.1 at τ=0 is used for reranking, while smaller and open-source rerankers remain untested because of budget constraints.The authors expect the annotation-conditioned protocol to be reranker-agnostic because candidate JSON exposes per-page evidence to instruction-tuned LLMs.
  • LongDocURL coverage: N=1,122 LongDocURL QAs with non-empty gold evidence support adaptive-K comparisons, but full five-pool isolation is replicated only on MMLongBench-Doc.End-to-end LongDocURL results confirm the broad pattern, without separately reproducing the full cross-pool study.
  • TRIDENT-S on extractive workloads: ∆≈0 across four LongDocURL pipelines shows TRIDENT-S adds no mean benefit on MCQ-style and short-extractive questions.The topical, entity, and structural lenses converge on the same extracted answer, motivating gating TRIDENT-S by question format.
  • Parser substitution: The PageIndex API parses section paths independently from PageIndex retrieval, but parser-substitution robustness remains unstudied.The authors note that public parsers such as PyMuPDF and GROBID could provide the required section tree.
Loading 2608.14841v1…