PaperlayerView exact revision on arXiv

Source-linked AI summary

Query-Driven Multimodal Information Extraction from Long Documents

Yikai Gao, Ding Xia, Xi Yang

arXiv:2608.22214v1cs.AIcs.MM

TL;DR

Existing document understanding methods do not adequately retrieve query-specific textual attributes and their corresponding images from complex multimodal long documents. The paper defines this task, builds the ITJoint benchmark, and proposes Q2IT; experiments show standalone VLMs struggle, while Q2IT improves performance but remains far from perfect.

  • Problem

    Existing DocVQA and image-text extraction methods do not provide query-driven textual attributes paired with corresponding images for complex multimodal long documents.

  • Method

    The paper defines a two-level query- and instance-level taxonomy, constructs ITJoint, and evaluates standalone VLMs alongside the three-agent Q2IT workflow.

  • Results

    Standalone VLMs struggle with query-driven image-text joint extraction, while Q2IT brings clear improvements but remains far from perfect.

  • Takeaways & Limitations

    The findings demonstrate that query-driven image-text joint extraction remains difficult and that both VLMs and multi-agent system designs require further improvement.

  • Takeaways & Limitations

    ITJoint’s scale and representativeness are limited by the high cost of manually annotating documents, queries, answer instances, and image bounding boxes.

Abstract

from arXiv · show

In domain-specific multimodal long documents, images and text jointly convey complex knowledge that cannot be fully captured by plain text alone. However, existing paradigms like DocVQA primarily focus on generating textual answers or localizing evidence regions, rather than outputting query-specific textual attribute values and corresponding images. To address this gap, we propose query-driven image-text joint extraction from long documents, requiring models to output query-requested textual attribute values and corresponding image bounding boxes. Based on challenges related to both user intent and document content, we designed a two-level taxonomy that operates at the query and instance levels. Further, we construct ITJoint, the first high-quality, manually annotated benchmark for this new task, comprising 2,455 pages of domain-specific documents with numerous non-decorative images, 316 queries, and 910 answer instances. Finally, we evaluate representative standalone Vision-Language Models from different providers and further design Q2IT, a multi-agent collaborative framework consisting of three progressively collaborating agents for evidence collection, page selection, and target-image localization. Using a joint evaluation approach that assesses both text extraction and image localization, our experiments show that standalone VLMs struggle with this task, while Q2IT significantly improves performance on ITJoint, although a substantial gap remains toward perfect results.

1 Introduction

The paper defines query-driven image-text joint extraction to retrieve query-requested textual attributes with corresponding image bounding boxes from domain-specific multimodal long documents. It introduces the ITJoint benchmark and Q2IT framework to address complex user intents and document layouts.

  • Motivation: Domain-specific multimodal documents tightly couple text and images, so plain text alone cannot fully capture their knowledge.Transforming this information into structured image-text pairs otherwise relies heavily on manual annotation.
  • Task Definition: Query-driven image-text joint extraction retrieves query-requested textual attributes together with corresponding image bounding boxes.Unlike DocVQA, the task constructs query-specific multimodal instances rather than returning only textual answers or evidence regions.
  • Challenges: The task must handle uncertain set-query cardinality, nonlinear layouts, cross-page evidence, and non-one-to-one image-text mappings.These challenges span both user intent and document content, making simple rule-based association fragile.
  • Approach: Q2IT uses progressively collaborating Evidence, Page, and Figure Agents for evidence collection, page selection, and target-image localization.The paper evaluates this workflow alongside standalone VLMs.
  • Benchmark: ITJoint contains 2,455 document pages, 316 queries, and 910 answer instances in the first manually annotated benchmark for this task.The benchmark targets real-world image-text coupling in domain-specific long documents and averages 1.77 non-decorative images per page.

2 Related Work

Related work spans document visual question answering, document image-text pair extraction, and multi-agent frameworks for decomposing complex tasks. The paper positions its task at their intersection while targeting query-driven extraction from domain-specific long documents.

  • Document Visual Question Answering and Benchmarks: DocVQA benchmarks evolved from single-page to multidomain, visually rich, long-context document settings, with methods progressing from OCR-based to OCR-free and document-oriented multimodal models.These approaches primarily address answering questions from document images.
  • Image-Text Pair Extraction from Documents: Document image-text extraction methods recover figures, captions, coordinates, and subfigure-subcaption alignments, mainly from academic papers and domain-specific catalogs.Representative systems include PDFFigures 2.0, PDFigCapX, DeepFigures, and PDFChatAnnotator.
  • Multi-Agent Frameworks for Complex Tasks: Multi-agent frameworks decompose complex tasks into collaborative subprocesses through role specialization, conversation, or procedural intermediate artifacts.General systems include CAMEL, AutoGen, and MetaGPT, while downstream applications adapt decomposition to specialized workflows.

3 Task Definition and Benchmark

The task requires extracting query-specific textual attributes together with corresponding image evidence from long documents, supported by taxonomies and the ITJoint benchmark. ITJoint combines diverse document layouts with manually annotated queries, answer instances, and image-text relationships.

  • Task Formulation: Each answer instance pairs a query-satisfying textual attribute value with its associated image set, including an explicitly empty set when no image corresponds.Image locations are represented by page indices and bounding boxes.
  • Challenge Taxonomy: The taxonomy distinguishes single-instance queries from set queries, while separating positive cases from text-supported instances whose corresponding images are missing.Set queries require retrieving all satisfying instances despite an unknown target count.
  • Challenge Taxonomy: Instances are categorized as 1-to-0, 1-to-1, or 1-to-N according to the number of associated images.Positive instances receive additional labels for visual layout, spatial span, and alignment clues.
  • ITJoint Benchmark: The benchmark spans medicine, agriculture, archaeology, biology, and art, with English and Chinese documents containing diverse image-text layouts.Annotation uses a human-LLM workflow and the ITLabel visual tool for query revision, answer annotation, and taxonomy labeling.
  • ITJoint Benchmark: ITJoint contains 2,455 domain-specific pages, 316 queries, and 910 answer instances, averaging 1.77 non-decorative images per page.Set queries account for 679 answer instances, while 166 1-to-N instances contain 450 images.

4 Method

Q2IT progressively constructs answer sets through evidence collection, page selection, and target-image localization. Its agents retrieve and verify textual and page evidence before selecting image regions and binding them to final answer instances.

  • Framework Overview: Q2IT uses Evidence, Page, and Figure Agents to perform evidence collection, page selection, and target-image localization sequentially.The three stages progressively construct the answer set R.
  • Evidence Agent: The Evidence Agent decomposes queries into subqueries, retrieves relevant chunks, and expands seed evidence to clustered document regions for set-query recall.It also handles section-range requests and checks whether textual values are missing or incomplete.
  • Page Agent: The Page Agent maps textual and image-reference clues to candidate pages, then visually verifies those pages and their neighbors.Fallbacks include entity-based full-document matching and source-page retrieval.
  • Figure Agent: The Figure Agent detects candidate image boxes, filters them, and selects target regions using page context such as captions, labels, and layout relations.The Figure Checker removes duplicates, merges complementary subquery results, and produces final answer instances.

5 Experiments

Experiments compare standalone VLMs with Q2IT on ITJoint using joint text-and-image evaluation. Standalone models struggle, while Q2IT improves localization and joint extraction but leaves substantial errors, especially in difficult binding cases.

  • Evaluation: The evaluation matches predicted and gold instances using image-box overlap, then assesses image localization, text extraction, and their joint correctness.Strict Joint requires exact image-box cardinality and IoU at least 0.75; query-level macro-F1 and instance-level accuracy are reported.
  • Standalone VLMs: Gemini leads standalone VLMs across most categories, while Qwen is the strongest open-source model and GPT and GLM struggle with positive image-bearing instances.Gemini reaches 0.447 on single positive queries and 0.482 on 1-to-1 instances; GPT reaches only 0.113 and 0.024 on 1-to-1 and 1-to-N instances, respectively.
  • Q2IT workflow: Under Q2IT, Gemini remains the most stable backbone, while GPT becomes competitive on set positive queries and 1-to-1 instances.Gemini reaches 0.692 on single positive queries and 0.563 on 1-to-1 instances; GPT achieves 0.571 on set positive queries and 0.534 on 1-to-1 instances.
  • Q2IT workflow: Closed-source models remain stronger under Q2IT, whereas Qwen and GLM show distinct but limited strengths across multi-instance, alignment, negative, and no-image categories.Qwen is relatively stronger on set positive, 1-to-N, and implicit-alignment cases; GLM is stronger on negative, 1-to-0, and some explicit-clue categories.
  • Limitations: Most categories remain below 0.60 even under Q2IT, leaving substantial room for improving model capability and workflow design.This limitation persists despite the workflow’s gains over standalone inference.
  • Decomposed Analysis: 0.518 to 0.793 and 0.187 to 0.725 are Gemini’s and GPT’s Image-score improvements under Q2IT, respectively, while Page gains are smaller.The results indicate that Q2IT’s key benefit is converting page-level evidence into bbox-level image localization.
  • Decomposed Analysis: 0.722 versus 0.787 and 0.793 shows Gemini’s Joint score remains below its Text and Image scores under Q2IT.The remaining difficulty is binding textual attributes to corresponding images, especially in multi-image, compound-layout, and implicit-alignment cases.
  • Decomposed Analysis: 0.226 to 0.085 and 0.324 to 0.113 are GPT’s and GLM’s reduced Text–Joint gaps on 1-to-0 instances under Q2IT.The residual gap reflects incorrect image assignment when textual evidence exists but no corresponding image should be returned.

6 Conclusion

The paper introduces query-driven image-text joint extraction for domain-specific multimodal long documents and supports it with the ITJoint benchmark and Q2IT workflow. Experiments find that standalone VLMs struggle, Q2IT improves performance, and substantial difficulty remains.

  • Conclusion: The task requires retrieving query-requested textual attributes together with corresponding images from domain-specific multimodal long documents.It differs from textual answering by constructing query-specific multimodal instances.
  • Conclusion: The experiments compare standalone VLMs with the task-specific multi-agent workflow Q2IT.Q2IT is evaluated as a representative workflow-based solution alongside direct VLM inference.
  • Conclusion: Standalone VLMs struggle with the task, whereas Q2IT brings clear improvements but remains far from perfect.The findings emphasize the continuing difficulty of jointly extracting textual attributes and corresponding images.

Limitations

The paper identifies limitations in ITJoint’s scale, annotation boundaries, and Q2IT’s staged coordination. These constraints affect benchmark representativeness, fine-grained localization evaluation, and end-to-end flexibility.

  • Limitations: Manual annotation limits ITJoint’s scale and representativeness across document sources, domains, and dataset size.The process is especially costly for exhaustive set-query annotation.
  • Limitations: Nearby-page cases comprise a relatively large share of cross-page samples, potentially reducing the visible contribution of multi-strategy page search.
  • Limitations: Ambiguous boundaries between domain-specific target images and backgrounds may affect fine-grained image-localization evaluation despite unified annotation guidelines.
  • Limitations: Q2IT’s structured three-stage workflow improves controllability and traceability but depends on the quality of intermediate results across stages.Future work could explore more flexible planning and feedback while maintaining reliability.

Ethical Considerations

The dataset is constructed from publicly accessible materials without personal information, with stated respect for copyright and intellectual-property rights.

  • Ethical Considerations: ITJoint contains no personal information, and its source materials come from publicly accessible channels.
  • Ethical Considerations: Dataset construction and refinement are conducted with respect for copyright and intellectual-property rights.

A.1 Annotation Details

The annotation pipeline combines a document-level tool, human-LLM query generation, and rules for faithful multimodal answer annotation. ITLabel supports complex cross-page and multi-instance image-text links.

  • Tooling and workflow: ITLabel provides document-level browsing, OCR, search, query revision, attribute entry, box annotation, and taxonomy labeling.It treats the document as a unified interface rather than isolated page images.
  • Tooling and workflow: ITLabel supports cross-page, multi-instance, and multi-image links, including 1-to-N mappings between textual values and image boxes.
  • Data and query construction: Documents are selected for image richness, domain diversity, and realistic domain-specific multimodal scenarios.
  • Data and query construction: Queries are generated through LLM proposals followed by human filtering, revision, and supplementation.The rules emphasize objective, clear, diverse, nonredundant, and balanced queries.
  • Answer annotation: Annotators record every matching entity’s textual attributes, image boxes, and query- and instance-level labels under explicit annotation rules.Text must be copied faithfully; missing images receive empty image sets, and independent visual entities receive separate boxes.
  • Answer annotation: Set-query annotation requires exhaustive identification of valid instances, including cross-page relations, implicit alignment, and weak visual clues.The ground-truth format records text answers, image boxes, and taxonomy labels, including cross-page cases.

A.2 Instance-Level Taxonomy Examples

ITJoint’s instance taxonomy describes modality cardinality, layout, spatial span, and alignment clues across diverse bilingual domains. The examples also show how Q2IT and its components address hallucination, page search, and image localization.

  • Taxonomy dimensions: The taxonomy labels instances by modality cardinality, visual layout, spatial span, and alignment clues.Examples span medicine, archaeology, agriculture, biology, and art in English and Chinese.
  • Instance examples: A 1-to-N compound explicit instance can link multiple target images and textual evidence across pages through explicit references.
  • Instance examples: A 1-to-1 cross-page implicit instance requires contextual inference because no explicit marker establishes the image-text correspondence.
  • Instance examples: An intra-page independent implicit instance associates one textual value with multiple standalone image regions through layout and local context.
  • Instance examples: A 1-to-0 instance requires returning the textual value with an empty image set rather than hallucinating an image region.
  • Evaluation and ablations: Q2IT preserves correct text while avoiding an image on a 1-to-0 example where direct GPT inference hallucinates an image region.
  • Evaluation and ablations: Soft Joint permits fewer correctly matched boxes than gold while still penalizing irrelevant or hallucinated images, and Q2IT improves all backbones across categories.
  • Evaluation and ablations: Removing the Page Checker drops 1-to-0 accuracy from 0.802 to 0.531, while removing multi-strategy page search lowers cross-page accuracy from 0.349 to 0.234.
Loading 2608.22214v1…