PaperlayerView exact revision on arXiv

Source-linked AI summary

DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents

Zhuoran Yu, Le Thien Phuc Nguyen, Jaden Park, Xinyi Gu, Zexue He, Soochahn Lee, Rogerio Feris, Yong Jae Lee

arXiv:2609.02059v1cs.AIcs.CVcs.LG

TL;DR

Existing benchmarks largely isolate chart and document understanding, leaving integrated reasoning over textual constraints and visual evidence underexplored. DocHop addresses this gap with a logic-first benchmark whose narrative defines multi-step constraints and whose charts provide numerical evidence. Humans exceed 90% accuracy, whereas the best model reaches 62.83%, with reasoning helping and greater complexity degrading performance.

  • Problem

    Existing benchmarks predominantly evaluate charts and documents in isolation, leaving models’ ability to bind textual constraints to visual evidence underexplored.

  • Method

    DocHop uses stochastic logic-first generation to create document images where narrative constraints define target entities and multiple charts supply evidence for six reasoning task categories.

  • Results

    DocHop exposes a substantial human–model gap: annotators exceed 90% accuracy, while GPT-5.2 Reasoning reaches 62.83% overall; reasoning variants improve over base models and complexity reduces performance.

  • Takeaways & Limitations

    DocHop provides a controlled testbed for evaluating multi-hop chart–context reasoning and diagnosing failures as reasoning depth and chart count increase.

  • Takeaways & Limitations

    DocHop is synthetic and controlled rather than representative of naturally occurring documents, and focuses on charts paired with narrative context in single-page layouts.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have achieved strong performance on structured visual understanding tasks such as chart and document question answering. However, existing benchmarks typically evaluate these domains in isolation, leaving underexplored a key capability: whether models can use textual context to determine how chart evidence should be selected, interpreted, and aggregated. We introduce DocHop, a benchmark for integrated chart--context reasoning in document-style images. In DocHop, the document narrative specifies multi-step compositional constraints, while charts provide the corresponding data values. Questions are grounded on a semantic reference label defined in the narrative, requiring models to resolve target entities from context before aggregating evidence across multiple charts. To enable systematic evaluation, we construct DocHop via a stochastic logic-first generation pipeline with controllable reasoning depth and visual density, covering 2,074 examples across six task categories. Experiments on a wide range of proprietary and open-source MLLMs show a substantial gap to human performance: annotators achieve over 90% accuracy, while the best model reaches only 62.83%. Reasoning-enhanced models consistently show improved results, but performance degrades as reasoning complexity increases. Overall, DocHop provides a controlled testbed for challenging multi-hop document reasoning.

1. Introduction

DocHop benchmarks whether models can bind narrative constraints to chart evidence in multi-hop document reasoning. Its controlled evaluation reveals substantial model–human gaps and worsening performance with greater reasoning complexity.

  • DocHop addresses the underexplored need to bind textual constraints to visual evidence for downstream reasoning.
  • The benchmark combines narrative-specified multi-step constraints with multiple charts, requiring target resolution from context before numerical aggregation.
  • DocHop uses stochastic logic generation and independent chart–narrative synthesis to support controlled reasoning complexity and semantic consistency.
  • 2,074 examples span 480 topics, 7 chart types, 6 task categories, and 41 question templates.
  • Over 90% human accuracy contrasts with 62.83% for the best-performing model, while reasoning-enhanced variants consistently outperform non-reasoning counterparts.
  • Performance degrades steadily as reasoning depth or chart count increases.

2. Related Work

Prior multimodal benchmarks evaluate charts, documents, and webpages largely in isolation. This leaves complex multi-modal multi-hop reasoning that integrates textual context with visual evidence underexplored.

  • MLLMs perform strongly across images, documents, and structured layouts, but complex multi-modal multi-hop reasoning remains underexplored.
  • Earlier chart question-answering datasets often rely heavily on questions generated from limited templates.
  • ChartQA adds human-authored questions and linguistic diversity but remains focused on single-chart reasoning without external textual context.

3. DocHop

DocHop constructs logic-first document images by separating symbolic reasoning specifications from chart values, then grounds questions on narrative-defined semantic reference labels. The benchmark covers varied multi-hop tasks and computes answers from the generated chart tables.

  • Overview: DocHop separates symbolic reasoning traces, narrative constraints, and numerical chart evidence to enforce joint chart–context reasoning.
  • Construction Pipeline: The pipeline curates metadata, samples chart schemas, generates symbolic traces, synthesizes charts and narratives independently, and renders QA documents.
  • Symbolic Reasoning Trace: Reasoning traces are logical trees with target depth D ∈{2, 3, 4, 5}, constraint nodes, and Boolean composition nodes using AND or OR.
  • Symbolic Reasoning Trace: The stochastic recursive generator produces structures ranging from linear chains to multi-branch trees across sampled chart schemas.
  • Chart and Context Generation: Narratives verbalize constraints without chart values or final entity assignments, while independently generated chart tables satisfy the instantiated trace.
  • Question Curation: Semantic reference labels identify trace-derived entity sets, forcing questions to require narrative resolution before chart evidence aggregation.
  • Task Definitions: The six task categories are Value Retrieval, Counting, Numeric Reasoning, Ranking, Hypothetical Reasoning, and Fact Checking.

4. Experiments

Experiments show a substantial human–model gap on DocHop, although explicit reasoning improves proprietary-model performance. Accuracy declines as chart count or reasoning depth increases, with qualitative errors spanning narrative grounding, entity attribution, and chart value extraction.

  • Main Results: Over 90% human accuracy contrasts with 62.83% for GPT-5.2 Reasoning, the best-performing evaluated model.Human annotators took approximately five minutes per question on average.
  • Main Results: GPT-5.2 improves from 40.36% to 62.83% overall with explicit reasoning, while Gemini-2.5-Flash rises from 24.88% to 32.02%.Gemini-2.5-Pro with dynamic reasoning reaches 40.60% overall accuracy.
  • Main Results: GPT-5.2 Reasoning reaches 62.83% overall accuracy, while Gemini-2.5-Pro Reasoning reaches 40.60%; open-source models generally score roughly 9–24%.The proprietary models generally outperform the open-source baselines in this evaluation.
  • Analysis: Accuracy decreases for both reasoning models as the number of charts and reasoning depth increase, with highest performance at k = 2 and D = 2.Increasing either axis makes cross-chart aggregation under narrative constraints more difficult.
  • Analysis: Qualitative failures include attributing values to the wrong entity, selecting the global maximum instead of the semantically qualified set, and misreading chart values.The examples show that models can fail at different stages of integrated reasoning.

Limitations and Future Work

DocHop is a controlled diagnostic benchmark rather than a replica of naturally occurring documents. Its chart-focused, single-page design limits visual and structural coverage, while synthetic generation may still produce occasional issues.

  • Scope: DocHop does not reproduce naturally occurring document distributions, so its performance measures a specific reasoning capability rather than real-world document performance.The authors propose extending the framework to reports, scientific articles, dashboards, and other domains.
  • Scope: DocHop currently pairs narrative context with chart-based numerical evidence, excluding tables, diagrams, forms, equations, and interface screenshots.The authors identify heterogeneous document structures as a target for future extension.
  • Rendering: Controlled single-page A4-style layouts reduce severe OCR, parsing, and layout confounds but do not capture naturally occurring visual variability.Future work may introduce multipage documents, noisier scans, and more diverse layouts.
  • Quality: Despite programmatic verification and human review, synthetic generation may leave occasional annotation, wording, or rendering issues.The benchmark will be maintained through versioned updates and validated community feedback.

5. Conclusion

DOCHOP evaluates integrated chart–context reasoning, requiring models to resolve narrative constraints and aggregate evidence across multiple charts. Results show persistent difficulty on dense multi-hop reasoning despite gains from reasoning-enhanced variants.

  • DOCHOP requires models to resolve narrative constraints and aggregate numerical evidence across multiple charts.Semantic reference labels prevent direct chart-only lookup and enforce joint reasoning over text and visual data.
  • Reasoning-enhanced MLLMs consistently improve accuracy over their base counterparts.
  • Accuracy steadily degrades as reasoning depth and chart count increase.This exposes persistent challenges in multi-hop evidence aggregation under dense visual contexts.
  • The benchmark reveals a substantial gap between MLLM performance and human performance.

Impact Statement

The work advances multimodal evaluation through DOCHOP rather than deployment-facing model components. Its impacts are indirect, with potential misuse if systems are overoptimized for benchmark performance without real-world generalization.

  • DOCHOP is intended to support more reliable assessment of multimodal reasoning capabilities.
  • The work does not introduce new deployment-facing model components.
  • Potential societal impacts are indirect, including improved robustness and transparency of document-understanding evaluation.
  • Benchmark overoptimization could produce gains without corresponding real-world generalization.The paper encourages broader document coverage and responsible evaluation practices.

A. LLM Prompts for Data Curation

The appendix describes LLM prompts for curating metadata, synthesizing chart tables, and generating narrative context under logic and data-design constraints.

  • A. LLM Prompts for Data Curation: The curation pipeline uses an LLM to synthesize chart tables and accompanying document narratives from sampled symbolic reasoning traces.Chart and context generation are performed independently to improve reliability and prevent information leakage.
  • A. LLM Prompts for Data Curation: Three prompt classes curate metadata, synthesize per-chart tables, and verbalize reasoning traces into narrative context.The narrative-generation prompt introduces the semantic reference label used for question grounding.
  • A. LLM Prompts for Data Curation: Metadata specifies entity types, entities, groups, metrics, units, data types, and timestamps.The example uses three bottling plants, three groups, three metrics, thousand-bottle units, integers, and monthly timestamps.
  • A. LLM Prompts for Data Curation: Metrics must be independent, naturally named, summable, and paired with specific measurement units.The rules prohibit rates and encourage scale prefixes such as Million USD, Thousand Liters, and Metric Tons.
  • A. LLM Prompts for Data Curation: The prompt rules enforce a consistent abstraction level by choosing either components or aggregates and excluding redundant formulas.Examples reject Revenue, Cost, and Profit together because Profit = Revenue - Cost.
  • A. LLM Prompts for Data Curation: Groups must be universal, orthogonal categories that apply meaningfully across all selected metrics.The rules also prohibit combining totals with their parts or including variables that are simple mathematical functions of others.

SAFETY & CONTENT GUIDELINES (STRICT)

The appendix specifies strict content, naming, chart-generation, constraint, and narrative rules for constructing synthetic benchmark documents. These rules aim to preserve safe fictional content and semantic consistency while producing visually distinguishable data and explicit reasoning traces.

  • SAFETY & CONTENT GUIDELINES (STRICT): Synthetic content excludes biological fluids, gore, violence, harm, sensitive real-world topics, and real-world entities.The rules substitute benign alternatives and require realistic fictional names instead of real countries, cities, companies, or famous people.
  • SAFETY & CONTENT GUIDELINES (STRICT): Chart prompts must follow assigned chart types, entities, units, metrics, timestamps, and constraints.Outputs use a JSON object containing valid CSV with entity names, exact metric or group headers, and values in the specified unit.
  • SAFETY & CONTENT GUIDELINES (STRICT): Constraint interpretation requires named entities to satisfy a rule and all unnamed entities to violate it.For example, if A and B must meet a threshold, every other entity must fall below that threshold.
  • SAFETY & CONTENT GUIDELINES (STRICT): The narrative generator translates decision trees into professional policy documents with explicit topology and logical-composition requirements.Serial nodes inherit eligibility, convergent nodes preserve operand order, and AND and OR receive distinct prose interpretations.
  • SAFETY & CONTENT GUIDELINES (STRICT): Narrative architecture may introduce the final label before prerequisites or only in the concluding summary.Policy prose must translate formulas faithfully, use the supplied entity type, and express maximum or minimum rules as comparative rankings.

B. Examples of Symbolic Reasoning Traces

DocHop represents sampled reasoning traces as logical trees that filter chart entities through instantiated constraints, from linear chains to branched AND/OR compositions. Question templates then query entities identified through these traces using semantic reference labels.

  • Symbolic Reasoning Traces: Reasoning traces are logical trees whose nodes encode parameterized constraints and whose edges propagate valid entity sets through filtering steps.The sampled structures range from simple linear chains to multi-branch compositions with Boolean operators.
  • Atomic Constraint Templates: Atomic templates express chart-grounded conditions such as thresholds, within-entity comparisons, temporal trends, and extremal selection.Templates are instantiated with concrete timestamps, metrics, groups, operators, and numeric thresholds.
  • Atomic Constraint Templates: Non-unique atoms support gradual filtering because multiple entities may satisfy them, whereas unique atoms deterministically select an extremal entity.Examples include “Revenue ≥70” for non-unique filtering and highest-value selection for unique constraints.
  • Question Templates: Questions use task-specific formats such as retrieval, counting, aggregation, ranking, hypothetical updates, and fact checking.Key fields including the target metric, timestamp, aggregation operator, and additional condition are filled from instance metadata.
  • Question Templates: Questions refer to a semantic reference label rather than explicit chart entities, requiring models to resolve the target set before retrieving or aggregating values.Across six task categories, 41 templates cover direct queries and compositional variants with additional conditions or counterfactual modifications.

E. Evaluation Details

Evaluation uses a unified single-image prompt and standardized answer formatting, followed by parsing, normalization, and task-independent correctness checks. The protocol supports exact matching for categorical answers and tolerance-based numeric comparison.

  • Evaluation Setup: All models receive a single document-page image containing narrative text and one or more charts, with a shared instruction enforcing answer-span and formatting conventions.The instruction also requires the fixed marker “The answer is:” for robust parsing.
  • Answer Parsing: Responses are parsed from the final occurrence of “the answer is:” and otherwise fall back to the last non-empty line.Extracted strings are normalized by removing prefixes, whitespace, and lightweight formatting tokens.
  • Correctness Criteria: Correctness is first determined by case-insensitive exact match, with numeric comparison used when exact matching fails.The same two-step procedure applies across task types, including fact checking and entity-name answers.
  • Output Formatting: Models are instructed to return a single pure value, using complete entity names or appropriately rounded numbers without units or symbols.Integers remain integers, while decimals are rounded to two places unless the question specifies otherwise.

F. Model Configuration

The evaluation reports configurations for proprietary APIs and open-source checkpoints, with visual-tokenization settings specified for Qwen models. Generation budgets are increased to accommodate DocHop’s multi-step reasoning requirements.

  • Model Configuration: Proprietary evaluations specify exact model versions, while open-source baselines identify their corresponding HuggingFace checkpoints.These configurations are reported for all models considered in the experiments.
  • Model Configuration: Qwen-2.5-VL and Qwen3-VL use minimum resolution 1280 × 28 × 28 and maximum resolution 16384 × 28 × 28.The paper follows the visual-tokenization settings recommended for these models.
  • Generation Settings: All supported models use a maximum generation length of 214 tokens to reduce premature truncation during multi-step aggregation.The budget is set because DocHop is reasoning-intensive.
Loading 2609.02059v1…