PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGE: From Direct Answering to Evidence-Grounded Inference for Chinese Ancient Document Understanding

Yuchuan Wu, Xuan Luo, Yinglian Zhu, Meng Fang, Xiangyang Xue, Bin Li

arXiv:2608.24011v1cs.CLcs.AI

TL;DR

Chinese ancient document understanding requires visual, linguistic, and historical reasoning, yet LVLMs often rely on opaque, single-pass generation. SAGE recasts the task as evidence-grounded multi-agent inference and consistently surpasses matched direct-answering baselines across three backbones on AncientDoc.

  • Problem

    Chinese ancient document understanding combines visual reading, classical Chinese interpretation, and knowledge-grounded reasoning, while direct LVLM generation is monolithic rather than evidence-grounded.

    Source: §1 · p. 1

  • Method

    SAGE coordinates role-specialized agents for planning, tool-mediated evidence acquisition, claim-level verification, and bounded replanning within a constrained shared-state runtime.

    Source: §1 · p. 2

  • Results

    Across three LVLM backbones, SAGE surpasses matched direct-answering baselines on all evaluated metrics; Qwen3.5-9B achieves state-of-the-art performance on seven of eight metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Making evidence acquisition and verification explicit provides a traceable basis for diagnosing unsupported or insufficiently grounded answers.

    Source: §5 · p. 8

    Evaluation is limited to AncientDoc’s tasks and data distribution; applicability to broader historical collections, document layouts, and ancient languages remains unvalidated.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Chinese ancient document understanding demands complex visual, linguistic, and historical reasoning. Current Large Vision-Language Models (LVLMs) typically rely on an opaque, single-pass generation paradigm, often producing overconfident and weakly grounded responses. To address this, we propose SAGE, an evidence-grounded multi-agent framework that reformulates Chinese ancient document understanding as evidence-grounded inference rather than direct answer generation. SAGE coordinates specialized agents for task-aware planning, tool-mediated evidence acquisition, claim-level verification, and bounded replanning under a constrained shared-state runtime. This design supports bounded evidence seeking, answer revision, and abstention when grounding is insufficient. Experiments on the AncientDoc benchmark show that SAGE consistently outperforms matched direct-answering baselines across three LVLM backbones. Remarkably, SAGE with Qwen3.5-9B surpasses much larger monolithic LVLMs on most evaluated metrics, highlighting the importance of structured, evidence-grounded inference beyond model scaling.

1 Introduction

Chinese ancient document understanding requires visual reading, classical Chinese interpretation, and knowledge-grounded reasoning. SAGE replaces direct answering with evidence-grounded inference that supports verification, replanning, and abstention when evidence is insufficient.

1 Introduction

  • Direct-answering LVLMs collapse reading, evidence acquisition, reasoning, and verification into one opaque pass, often yielding overconfident, weakly grounded answers that are difficult to diagnose.Ancient documents combine vertical layouts, variant characters, missing punctuation, dense annotations, and implicit historical-cultural semantics, requiring visual reading, classical Chinese interpretation, and knowledge-grounded reasoning.Source: §1 · p. 1
  • SAGE coordinates planning, tool-mediated evidence acquisition, and claim verification within a constrained shared-state runtime that regulates tools and budgets and can stop, replan, or abstain.Tools include page reading, text normalization, term extraction, and local evidence retrieval.Source: §1 · pp. 1–2
  • Across three LVLM backbones, SAGE surpasses matched direct-answering baselines on all evaluated metrics; Qwen3.5-9B achieves state-of-the-art on seven of eight metrics, outperforming much larger monolithic LVLMs.SAGE’s largest gains are on knowledge-intensive and linguistic tasks; trace analyses find task-adaptive tool use and verifier-guided replanning improve evidence support and verifier confidence.Source: §1 · p. 2
  • SAGE reframes ancient-document understanding as evidence-grounded inference, exposing supporting evidence, verification reports, abstention decisions, and execution traces alongside answers.This approach reflects scholarly practice of checking visual clues and references and expressing uncertainty when evidence is lacking.Source: §1 · pp. 1–2

2 Related Work

Prior benchmarks and ancient-Chinese models address document reading and language understanding, but mostly evaluate final answers or focus on text-centric tasks. SAGE instead frames image-based ancient-document understanding as model-agnostic, evidence-grounded inference with verification, abstention, and trace diagnostics.

2.1 Document and Ancient-Document Understanding Benchmarks

  • General document benchmarks assess reading, localization, and reasoning, but mainly cover contemporary documents, scene text, or general multimodal inputs (Mathew et al., 2020; Singh et al., 2019; Liu et al., 2024; Fu et al., 2024; Ma et al., 2024).Source: §2.1 · p. 2
  • AncientDoc evaluates LVLMs across page-level OCR, vernacular translation, reasoning-based QA, knowledge-based QA, and linguistic variant QA.Ancient Chinese documents also feature vertical layouts, variant characters, dense annotations, missing punctuation, classical expressions, implicit semantics, and historical-cultural references.Source: §2.1 · pp. 2–3
  • Most ancient-document benchmarks evaluate final outputs under direct answering, whereas SAGE focuses on evidence acquisition, tentative answers, claim verification, abstention, and trace-based diagnostics.Source: §2.1 · p. 3

2.2 Ancient Chinese Models and Historical Document Processing

  • SAGE complements ancient-Chinese pretrained and LLM-based approaches by offering a model-agnostic inference-time framework rather than training a new ancient Chinese model (Tian et al., 2021; Yao et al., 2025; Cao et al., 2024a).Source: §2.2 · p. 3
  • SAGE coordinates visual reading, text normalization, tool-mediated evidence acquisition, answer generation, and verification for multitask understanding from page images.Its process supports claim verification, bounded replanning, abstention, and trace-based component analysis.Source: §2.2 · p. 3

3 SAGE Framework

SAGE turns Chinese ancient document understanding into a stateful evidence-seeking process coordinated by planning, tool-mediated execution, and verification. A constrained shared-state runtime tracks evidence and budget, controls stopping, replanning, or abstention, and records traces.

3.1 Overview

  • SAGE coordinates a Scholarly Planning Agent, Evidence-Grounded Execution Agent, and Scholar Verifier Agent through a constrained runtime rather than a fixed cascade.Its outputs include a final answer, collected evidence, an evidence report, an abstention indicator, and an execution trace to support grounding and diagnosis.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Scholarly Planning Agent

  • The planner predicts task type, required grounding sources, and an evidence-seeking action sequence before answer synthesis.It sets evidence requirements and verification criteria under an initial budget; translation-oriented tasks emphasize page reading and normalization, while knowledge-intensive tasks add concept extraction and source-constrained retrieval. It can update the plan during bounded replanning based on current evidence and remaining budget.Source: §3.2 · pp. 3–4

3.3 Evidence-Grounded Execution Agent

  • The execution agent performs runtime-allowed reading, normalization, extraction, retrieval, and synthesis, producing a candidate answer for later verification.Accumulated evidence always includes page-derived evidence; retrieved evidence may supplement it for knowledge-intensive questions, but serves as contextual support rather than an answer source by itself.Source: §3.3 · pp. 4–5

3.4 Scholar Verifier Agent

  • The verifier assesses candidate answers against page content and collected evidence by decomposing QA answers into claims and judging their support.Its report summarizes evidence coverage, unsupported and contradicted claim rates, support confidence, and an abstention recommendation. Knowledge-intensive questions require stricter claim-evidence grounding, while page-grounded tasks emphasize validity, consistency, adequacy, and unsupported claims.Source: §3.4 · p. 5

3.5 Constrained Shared-State Runtime

  • The runtime updates shared state after tool calls and verification, then applies control rules to stop, replan, or abstain instead of enforcing a fixed action sequence.It stops when support or task-specific adequacy is sufficient, replans when evidence is missing and budget remains, and abstains when evidence is insufficient, verification fails, or the budget is exhausted. It also constrains tool access and records execution traces under tool-call and replanning limits.Source: §3.5 · pp. 5–6

4 Experiments

Experiments assess SAGE against direct-answering LVLMs on AncientDoc’s understanding tasks, then examine component ablations and execution traces. Results show gains across matched backbones, while trace analyses characterize task-specific evidence use and verifier-guided replanning.

4.1 Experimental Setup

  • On AncientDoc (Yu et al., 2025), evaluation covers translation, reasoning-, knowledge-, and linguistic QA with CHRF++ and BERTScore, excluding OCR from the main comparison.Matched comparisons use InternVL3-8B (Zhu et al., 2025), Qwen2.5-VL-7B (Team, 2025b), and Qwen3.5-9B; SAGE uses task prediction, evidence acquisition, verification, and bounded replanning under shared constraints.Source: §4.1 · pp. 6–7

4.2 Main Results

  • Across three backbones and eight task metrics, SAGE improves every matched direct baseline; Qwen3.5-9B SAGE leads seven of eight metrics, with Reasoning QA BS-F1 the exception.With InternVL3-8B, SAGE gains +6.09 CHRF++ and +6.80 BS-F1 on Knowledge QA, and +4.09 CHRF++ and +8.55 BS-F1 on Linguistic QA. Qwen3.5-9B SAGE also surpasses all direct LVLM baselines on Translation, Knowledge QA, and Linguistic QA.Source: §4.2 · p. 7

4.3 Ablation Study

  • Full SAGE achieves the best results on reasoning and knowledge QA; all SAGE variants beat Direct LVLM, while removing planning or verification causes consistent drops, especially in BS-F1.Local retrieval has a positive but modest effect on Knowledge QA; the reported ablations keep the constrained runtime and execution agent fixed.Source: §4.3 · pp. 7–8, Table 3 · p. 7

4.4 Trace-based Behavior Analysis

  • Knowledge QA invokes extraction and retrieval in 90.39% of examples, versus 9.42% in reasoning QA; page reading, normalization, and verification are used for almost all examples.Translation and linguistic QA rarely trigger retrieval, and verifier confidence remains comparable across tasks; the authors characterize these patterns as task-adaptive evidence paths rather than a fixed pipeline.Source: Figure 3 · p. 8, §4.4 · p. 8
  • After replanning on Knowledge QA examples, trace-derived support rate rises from 69.23% to 74.45%, and verifier confidence rises from 67.99% to 71.59%.These measures come from the system’s own verification reports, not human factuality annotations, so the authors present them as behavioral diagnostics.Source: Figure 3 · p. 8, §4.4 · p. 8

5 Conclusion

SAGE reframes Chinese ancient document understanding as evidence-grounded inference, coordinating specialized agents to acquire and verify evidence. It improves over matched direct-answering baselines across multiple backbones, with gains linked to structured page-grounded evidence use, tool-mediated execution, and verification.

5 Conclusion

  • SAGE improves over matched direct-answering baselines across multiple backbones, especially on knowledge-intensive and linguistic tasks.The conclusion reports that ablation and trace-based analyses link these gains to structured page-grounded evidence use, tool-mediated execution, and verification—not larger backbones or external knowledge alone.Source: §5 · p. 8
  • SAGE coordinates role-specialized scholarly agents under a constrained shared-state runtime for task-aware planning, tool-mediated evidence acquisition, scholarly verification, and bounded evidence seeking.Making evidence acquisition and verification explicit provides a traceable basis for diagnosing unsupported or insufficiently grounded answers.Source: §5 · p. 8
  • The conclusion frames ancient document understanding as an evidence-grounding and reliability challenge, not only a model-scaling problem.Source: §5 · p. 8

Limitations

SAGE is an initial exploration whose broader applicability and answer correctness remain unvalidated, and whose planning, tool use, verification, and replanning add inference cost.

Limitations

  • SAGE’s evaluation covers only AncientDoc’s tasks and data distribution; applicability to broader historical collections, different layouts, and other ancient languages remains unvalidated.The paper characterizes SAGE as an initial exploration of evidence-grounded Chinese ancient document understanding.Source: Limitations · p. 9
  • Verifier outputs and trace-based support statistics are system-generated diagnostics, not human factuality annotations or definitive proof of answer correctness.They should be interpreted as indicators of runtime behavior.Source: Limitations · p. 9
  • Planning, tool use, verification, and replanning introduce additional inference cost; future work may explore stronger evidence sources, human-grounded verification, and more efficient execution.Source: Limitations · p. 9

A Implementation Details

The section outlines the experimental implementation setup, covering model and baseline choices, SAGE runtime configuration, decoding, and execution budgets.

A Implementation Details

  • Implementation details cover model backbones, direct-answering baselines, SAGE runtime configuration, decoding settings, and execution budgets.The passage identifies these setup categories but does not provide their specific configurations.Source: A Implementation Details · p. 10

A.1 Backbone Models

SAGE is evaluated with three LVLM backbones using matched direct-answering baselines. Backbone parameters remain fixed, so the comparison isolates the effect of SAGE’s inference procedure from backbone choice.

A.1 Backbone Models

  • SAGE uses InternVL3-8B, Qwen2.5-VL-7B, and Qwen3.5-9B as its three LVLM backbones.Source: §A.1 · p. 10
  • Each SAGE configuration is compared with a direct-answering baseline using the same underlying model, isolating the effect of the evidence-grounded runtime from backbone choice.Source: §A.1 · p. 10
  • Backbone parameters are kept fixed during evaluation; SAGE changes the inference procedure through planning, tool-mediated evidence acquisition, verification, and bounded replanning.Source: §A.1 · p. 10

A.2 Direct-Answering Baselines … A.5 Execution Budgets

The appendix specifies how direct-answering baselines and SAGE are configured, including SAGE’s runtime, decoding settings, and bounded execution budgets.

A.2 Direct-Answering Baselines

  • The direct-answering baseline answers from the document image and question in one pass, using task-specific prompts for translation, reasoning-based QA, knowledge-based QA, and linguistic QA.It receives no retrieved evidence, intermediate verification reports, or replanning feedback.Source: §A.2 · p. 10

A.3 SAGE Runtime Configuration

  • SAGE uses the same input as the baseline but runs task prediction, evidence acquisition, evidence-grounded answer generation, and scholarly verification that may trigger bounded replanning.Evidence tools include page reading, normalization, extraction, and local evidence retrieval when applicable.Source: §A.3 · p. 10
  • SAGE variants share a constrained tool interface, frozen local evidence pool, verification protocol, and budget constraints; ablations change a component while holding the remaining configuration fixed.Source: §A.3 · p. 10

A.4 Decoding Settings

  • Direct answering and SAGE answer generation use the same decoding configuration under each backbone, with deterministic or low-temperature decoding to reduce answer variance.The same decoding settings also apply to planning and verification unless otherwise specified.Source: §A.4 · p. 10
  • The maximum generation length is set to cover the longest expected outputs in translation and knowledge-intensive QA.Source: §A.4 · p. 10

A.5 Execution Budgets

  • SAGE caps tool calls, retrieved evidence snippets, verification passes, and replanning rounds, using the same budgets across backbones and tasks.Source: §A.5 · p. 10
  • Replanning occurs only when the verifier finds insufficient evidence support or low confidence; if the budget is exhausted, SAGE returns the best-supported answer so far.Source: §A.5 · pp. 10–11

A.6 Reproducibility Notes … C.4 Term Extraction

SAGE uses specialized prompts and a constrained evidence-acquisition interface to guide task planning, answer synthesis, and verification; its appendix also specifies reproducibility settings and tool implementations. The prompts are written in Chinese and presented in lightly translated form for readability.

A.6 Reproducibility Notes

  • The official AncientDoc protocol reports CHRF++ and BERTScore F1 on four understanding tasks, excludes OCR, and freezes the evidence pool with matched retrieval and verification settings.The four tasks are translation, reasoning-based QA, knowledge-based QA, and linguistic QA.Source: §A.6 · p. 11

B Prompts; B.1 Prompt Organization; B.2 Task Classification Prompt

  • SAGE uses separate prompts for task classification, budget-aware tool planning, task-specific answers, and verification instead of one monolithic instruction.The templates are written in Chinese and presented in lightly translated form; the classifier maps questions to five task types.Source: §B.1 · p. 11, B Prompts · p. 11, §B.2 · p. 11

B.3 Scholarly Planning Prompt

  • The planner produces a constrained JSON tool strategy, respects the task budget and tool registry, calls verification after solving, and can replan or allow abstention when evidence is insufficient.Recommended workflows vary by task; Knowledge QA adds term extraction and local evidence retrieval.Source: §B.3 · p. 11

B.4 Page Reading Prompt; C.2 Page Reader

  • Page-reading instructions require right-to-left columns and top-to-bottom reading, preserve original wording, and prohibit guessing; the Page Reader supplies page-level OCR text and source metadata.The Page Reader is implemented with Qwen-VL-OCR.Source: §B.4 · p. 12, §C.2 · p. 13

B.5 Task-Specific Answer Prompts

  • Answer prompts favor direct, concise responses and task-specific grounding, with cautious handling of missing OCR, relevant page or knowledge evidence, and explicit insufficiency when warranted.Linguistic answers should cite concrete textual features; the common style constraint permits abstention only when information is genuinely insufficient.Source: §B.5 · p. 12

B.6 Verifier Prompt

  • The verifier checks rather than reanswers, applying evidence-consistency checks to knowledge and reasoning QA and format or completeness checks to translation and OCR.It returns claim-level verification, an evidence report, an abstention flag, confidence, and whether more evidence is needed.Source: §B.6 · p. 12

C Tool Interface, Evidence Pool, and Retrieval; C.1 Tool Registry

  • SAGE’s fixed evidence-tool registry uses predefined schemas and runtime validation to keep tool use constrained and evidence paths traceable; synthesis and verification are handled separately.The interface covers page reading, character normalization, term extraction, and local evidence retrieval.Source: §C.1 · p. 13, C Tool Interface, Evidence Pool, and… · p. 13, Table 4 · p. 13

C.3 Character Normalization; C.4 Term Extraction

  • The Char Normalizer converts OCR text from traditional to simplified Chinese while retaining the original, and a rule-based Term Extractor deduplicates candidate terms for retrieval queries.Extracted terms are especially used for Knowledge QA.Source: §C.3 · p. 13, §C.4 · p. 13

C.5 Local Evidence Retriever · C.6 Budgets and Traceability

SAGE uses a frozen offline evidence pool for local retrieval and constrains tool use with task-specific budgets and execution traces. The design keeps evidence access reproducible and inspectable while recording how evidence is acquired before answer synthesis and verification.

C.5 Local Evidence Retriever

  • Before evaluation, SAGE builds a frozen offline pool by searching AncientDoc QA terms, cleaning and consolidating results into keyed snippets with available source metadata.Keys can represent terms, entities, concepts, book titles, or historical-cultural expressions.Source: §C.5 · p. 13
  • At inference, the pool excludes test examples and model outputs, keeping evidence access reproducible and holding the external knowledge source constant across main and ablation experiments.The fixed pool is intended to prevent uncontrolled external access during evaluation.Source: §C.5 · p. 13
  • The retriever preserves page text and adds up to eight Knowledge QA snippets using transparent matching, then ranks and deduplicates results while recording evidence IDs, match basis, scores, and source metadata.Matching includes exact keys, substrings, year-equivalent forms, key occurrence in the question, and Chinese-character overlap.Source: §C.5 · pp. 13–14
  • The local evidence retriever applies transparent matching rules and records how each result matched, making retrieval decisions inspectable.The matching rules and retrieval records include match basis, scores, and evidence IDs.Source: §C.5 · pp. 13–14

C.6 Budgets and Traceability

  • Tool-call limits are 3 for OCR, 5 for translation, 5 for reasoning QA, 7 for Knowledge QA, and 6 for linguistic QA; over-budget plans and unregistered tools are rejected.Each local evidence call returns at most eight KB snippets in addition to page context.Source: §C.6 · p. 14
  • Execution traces record tool names, shared-state-implied inputs, outputs, retrieved evidence, retrieval statistics, and errors, supporting behavior analysis and diagnosis of evidence acquisition before synthesis and verification.These traces are used to examine how SAGE acquires evidence before answer synthesis and verification.Source: §C.6 · p. 14

D Qualitative Case Study

A representative Knowledge QA trace shows SAGE revising an overextended answer after verifier feedback, grounding the final explanation in page evidence. The trace combines task-aware planning, page reading and evidence retrieval, verification, and replanning.

D Qualitative Case Study

  • For the Knowledge QA example, the planner combines page reading, character normalization, term extraction, local evidence retrieval, answer synthesis, and verification.The execution agent reads the page with Qwen-VL-OCR, normalizes the text, extracts key terms, and retrieves local evidence with page context.Source: D Qualitative Case Study · pp. 14–15
  • The verifier rejected the initial candidate’s claims about official historiography, civil-service selection, and examination systems as unsupported, assigning support rate 0.00 and confidence 0.50.The candidate’s institutional expansion was not directly supported by the retrieved evidence.Source: D Qualitative Case Study · p. 15
  • After verifier feedback triggered replanning, the revised answer grounded its explanation in page evidence; support rate rose from 0.00 to 1.00 and confidence from 0.50 to 0.90.The practice was framed as a scholarly activity of collecting, organizing, and examining earlier documents to support moral cultivation and textual judgment.Source: D Qualitative Case Study · p. 15

Related papers

Loading 2608.24011v1…