Source-linked AI summary
SAGE: From Direct Answering to Evidence-Grounded Inference for Chinese Ancient Document Understanding
Yuchuan Wu, Xuan Luo, Yinglian Zhu, Meng Fang, Xiangyang Xue, Bin Li
TL;DR
Chinese ancient document understanding combines difficult visual, linguistic, and historical reasoning, while direct LVLM generation is often opaque and weakly grounded. SAGE addresses this gap with a multi-agent, evidence-grounded inference process that plans evidence seeking, uses constrained tools, verifies claims, and replans or abstains when needed. On AncientDoc, it improves over matched direct-answering baselines across backbones, while its broader applicability and diagnostic validity remain bounded by benchmark scope and system-generated verification statistics.
Problem
Chinese ancient document understanding requires integrated visual, linguistic, and historical reasoning, but direct LVLM systems often produce overconfident, weakly grounded, and hard-to-diagnose answers.
Method
SAGE coordinates planning, constrained tool-mediated evidence acquisition, claim-level verification, bounded replanning, and abstention under a shared-state runtime.
Results
SAGE consistently surpasses matched direct-answering baselines across three LVLM backbones on AncientDoc, with the strongest gains on knowledge-intensive and linguistic tasks.
Takeaways & Limitations
The results support treating ancient-document understanding as an evidence-grounding and reliability challenge rather than solely an LVLM scaling problem.
Takeaways & Limitations
Evaluation is limited to AncientDoc’s tasks and data distribution, while verifier and trace statistics are system-generated diagnostics rather than human factuality annotations.
Abstract
from arXiv · showhide
Chinese ancient document understanding demands complex visual, linguistic, and historical reasoning. Current Large Vision-Language Models (LVLMs) typically rely on an opaque, single-pass generation paradigm, often producing overconfident and weakly grounded responses. To address this, we propose SAGE, an evidence-grounded multi-agent framework that reformulates Chinese ancient document understanding as evidence-grounded inference rather than direct answer generation. SAGE coordinates specialized agents for task-aware planning, tool-mediated evidence acquisition, claim-level verification, and bounded replanning under a constrained shared-state runtime. This design supports bounded evidence seeking, answer revision, and abstention when grounding is insufficient. Experiments on the AncientDoc benchmark show that SAGE consistently outperforms matched direct-answering baselines across three LVLM backbones. Remarkably, SAGE with Qwen3.5-9B surpasses much larger monolithic LVLMs on most evaluated metrics, highlighting the importance of structured, evidence-grounded inference beyond model scaling.
1 Introduction
Chinese ancient document understanding requires integrated visual, linguistic, and historical reasoning, but direct LVLM answering often produces opaque, overconfident, weakly grounded responses. SAGE instead structures inference around evidence acquisition, verification, bounded replanning, and abstention, improving benchmark performance over matched direct-answering baselines.
- Chinese ancient documents require models to integrate visual reading, classical Chinese interpretation, and knowledge-grounded reasoning.
- Direct-answering LVLM systems collapse reading, evidence acquisition, reasoning, and verification into one opaque generation step.This often yields overconfident, weakly grounded, and difficult-to-diagnose answers.
- SAGE consistently improves over matched direct-answering baselines across three LVLM backbones on the AncientDoc benchmark.Ablation and trace analyses examine the roles of planning, tool-mediated execution, and scholarly verification.
- SAGE reformulates ancient document understanding as evidence-grounded inference, exposing supporting evidence, verification reports, abstention decisions, and execution traces.
- SAGE coordinates task-aware planning, constrained tool-mediated evidence acquisition, claim-level verification, and bounded replanning under a shared-state runtime.The runtime regulates tool access and budget, records traces, and can stop, replan, or abstain when grounding is insufficient.
2 Related Work
Prior document and ancient-Chinese benchmarks evaluate reading, translation, reasoning, knowledge, and linguistic variation, but mainly judge final outputs under direct answering. SAGE shifts attention to the inference process by operationalizing evidence acquisition, claim verification, abstention, and trace-based diagnosis for multimodal ancient-document understanding.
- Modern document benchmarks evaluate visual question answering, OCR, localization, and reasoning over document images or long documents.
- Chinese ancient-document settings add vertical layouts, variant characters, dense annotations, missing punctuation, implicit semantics, and historical-cultural references.
- Existing benchmarks primarily evaluate final outputs under direct answering, whereas SAGE evaluates how systems acquire evidence, verify claims, abstain, and provide trace-based diagnostics.
- SAGE is a model-agnostic, inference-time framework for coordinating visual reading, normalization, evidence acquisition, answer generation, and verification.It complements model-centric and translation-centric approaches rather than training a new ancient Chinese model.
3 SAGE Framework
SAGE organizes Chinese ancient document understanding as a stateful, evidence-seeking process rather than a fixed cascade or direct answer. Three specialized agents operate through a constrained runtime that manages evidence, tools, budgets, traces, verification, and control decisions.
- Overview: SAGE coordinates Scholarly Planning, Tool-Augmented Execution, and Scholar Verifier agents over a shared evidence-grounded state.The runtime regulates actions, tracks evidence and budget, and records execution traces.
- Scholarly Planning Agent: The planner converts each question and initial budget into a task-aware evidence-seeking action strategy before answer synthesis.It identifies grounding sources, selects allowed actions, and specifies evidence requirements and verification criteria.
- Evidence-Grounded Execution Agent: Execution performs constrained read, normalize, extract, retrieve, and synthesize actions, updating page text, evidence, and candidate answers.Retrieved evidence supports synthesis and verification but is not treated as an answer source by itself.
- Scholar Verifier Agent: Candidate answers remain hypotheses until the verifier decomposes them into claims and assesses support, contradiction, insufficiency, or inapplicability.The verifier aggregates claim judgments into an evidence report covering evidence coverage, unsupported and contradicted claims, confidence, and abstention recommendation.
- Constrained Shared-State Runtime: The runtime updates shared state after actions and verification, then applies predefined rules to stop, replan within budget, or abstain when grounding is insufficient.Verification criteria vary by task: knowledge-intensive questions require stricter claim-evidence grounding than translation, reasoning, and linguistic analysis.
4 Experiments
Experiments evaluate SAGE on four understanding-oriented AncientDoc tasks, comparing matched backbones, component ablations, and trace-based runtime behavior. SAGE improves matched direct-answering baselines across all eight task metrics, while replanning increases evidence support and verifier confidence.
- Benchmark and Metrics: SAGE is evaluated on translation, reasoning-based QA, knowledge-based QA, and linguistic QA using CHRF++ and BERTScore F1.OCR is excluded from the main comparison because it primarily measures page-reading or OCR tool quality.
- Main Results: Across three backbones and eight task metrics, SAGE improves every corresponding direct-answering baseline.The gains are especially large on Knowledge QA and Linguistic QA.
- Main Results: Qwen3.5-9B with SAGE achieves the best score on seven of eight metrics and surpasses all direct LVLM baselines on Translation, Knowledge QA, and Linguistic QA.The only exception is Reasoning QA BS-F1, where Qwen2.5-VL-72B Direct is slightly higher.
- Ablation Study: The full SAGE achieves the best results on reasoning-based QA and knowledge-based QA, while removing the planner or verifier causes consistent drops, especially in BS-F1.Local retrieval has a positive but modest effect on Knowledge QA.
- Trace-based Behavior Analysis: Knowledge QA invokes extraction and local evidence retrieval in 90.39% of examples, compared with 9.42% for reasoning QA; translation and linguistic QA rarely trigger retrieval.Page reading, normalization, and verification occur in almost all examples.
- Trace-based Behavior Analysis: After verifier-triggered replanning on Knowledge QA, trace-derived support rate rises from 69.23% to 74.45%, while verifier confidence rises from 67.99% to 71.59%.These statistics describe cases initially judged insufficiently grounded.
- Trace-based Behavior Analysis: Trace statistics are derived from the system’s verification reports rather than human factuality annotations.They function as behavioral diagnostics of tool invocation, replanning, and evidence-support assessment.
5 Conclusion
SAGE reframes Chinese ancient document understanding as evidence-grounded inference, coordinating specialized agents under a constrained runtime. Experiments and analyses attribute its gains to explicit evidence use, tool execution, and verification rather than model scale alone.
- 5 Conclusion: SAGE coordinates role-specialized scholarly agents under a constrained shared-state runtime for evidence-grounded inference.The framework supports task-aware planning, tool-mediated evidence acquisition, scholarly verification, and bounded evidence seeking.
- 5 Conclusion: SAGE provides a traceable basis for diagnosing unsupported or insufficiently grounded answers by making evidence acquisition and verification explicit.
- 5 Conclusion: Ablation and trace-based analyses indicate that SAGE’s gains arise from structured page-grounded evidence use, tool-mediated execution, and verification rather than larger backbones or external knowledge alone.
Limitations
SAGE’s current evidence is bounded by the AncientDoc benchmark and by system-generated verification diagnostics. The framework also introduces additional inference cost through its multi-stage execution.
- Limitations: The evaluation is limited to AncientDoc’s tasks and data distribution, leaving broader historical collections, layouts, and ancient languages unvalidated.
- Limitations: Verifier and trace-based support statistics are system-generated diagnostics rather than human factuality annotations.They indicate runtime behavior but do not definitively prove answer correctness.
- Limitations: SAGE introduces additional inference cost through planning, tool use, verification, and replanning.
A Implementation Details
The implementation-details section covers the experimental configuration used to evaluate the models and SAGE runtime. It specifies backbones, baselines, decoding, and execution budgets.
- A Implementation Details: The experimental setup includes model backbones and direct-answering baselines.
- A Implementation Details: The setup documents SAGE runtime configuration and decoding settings.
- A Implementation Details: The setup includes execution budgets for the evaluated procedures.
A.1 Backbone Models
SAGE is evaluated with three LVLM backbones against matched direct-answering baselines using the same underlying models. This isolates the evidence-grounded runtime’s effect from backbone choice.
- A.1 Backbone Models: SAGE uses InternVL3-8B, Qwen2.5-VL-7B, and Qwen3.5-9B as LVLM backbones.
- A.1 Backbone Models: Each backbone is compared with a matched direct-answering baseline using the same underlying model.
- A.1 Backbone Models: The matched comparisons isolate inference-procedure effects because backbone parameters remain fixed while SAGE adds planning, tools, verification, and bounded replanning.
A.2 Direct-Answering Baselines
The direct-answering baseline generates responses from the document image and question in one pass, while SAGE inserts task-aware evidence acquisition and verification before final answering. Both use matched decoding and fixed runtime settings for fair comparison.
- The direct-answering baseline receives the document image and question, then generates the final answer in a single pass.
- SAGE adds task prediction, evidence acquisition, grounded answer generation, and verification before producing the final answer.
- All SAGE variants retain the same tool interface, frozen evidence pool, verification protocol, and budget constraints across ablations.
- Direct answering and SAGE use the same decoding configuration under each backbone, with deterministic or low-temperature decoding to reduce variance.
- SAGE limits tool calls, retrieved snippets, verification passes, and replanning rounds; exhausted budgets return the best supported answer so far.
A.6 Reproducibility Notes
The reproducibility setup fixes evaluation data, prompts, tool access, task workflows, and verification behavior. It operationalizes SAGE as a structured, task-sensitive runtime rather than a single monolithic instruction.
- Experiments follow the official AncientDoc protocol across four understanding tasks, reporting CHRF++ and BERTScore F1 while excluding OCR from the main comparison.
- SAGE uses Chinese implementation prompts, presented with light translations while preserving task labels, tool names, and runtime placeholders.
- The prompt family separates task classification, budget-aware planning, task-specific answer synthesis, and structured verification.
- The classifier maps questions to five fixed task types: OCR, translation, reasoning_qa, knowledge_qa, and linguistic_qa.
- Planning produces executable JSON strategies using registered tools, task budgets, current state, prior strategies, and verification feedback.
- Recommended workflows differ by task, with Knowledge QA adding term extraction and local evidence retrieval before solving and verification.
- Answer prompts enforce direct outputs, preserve key terms, and permit abstention when available information is genuinely insufficient.
- Verification checks candidate answers rather than re-solving tasks, applying stricter claim-evidence criteria to knowledge and reasoning questions.
C.5 Local Evidence Retriever
SAGE retrieves supporting knowledge from a frozen offline pool while preserving page text as evidence. Retrieval is transparent, traceable, and bounded by fixed snippet and tool-use limits.
- The local retriever searches a frozen offline pool built from filtered and consolidated evidence gathered before evaluation.
- The fixed pool prevents uncontrolled external access and keeps evidence sources identical across experiments and ablations.
- Retrieval preserves page text and adds up to eight local snippets for Knowledge QA using transparent matching rules.
- A representative Knowledge QA plan combines page reading, normalization, term extraction, local evidence retrieval, synthesis, and verification.
- Retrieved candidates are ranked, deduplicated, and recorded with evidence ids, matched queries, match bases, scores, and source metadata.
- Task-specific budgets cap tool calls at 3 for OCR, 5 for translation and reasoning QA, 7 for Knowledge QA, and 6 for linguistic QA.
- Execution traces record tool calls, inputs, outputs, retrieved evidence, statistics, and errors for reproducibility and behavior analysis.
D Qualitative Case Study
The case study shows SAGE revising a Knowledge QA answer after verification detects unsupported institutional claims. Replanning removes the unsupported expansion and restores evidence support.
- The Knowledge QA example plans a multi-stage path combining page reading, normalization, term extraction, retrieval, synthesis, and verification.
- The initial answer invokes official historiography, civil-service selection, and examination systems without direct support from retrieved evidence.
- The verifier triggers replanning, after which the answer frames “zhengwen kaoxian” as scholarly collection and examination of earlier documents.
- Support rate rises from 0.00 to 1.00 and confidence from 0.50 to 0.90 after revision.