PaperlayerView exact revision on arXiv

Source-linked AI summary

LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger

Enjun Du, Hange Zhou, Chenxu Du, Siyi Liu, Zirong Chen, Ziyu Zheng, Yongqi Zhang

arXiv:2607.28374v1cs.LG

TL;DR

Final-answer accuracy cannot reveal whether multimodal agent trajectories are grounded, so LedgerMind enforces provenance through a Structured Evidence Ledger, grounding checks, adaptive execution, and typed repair. Across multiple multimodal benchmarks and backbone MLLMs, it consistently improves answer accuracy and trajectory-level faithfulness.

  • Problem

    Final-answer accuracy cannot distinguish grounded evidence from language priors or accidental error cancellation in multi-step multimodal agent trajectories.

  • Method

    LedgerMind uses a Structured Evidence Ledger to constrain claims to active tool-produced evidence, verify entity and numeric grounding, adapt reasoning depth, and perform typed repair.

  • Results

    LedgerMind consistently improves answer accuracy and trajectory-level faithfulness across multimodal benchmarks and multiple backbone MLLMs.

  • Takeaways & Limitations

    Trajectory-level evaluation indicates that LedgerMind’s gains reflect more grounded reasoning behavior rather than answer accuracy alone.

  • Takeaways & Limitations

    The study focuses on static image-and-text visual question answering and does not evaluate long-horizon agents with persistent memory, video streams, or embodied interaction.

Abstract

from arXiv · show

Multimodal agents for visual question answering increasingly operate as multi-step trajectories that interleave perception, retrieval, and reasoning, yet evaluation still largely reduces to final-answer accuracy. This aggregate signal cannot tell whether a correct answer was reached through grounded evidence, language priors, or accidental error cancellation. We propose to treat a multimodal agent trajectory as a provenance-constrained state machine: tool outputs are normalized into a Structured Evidence Ledger that serves as the trajectory state, downstream reasoning and decision claims may cite only active ledger entries, grounding is checked at the entity and numeric level, and repair is realized as typed state transitions that cannot introduce content without tool-produced provenance. We instantiate this design as LedgerMind (Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger), augmented by a Three-Layer Grounding Protocol, an Adaptive Dual-Path Dispatcher that matches reasoning depth to question complexity, and an Event-Triggered Verification-and-Repair engine with a formal provenance non-amplification guarantee. We use LedgerMind to target four recurring failure patterns that final-answer accuracy tends to obscure: unsupported intermediate reasoning, citation-backed entity hallucination (Phantom Grounding), over-reasoning on simple queries, and repair-time amplification. Experiments across multiple multimodal reasoning benchmarks and backbone MLLMs show that LedgerMind improves both answer accuracy and trajectory-level faithfulness.

1 INTRODUCTION

LedgerMind reframes multimodal agentic reasoning as a provenance-constrained process whose intermediate claims must remain auditable against active tool-produced evidence. Its Structured Evidence Ledger, grounding checks, adaptive reasoning, and typed repair target failure patterns that final-answer accuracy obscures and improve answer accuracy and grounded reasoning behavior.

  • Motivation: Multimodal agents produce multi-step trajectories interleaving observations, intermediate claims, tool calls, and final decisions, making single-answer accuracy insufficient for evaluating reasoning faithfulness.The trajectories arise as MLLM systems combine perception, retrieval, and language-based reasoning in agentic workflows.
  • Motivation: Free-form trajectory buffers mix tool outputs, paraphrases, inferred facts, and repaired claims, preventing reliable auditing of intermediate reasoning against supporting evidence.This motivates moving from aggregate accuracy toward trajectory-level faithfulness.
  • Failure patterns: LedgerMind addresses four accuracy-obscured failures: unsupported intermediate claims, Phantom Grounding, over-reasoning that overwrites correct answers, and repair-time amplification.Phantom Grounding occurs when valid evidence IDs accompany fabricated entities or numerical values absent from the cited source.
  • LedgerMind: LedgerMind normalizes tool outputs into a Structured Evidence Ledger with provenance metadata, allowing downstream reasoning and decision claims to cite only active ledger entries.Entries carry source, type, confidence, lifecycle status, and dependencies, enabling checks against active provenance rather than citation instructions alone.
  • Contributions and evaluation: The framework combines entity and numeric consistency checks, complexity-adaptive reasoning depth, and event-triggered typed repair with a provenance non-amplification guarantee, and evaluates them across multimodal benchmarks and Hard-200.The evaluation covers five answer-level multimodal benchmarks, one chain-level multimodal search benchmark, and 200 complex image-grounded queries, reporting gains in answer accuracy and grounded reasoning behavior.

2 RELATED WORKS

Related work spans multimodal agents that interleave reasoning with external tools, established provenance and tracing systems, multi-step reasoning methods, and verification-based reliability improvements. LedgerMind builds on these lines while focusing on structured evidence states for multimodal agent grounding.

  • Multimodal agents and tool-augmented reasoning: Multimodal agents extend MLLMs beyond single-pass perception by interleaving reasoning with external tool calls.ReAct alternates thought and action traces, while Toolformer trains models to invoke APIs.
  • Structured execution records and provenance tracking: Database provenance, workflow provenance, and distributed tracing record data origins, causal histories, or cross-component execution for diagnosis.The paper positions its contribution as their operational use in a shared evidence state for multimodal grounding, not as isolated structured tracing or dependency metadata.
  • Adaptive reasoning: Chain-of-thought, Self-Consistency, and Tree-of-Thoughts elicit multi-step reasoning, but unnecessarily long chains can waste computation and degrade accuracy.This overthinking phenomenon motivates adaptive inference methods that avoid applying deeper reasoning uniformly.
  • Self-correction and verification: Prior reliability methods use critique, verbal feedback, fact-check questions, or trajectory revision to verify and repair reasoning outputs.Examples include Self-Refine, Reflexion, Chain-of-Verification, and SE-Agent.
  • Hallucination mitigation and trajectory-level evaluation: Hallucination mitigation combines training-time changes with inference-time correction, while trajectory-level evaluation addresses failures beyond caption-level object hallucinations.Woodpecker verifies extracted concepts with external detectors, and OPERA adds an over-trust penalty during decoding.

3 METHOD

LedgerMind models multimodal reasoning as a provenance-constrained state machine whose Structured Evidence Ledger governs evidence acquisition, claim validation, adaptive execution depth, and typed repair. Its grounding checks license entities and numbers from cited leaf evidence, while repair operators guarantee that newly added entries retain tool-produced provenance.

  • Ledger state: The Structured Evidence Ledger is the canonical runtime state: tool returns become normalized evidence, and downstream claims must cite active entries.LedgerMind replaces free-form transcript state with ledger-facing operations that unify grounding, dispatch, and repair.
  • Grounding protocol: Grounding requires support coverage plus entity and numeric consistency, with derivation entries recursively resolved to perception and retrieval leaves.ECC checks conclusion-level entities up to aliases, while NCC uses exact matching for years, counts, dates, labels, and identifiers and tolerance for continuous visual readings.
  • Adaptive evidence-depth control: LedgerMind selects a DIRECT or FULLPIPE path before expensive acquisition, matching evidence depth to query complexity while writing both paths through the same ledger interface.DIRECT targets single-step visual or shallow factual queries; FULLPIPE adds planning, extraction, optional crop-zoom and Dual-Read Verification, retrieval, grounded reasoning, and decision checks.
  • Event-triggered typed repair: Verification fires on ledger-level risk events and restricts repair to seven typed operators spanning evidence, action, and trajectory layers.The operators are DROP and REFRESH; RETRY, SWITCH, and ACQUIRE; and STOPANDANSWER and ABSTAIN.
  • Provenance guarantee: Every new ledger entry produced by repair has tool-produced provenance, preventing repair from fabricating provenance-less content.The guarantee covers direct tool outputs or deterministic template images of tool outputs; it does not guarantee that tool outputs are factually correct.

4 EXPERIMENTS

LedgerMind is evaluated across diverse multimodal benchmarks, frontier MLLM backbones, and trajectory-level faithfulness settings. It improves benchmark accuracy while testing grounded reasoning, provenance-sensitive design choices, and robustness across difficult subsets and backbones.

  • Evaluation setup: LedgerMind is evaluated on six public benchmarks, Hard-200, and trajectory-level faithfulness audits using six frontier MLLMs from four vendors.The evaluated systems include GPT-4o/GPT-5.5, Gemini-3-Flash/3.1-Pro, Claude-Sonnet-4.6/Opus-4.7, and Kimi-K2.6.
  • VTC-Bench and general VLM benchmarks: 58.9%: Gemini-3-Flash with LedgerMind achieves a new VTC-Bench state of the art, while gains reach +23.3 points on GPT-4o.The framework also lifts Gemini-3.1-Pro by +11.8, Gemini-3-Flash by +12.4, and ranks first on MMStar, MMMU, and MMMU-Pro against eleven baselines.
  • EMMA and Hard-200: 58.29% overall: LedgerMind leads EMMA by +9.58 over the strongest thinking-mode baseline, with Math and Physics gains of +16.15 pp and +16.02 pp.Coding changes by −0.18 because that subset emphasizes multiple-choice code–visualization alignment rather than image-centric reasoning.
  • EMMA and Hard-200: LedgerMind improves every backbone on Hard-200 overall and across RealCAR, TIR-Bench, and BrowseComp-VL, with no negative cells in the baseline-difference heatmap.Hard-200 is an in-house stress set, and the figure reports results across six frontier backbones.
  • Chain-aligned reasoning on MC-Search: MC-Search evaluates both final-answer quality and intermediate retrieval chains through F1, LJ, HPS, RD, and a Golden F1 upper bound.HPS measures the fraction of golden steps recovered, while RD measures absolute deviation in step count.
  • Ablation study: Removing the Structured Evidence Ledger is the most damaging MMMU-Pro ablation, showing that prompt-level citation instructions cannot replace structural provenance constraints.The ablations use Gemini-3-Flash and share the same tool budget, isolating the effect of the design choices rather than extra compute.

5 CONCLUSION

LedgerMind is a training-free, provenance-constrained runtime that uses a Structured Evidence Ledger to coordinate claim grounding, adaptive execution, and typed repair under a formal provenance non-amplification guarantee. Across five answer-level multimodal benchmarks, it consistently improves answer accuracy and trajectory-level faithfulness.

  • LedgerMind coordinates claim grounding, adaptive execution, and typed repair through a Structured Evidence Ledger.The runtime is training-free and provenance-constrained.
  • A formal provenance non-amplification guarantee constrains LedgerMind’s typed repair process.
  • LedgerMind consistently improves answer accuracy and trajectory-level faithfulness across five answer-level multimodal benchmarks.The benchmarks are VTC-Bench, MMStar, MMMU, MMMU-Pro, and EMMA.

A STRUCTURED REASONING PIPELINE: FULL SPECIFICATION … D.3 OC SEMANTIC CATEGORIES

LedgerMind specifies a provenance-constrained multimodal pipeline that gathers evidence, performs citation-grounded reasoning, and applies defended decision-making. Its ledger schema, lifecycle graph, thresholds, and dispatcher formalize localized auditing and complexity-aware routing.

  • A STRUCTURED REASONING PIPELINE: FULL SPECIFICATION: The FULLPIPE procedure comprises evidence gathering, grounded reasoning, and decision with defense, expanding verification, grounding, search, and defense branches.Evidence is planned from the question, extracted from images, grounded through citation checks, and finalized with three defenses.
  • A STRUCTURED REASONING PIPELINE: FULL SPECIFICATION: Every noninitial claim must cite active ledger entries; unsupported nonjudgment claims are dropped, while unsupported judgments are degraded to σ = 0.55.Judgments additionally undergo support coverage, entity-level ECC, and numeric NCC checks, with ECC and NCC failures lowering confidence to 0.50 and 0.52.
  • B.1 EVIDENCE ENTRY SCHEMA: The 11-field evidence-entry schema records provenance, normalized facts, confidence, lifecycle status, spatial and temporal metadata, supersession, and reverse dependencies.Entries encode the source tool, epistemic type, creation timestamp, normalized fact, confidence σe, status ωe, bounding box be, TTL ℓe, superseding ID ξe, and dependency set De.
  • B.2 LIFECYCLE OPERATIONS: The ledger finite state machine supports APPEND, MARKSTALE, SUPERSEDE, and DROP transitions across ACTIVE, STALE, CONFLICTED, and DROPPED states.Staleness triggers re-verification of dependent claims, supersession marks old evidence stale, and dropping propagates invalidation.
  • B.3 DEPENDENCY GRAPH: The dependency graph limits re-auditing to affected claims, with |Affected(e)| ≤ ∆max ≪ |Cτ|, where ∆max is the maximum evidence-node degree.Edges connect evidence entries to claims that cite them, enabling localized impact analysis when evidence status changes.
  • C GROUNDING THRESHOLD SENSITIVITY: Grounding degradation values 0.50, 0.52, and 0.55 all remain below σverify = 0.6, triggering hypothesis verification after ECC, NCC, or structural failures.These values were selected by grid search over {0.40, 0.45, 0.50, 0.55, 0.60} on 50 held-out questions.
  • D.2 ANSWER GRANULARITY CONSTRAINTS: Answer granularity, OC semantic categories, and dynamic reasoning methods are specified through Tables 4, 5, and 6, completing the dispatcher and grounded-reasoning configuration.The supplied passages reference these tables rather than reproducing their definitions.
  • D.1 COMPLEXITY CLASSIFICATION: The dispatcher routes questions to complex processing for precise perception, visual–knowledge fusion, verification or multi-entity reasoning, visually grounded MCQs, or non-English input.Deep-knowledge markers such as “according to” and “historically” trigger a second knowledge-search round.

D.4 REASONING METHOD ADAPTATION … E.2 REPAIR OPERATOR DEFINITIONS

The method adapts verification and search to query conditions, then uses explicit uncertainty markers and event-triggered typed repairs to detect and address evidence failures. Repair operators escalate locally, with bounded retries and abstention when evidence remains insufficient.

  • D.5 DUAL-READ VERIFICATION: Dual-read verification repeats each reading task twice and lowers confidence when independent readings disagree.After a 2.5× crop-zoom, two context-independent API readings yield σ(ec) = 0.97 when ∆v ≤0.15 and σ(ec) = 0.65 otherwise; disagreement triggers hypothesis verification and adopts v2.
  • D.6 KNOWLEDGE SEARCH STRATEGY: Knowledge search uses Serper.dev primarily and SerpAPI as a timeout-or-error fallback, with exponential backoff and jitter capped at 5 retries and 32 s.Each call combines an Answer Box, Knowledge Graph, and top-3 Organic Results snippets, truncating the concatenation to 600 characters; deep-knowledge queries receive a second refined round.
  • D.7 UNCERTAINTY MARKERS: Visual fallback is triggered by English uncertainty phrases such as “cannot determine” and system markers including “api_error”, unknown, and need_more.These markers define the set U used to trigger visual fallback.
  • E.1 TRIGGER CONDITIONS: The verifier fires on six conditions covering tool anomalies, stale references, conflicts, confidence drops, unsupported decisions, and phantom grounding.The protocol audits grounding, scans conflicts and citation staleness, then recommends a repair operator; β = 0.15, σfloor = 0.3, and σverify = 0.6 serve distinct decision, failure, and grounding thresholds.
  • E.2 REPAIR OPERATOR DEFINITIONS: Repair operators span evidence, action, and trajectory layers, including retry, action switching, evidence acquisition, stop-and-answer, and abstention.DROP invalidates dependent claims without creating entries; REFRESH and RETRY issue updated tool calls, while STOPANDANSWER selects the highest-confidence active evidence and ABSTAIN returns ⊥ after exhausted repair.
  • E.2 REPAIR OPERATOR DEFINITIONS: The repair policy follows locality-first escalation, allowing up to M0 = 2 repairs per trigger before bounded termination or abstention.T4 retries with modified parameters, T5 acquires evidence when M > 0 or abstains, and T6 injects a warning before acquisition.

F FULL METRIC DEFINITIONS · G PROMPT TEMPLATES

Section F defines diagnostic metrics for separating perception noise, evidence use, recovery, repair cost, efficiency, and Phantom Grounding failure modes. Section G provides stage-specific prompts for planning, observation extraction, grounded reasoning, verification, entity rechecking, and concise direct answering.

  • F FULL METRIC DEFINITIONS: The diagnostic suite isolates perception disagreement, post-repair grounding, evidence utilization, recovery, repair cost, and step efficiency.OCErr prevents tool noise from confounding reasoning-faithfulness assessment, while EUR diagnoses unused evidence collection.
  • F FULL METRIC DEFINITIONS: A high WDG with low ECC pass rate signals Phantom Grounding, whereas high WDG with high ECC pass rate indicates a capability limit.The diagnostic rule distinguishes hallucination from genuine inability using the combination of WDG and ECC outcomes.
  • G PROMPT TEMPLATES: Representative templates cover the main stages, while complete prompt templates are provided in supplementary code.The section presents abbreviated examples rather than the full implementation templates.
  • G.1 TASK PLANNING: Task planning uses only the question text to specify observation targets, reasoning method, judgment criteria, answer form, and dynamically inserted answer granularity.The planner is instructed to produce a structured analysis plan before receiving the image.
  • G.2 OBSERVATION CLAIM EXTRACTION: Observation extraction produces up to 15 fine-grained claims tagged as TEXT, NUMBER, OBJECT, SPATIAL, or task-injected categories.TEXT claims transcribe visible text exactly, while NUMBER claims include values with units.
  • G.3 STATE CLAIM REASONING: State reasoning follows [E]/[I]/[J] Evidence Integration, Inference, and Judgment steps and must cite specific observation claims.Each state claim records the observation identifiers supporting its conclusion.
  • G.4 HYPOTHESIS VERIFICATION (ECC FAILURE): ECC verification treats hypotheses as falsifiable, requiring supporting evidence, contradicting evidence, and confirmation or revision when entities are absent from visible text.The prompt explicitly warns that the conclusion may be a language-model hallucination.
  • G.5 ENTITY RECHECK DEFENSE: Entity rechecking restricts the answer to confirmed visible text and numbers and forbids introducing unlisted entities, names, or numbers.The defense supplies a formatted list of confirmed image content before requesting the answer.

H HARD-200 DATASET CONSTRUCTION … I.2 FULL RESULTS OF VLM BENCHMARKS

LedgerMind constructs Hard-200 through committee-based, cross-vendor hardness mining and diversity-constrained selection, combining public benchmark examples with self-constructed real-world challenges. The paper then reports complete Hard-200 and three-multimodal-benchmark results in Tables 7 and 8.

  • H HARD-200 DATASET CONSTRUCTION: Hard-200 uses committee-based hardness mining across heterogeneous frontier MLLMs, ranking candidates by cross-model failure consistency before diversity-constrained selection.Vendor-level aggregation prevents multiple models from one provider from being over-counted.
  • H HARD-200 DATASET CONSTRUCTION: BrowseComp-VL selection emphasizes image-grounded retrieval, multi-hop aggregation, and evidence composition, with 75 Level 2 and 25 Level 1 examples.Selection also applies domain-level diversity constraints.
  • H HARD-200 DATASET CONSTRUCTION: RealCAR is self-constructed to require multi-step reasoning over visual evidence, implicit constraints, and compositional evidence integration.It targets real-world cases beyond isolated recognition or surface-level text extraction.
  • H HARD-200 DATASET CONSTRUCTION: Hard-200 answers are scored by an LLM-based judge against verified answers and minimal evidence rationales using scores of 0, 0.5, or 1.Partial credit applies when a question requires multiple answer elements and only some are correct.
  • H HARD-200 DATASET CONSTRUCTION: The final Hard-200 composition is 100 BrowseComp-VL, 75 TIR-Bench, and 25 RealCAR examples.Diversity constraints prevent selection from concentrating in narrow domains or task types.
  • I.1 FULL RESULTS OF HARD-200: Table 7 gives the full Hard-200 results and reports LedgerMind scores with absolute improvements over baseline in percentage points.The table caption specifies the format as LedgerMind score followed by the delta over baseline.
  • I.2 FULL RESULTS OF VLM BENCHMARKS: Table 8 gives the full results for three multimodal benchmarks.This reports the VLM benchmark results referenced in the paper.

I.3 FULL RESULTS OF VTC-BENCH · I.4 FULL RESULTS OF EMMA

The sections present full results for VTC-Bench and EMMA, with VTC-Bench using exact-match or partial-credit scoring aggregated into sample-weighted category and overall means. Results are reported in Tables 9 and 10, while Table 8 summarizes comparisons on MMStar, MMMU, and MMMU-Pro.

  • I.3 FULL RESULTS OF VTC-BENCH: VTC-Bench multiple-choice responses receive binary exact-match credit of 0 or 1.Each example is evaluated once.
  • I.3 FULL RESULTS OF VTC-BENCH: VTC-Bench open-ended responses receive credit of 0, 0.5, or 1, with 0.5 indicating partial correctness.Partial credit applies when a question requires multiple answer elements.
  • I.3 FULL RESULTS OF VTC-BENCH: Category and overall VTC-Bench scores are sample-weighted means of per-example credits, allowing fractional effective counts.For example, 36.67% on 45 examples corresponds to total credit 16.5/45.
  • I.3 FULL RESULTS OF VTC-BENCH: Table 8 compares performance on MMStar, MMMU, and MMMU-Pro, identifying LedgerMind as the proposed method.Best results are bold and underlined, while best baseline results are italicized.
  • I.3 FULL RESULTS OF VTC-BENCH: The full VTC-Bench results are reported in Table 9, whose colored subscripts show absolute percentage-point changes versus the corresponding native backbone.Green denotes improvement and red denotes regression; bold and underlining identify best and second-best results per column.
  • I.4 FULL RESULTS OF EMMA: The full EMMA results are reported in Table 10.The supplied passage identifies the table but provides no numerical results.

I.5 FULL RESULTS OF MC-SEARCH

This section presents LedgerMind’s full MC-SEARCH results through a topology-wise comparison in Table 11. The table compares full-benchmark LedgerMind results with metric-wise best official baselines.

  • Full Results of MC-SEARCH: Table 11 reports the full-benchmark, topology-wise comparison on MC-SEARCH.The passage identifies Table 11 as the source of the full MC-SEARCH results.
  • Full Results of MC-SEARCH: Each topology’s official baseline row gives the best official baseline for each metric, serving as a strong ceiling reference.LedgerMind rows are full-benchmark results under the proposed framework and are labeled with their base model.
  • Full Results of MC-SEARCH: The table marks the best results within each topology in bold, second-best results with underlining, and the best official baseline in italics.These formatting conventions support direct comparison between LedgerMind and official baselines.

J BROADER IMPACT AND LIMITATIONS

LedgerMind frames multimodal agent trajectories as provenance-constrained ledgers to improve auditability in safety-sensitive applications. The study is limited to static image-and-text visual question answering, leaving persistent-memory, video, and embodied-agent extensions open.

  • Broader impact: LedgerMind makes tool-output support for downstream claims easier to inspect by representing trajectories as provenance-constrained ledgers.The paper positions this auditability as useful where opaque reasoning traces are a longstanding concern.
  • Broader impact: The proposed auditability may benefit safety-sensitive applications including educational tutoring, scientific diagram interpretation, and document understanding.These settings are identified as examples where inspecting reasoning support could matter.
  • Limitations: The evaluation focuses on static image-and-text visual question answering rather than long-horizon agents with persistent memory, video streams, or embodied interaction.Extending the ledger to these settings is presented as an open problem.
  • Limitations: Applying LedgerMind to long-horizon settings may require revisiting its time-to-live policy.The paper specifically identifies persistent memory, video streams, and embodied interaction as contexts where this policy may need reconsideration.
Loading 2607.28374v1…