PaperlayerView exact revision on arXiv

Source-linked AI summary

ActiveEvidence: Decision-Focused Evidence Graphs with Verifiable Acquisition Gaps

Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Peng Zhang, Daren Zha, Jun Xiao

arXiv:2610.20532v1cs.AIcs.CVcs.LG

TL;DR

Multimodal agents need to decide whether, which, and when to acquire visual evidence under resource constraints. ActiveEvidence learns evidence interactions and refines only decision-relevant uncertainties, then certifies decisions relative to its pinned controller. On held-out cases, it reports task score 0.842 and dense replay action agreement of 0.992 while evaluating fewer expensive interaction terms per decision.

  • Problem

    Agents need to decide whether, which, and when to acquire visual evidence, but existing systems leave the measurable decision across multiple candidates and resource budgets unresolved.

    Source: §1 · p. 1

  • Method

    A decision-certifiable CEIG models evidence interactions with score enclosures, and gap-directed refinement evaluates only unresolved interactions that could change the selected action.

    Source: §1 · p. 1, §3.1 · p. 2

  • Results

    Task score rose from 0.828 to 0.842 while accepted calls fell from 1.82 to 1.74 and visual tokens from 299.4K to 286.5K; dense replay showed 0.992 action agreement.

    Source: §4.2 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    The certificate verifies decisions relative to the pinned learned controller, while semantic quality is evaluated separately through post-freeze labels.

    Source: §1 · p. 1, §6 · p. 8

    Semantic evidence rests on 128 held-out cases from four domains, and offline supervision breaks even only after 580–1290 queries.

    Source: §5 · p. 8

Abstract

from arXiv · show

A multimodal agent must decide whether visual evidence is worth acquiring, which candidate or resolution to query, and when further interaction can no longer change its decision. We study this problem under a strict free-metadata/paid-observation boundary, stochastic vector costs, and a frozen execution trace, and treat it as decision-certifiable active evidence acquisition. A Counterfactual Evidence Interaction Graph (CEIG) represents unary evidence value, signed second-order interaction, and directed conditional harm while maintaining explicit score enclosures for unresolved interactions; a decision-weighted enclosure objective tightens the bounds that can change action identity. Gap-directed refinement evaluates only interactions that can displace the incumbent acquisition and stops once an $\epsilon$-decision certificate shows that unresolved terms cannot change the selected action beyond a declared tolerance. The certificate is model-relative: it verifies policy-consistent optimality under the pinned learned score, and semantic quality is measured separately through post-freeze labels. Paired continuation rollouts learn the downstream value of paid previews, an online operational cost filter is separated from independent finite-policy-family trajectory certification, and an acquisition monitor with an independently implemented verifier binds every certificate to the frozen trace. On 640 cases split into 384/128/128 training, validation, and test cases, the controller reaches task score 0.842, grounding F1 0.816, evidence recall 0.873, 1.74 accepted calls, 286.5K visual tokens, and 742ms p95 latency. Dense replay of issued certificates shows 0.992 action agreement and a 0.988 certificate rate while evaluating 19.6 instead of 191.4 expensive interaction terms per decision.

1 Introduction

ActiveEvidence frames visual acquisition as a sequential decision under resource and information constraints, asking whether, which, and when evidence should be acquired. It combines learned evidence interactions with decision-focused refinement and certification to justify acquisitions without exhaustive interaction evaluation.

1 Introduction

  • The remaining gap is a measurable whether–which–when decision over visual candidates and resource budgets, with an execution trace that preserves the information boundary.Prior tool-using and grounded web agents make action–observation loops, realistic interfaces, and functional evaluation explicit.Source: §1 · p. 1
  • ActiveEvidence treats acquisition as decision certification: it learns counterfactual interactions, encloses unresolved scores, refines interactions that could change the top action, and stops at a declared tolerance.The certificate is relative to the pinned controller; semantic quality is evaluated separately using post-freeze labels.Source: §1 · p. 1
  • The decision-certifiable CEIG separates unary utility, signed second-order interaction, directed conditional harm, and semantic contradiction, while decision-weighted enclosure training prioritizes bounds that can change the acquisition decision.Gap-directed refinement evaluates only interactions that could displace the incumbent and terminates with an ε-decision certificate relative to the pinned dense controller.Source: §1 · p. 1
  • The framework separates paid-preview value learning, online operational cost filtering, finite-policy-family trajectory certification, trace monitoring, and post-freeze evaluation.Its evaluation covers semantic quality, interaction attribution, certificate validity, preview value, cost shifts, certificate attacks, candidate-pool scaling, and transfer under matched contracts.Source: §1 · p. 2

2 Related Work

Prior work studies tool use, grounding, information acquisition, selective routing, and adaptive visual tools. ActiveEvidence positions itself at the visual-acquisition boundary and couples a learned controller with decision certificates and replayable execution traces.

Tool use and grounding

  • Tool-use work models action–observation trajectories [40], while grounded web and agent benchmarks emphasize realistic environments, tool contracts, and functional evaluation [16] [38] [39].ActiveEvidence instead isolates whether to acquire a visual observation and records the candidate, cost, and stop event for replay.Source: §2 · p. 2

Agent evaluation and information acquisition

  • Information-acquisition research spans value of information, adaptive submodularity [22], constrained decision processes, metareasoning [9, 24, 43], and cost-sensitive or nonmyopic evidence selection.ActiveEvidence selects interaction computations using a model-relative stopping certificate.Source: §2 · p. 2

Selective prediction and cost-aware routing

  • Selective prediction and routing study reject options [19], risk control, and quality–cost model selection [11] [41]; ActiveEvidence applies decision-making to visual observations with trace and integrity records.Its reported selective-risk measures include AURC 0.118, risk at 80% coverage 0.134, ECE 0.031, and abstention rate 0.064.Source: §2 · p. 2

Sequential decision and verification

  • Offline planning and reinforcement-learning methods learn from fixed traces or counterfactual rollouts [10] [18] [29] [31] [32] [48], while verifier research emphasizes explicit checks and trace audits [13] [33] [36].ActiveEvidence applies these ideas to counterfactual utility, calibration, and ledger checks around visual acquisition.Source: §2 · p. 2

Adaptive visual acquisition and visual-tool invocation

  • Adaptive visual systems request resolutions, crops, or tools [15] [28] [37]; ActiveEvidence certifies that unevaluated interactions cannot change the selected acquisition under its pinned controller.Its positioning couples an observed-only interface and independent verifier, distinguishing controller-relative certificates from post-freeze semantic evaluation.Source: §2 · p. 2

3 Method

ActiveEvidence separates learned evidence valuation, decision-focused refinement, and execution verification under a free-metadata/paid-observation boundary. It stops when unresolved score bounds cannot change the selected action beyond tolerance and independently verifies the frozen trace.

3.4 Value-Aware Gate and Candidate Ranker

  • The controller uses a Need Gate, a CEIG with score enclosures, and paired continuation decisions to estimate the downstream value of paid previews.A frozen decision-sensitivity weight emphasizes incumbent and near-optimal-challenger interactions; at inference, preview value competes with acquisition and termination scores in the feasible-action set.Source: §3.1 · p. 2, §3.4 · p. 5

3.1 Method Overview

  • The controller refines only interactions that can reduce incumbent–challenger ambiguity and stops when the best unresolved upper score is within ε_dec of the incumbent.Its certificate is model-relative to the pinned dense controller; at ε_dec = 0, the selected action maximizes the dense pinned score up to deterministic tie-breaking.Source: §3.1 · p. 2, §3.4 · p. 5

3.2 Problem Setup and Information Boundary

  • Selection-time inputs exclude pixel-derived quality, relevance, captions, thumbnail embeddings, and task labels; paid preview fields become available only after an accepted preview.Evaluation labels are joined only after the trace digest is committed.Source: §3.2 · p. 3

3.3 State, Actions, and Resource Ledger

  • STOP answers from the current state, ABSTAIN emits no task answer, and an acquisition bundle executes atomically at one resolution with per-image calls and costs.A bundle contains at most K candidate IDs and scores within-bundle interactions before execution; rejected actions remain in the ledger without reducing the budget.Source: §3.3 · p. 3

3.4 Value-Aware Gate and Candidate Ranker

  • 0.071 held-out interaction error, 0.914 sign AUROC, and 0.861 complementary-evidence recovery characterize the CEIG results.The CEIG models unary changes, signed pair interactions, and directed conditional harm; its enclosure objective gives greater weight to interactions that may affect the acquisition decision.Source: §3.4 · pp. 4–5

3.5 Budget-Conditioned Controller and Audit Quantities

  • Online risk-bounded cost filtering is operational; complete-policy trajectory certification is the separate safety statement for the frozen policy family.The conformal predictor's marginal calibration is weaker than conditional coverage for adaptively selected state–action pairs.Source: §3.5 · p. 6

3.6 Verifiable Decision Certificates

  • An independently implemented verifier reconstructs the feasible set, certificate, ledger, and terminal semantics from monitor receipts and the committed trace.The trusted execution boundary comprises the monitor, metering interface, and commitment store; verification precedes the post-freeze label join.Source: §3.1 · p. 3, §3.6 · p. 6

3.7 Execution and Trace-Freeze Contract

  • 0.985 exact replay, 0.972 action-equivalent replay, and 0.957 metric-equivalent replay were reported for the trace-freeze procedure.Access-injection detection was 0.973 and mutation detection was 0.967 over the 2000-case taxonomy.Source: §3.7 · pp. 6–7

4 Experiment

The experiment evaluates the controller under shared test conditions, reports held-out semantic and resource results, and examines ablations and audit outcomes.

4.1 Experimental Setup

  • The 640 cases are divided into 384 training, 128 validation, and 128 test cases, with policies evaluated under shared prompts, action limits, preview terms, and vector budgets.The setup specifies matched baselines and reports resource, quality, and STOP/ABSTAIN metrics.Source: §4.1 · p. 7

4.2 Main Results: Held-out Semantic Evaluation

  • On the 128-case held-out test split, under the common evaluation setup, the full controller scored 0.842 versus recurrent acquisition’s 0.828, with accepted calls dropping from 1.82 to 1.74 and visual tokens from 299.4K to 286.5K.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 Analysis and Ablations

  • 0.988 certificate rate is reached with 19.6 rather than 34.2 edges under decision-weighted training; per-compute refinement reaches 0.994 dense agreement with 16.8 edges.The analysis also reports matched prior-mechanism task scores of 0.818 for VisionThink, 0.823 for AdaptVision, and 0.810 for AdaTooler-V, versus 0.842 for ActiveEvidence.Source: §4.3 · pp. 7–8

4.4 Held-out Evaluation and Audit Results

  • On 128 test cases, the controller abstains on 0.078, uses 15.8GB peak VRAM, and fails on 0.023.Access-injection detection is 0.973 with clean-trace FPR 0.012; paired 95% intervals are [0.830,0.854] for task, [0.801,0.831] for grounding, and [0.856,0.890] for evidence recall.Source: §4.4 · p. 8

5 Limitations, Ethics, and Reproducibility

The paper bounds its semantic evidence and certificate claims, documents deployment requirements, and reports reproducibility checks.

Limitations

  • Semantic evidence comes from 128 held-out cases across four domains and three controller seeds, while certificates verify consistency with the pinned learned score rather than ground-truth utility.Ground-truth utility is evaluated separately through post-freeze labels, and offline supervision breaks even only after 580–1290 queries.Source: §5 · p. 8

Reproducibility

  • The execution contract binds configuration, seeds, source revision, manifests, traces, labels, verifier, environment, and regeneration command through content digests.Cross-implementation replay is 0.985 exact, 0.972 action-equivalent, and 0.957 metric-equivalent; row-level ledger auditing records a 0.018 budget-violation rate.Source: §5 · p. 8

Ethics

  • Visual acquisition is limited to hash-bound assets under declared license and access policies, with abstention or stopping for missing, conflicting, stale, or out-of-root evidence.Assets require a documented license or consent basis, PII handling, and a deletion policy for sensitive traces.Source: §5 · p. 8

6 Conclusion

ActiveEvidence combines a structured evidence model, decision-focused refinement, and separate execution checks; held-out dense replay reports high action agreement with fewer interaction evaluations.

Conclusion

  • Dense replay confirms 0.992 action agreement while using 19.6 instead of 191.4 expensive interaction evaluations per decision.The framework separates unary value, second-order interaction, directed harm, and semantic contradiction, and certifies decisions relative to the pinned dense controller.Source: §6 · p. 8

A Mechanism-Level Unit Test

The mechanism-level fixture tests acquisition accounting under an abstract budget, alongside selector behavior and post-freeze audit quantities.

A Mechanism-Level Unit Test

  • Active selection accepts 1.03 image calls and lowers fixture visual tokens from 13.00 to 10.00, latency from 4.25 to 3.50, and exhaustion from 0.625 to 0.375 versus always-on.The static selector has the highest ID hit at 0.625, so the fixture isolates acquisition economy rather than semantic quality.Source: A Mechanism-Level Unit Test · p. 11, Table 3 · p. 11

A.1 Fixture Budget Vector

  • The fixture uses an abstract budget vector, while held-out results use native token, latency, and call units.The stated preview cost is (0, 2, 1, 0), and image cost is (0, 8|J|, 2|J|, |J|).Source: §A.1 · p. 11

A.2 Fixed Gate and Baseline Selectors

  • The fixed screen acquires when its gate value is at least 0.45, while the static selector orders free metadata by byte size and image ID.The Weighted Metadata Baseline pays for requested preview fields before ranking by relevance, normalized byte size, and image ID.Source: §A.2 · p. 11

A.3 Audit Quantities

  • The row-level held-out audit reports accepted-call means of 1.63, 1.50, and 1.03 and exhaustion rates of 0.625, 0.500, and 0.375 for always-on, static, and active policies.These diagnostics are computed only after trace freeze; semantic task quality is reported separately.Source: §A.3 · p. 11

B.1 Execution and Trace-Freeze Procedure

The procedure meters accepted previews and acquisitions, freezes a canonical execution trace, then independently checks its witnesses and receipts. The held-out protocol fixes splits and evaluation conditions before a single test evaluation.

B.1 Execution and Trace-Freeze Procedure

  • The policy initializes a ledger, scores feasible actions, records rejected attempts, executes accepted actions, and re-scores until STOP or ABSTAIN.It then freezes the ordered trace and selection-visible metrics for independent verification.Source: §B.1 · p. 11
  • Replay yields 0.985 exact, 0.972 action-equivalent, and 0.957 metric-equivalent rates; the trace stores action, cost, budget, terminal, and digest fields.The manifest has 12480 unique assets and 12480/12471 byte-hash matches; nine mismatches are controlled perturbations.Source: §B.1 · p. 11

B.2 Held-out Study Protocol

  • The held-out study partitions examples before threshold selection and evaluates once under matched prompt, decoder, action-cap, and vector-budget conditions.It tests active, static, random, always-on, and text-only controls; the ledger records semantic, resource, and audit outcomes from the frozen receipt.Source: §B.2 · p. 12

B.3 Release Checklist

  • The evidence ledger binds fields, events, budgets, post-freeze labels, and replay receipts; each receipt supports only its designated interpretation, not semantic quality.Semantic quality comes from the held-out evaluation; hash, accounting, replay, and injection receipts support identity, conservation, equivalence, and threat-model detection, respectively.Source: §B.3 · p. 12

C.1 Experiment Ledger

The experiment ledger separates mechanism proxies and provenance records from semantic results on the held-out visual contract. Evidence receipts support narrowly defined claims rather than serving as interchangeable evidence.

C.1 Experiment Ledger

  • The ledger distinguishes metadata-fixture mechanism tests, provenance records, and held-out test results under frozen splits and post-freeze labels.Held-out rows carry the semantic, cost, and audit statements, including task drops of 0.021, 0.028, and 0.034 under matched transfer, hard negatives, and conflicting evidence.Source: §C.1 · p. 12

C.2 Evidence Receipts

  • Receipts support bounded interpretations: hashes establish asset identity, accounting and replay establish budget conservation and trace equivalence, and injection tests establish detection under the declared threat model.Semantic quality is supported by the held-out evaluation, not these receipts.Source: §C.2 · p. 12, Table 5 · p. 12

C.3 Robustness and Audit Stress

Stress tests, audits, and ablations assess semantic quality, resource use, certification, and trace integrity under matched settings. The results span robustness, interaction refinement, candidate scaling, and offline training cost.

C.3 Robustness and Audit Stress

  • Conflicting evidence and hard negatives lower task scores to 0.808 and 0.814, while metadata corruption lowers it to 0.832; worst budget-shift violations are 0.029.The large pool reaches task score 0.846 at 338.2K visual tokens; access-injection detection remains 0.969–0.981 across stresses.Source: §C.3 · pp. 12–13, Table 6 · p. 13

C.4 Row-Level Budget Audit; C.5 Gate–Ranker Attribution; C.6 Held-out Operational Outcomes

  • Active evidence attempts 1.42 and accepts 1.03 calls per row, leaves 0.97 calls, and exhausts budget in 0.375 of rows; the learned gate, ranker, and stopper reach held-out quality 0.842.The learned combination selects 1.125 candidates, while active evidence has 742 ms p95 latency, 0.023 failure rate, and 0.005 seed dispersion.Source: §C.4 · p. 13, §C.5 · p. 13, §C.6 · p. 13

D.1 Interaction Attribution; D.2 Decision-Certificate Attribution

  • Explicit interaction supervision lowers interaction MAE to 0.071 and raises sign AUROC to 0.914; decision-weighted tightness reaches a 0.988 certificate rate with 19.6 evaluated edges and 0.003 regret.The interaction results compare against unary-only and latent-pairwise controls; certificate results compare training objectives while holding model capacity and refinement fixed.Source: §D.1 · p. 14, §D.2 · p. 14

D.3 Validation-Calibrated Budget Shift; D.4 Independent Trajectory-Risk Certification

  • Under +50% latency, violations are 0.319 for the hard filter and 0.041 for the full controller; the mixed-shift figures are 0.274 and 0.029, respectively.Independent certification approves the adaptive upper-cost filter and complete policy, while rejecting the point-cost filter whose bound 0.082 exceeds target.Source: §D.3 · p. 14, §D.4 · p. 14

D.5 Decision-Certificate Attack Taxonomy; D.6 Access Boundary and Coordinated Attacks

  • Post-freeze trace edits have the lowest detection rate, 0.944, and highest metric impact, 0.068; coordinated tests reject 198/200 pre-freeze label-access attempts and 189/200 post-freeze chain-reconstruction attacks.Five of seven incorrectly accepted chain-reconstruction traces change a reported metric.Source: §D.5 · p. 15, §D.6 · p. 15

D.7 Gap-Directed Interaction Refinement; D.8 Candidate-Pool Scaling

  • Per-compute refinement reaches 0.994 dense agreement, 0.991 certificate rate, and 16.8 expensive edges; task score rises from 0.821 at pool size 4 to 0.846 at size 64.With pool growth, accepted calls increase from 1.48 to 1.81, controller time from 5.6 to 31.4 ms, and peak memory from 15.0 to 16.1 GB.Source: §D.7 · p. 16, §D.8 · p. 16

D.8 Candidate-Pool Scaling; D.9 Offline Supervision and Online Amortization

  • Evaluated-edge fraction falls from 0.800 at four candidates to 0.012 at 64, while certificate rate remains 0.996–0.982; evaluated edges rise from 4.8 to 24.5 despite action materialization growth.Offline supervision packages break even after 580–1290 queries; the certificate-oriented package takes 16.4 GPU-hours and breaks even after 1290 queries.Source: §D.8 · p. 16, §D.9 · pp. 17–18

D.10 Preview Value and Leakage

The preview study compares paired decision-value supervision with alternative preview rules and tests whether preview scores use only paid information. The section also evaluates transfer and certificate agreement under dense replay.

D.10 Preview Value and Leakage

  • Paired decision-value supervision reaches task 0.842 and net benefit 0.124, with 0.043 harmful previews, 62.4K preview tokens, and decision regret 0.021, versus 0.836, 0.098, 0.106, 73.1K, and 0.038 for unpaired regression.The fixed schedule uses 118.6K preview tokens for net benefit 0.047; the immediate-utility gate lies between fixed scheduling and unpaired regression on every column.Source: Table 20 · p. 18, §D.10 · p. 18
  • Permutation sensitivity is 0.007 for paired supervision versus 0.016 for unpaired regression; pre-preview AUROC is 0.502 versus 0.508, and post-preview AUROC is 0.816 versus 0.724.The measured separation occurs after the paid-preview event.Source: Table 21 · p. 18, §D.10 · p. 18

D.11 Cross-Model and Cross-Domain Transfer

  • Across three transfer targets, zero-shot task drops stay within 0.027 and recalibration-only drops within 0.002, with ECE at most 0.059 and violation rates at most 0.028.The new task domain is the hardest target on every column; for the second VLM family, task success is 82.4% zero-shot, 84.1% with few-shot adaptation, and 85.0% after lightweight tuning.Source: §D.11 · p. 18, Table 22 · p. 18

D.12 Certificate Validity under Dense Replay

  • Dense replay of 458 issued decisions yields pooled agreement 0.992, disagreement 0.008, and regret 0.003.The decisions include 118 STOP and 10 ABSTAIN actions.Source: §D.12 · pp. 18–19, Table 23 · p. 19

D.13 Stratified Held-out Analysis

The held-out analysis reports task scores by domain, candidate-pool size, and interaction strength, alongside calls, visual tokens, and bootstrap intervals.

D.13 Stratified Held-out Analysis

  • Domain task scores range from 0.838 to 0.846, with every domain interval containing aggregate score 0.842; high-candidate cases score 0.847 versus 0.837 for low-candidate cases.Strong-interaction cases score 0.856 and weak-interaction cases 0.831; these are within-controller strata.Source: Table 24 · p. 19, §D.13 · p. 19

D.14 Dataset and Runtime Configuration

The configuration passage specifies the study's dataset, evaluation, and runtime setup. The held-out task-score aggregate is reported with a hierarchical-bootstrap interval.

D.14 Dataset and Runtime Configuration

  • The study evaluates 640 cases using exact matching and semantic equivalence, with grounding and evidence recall measured against frozen labels and runtime identifiers tying measurements to the same model and execution configuration.Source: §D.14 · p. 19
  • The aggregate held-out task score is 0.842, with a hierarchical-bootstrap 95% interval of [0.830, 0.854].Figure 8 marks the aggregate score and interval alongside stratified scores.Source: Figure 8 · p. 19
Loading 2610.20532v1…