TAEC: Trajectory-Aware Evidence Coordination for Multi-Step Visual RAG

1 INTRODUCTION

• Multi-step visual RAG extends retrieve-then-read systems with successive searches, query reformulation, evidence inspection, and updates to a reasoning state.The interaction continues until the model answers or exhausts its interaction budget.Source: §1 · p. 1
• ReAct analysis on ViDoSeek found that accuracy among trajectories retrieving annotated evidence fell from 91.7% for two searches to 52.0% for nine or more.The paper terms the gap between retrieving relevant evidence and using it effectively over a trajectory trajectory-level evidence utilization degradation.Source: §1 · p. 2
• As reasoning progresses, redundant sources can consume context needed for missing evidence, while resolved or unproductive observations linger and retained images may lack needed detail.The paper argues these decisions should be coordinated around an evolving account of unresolved requirements.Source: §1 · p. 2
• TAEC is a training-free coordination layer that uses a shared state of unresolved answer requirements to guide evidence admission, memory exposure, and visual detail allocation.Search planning, visual interpretation, state integration, stopping decisions, and answer generation remain the acting model’s responsibility.Source: §1 · p. 2
• TAEC achieves the highest average accuracy against leading training-free visual RAG baselines on ViDoSeek, SlideVQA, and MMLongBench-Doc under a unified evaluation protocol.The protocol uses a common retriever, shared backbone models, the same judge, and matched interaction budgets.Source: §1 · p. 3

Visual and multi-step multimodal RAG

• Visual RAG includes systems retrieving natural images or rendered document pages, while multi-step systems add iterative retrieval and reasoning, including multi-agent workflows and learned retrieval or perception actions.Examples include ViDoRAG (Wang et al., 2025a) and VRAG-RL (Wang et al., 2025b).Source: §2 · p. 3

Adaptive retrieval and evidence selection

• Adaptive RAG methods vary retrieval decisions using reflection, confidence, question complexity, retrieval quality, evidence sufficiency, or information gaps; other methods select sources by relevance and redundancy.Examples include Self-RAG (Asai et al., 2024), FLARE (Jiang et al., 2023), GRO-RAG, and AdaGReS.Source: §2 · p. 3

Context and memory management

• Prior context and memory approaches compress prompts, manage memory tiers, structure evidence, apply semantic priority or temporal decay, and construct query-specific evidence subgraphs.TAEC focuses on criteria for evidence admission, memory exposure, and visual detail allocation; its mechanisms also condition history presentation on resolution status and branch outcomes, and visual allocation on detail demand and context pressure.Source: §2 · pp. 3–4

3 TRAJECTORY-AWARE EVIDENCE COORDINATION

• TAEC represents the reasoning trajectory as a directed acyclic graph of searches and observations, with dependencies that support requirement-resolution assessment, evidence tracing, and abandoned-branch identification.The graph is denoted Gt, and memory items are indexed by their originating nodes.Source: §3 · p. 4

3.1 UNIFIED FORMULATION

• The unified formulation represents the question’s answer needs as weighted requirements and uses their current support in context to form a shared state emphasizing unresolved requirements.At each step, newly admitted sources, exposed historical observations, and retained images shape the evidence presented under limited processing budgets; K, Lt, and Bt constrain source count, memory context length, and visual capacity.Source: §3.1 · p. 4

3.2 EVIDENCE ADMISSION

• Evidence Admission addresses redundancy by selecting retrieved sources according to their coverage of unresolved requirements, retrieval relevance, and redundancy with already selected candidates.A greedy procedure approximates the objective while retaining the top-ranked retrieval result as an anchor.Source: §3.2 · pp. 4–5

3.3 ADAPTIVE MEMORY EXPOSURE

• Adaptive Memory Exposure adjusts the persistence and presentation of accumulated observations according to their current trajectory role and information granularity.Its rendering preserves active or uncertain observations in full, condenses resolved observations, and gives failed branches brief terminal traces; the stored trajectory remains unchanged.Source: §3.3 · p. 5

3.4 VISUAL DETAIL ALLOCATION

• Visual Detail Allocation separates image relevance from the capacity assigned to inspect each retained image and allocates detail according to expected benefit and support for current requirements.It preserves a share based on historical relevance, uses the remaining share to favor images whose finer detail is expected to support current reasoning, and adapts total capacity to context pressure.Source: §3.4 · pp. 5–6

4.1 EXPERIMENT SETTING

• The evaluation uses four frozen proprietary vision-language models and three benchmarks, with shared retrieval indexes, interaction and visual budgets, and a fixed GPT-4.1 judging prompt.Baselines include single-pass, iterative, and agentic systems.Source: §4.1 · p. 6

4.2 EVIDENCE UTILISATION OVER LONG TRAJECTORIES

• Among ReAct trajectories that retrieve a gold page, accuracy decreases with more searches; even when the gold page appears among the first five results, accuracy is 96.0% after one search and 39.2% at nine or more.The analysis uses ReAct on ViDoSeek with Gemini-3.5-Flash, retaining retrieved images in context.Source: §4.2 · pp. 6–7

4.3 MAIN RESULTS

• TAEC has the highest accuracy in all twelve Table 1 settings; with Gemini-3.5-Flash, its gains over ViDoRAG are 7.8, 11.1, and 18.6 percentage points across ViDoSeek, SlideVQA, and MMLongBench-Doc.Across four acting models, TAEC leads in ten of twelve comparisons; both exceptions are on MMLongBench-Doc.Source: Table 1 · p. 7, §4.3 · p. 7

4.4 COMPONENT ABLATION

• Each TAEC component improves accuracy over the DAG agent on all three benchmarks, achieving approximately 41%–88% of the full system’s gain.The complete system achieves the highest accuracy throughout; admission gives the largest individual gains on ViDoSeek and MMLongBench-Doc.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5 ANALYSIS

• On questions requiring at least four ReAct searches, TAEC exceeds ReAct by 28.1, 26.6, and 13.6 percentage points on ViDoSeek, SlideVQA, and MMLongBench-Doc; on multi-search subsets, TAEC’s gains over the DAG agent are 14.0, 13.9, and 5.6 points.The largest positive contributions come from questions where only TAEC retrieves a gold page.Source: §4.5 · p. 8
• On ViDoSeek, distinct pages make up 99.6% of TAEC page admissions, while coverage on the fixed multi-search subset rises from 74.6% with ReAct to 90.8% with TAEC.Across benchmarks, TAEC averages 1.28–2.58 searches per question versus 3.00–6.00 for ReAct.Source: §4.5 · p. 8, Figure 5 · p. 9

4.6 EFFICIENCY

• On ViDoSeek, TAEC transmits 0.61 MB of context per question versus 3.73 MB for ReAct and reduces cumulative image transmissions from 92.8 to 14.7 per question.These savings accompany higher answer accuracy.Source: §4.6 · p. 9

4.7 LIMITATIONS

• TAEC relies on the acting model’s agentic capabilities; the main comparisons use one retriever family, and all three benchmarks contain English document collections.The authors note that TAEC cannot fully compensate for deficiencies in those capabilities.Source: §4.7 · p. 9

• The conclusion frames requirement-aware evidence coordination as an effective approach to sustaining evidence use in multi-step visual RAG.Source: §5 · p. 9

A.1 SUBSTRATE: THE DAG AGENT

• The DAG substrate records searches and answers as a trajectory graph, stores visual memories under their admitting nodes, and uses graph structure to assess their continuing contribution.Its tool loop retrieves pages, summarizes them, and rebuilds the prompt with the action graph and re-presented visual memories.Source: §A.1 · pp. 12–13

A.2 REQUIREMENT STATE Ut

• The evaluated requirement set is extracted once per question as two to four slot keywords, capped at four with uniform weights; coverage updates as pages are admitted.Memory exposure reads resolution cues from summaries, while admission passes slot-coverage scores to visual allocation.Source: §A.2 · p. 13

A.3 EVIDENCE ADMISSION

• Evidence Admission keeps the retriever’s top result, then greedily fills remaining slots from an oversampled pool using slot coverage and relevance while penalizing caption overlap.Admitted pages pass through a seen-page filter; no images are cropped in the main runs.Source: §A.3 · p. 14

A.4 ADAPTIVE MEMORY EXPOSURE

• Adaptive Memory Exposure renders older graph nodes as resolved, failed, or active based on summary cues, using compact sentences, short terminal traces, or full summaries, respectively.The persistence term applies granularity-specific decay.Source: §A.1 · p. 13, §A.4 · p. 14

A.5 VISUAL DETAIL ALLOCATION

• Visual detail allocation acts only when historical images are re-presented: it selects images by memory energy and adjusts their pixel shares using detail proxies, with region crops receiving more when present.In the evaluated runs, the requirement-support input is constant; requirement information reaches allocation through the detail proxy.Source: §A.5 · p. 14

A.6 COMPOSITION AND COST

• TAEC composes the three components over separate substrate methods and adds one requirement-slot extraction call per question, with no extra calls per step.Other added computation is string matching and arithmetic on the stored graph.Source: §A.6 · p. 14

A.7 READING THE OVERVIEW FIGURE

• The overview’s admission panel depicts retaining the top retrieval result and favoring pages that cover uncovered slots while penalizing overlap; its memory panel maps active, resolved, retired, and faded representations.The evaluated visual-allocation support row is constant, while its detail-gain row combines caption length with admission’s slot-coverage score.Source: §A.7 · pp. 14–15

A.8 BASELINES

• Vanilla retrieves once, while ReAct retains retrieved images in message history and uniformly reallocates pixels under a visual budget; the same call ceiling and fallbacks apply.ViDoRAG is run from its public implementation with the shared index, backbone, and judge.Source: §A.8 · p. 15

B EXPERIMENTAL DETAILS

• The evaluation uses ViDoSeek, SlideVQA, and MMLongBench-Doc, with the main MMLongBench-Doc results scoring 847 questions annotated as answerable.The benchmarks differ in document type and evidence needs; MMLongBench-Doc is the longest and most often requires evidence from multiple pages.Source: B EXPERIMENTAL DETAILS · p. 15
• Systems share a retriever within each comparison; the main index pools 36,233 pages, and all systems admit at most five pages per search.The protocol records retriever fingerprints and uses fixed VLM-generated captions; four proprietary acting models span deployment tiers.Source: B EXPERIMENTAL DETAILS · pp. 15–16
• Accuracy counts unanswered failures as incorrect, and evidence coverage measures whether a gold page enters context during a trajectory; post-hit accuracy is accuracy on that subset.The fixed GPT-4.1 binary judge is used for all reported results; MMLongBench-Doc values are not its official generalized accuracy and are comparable within this paper.Source: B EXPERIMENTAL DETAILS · pp. 16–17

C FULL TABLES AND FIGURES FOR THE MAIN TEXT

• Table 2 groups ReAct outcomes by search count and identifies the found-unused share; Table 3 groups accuracy by ReAct difficulty, while Tables 4–7 cover matched subsets, coverage and use, cost, and ablations.Figure 6 shows closed-source backbone comparisons and matched ViDoSeek question groups routed by gold-page-found status under ReAct and TAEC.Source: C FULL TABLES AND FIGURES FOR THE MAIN… · p. 17, Table 2 · p. 18, Table 3 · p. 18
• Figure 7 compares accuracy with search budgets and transmitted context; it marks the smallest saturated search budget and scales point area by images sent.Its caption reports that TAEC answers eight points above the unmanaged agent while sending one-sixth of its context and images.Source: Figure 7 · p. 20

D M3RAG, REIMPLEMENTED

• The evaluated M3RAG is an adaptation for closed backbones, replacing unavailable model-access mechanisms with acting-model verdicts, reported bounding boxes, and model-assigned relevance.The paper retains five candidate pages per query, 144-token regions, and seven rounds, while specifying settings for parameters the original paper leaves open.Source: D M3RAG, REIMPLEMENTED · p. 21
• TAEC leads M3RAG by 5.0, 2.1, and 2.5 points on Gemini-3.5-Flash, and is ahead in ten of twelve comparisons across the acting models.The comparison uses the same index, budgets, and judge.Source: D M3RAG, REIMPLEMENTED · p. 21, Table 9 · p. 21
• Across all 1,091 MMLongBench-Doc questions, TAEC leads M3RAG by 1.1 points with GPT-4o-mini, while its GPT-5.6-Sol deficit widens to 4.2 points; the main evaluation scores 847 answerable questions.Source: D M3RAG, REIMPLEMENTED · pp. 21–22

E RESULTS BY QUESTION TYPE

• 9.2 points on single-hop and 5.0 on multi-hop ViDoSeek questions: TAEC improves over the substrate in both groups.The respective accuracies are 77.8 → 87.0 and 83.1 → 88.1 on Gemini-3.5-Flash.Source: E RESULTS BY QUESTION TYPE · p. 22, Table 10 · p. 23
• On MMLongBench-Doc, gains over the substrate are +10.4 on text, +9.4 on charts, +4.2 on tables, and +3.3 on figure evidence.Figure questions depend on reading a picture rather than on which pages are present; every system is weakest there.Source: E RESULTS BY QUESTION TYPE · p. 22, Table 10 · p. 23

E.1 MMLONGBENCH-DOC WITH ITS UNANSWERABLE QUESTIONS

• On the full 1,091-question MMLongBench-Doc set, TAEC leads on three backbones, while M3RAG leads by 4.2 points with GPT-5.6-Sol.The main-text comparison scores the benchmark’s 847 answerable questions; on the full set, declining on unanswerable questions is a capability none of the three components is designed to encourage.Source: §E.1 · p. 22, Table 11 · p. 23

F A TRAJECTORY IN DETAIL

• In the worked renal-registry question, no baseline retrieves page 7, and no single retrieval could have found it among the systems’ top-three results.Vanilla estimates 60%, ReAct returns nothing, and the DAG agent reads 55% for all patients rather than the transplant group.Source: F A TRAJECTORY IN DETAIL · pp. 23–24
• With five searches, TAEC retrieves the answer page on its eleventh position by querying against still-open requirements, unlike ReAct’s rewrites.TAEC admits 25 of 48 candidates across four calls, and the answer cites the transplant group of 461 patients.Source: F A TRAJECTORY IN DETAIL · p. 24

G CONTEXT PRESSURE WITH RETRIEVAL HELD FIXED

• Accuracy falls by 4.0 points over the first 16K tokens and by 5.2 over 32K, although retrieval and the gold page remain fixed.The added text is neutral, unrelated to the question, and carries no relevant evidence.Source: G CONTEXT PRESSURE WITH RETRIEVAL HELD… · pp. 24–25, Table 12 · p. 24
• The fixed-retrieval result shows that evidence can remain present yet be read less reliably as unrelated context grows.The evidence is unchanged across the configurations, supporting a context-pressure diagnosis separate from retrieval differences.Source: G CONTEXT PRESSURE WITH RETRIEVAL HELD… · pp. 24–25

H WHERE THE MULTI-STEP BASELINES LOSE

• On deeper ReAct trajectories, it loses 8.6 points on ViDoSeek with GPT-4o-mini and 8.2 on SlideVQA with Gemini-3.5-Flash against single-pass retrieval.ReAct re-sends its observation history on every call and reaches 93 images per question; accuracy falls with the number of images even when the gold page is present.Source: H WHERE THE MULTI-STEP BASELINES LOSE · p. 25, Table 13 · p. 25
• The sign of ReAct’s deeper-trajectory comparison varies by acting model: on Kimi-K3 it gains 10.2 points on SlideVQA.The passage links outcomes to whether the model issues useful follow-up searches and still reads accumulated evidence.Source: H WHERE THE MULTI-STEP BASELINES LOSE · p. 25

I HOW THE ACTING MODEL DETERMINES THE SIZE OF THE GAIN

• The section distinguishes perceptual ability—answering after a gold page is retrieved—from agentic ability: judging whether accumulated evidence answers the question at each step.Agentic errors include answering without a gold page in context or searching again when single retrieval had already answered the question.Source: I HOW THE ACTING MODEL DETERMINES THE… · pp. 25–26, Table 14 · p. 26
• Perceptual accuracy is 85.0, 85.3, and 82.5 for Gemini-3.5-Flash, Kimi-K3, and GPT-5.6-Sol, while their premature-commitment rates are 22.6%, 12.2%, and 14.8%.Their over-search rates on questions answered by single retrieval are 6.0%, 15.5%, and 12.8%, respectively.Source: I HOW THE ACTING MODEL DETERMINES THE… · p. 26
• +6.6 points on Gemini-3.5-Flash, +3.9 on GPT-5.6-Sol, and +2.7 on Kimi-K3 are the accuracy gains as coordination repairs sufficiency-judgment errors.Premature commitment and failure to recognize sufficient evidence converge to about one point each across these backbones.Source: I HOW THE ACTING MODEL DETERMINES THE… · p. 26
• GPT-4o-mini reads a retrieved gold page at 64.2 accuracy and over-searches answerable questions 47.9% of the time; coordination adds 4.7 points, chiefly through evidence admission.Its over-search rate falls to 40.0%, whereas rates for the other backbones fall to near 6%.Source: I HOW THE ACTING MODEL DETERMINES THE… · p. 26
• On ViDoSeek and SlideVQA, coordination raises the share of single-search answers with a gold page in context to 96.4% and 92.7%, from 85.9% and 80.4%, respectively, without more pages in context.On Gemini-3.5-Flash, coordination also lengthens the trajectory from 1.33 to 1.54 searches, while the evidence gain holds at matched length.Source: I HOW THE ACTING MODEL DETERMINES THE… · p. 26
• TAEC leads every Table 1 cell, with its largest margins over single-pass retrieval on ViDoSeek and MMLongBench-Doc and its smallest on mostly single-slide SlideVQA.The spread across backbones for ViDoRAG is of the same order as for TAEC.Source: I HOW THE ACTING MODEL DETERMINES THE… · p. 27

J OPEN-WEIGHT BACKBONES

• The open-weight evidence indicates that TAEC requires an acting model able to run its own trajectory, a property distinct from reading a page well.The main-text results use proprietary API backbones.Source: J OPEN-WEIGHT BACKBONES · p. 27

J.1 RESULTS ON OPEN-WEIGHT BACKBONES

• On Qwen2.5-VL-7B, TAEC adds 2.4 points on ViDoSeek and 0.7 on the RL-tuned checkpoint, but single-pass retrieval is strongest on both benchmarks.As ReAct, the model answers 24.2% of ViDoSeek questions without retrieval; under TAEC, it continues searching on 58.1% of questions, versus 6.2% for Gemini-3.5-Flash.Source: §J.1 · pp. 27–28
• TAEC is strongest on ViDoSeek and MMLongBench-Doc with Qwen3-VL-4B, while ViDoRAG leads on SlideVQA and TAEC is one question behind the substrate.Under TAEC, Qwen3-VL-4B answers after one search on 97.4% of ViDoSeek, 84.8% of SlideVQA, and 62.2% of MMLongBench-Doc questions.Source: §J.1 · p. 28
• Table 16 groups 989 ViDoSeek questions by TAEC search count; evidence was present after the first search in every group, while accuracy falls with each additional search.Single-pass accuracy is almost flat across groups, indicating comparable difficulty.Source: §J.1 · p. 28, Table 16 · p. 28

J.2 TRAJECTORY DEPTH AND COMPONENT FOOTPRINT ON AN OPEN-WEIGHT BACKBONE

• Across benchmarks, components add 0.4 points on the open-weight backbone, versus 2.7 to 6.6 on the closed-source backbones.The authors attribute the smaller gain to a model that answers a quarter of questions without searching being unable to use a better-managed trajectory.Source: §J.2 · p. 28

J.3 COMPARISON WITH RL-TRAINED SYSTEMS

• RL-trained systems improve the search policy, whereas TAEC changes the evidence shown, memory exposure, and pixel allocation while leaving the policy fixed.Against the strongest training-free system, the reported RL-system results range from 2.6 points behind to 5.5 ahead; composition on the released checkpoint adds 0.7 points.Source: §J.3 · p. 28
• The RL and training-free table blocks are not comparable cell by cell because they use different judges, retrievers, and page collections.The paper therefore makes no claim that coordination is worth less than a trained search policy.Source: §J.3 · p. 29, Table 17 · p. 29

K RETRIEVER CONTROL

• TAEC leads over the stronger baseline by 7.4 points on the dense retriever, 0.5 on BM25, and 4.4 on the hybrid retriever.The authors report that the lead tracks how much selection work the retriever leaves available.Source: K RETRIEVER CONTROL · pp. 29–30, Table 18 · p. 29

L SEARCH DEPTH

• The depth table reports accuracy attained using at most k searches, showing when each recorded system realizes its final accuracy rather than behavior under an imposed limit.At small k, results combine reading accuracy and how early the system stops; the controlled depth comparison is reported elsewhere.Source: L SEARCH DEPTH · p. 30
• TAEC and the substrate are saturated by eight searches, while ReAct continues gaining through nineteen and remains 2.5 points short at ten.No TAEC trajectory exceeds eight searches, and no substrate trajectory exceeds seven.Source: Table 19 · p. 30

M WHY POST-HIT ACCURACY CANNOT BE COMPARED ACROSS SYSTEMS

• Post-hit accuracy cannot compare systems because each denominator contains a different set of gold-page questions; ViDoSeek scores were 90.1 and 90.6 despite a 7.4 accuracy gap.The passage characterizes the less-than-one-point post-hit gap as almost uninformative about the mechanism.Source: M WHY POST-HIT ACCURACY CANNOT BE… · p. 30
• Among 118, 131, and 62 retrieval-disagreement questions, TAEC alone retrieved 115, 110, and 47; without gold pages, substrate/TAEC accuracy was 28/28%, 23/28%, and 9/10%.The reverse-only groups were small, including three questions on ViDoSeek; retrieval misses did not always mean an incorrect answer.Source: M WHY POST-HIT ACCURACY CANNOT BE… · p. 30
• Figure 9 compares four disjoint retrieval-outcome groups across three benchmarks; their size-weighted accuracy differences sum exactly to the overall accuracy gap.The decomposition is an identity, not an approximation.Source: M WHY POST-HIT ACCURACY CANNOT BE… · p. 30, Figure 9 · p. 31

N PROTOCOL DIFFERENCES BEHIND THE BASELINE GAP

• Protocol differences leave within-table comparisons like-for-like, but make this paper's values not directly comparable with published values.Table 20 summarizes the protocol differences; rows with different retriever fingerprints are never placed in one table.Source: N PROTOCOL DIFFERENCES BEHIND THE… · p. 31, Table 20 · p. 31
• A search URL alone does not identify the retriever; without a behavioral fingerprint, runs a week apart can differ by more than any method effect reported here.The authors identify the retriever finding as the most consequential protocol difference.Source: N PROTOCOL DIFFERENCES BEHIND THE… · p. 31