ReTAC: Residual Test-Time Agentic Computation for Video Spatial Reasoning

1 INTRODUCTION

• Residual computation targets requirements unresolved by native inference while retaining compatible native evidence, rather than repeating already resolved work.The motivation is that additional processing can reacquire supported evidence, introduce noisy observations, and perturb an otherwise correct prediction.Source: §1 · pp. 1–2
• ReTAC builds a structured native state, plans targeted perception for unresolved requirements, combines new and native evidence, and verifies planning, perception, and reasoning.After a localized failure, it replans, reperceives, or rereasons while reusing unaffected intermediate states.Source: §1 · p. 2
• ReTAC achieves the highest overall score among compared methods for all three MLLMs on both video spatial reasoning benchmarks.The benchmarks are VSI-Bench-Debiased and STI-Bench; the models are Qwen3-VL-8B, Gemini 3.1 Pro, and GPT-5.5.Source: §1 · pp. 2–3
• Forcing refinement on direct-return instances or bypassing it on refinement instances lowers performance, while selective recovery uses fewer mean calls than full restart across all three models.Selective recovery yields higher overall scores for Qwen and GPT and a slight 0.08-point decrease for Gemini.Source: §1 · pp. 2–3

Video Spatial Reasoning with MLLMs

• Spatial-reasoning methods enrich evidence with scene representations or additional gathering, while ReTAC invokes supplementary processing only when native evidence is insufficient.Examples include bird’s-eye views and object markers in GPT4Scene [Qi et al., 2026], structured scene text in TRACE [Hua et al., 2026], and query-conditioned representations in AlloSpatial [Ruan et al., 2026].Source: §2 · p. 3

Adaptive Test-Time Agentic Computation

• Adaptive agents use task context, intermediate outcomes, disagreement, or prior tool-use experience for allocation, whereas ReTAC uses the current model–instance residual state.AdaTooler-V learns selective invocation [Wang et al., 2026], and AdaReasoner selects tools across multiple steps [Song et al., 2026].Source: §2 · p. 3

Multimodal Reflection and Selective Recovery

• Reflection methods revisit visual evidence or reasoning, while ReTAC localizes failures to planning, perception, or reasoning and limits recovery to invalidated dependencies.Dependency tracking preserves unaffected states during recomputation.Source: §2 · p. 3

3.1 NATIVE STATE AND RESIDUAL ROUTING

• The native state records task requirements, source-linked evidence, unresolved requirements, and a candidate answer; ReTAC routes unresolved requirements to refinement while retaining supported observations.The direct branch returns the candidate when requirements are supported; refinement comprises residual planning, perception, reasoning, and verification.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 RESIDUAL PLANNING AND EVIDENCE ACQUISITION

• Residual planning orders supplementary steps only for unresolved requirements or their prerequisites, and targeted perception executes those steps in dependency order.Observation records retain source, object identity, temporal reference, coordinate frame, scale, and dependencies; supplementary observations are merged with native evidence for reasoning.Source: §3.2 · pp. 4–5

3.3 STAGE-ALIGNED VERIFICATION

• Stage-aligned verification checks planning, perception, and reasoning in order, stopping at the first non-passing stage and returning a localized diagnosis and revision target.Planning checks completeness and operation feasibility; perception checks source validity and requirement coverage; reasoning checks evidence support, inference validity, and answer consistency.Source: §3.3 · pp. 5–6

3.4 SELECTIVE RECOVERY

• Dependency-based recovery recomputes changed or invalidated states and their transitive dependents while reusing compatible evidence, unaffected plan steps, and still-valid checks.ReTAC regenerates the reasoning trace and answer together; VERIFIED means acceptance under the specified criteria, not ground-truth correctness.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• Evaluation covers Qwen3-VL-8B, Gemini 3.1 Pro, and GPT-5.5 on both benchmarks, with a default recovery budget of T = 2.Main experiments, component ablations, and recovery cost and budget analyses report means over three runs.Source: §4.1 · p. 7

4.2 MAIN RESULTS

• ReTAC achieves the highest score among compared methods in all six model–benchmark settings.The evaluation uses VSI-Bench-Debiased and STI-Bench.Source: §4.2 · p. 7
• On VSI-Bench-Debiased, relative-direction gains span all models; on STI-Bench, displacement and path length improve for all, while trajectory-description gains peak for Qwen and Gemini.Absolute distance and room size are less consistent across models, while several STI-Bench categories regress slightly.Source: §4.2 · p. 7

4.3 RESIDUAL COMPUTATION ALLOCATION

• Under a limited budget, native-state conditioning outperforms a general adaptive planner that can reacquire evidence for the full task.Removing native support labels and evidence lowers scores in 15 of 16 task-level comparisons.Source: §4.3 · p. 7
• Forcing acquisition on direct-return instances or suppressing refinement on refinement instances lowers subset performance for every model.The larger penalty differs by model: Qwen and Gemini lose more when refinement is suppressed, while GPT loses more when acquisition is forced.Source: Figure 3 · p. 8, §4.3 · p. 8

4.4 RESIDUAL RECOVERY

• Replacing localized verification and selective recovery with general reflection and full restart lowers task scores; among recovered cases, about two-thirds go wrong-to-correct and under 3% go correct-to-wrong.Recovery is invoked for roughly 30% of each model’s refinement subset.Source: Figure 4 · p. 8, §4.4 · p. 8

4.5 RECOVERY COST AND BUDGET

• At T = 2, selective recovery reduces calls versus full restart for all models; raising T to 3 adds calls but yields only a small Qwen gain and lower Gemini and GPT scores.Qwen and GPT improve in both cost and score versus full restart, while Gemini shows a small cost–accuracy trade-off.Source: Figure 5 · p. 8, §4.5 · pp. 8–9

4.6 QUALITATIVE ANALYSIS

• Examples show model-specific routing for one query and reasoning-only REREASON recovery that reuses the verified plan and evidence.The recovery corrects an appearance-order error without new perception.Source: Figure 6 · p. 9, §4.6 · p. 9

• Across three MLLMs and two benchmarks, ReTAC achieves the highest overall score among the compared methods.Source: §5 · p. 9
• ReTAC targets supplementary computation to requirements unresolved by native inference and limits recovery to invalidated states and their dependencies.Source: §5 · p. 9
• Controlled interventions lower performance when the selected route is forced in the opposite direction, while selective recovery reduces calls versus full restart with comparable overall performance.Source: §5 · p. 10

6 FUTURE WORK

• An incorrect native estimate can mark incomplete evidence as supported and cause ReTAC to terminate early.The proposed response is calibrated support estimates and uncertainty-aware routing criteria.Source: §6 · p. 10
• VGGT-based geometry lacks metric scale, coinciding with less uniform gains on metric tasks.Future work will extend the operation interface with more robust spatial operations.Source: §6 · p. 10
• ReTAC’s practical scope is bounded by both state estimates and supplementary operations.Source: §6 · p. 10

A.1 STATE REPRESENTATION AND EVIDENCE INTEGRATION

• The native state labels requirements as supported or unresolved; residual plans target unresolved requirements or their prerequisites, and evidence records retain source and dependency information.The Reasoner must ground critical claims in active evidence or preceding supported claims; plans specify acquisition targets but are not evidence.Source: §A.1 · p. 14

A.2 STAGE-ALIGNED VERIFICATION DETAILS

• Verification checks planning, perception, then reasoning; a stage passes only when every criterion passes, and UNKNOWN is non-passing.Verification stops at the first non-passing stage, leaving later stages unevaluated.Source: §A.2 · p. 15
• Planning verification checks plan completeness and operation feasibility, including whether documented operation outputs can support target requirements.A feasibility pass does not establish that the acquired observation is correct.Source: §A.2 · p. 15
• Perception verification checks source validity and whether valid active evidence resolves every requirement; reasoning verification checks the derivation from active evidence to answer.Source: §A.2 · p. 15

A.3 SELECTIVE RECOVERY AND CACHE INVALIDATION

• Recovery routes to ACCEPT only when all stages pass; concrete diagnoses select an operator, while implicated upstream criteria are rechecked before recovery.REPLAN revises acquisition definitions, REPERCEIVE repeats an unchanged acquisition, and REREASON regenerates the derivation and answer while preserving plan and evidence.Source: §A.3 · p. 16
• Recovery invalidates changed items and their dependency descendants, reuses only still-valid criteria, and never reuses checks on the previous reasoning or answer.The budget counts recovery actions; a final passing state is VERIFIED, while an unsupported revision or non-passing final state is returned as UNVERIFIED.Source: §A.3 · p. 16

A.4 COMPLETE INFERENCE PROCEDURE

• Algorithm 1 combines native routing, residual refinement, ordered verification, and bounded recovery, reusing only valid cache entries.Recovery iterations also condition the Reasoner on accumulated history and current feedback.Source: §A.4 · p. 16

B.1 SHARED EVALUATION PROTOCOL

• Qwen3-VL-8B, Gemini 3.1 Pro, and GPT-5.5 are evaluated on VSI-Bench-Debiased and STI-Bench using 32 uniformly sampled frames per video.The native call receives the sampled video, question, and optional answer choices, but no tool descriptions or outputs.Source: §B.1 · p. 16
• Unless stated otherwise, the recovery budget is T = 2 and counts recovery actions rather than verification rounds.The analyses of routing, component replacement, answer transitions, and recovery budgets use VSI-Bench-Debiased.Source: §B.1 · p. 16
• Model-call cost averages invocations over all 2,362 VSI-Bench-Debiased questions, charging failed calls and retries while excluding deterministic spatial-tool executions.Direct-return instances still contribute their native call.Source: §B.1 · p. 17
• Main tables report means over three runs, rounding each run-level percentage before averaging and rounding the resulting mean.Parenthesized differences use displayed means.Source: §B.1 · p. 17

B.2 BENCHMARKS AND METRICS

• VSI-Bench-Debiased has 2,362 indoor-video questions across eight tasks, while STI-Bench has 2,064 five-choice questions across three static and five dynamic tasks.VSI overall score equally averages task scores; STI overall accuracy weights by sample count.Source: §B.2 · pp. 17–18
• VSI numerical tasks use MRA and multiple-choice tasks use accuracy; STI reports task and overall accuracy.VSI MRA averages satisfaction across ten relative-error thresholds, and relative-direction accuracy first macro-averages difficulty subsets.Source: §B.2 · pp. 17–18

C SPATIAL TOOLS AND OPERATION INTERFACE

• The operation interface uses sampled-video references or prior-step outputs, preserves traceable records, and requires a common coordinate frame for combined 3D quantities.Without calibration, reconstructed geometry has unknown metric scale and gravity.Source: C SPATIAL TOOLS AND OPERATION INTERFACE · p. 18

Object Detection (object detection)

• Frame inspection retrieves selected images; GroundingDINO detects requested objects, but misses do not prove absence and detections alone do not establish cross-frame identity.Geometry reconstruction estimates dense 3D points and camera parameters, while localization lifts boxes into 3D and may retain background points.Source: Frame Inspection (inspect frames) · p. 18, Object Detection (object detection) · p. 18, Geometry Reconstruction (reconstruct… · p. 19

Euclidean Distance (compute distance)

• Distance operations compute Euclidean or closest-point separation in input coordinates, without metric conversion for uncalibrated geometry.Planar projection is a horizontal bird’s-eye plane only when the supplied up direction is supported as gravity.Source: Euclidean Distance (compute distance) · p. 19, Closest-Point Distance (compute… · p. 19, Planar Projection (project to bev) · p. 20

Observer Pose (observer pose)

• Observer-pose and projection operations construct observer-centered coordinates from supplied references and basis vectors; direction classification uses the supported forward axis.Spatial clustering returns transitive connected components, not direct object counts; cross-frame association is a geometric cue, not an identity guarantee.Source: Observer Pose (observer pose) · p. 20, Observer-Frame Projection (project to… · p. 20, Relative-Direction Classification… · p. 20

C.5 COMPOSITE OPERATIONS AND STEP ACCOUNTING

• Composite operations are planned from registered primitives, execute internal steps in dependency order, and count as one plan step regardless of the number of primitives.The Planner is encouraged to use at most five steps, but no fixed upper bound is imposed.Source: §C.5 · p. 21

D.1 IMPLICATIONS FOR RESIDUAL TEST-TIME COMPUTATION

• On fixed routing subsets, forcing acquisition for direct-return instances or suppressing refinement for refinement instances reduced performance; selective recovery used fewer calls than full restart, while larger budgets did not yield monotonic gains.The analysis supports acquiring missing evidence and recomputing only states invalidated by a localized failure.Source: §D.1 · p. 21

D.2 SCOPE AND EXTENSIONS

• ReTAC’s scope depends on state-estimate and operation quality; incorrect support labels can end refinement early, shared-model roles can preserve correlated errors, and VGGT geometry lacks metric scale.The study covers three MLLMs and two benchmarks, uses mean model calls as its compute proxy, and uses the original STI-Bench release, whose reported annotation issues bound interpretation.Source: §D.2 · p. 22

E.2 NATIVE STATE ELICITATION AND RESIDUAL ROUTING

• Fixed role prompts share conventions for structured output and evidence bookkeeping, while the native state records answer requirements, observations, support status, and unresolved requirements.The native prompt asks for a candidate answer and source-linked observations, and labels a requirement supported only when its observations resolve it.Source: E AGENT AND ABLATION PROMPTS · p. 22, §E.1 · p. 22, §E.2 · p. 22

E.3 RESIDUAL PLANNING, PERCEPTION, AND REASONING

• On the refinement branch, the planner turns unresolved requirements into acquisition steps, the perceiver registers source-linked evidence, and the reasoner derives an answer from active evidence.The planner preserves supported native state, while the perceiver targets planned acquisitions rather than producing the final task answer.Source: §E.3 · p. 23, Prompt: PLANNER · p. 23, Prompt: PERCEIVER · p. 24

E.4 STAGE-ALIGNED VERIFICATION

• Stage-aligned verification checks planning, perception, and reasoning separately, with targeted actions such as replanning, reperceiving, or rereasoning for supported failures.The shared instructions distinguish definite failure from insufficient support and specify recovery targets.Source: §E.4 · p. 24, Prompt: CHECK COMMON · p. 24, Prompt: PERCEPTION CHECK · p. 25

E.5 COMPONENT-REPLACEMENT ABLATION PROMPTS

• The general-planning replacement chooses whether to supplement without native support labels or evidence, whereas the general-reflection replacement makes one accept-or-retry decision over the full trajectory.The latter does not localize failure to a specific stage.Source: §E.5 · p. 25, Prompt: GENERAL PLANNING · p. 26

F WORKED EXAMPLE: PERCEPTION ERROR AND SELECTIVE RECOVERY

• The worked trace follows a perception error that triggers REPERCEIVE, contrasting with a separate reasoning-error case that uses REREASON.Its query asks which of a table, lamp, or stool appears first in a sequence of 32 sampled frames.Source: F WORKED EXAMPLE: PERCEPTION ERROR AND… · p. 26, §F.1 · p. 27

F.2 NATIVE STATE AND INITIAL RESIDUAL REFINEMENT

• The example’s native state resolves the table and lamp appearances but leaves the stool appearance unresolved.The first appearances in the trace are sampled frames 8, 24, and 4 for the table, lamp, and stool, respectively.Source: §F.2 · p. 27, §F.1 · p. 27

1. Native state and routing

• The native state leaves only the stool’s first appearance unresolved, so the residual plan reacquires only that requirement.The native controller enters refinement with no established candidate answer; the plan adds no requirements and retains the native evidence.Source: §1 · pp. 27–28, §2 · p. 28

3. Supplementary perception and evidence merge

• Supplementary perception reports the stool at frame 10 despite a cited raw earliest-detection value of 4.The raw record remains in history, and empty unresolved or conflict lists do not validate the observation; source validity and requirement coverage are checked separately.Source: §3 · pp. 28–29

4. Initial reasoning

• Initial reasoning provisionally answers “A: table” using the active comparisons, although the stool premise is source-inconsistent.The reasoning trace marks no unresolved requirements or conflicts, but the candidate remains provisional because its stool evidence conflicts with its cited source.Source: §4 · pp. 29–30

F.3 STAGE-ALIGNED VERIFICATION AND ERROR LOCALIZATION

• Verification proceeds from planning to perception to reasoning, and the declared plan passes feasibility checks without validating the stool observation.The planning verifier uses the query, objects, plan, and tool definition rather than video or reasoning contents; source support is checked later.Source: §F.3 · p. 30, 5a. Planning verification · p. 30

5b. Perception failure and recovery routing

• Perception verification fails because the stool claim reports 10 while the raw field and sampled frame support an occurrence at 4.Reasoning verification is not run after the perception failure; ReTAC repeats the same acquisition, replaces the invalid observation, and regenerates dependent reasoning without rechecking planning.Source: 5b. Perception failure and recovery… · p. 31

F.4 SELECTIVE RECOVERY, REVERIFICATION, AND FINAL RETURN

• Selective recovery replaces the invalid stool observation and recomputes its dependents while reusing the plan and unaffected native evidence.Because the same acquisition can repair the error, the plan and checks on table and lamp evidence remain reusable; checks depending on the invalid stool observation are invalidated.Source: §F.4 · p. 31, 5b. Perception failure and recovery… · p. 31

6. Reperception and dependency update

• Reperception corrects the stool’s first-appearance claim to frame 4, while the raw output is unchanged and the native observations remain active.Reasoning regenerated from the corrected evidence concludes that the stool appears before the table and lamp, yielding candidate answer “C: stool.”Source: §6 · pp. 32–33

7. Reverification and return

• Reverification reuses valid planning and native-evidence checks, then checks the replacement stool observation and coverage before reasoning.The provided trace describes the replacement observation as matching the raw earliest-detection field and sampled frames; it does not include the subsequent reasoning-verification result.Source: §7 · p. 33
• The corrected state passes verification and is accepted after one recovery action, with answer C marked VERIFIED.An unused recovery budget does not prevent acceptance at t = T, because T limits recovery actions rather than final verification.Source: §7 · pp. 33–34
• Recovery is localized to the erroneous stool observation, requirement coverage, and dependent reasoning; the plan and unrelated native evidence remain reusable.Source: §7 · p. 34