PaperlayerView exact revision on arXiv

Source-linked AI summary

EKT-Router: Expert-Evidence Alignment for Embodied Vision-Language Models

Weilin Ruan, Yingjie Xu, Yangkai Wei, Bohan Zhou, Shuaibo Li, Harold Haodong Chen, Kaiwen Zhou, Yinchuan Li, Ying-Cong Chen, James Cheng

arXiv:2610.33044v1cs.CV

TL;DR

Embodied vision–language models must reason from partial observations, but expert selection and evidence acquisition are often treated separately. EKT-Router jointly chooses an expert and bounded evidence, then verifies returned packets before one answer. Across six embodied QA benchmarks, it achieves the highest compared mean score, 61.4.

  • Problem

    Embodied vision–language models must answer from partial observations, yet selecting a suitable expert and its evidence jointly remains a challenge.

    Source: §1 · p. 1

  • Method

    EKT-Router learns outcome-derived expert signatures and complete-route utility, then verifies acquired packets and keeps evidence only when its predicted expert-conditioned gain is positive.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    61.4 mean score across six embodied QA benchmarks, the highest among compared systems.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    With external evidence disabled, question-conditioned expert selection reaches 60.24 versus 58.49 for the best fixed expert, a +1.75 gain.

    Source: §4.3 · p. 7

    The experiments do not establish the stability of online adaptation to updated signatures or utilities.

    Source: D Limitations and Future Work · p. 15

Abstract

from arXiv · show

Embodied vision-language models must answer questions from partial, evolving observations. Their success depends on both the reasoning capability selected and the evidence available to it. Expert routers adapt internal computation, while tool-augmented systems acquire external evidence; choosing these independently can leave a suitable expert without the evidence it needs. We introduce EKT-Router, a bounded framework that jointly selects a capability expert, optional knowledge, and perceptual tools. Outcome-derived signatures represent the overlapping strengths of General, Spatial Reasoning, and Spatiotemporal experts. A joint utility scores complete expert-evidence routes under a call budget. After acquisition, a packet verifier admits valid evidence with positive predicted expert-conditioned gain. The selected expert then answers once, reverting to the unchanged observation if all packets are rejected. Across 6 embodied QA benchmarks, EKT-Router achieves a mean score of 61.4, the highest among the compared systems, and averages 74.7 on 10 general vision-language benchmarks. With all external evidence disabled, question-conditioned expert selection alone improves over the best fixed expert by 1.75 points. Ablations show that learned routing is the dominant component and that knowledge and perception are complementary.

1 Introduction

Embodied vision–language models must answer from partial observations, and their success depends on matching both reasoning capability and evidence to each query. EKT-Router jointly selects a bounded expert–evidence route, then verifies acquired packets before the selected expert answers.

1 Introduction

  • Partial, evolving observations require semantic knowledge, geometric reasoning, or temporal tracking, sometimes from cues spanning only a few pixels or frames.A relative-position question may benefit from localization, whereas an event-order question requires temporal context.Source: §1 · p. 1
  • Expert routers select parameter paths, while tool-augmented systems acquire external evidence; EKT-Router makes their dependence part of one routing decision.The route combines one expert with optional knowledge and a budget-feasible tool subset before acquisition, verification, and one answer.Source: §1 · p. 1, Figure 1 · p. 2
  • Expert and evidence choices are coupled: the same external information can help different experts differently, while returned packets may be incomplete or ambiguous.EKT-Router selects a complete route before acquisition, then revises only the evidence decision after inspecting packets.Source: §1 · p. 1
  • EKT-Router forms General, Spatial Reasoning, and Spatiotemporal experts through independent capability post-training, then represents their overlapping strengths with outcome-derived signatures.A complete-route utility guides the router, and a bounded verifier retains valid evidence with predicted expert-conditioned gain.Source: §1 · pp. 1–2

2 Related Work

Prior work develops embodied generalists, spatially specialized models, routed parameter paths, and tool-augmented reasoning. EKT-Router differs by scoring a complete route that pairs a capability path with the evidence it will consume.

Embodied question answering

  • Embodied QA benchmarks cover episodic memory, spatial representation, and interactive or long-horizon reasoning; existing models remain limited in reliable spatial and cross-frame evidence from partial observations.The diversity of image, multi-frame, and video questions motivates measuring where specialists are complementary.Source: §2 · p. 2

Routed parameter-efficient specialization

  • Routed specialization methods select models, parameter paths, or internal strategies, whereas EKT-Router scores a capability path together with its non-parametric evidence.The cited approaches include parameter-efficient routing, preference-based model selection, budgeted model-and-strategy selection, and spatial adapter routing.Source: §2 · p. 2

Tool-augmented multimodal reasoning

  • Tool-augmented systems recover missing information through programs, retrieval, or vision APIs; EKT-Router evaluates acquired evidence for the selected expert and separates acquisition from acceptance.Its utility is evaluated for the selected expert, beyond simply selecting tools.Source: §2 · p. 3

3 Method

EKT-Router jointly selects an expert and evidence under a bounded route, then checks returned packets while keeping the expert fixed. Its method derives expert signatures and route utility from outcomes, supervises routing with answer comparisons, and retains evidence only when verified packets beat the null reference.

3 Method

  • The two-stage procedure first selects an expert and typed evidence actions, then revises only the evidence decision after packets return; the expert answers once.The packet verifier scores valid subsets against the null reference, while Figure 2 links online routing to offline supervision.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Joint Expert–Evidence Formulation

  • The route pairs one reasoning expert with optional knowledge and a tool subset under a cost budget; the null action preserves the original-input path.Returned packets separate provider-specific content from attributes used to assess coverage, confidence, ambiguity, provenance, and cost.Source: §3.1 · p. 3

3.2 Outcome-Derived Expert Signatures

  • Outcome-derived capability signatures encode overlapping expert strengths from evidence-free diagnostic probes, with probe-wise normalization making different outcome scales comparable.The expert names remain descriptive labels for the current paths; extending the portfolio requires new diagnostics and fitting affected routing components.Source: §3.2 · p. 3

3.3 Interaction-Factorized Route Utility

  • Route utility combines evidence-free competence, an action value shared across experts, expert–evidence compatibility, and acquisition cost.The centering constraint represents broadly shared action value in the shared term and expert-specific deviations in the interaction term; this is a modeling constraint, not a claim of separately identified causal effects.Source: §3.3 · p. 4

3.4 Route Supervision from Answers

  • Paired route outcomes supervise route ranking and acquisition-gain calibration, while packet outcomes train a residual correction for valid non-null evidence subsets.The preference loss compares routes with distinguishable outcomes; a lower-weight Huber term calibrates predicted gain, and the verifier’s residual predicts realized-versus-predicted utility discrepancy.Source: §3.4 · pp. 4–5

3.5 Bounded and Verified Inference

  • Inference proposes one route, runs each selected provider at most once, and retains a valid packet subset only if its predicted margin over null exceeds γ.If no subset clears the margin, the selected expert answers from the original observation; the procedure uses one verification pass and one answer generation.Source: §3.5 · p. 5

4 Experiments

The experiments compare EKT-Router on six embodied QA benchmarks and a separate general vision–language suite, then test expert complementarity, evidence use, and routing mechanisms. The reported results favor expert–evidence alignment, while also showing selective losses and bounded sensitivity claims.

4.2 Q1: Embodied QA Results

  • EKT-Router, activating one 8B expert per query, achieves the highest six-benchmark average (61.4) among compared systems and ranks first on all six benchmarks.Its largest leads are ERQA (46.5 vs. 42.8) and VSI-Bench (64.1 vs. 61.2).Source: §4.2 · p. 6
  • Against its General path, EKT-Router gains +1.78 on average, but loses on UniEQA (−3.37) and OpenEQA (−2.43), where General remains stronger.The reported losses occur on tasks dominated by open-vocabulary semantics and episodic recall.Source: §4.2 · p. 6

4.3 Q2: Expert Complementarity

  • General leads UniEQA and OpenEQA, Spatial Reasoning leads RefSpatial and VABench-Point, and Spatiotemporal leads RoboVQA and VSI-Bench; reversals persist from 2K to 8K steps.The Pointing/Grounding candidate overlaps with Spatial Reasoning, so localization becomes a Tool action available to any expert.Source: §4.3 · pp. 6–7, Figure 4 · p. 7
  • 60.24 versus 58.49 for the best fixed expert gives question-conditioned expert selection a +1.75 gain with external evidence disabled.The media-grouped 95% CI is [1.25, 2.26].Source: §4.3 · p. 7

4.4 Q3: Mechanism Attribution and Sensitivity

  • Removing the router drops the separate frozen-protocol score by −6.1 to 54.1; random routing drops it by −7.8 to 52.4.Heuristic capability labels score 58.3 (−1.9), while joint end-to-end training scores 57.1 (−3.1) and joint expert updates 56.2 (−4.0).Source: §4.4 · pp. 7–8
  • Restoring Knowledge and perception to the evidence-free variant recovers 4.4 points, 0.7 more than the sum of their separate recoveries.Knowledge supplies procedures, perception supplies observation-bound measurements, and a verifier can reject harmful returned packets before answering.Source: §4.4 · pp. 7–8
  • In the diagnostic sweep, scores rise from 55.3 at LoRA rank 4 to 60.2 at rank 16; three experts improve over two by 2.3 points, while a fourth adds 0.2.Scores stay within 59.8–60.5 across answer judges; halving route labels costs 1.1 points to 59.1, but these settings do not establish robustness to arbitrary provider failures or distribution shifts.Source: §4.4 · p. 8, Figure 5 · p. 8

4.5 Q4: General Vision–Language Performance

  • 74.7 is EKT-Router’s average on the general vision–language suite, on par with the strongest open general VLM; it scores best on MMStar (63.2) and MMMU (57.8).Table 1 marks EKT-Router in bold and column maxima, including ties, with underlining.Source: Table 1 · p. 7, §4.5 · p. 8

5 Conclusion

EKT-Router jointly aligns an expert with suitable evidence, and the reported comparisons favor learned routing while showing complementary contributions from knowledge and perception. The conclusion also bounds these findings to the expert portfolio, evidence interfaces, and evaluation protocol.

5 Conclusion

  • EKT-Router achieves the best embodied QA average among compared systems while matching strong general VLMs.The conclusion also reports query-conditioned expert selection gains over the best fixed expert and complementary contributions from knowledge and perception.Source: §5 · p. 9
  • Matched controls show query-conditioned expert selection pays off alone, while ablations show learned routing dominates and knowledge and perception are complementary.Together, these comparisons support treating which expert should see which evidence as an explicit decision.Source: §5 · p. 9
  • The conclusion bounds its findings to the expert portfolio, evidence interfaces, and evaluation protocol, while describing bounded execution and original-input fallback as structural properties.Appendix D outlines future directions.Source: §5 · p. 9

AI use statement

  • Generative AI assisted with literature discovery, language polishing, and LaTeX formatting, but did not conduct the experiments or generate the measurements.The authors take responsibility for the final text, claims, code, and results, including verification of primary sources, experimental artifacts, equations, citations, and AI-assisted edits.Source: AI use statement · p. 9

A Expert–Evidence Routing Details

The appendix describes what the router and verifier can observe at each decision stage and how evidence packets are validated and retained. Verification keeps the selected expert and original observation fixed.

A Expert–Evidence Routing Details

  • The selected expert and original observation remain fixed throughout evidence verification.The appendix frames its discussion around information available at each decision stage and the relation between acquisition and retention scores.Source: A Expert–Evidence Routing Details · p. 11

A.1 Information Available to the Two Decisions

  • The portfolio has General, Spatial Reasoning, and Spatiotemporal experts whose labels denote principal supervision, not exclusive task assignments.Evidence-free probe outcomes characterize their overlapping strengths; every expert permits a null action, while General can also use Knowledge or Tools.Source: §A.1 · p. 11
  • Before acquisition, the router sees the query, original visual input, expert signatures, and candidate action embeddings, but not provider outputs.After acquisition, the verifier sees returned packet content and attributes and selects a subset without changing the expert or acquiring more evidence; reference answers and answer-judge outputs serve only offline supervision.Source: §A.1 · p. 11

A.2 Packet Semantics and Valid Subsets

  • Packets pair evidence content with coverage, confidence, ambiguity, provenance, and cost attributes, which are inspected before deterministic verbalization.Rejecting a packet removes its added evidence without altering the original observation.Source: §A.2 · p. 11
  • Retention considers only valid subsets of the proposed action, including the null subset, and validity checks schema consistency, target coverage, finite confidence, and provenance.The learned verifier ranks subsets that pass these checks, separating validity from predicted utility.Source: §A.2 · p. 12

A.3 Null Cost, Residual Training, and Retention Scores

The appendix distinguishes costs incurred when evidence is acquired from scores for retained packets, and explains how residual training and null calibration affect the verifier's predicted gain.

A.3 Null Cost, Residual Training, and Retention Scores

  • Acquisition incurs provider cost even when the verifier later retains only the null subset, whereas selecting the null action before acquisition costs zero.The cost adjustment in Equation 13 implements this distinction.Source: §A.3 · p. 12
  • The packet target cancels the proposal-cost terms, so Hψ learns the discrepancy between realized and predicted utility for retained non-null evidence.Inference separately accounts for the original proposal's cost; the verifier objective uses a Huber loss over collected valid non-null subsets, with a fixed zero null correction excluded from residual loss.Source: §A.3 · p. 12
  • The retention score decomposes into an expert baseline, a general evidence term, and an expert-conditioned interaction term.The shared acquisition cost and null anchors yield the predicted score margin identity.Source: §A.3 · p. 12
  • Exact fitting of the verifier's residual yields realized gain only when the evidence-free score is calibrated; shared offsets do not absorb query-dependent miscalibration.The passage identifies held-out calibration of the evidence-free score against evidence-null outcomes, per expert, as part of verifying retention.Source: §A.3 · p. 12

A.4 Finite Search and Original-Input Fallback

The appendix specifies a finite route-and-retention search and an original-input fallback when no evidence subset clears the commitment margin. It also identifies the scope of the configuration table.

A.4 Finite Search and Original-Input Fallback

  • The deployed menu permits at most four provider invocations and sixteen retained subsets per proposal, with configured budgets able to reduce these bounds.Scoring additional subsets invokes no provider and generates no candidate answer.Source: §A.4 · p. 12
  • If no valid subset clears the commitment margin, the selected expert answers from the unchanged original input, even if providers were proposed.The always-valid null subset guarantees a defined fallback when non-null packets fail validation.Source: §A.4 · p. 12
  • The algorithm selects an expert and feasible action by predicted utility, executes non-null providers once, then enumerates and scores valid retained subsets.It retains the highest-scoring subset only if its margin over null exceeds γ; otherwise it selects the null subset and generates one answer.Source: §A.4 · p. 13
  • Table 2 covers model, adaptation, and routing configuration, with spatial and temporal settings separate from Q3's diagnostic LoRA-rank sweep.Source: Table 2 · p. 13

B Training and Evaluation Details

EKT-Router forms and freezes three specialized experts before fitting the router. Outcome-derived diagnostics and paired executions supervise capability profiles, route utility, and packet correction.

B.1 Adaptation and Routing Stages

  • General uses continued pretraining, while Spatial Reasoning and Spatiotemporal use spatial and video supervision, respectively.The completed experts and query descriptor remain frozen during router fitting.Source: §B.1 · p. 13
  • Evidence-free diagnostic outcomes define capability profiles, while paired route executions provide answer-derived utility and packet-correction targets.Table 2 collects the main-text configuration.Source: §B.1 · p. 13
  • Paired evidence-on and null executions use the same prompts and decoding; Equation 9 combines normalized scores with action costs.Preference ranking uses route pairs separated by more than ϵ, while a lower-weight Huber term fits realized acquisition gains.Source: §B.1 · p. 13
  • Held-out routing cases set operating costs and margins, while benchmark test scores do not select these operating points.Media-group separation prevents closely related questions from the same scene appearing across the validation boundary.Source: §B.1 · p. 13

B.2 Comparison Units

The evaluation reports suite means across benchmark scores, not pooled question accuracy, and distinguishes the full system from an evidence-null expert-selection control. Matched decision ablations and media-grouped uncertainty estimates qualify these comparisons.

B.2 Comparison Units

  • 61.40 is the six-task full-system mean in Q1.Q1 equally weights the six primary embodied QA benchmark scores using each benchmark’s native higher-is-better metric.Source: §B.2 · p. 13
  • In the nine-task evidence-null control, question-conditioned selection scores 60.24 versus 58.49 for the best fixed expert, a reported 1.75-point improvement.Q2 disables both external evidence channels for every compared expert.Source: §B.2 · p. 13
  • Table 3 contrasts evidence channels and representative interpretation issues, emphasizing that channels supply different information and require different consistency checks.Source: Table 3 · p. 14
  • The Q2 media-grouped bootstrap interval for the selection gain is [1.25, 2.26], not an estimate of variation across independently retrained portfolios.Decision ablations hold expert checkpoints, instances, decoding, and evaluators fixed.Source: §B.2 · p. 14

B.3 Mechanism-Level Interpretation of Q3

Q3 interprets routing and evidence interventions at distinct decision interfaces, while channel semantics and packet retention clarify what those interventions change. The component removals do not independently identify the interaction term or Packet Verifier.

B.3 Mechanism-Level Interpretation of Q3

  • With both evidence channels, recovery is 4.4 points, exceeding the 3.7-point sum of single-channel recoveries by 0.7 points.The interpretation treats routing and evidence interventions as changes to distinct decision interfaces; knowledge and perception preserve the selected expert while varying available information.Source: §B.3 · p. 14
  • The frozen-expert, outcome-derived setting keeps the answer path fixed while learning route utility, making comparisons with heuristic routing and joint expert updates more interpretable.The authors frame this as an attribution rationale, not proof that the controls isolate all optimization effects.Source: §B.3 · p. 14
  • The reported component removals do not independently identify Γ or the Packet Verifier.A proposed audit would compare additive utility, utility with Γ but without packet correction, and a sequential expert-then-evidence selector under matched conditions.Source: §B.3 · p. 14

C.1 Channel Semantics

  • Knowledge supplies reasoning procedures and semantic context, whereas perception extracts information tied to the observed image or frames.Relative-depth evidence can support ordinal comparisons, but metric distance requires additional calibration; the original visual input remains available whether a packet is retained or rejected.Source: §C.1 · p. 14

C.2 Interpreting Acquisition and Retention

  • An evidence-on/null comparison holds expert, question, observation, and decoding fixed, whereas expert comparisons change the reasoning path.Validity, usefulness, and answer reasoning are separate stages: invalid subsets are excluded, while valid but irrelevant or conflicting packets are assessed by predicted expert-conditioned margin.Source: §C.2 · pp. 14–15

D Limitations and Future Work

The authors delimit EKT-Router by its current portfolio, single-round observation, answer-based supervision, and calibration requirements. They identify transfer, active perception, action outcomes, and stable online updating as areas needing further evaluation or extension.

Portfolio coverage and transfer

  • The three-expert, three-tool portfolio’s outcome-derived signatures depend on diagnostic coverage and score normalization.New adapters require fresh diagnostics and fitting of affected controller components; transfer to new scenes, tasks, adapters, and providers remains to be evaluated.Source: D Limitations and Future Work · p. 15

Single-round observation limits

  • The single-round route cannot reveal occluded objects or recover events absent from observed frames.Sequential acquisition or viewpoint changes are proposed as extensions.Source: D Limitations and Future Work · p. 15

Outcome validity beyond question answering

  • Answer-based utility is not a validated proxy for action success or physical safety, and packet validity does not establish action safety.Extending the formulation to task success, plan feasibility, or navigation requires outcomes and budgets defined for those settings.Source: D Limitations and Future Work · p. 15

Calibration and continual updates

  • Online updates require a reliable reference, task feedback, or an independently validated scorer; provider confidence scales may also shift across models and domains.The experiments do not establish online-adaptation stability, and component ablations do not isolate a separate causal contribution from every verifier and interaction term.Source: D Limitations and Future Work · p. 15

Related papers

Loading 2610.33044v1…