PaperlayerView exact revision on arXiv

Source-linked AI summary

MIND: Instruction-Conditioned Incremental Grounding for Robots

Mohamed Abouagour, Junhyun Nam, Wonse Jo, Byung-Cheol Min

arXiv:2610.19262v1cs.ROcs.AI

TL;DR

Robots following persistent instructions must decide which cached room judgments remain valid as their environment changes. MIND compiles instruction-specific dependencies and combines them with event footprints to maintain judgments, reachability, and selection. In the reported ProcTHOR sequences, it reduces semantic evaluations while preserving all post-event decisions under the stated setup.

  • Problem

    After a world change, it is unclear which cached room judgments must be checked again, even though reevaluating every room discards reusable computation and retaining every judgment can leave a stale target.

    Source: §I · p. 1

  • Method

    MIND compiles separate qualification and selection dependencies for each instruction, intersects semantic dependencies with event footprints, and maintains reachability and selection separately.

    Source: §I · p. 1

  • Results

    30.6 to 11.2 mean semantic evaluations on 240 five-event ProcTHOR sequences, with all 1,200 post-event decisions unchanged.

    Source: A. Ablation: the same targets, fewer… · p. 5

  • Takeaways & Limitations

    Under fixed deterministic evaluation, planning, and selection with conservative dependency and footprint coverage, MIND preserves full-recomputation decisions.

    Source: §I · p. 1, §VII · p. 8

    The controlled benchmarks assume populated semantic maps and a restricted state vocabulary, while hardware evidence covers four scenarios on one layout.

    Source: §VII · p. 8

Abstract

from arXiv · show

Robots following persistent language instructions must decide which cached room judgments remain valid as their environment changes. We present MIND (Maintenance via INstruction-conditioned Dependencies), a framework for maintaining target-room decisions without re-evaluating every room after each event. A language model compiles each instruction into dependencies for semantic qualification and target selection. The controller intersects semantic dependencies with event footprints, refreshes affected judgments, and updates reachability and selection separately. For fixed deterministic evaluation, planning, and selection, conservative dependency and footprint coverage preserves the decisions of full recomputation. On 240 five-event ProcTHOR sequences with an exact benchmark predicate, MIND reduces mean semantic evaluations from 30.6 to 11.2 while preserving all 1,200 post-event decisions. Frozen-evaluator experiments with Phi-4 and Qwen3-14B also preserve all 1,200 decisions, without improving the evaluators' underlying task accuracy. In 32 closed-loop AI2-THOR episodes using exact-predicate qualification and metadata-derived footprints, MIND matches the final target decision of full recomputation in every episode. Unitree G1 experiments demonstrate target reuse and reselection across room transitions without additional semantic evaluations under MIND, and abstention after a barrier and an object removal at 4 evaluations against 15 to 18. Additional materials are available at https://mind-icra27.github.io/.

I. INTRODUCTION

Persistent instructions leave room judgments cached while the world changes, creating a choice between redundant reevaluation and stale targets. MIND uses instruction-specific dependencies to maintain those judgments and preserve full-recomputation decisions under stated coverage assumptions.

I. INTRODUCTION

  • Each room judgment consumes a language-model call, sharing a robot’s GPU with perception or incurring hosted-model billing.The question is which cached judgments must be checked again after a world change.Source: §I · p. 1
  • The same variable can affect qualification or selection depending on the instruction, determining whether motion requires semantic reevaluation.For “nearest chair,” motion can change the selected room without changing which rooms qualify; for “a chair in the room I am in,” motion changes qualification.Source: §I · p. 1
  • MIND compiles separate qualification and selection dependencies, intersects semantic dependencies with event footprints, and maintains reachability and selection separately.This allows target reselection without unnecessary semantic evaluation.Source: §I · pp. 1–2
  • Under fixed-evaluator and conservative-coverage assumptions, MIND preserves full-recomputation decisions; the guarantee does not certify compiler predictions or evaluator accuracy.Over-approximation may add work, while missing coverage can leave stale judgments.Source: §I · pp. 1–2
  • 30.6 to 11.2 mean semantic evaluations across 240 five-event ProcTHOR sequences, with all 1,200 post-event decisions unchanged.The introduction also reports 32 closed-loop AI2-THOR episodes and 26 Unitree G1 runs across four scenarios.Source: §I · p. 2

II. RELATED WORK

Prior work addresses grounding, replanning, memory, and dependency-based reuse, while MIND targets cached regional qualification judgments under persistent instructions. The problem formulation models room-level state changes, reachability, cached semantic verdicts, and target selection as distinct parts of the decision process.

II. RELATED WORK

  • MIND differs from systems that cache model states or plan transitions by maintaining regional semantic verdicts for a persistent target decision.Its instruction-conditioned signatures separate qualification from selection, and regional footprints scope invalidation.Source: §II · p. 2

III. PROBLEM FORMULATION

  • The formulation represents each episode as an instruction, building regions, and events that update inventories, labels, robot region, history, and adjacency.The five state categories are inventory, region label, position, history, and topology.Source: §III · p. 2
  • The planner computes reachable regions, while the selector chooses from feasible regions or returns NONE when none remain.The controller stores room verdicts, refreshes a selected set after events, and reruns selection when feasibility or selection inputs may have changed.Source: §III · pp. 2–3
  • The scope assumes qualification is unary in the candidate region and inventory dependence is local; cross-region inventory predicates and future-state prediction are excluded.Perception is outside the formulation, and the controller is assumed to have current state values available.Source: §III · pp. 2–3

IV. METHODOLOGY

MIND compiles dependencies once per instruction and uses event footprints to maintain cached room judgments, reachability, and target selection.

IV. METHODOLOGY

  • MIND compiles dependencies once per instruction and uses event footprints to maintain cached room judgments, reachability, and target selection.The workflow reuses unaffected judgments rather than reevaluating every room after each event.Source: §IV · p. 3

A. Instruction dependency map

The instruction dependency map estimates which state variables affect room qualification and which affect target selection. It is compiled before execution and can be widened conservatively, but cannot recover dependencies absent from every compiler output.

A. Instruction dependency map

  • True qualification and selection read sets are defined by whether changing a state category can change the semantic verdict or selected region, respectively.The sets may overlap; a selection-only change requires deterministic reselection but no fresh semantic verdict.Source: A. Instruction dependency map · p. 3
  • The compiler emits two variable lists after defining the five categories and explaining the two dependency slots, using development examples for each request family.Greedy decoding and unions over cyclic example orders are used in the frozen downstream configuration.Source: A. Instruction dependency map · p. 3
  • Sample union and stage widening expand dependency signatures monotonically, with widening treating reported variables as possibly belonging to either stage.Widening can add semantic evaluations; it does not recover a variable absent from every compiler output.Source: A. Instruction dependency map · p. 3
  • Promoting a selection dependency into the semantic stage adds reevaluation, whereas demoting or omitting one can leave stale support and void the guarantee.The selector never reads inventory, and adding dependencies or regions preserves deterministic outputs.Source: A. Instruction dependency map · p. 3

B. Event footprint

Observed events are represented by changed state variables and estimated room scopes; several state changes conservatively use all rooms as their scope.

B. Event footprint

  • An event footprint pairs each reported changed variable with an estimated set of rooms to invalidate.Reference footprints use changed rooms for inventory changes: one room for a removal, and source and destination rooms for a relocation.Source: B. Event footprint · p. 3
  • Position, topology, history, and region-label changes use all rooms as the invalidation scope, conservatively allowing nonlocal effects on qualification.This convention does not claim every room physically changes.Source: B. Event footprint · p. 4
  • The correctness guarantee requires detector footprints to cover the reference footprint, although estimates may include extra variables or rooms.The stated convention uses changed rooms for inventory and all rooms for other categories.Source: B. Event footprint · pp. 3–4

C. Scoped update and robot target selection

MIND intersects semantic dependencies with event footprints to choose which room judgments to refresh, while maintaining reachability and target selection separately.

C. Scoped update and robot target selection

  • The controller sets the semantic update set to the intersection-derived invalidation set for the event and instruction.Only the semantic update step invokes the evaluator.Source: C. Scoped update and robot target… · p. 4
  • MIND refreshes planner outputs when topology or robot position changes, updates room judgments only in the selected set, and recomputes reachability when needed.Reachability is recomputed if it changed or if the semantic update set is nonempty.Source: C. Scoped update and robot target… · p. 4
  • Selection reruns when reachability is recomputed or a reported change intersects qualification or selection dependencies, even if no room judgment was refreshed.A selection dependency promoted into the semantic slot can trigger reselection without semantic evaluation.Source: C. Scoped update and robot target… · p. 4

D. Conservative estimates and correctness

Decision preservation depends on conservative dependency and footprint coverage under stated locality and determinism assumptions; extra coverage costs work, while omitted dependencies can invalidate the guarantee.

D. Conservative estimates and correctness

  • Conservative overestimates add work, whereas omitting a true dependency can leave stale support and void the guarantee without necessarily changing every target.Promoting a selection dependency into the semantic stage can cause unnecessary re-evaluation while still triggering reselection.Source: D. Conservative estimates and… · p. 4
  • Under dual conservative containment and the stated assumptions, MIND preserves full-recomputation decisions after every event.The result assumes initialization matches the fixed evaluator, inventory dependencies are local to each candidate region, all task-relevant changes are represented, and evaluation, planning, and selection are deterministic.Source: D. Conservative estimates and… · p. 4
  • The guarantee preserves a fixed evaluator’s decisions, not task correctness; savings vanish with global refreshes, and maintaining all verdicts is not minimum-cost.Semantic cost is initialization plus later regional refreshes.Source: D. Conservative estimates and… · p. 4

V. BENCHMARKS AND EXPERIMENTAL SETUP

The evaluation combines benchmark sequences and request tests with frozen-evaluator and policy comparisons to assess maintenance, compiler dependencies, and task accuracy separately.

V. BENCHMARKS AND EXPERIMENTAL SETUP

  • The primary dependency benchmark uses 240 five-event ProcTHOR sequences across 57 houses, 11 instructions, and six request families.Events include doorway closures, object removal or relocation, robot motion, and referent consumption; changed regional verdicts define invalidation recall and precision.Source: §V · p. 4
  • The request benchmark holds out whole templates, while frozen Phi-4 and Qwen3-14B verdict tapes separate maintenance agreement from repeated inference.The benchmark also uses minimal pairs to test changes to specified stage-specific dependency edges.Source: §V · pp. 4–5
  • An unseen-placement Adjacent benchmark tests 80 sequences in 80 houses, where closing a doorway can change qualification despite physical adjacency.Its semantic dependencies are inventory, position, and topology.Source: §V · p. 5
  • Policies share the evaluator, planner, selector, and composition rule but differ in maintenance schedules; the exact predicate isolates maintenance errors from model errors.Reference signatures serve as oracle controls, and all policies rerun the selector after every event.Source: §V · p. 5
  • Episode accuracy requires all five post-event targets to match the reference, while incremental agreement compares targets against full recomputation under the same evaluator and interface.Semantic-evaluation counts include initialization; with frozen verdicts, they measure logical refresh costs rather than repeated inference calls.Source: §V · p. 5

VI. RESULTS

MIND's ablations reduce semantic evaluations while preserving decisions, and compiler tests show that conservative dependency coverage can trade exact recovery for containment. On an unseen dependency placement, widening repairs demotions while retaining fewer evaluations than full recomputation.

A. Ablation: the same targets, fewer rooms rechecked

  • 30.6 to 11.2 mean semantic evaluations per sequence, as regional footprints, request-specific variables, and stage separation are added; all containing signatures preserve 1,200 decisions.The sequential ablation values are 30.6, 23.4, 14.1, and 11.2 evaluations; these differences are not order-independent contributions.Source: Table I · p. 5, A. Ablation: the same targets, fewer… · p. 5
  • Event-wise relevance reaches 0.658 episode accuracy at 13.1 evaluations, while compile-once signatures reach 1.000 episode accuracy at 11.2 evaluations.Event-wise relevance omits required semantic dependencies in 184 of 1,200 decisions and adds spurious ones in 165.Source: A. Ablation: the same targets, fewer… · pp. 5–6

B. Instruction-conditioned compilation for new requests

  • On 69 held-out requests, Qwen3-14B has containment 1.000 ± 0.000 but exact recovery 0.454 ± 0.097; gpt-oss-120b has the highest exact recovery at 0.935.The compiler evaluation averages over six demonstration orders.Source: Table II · p. 6, B. Instruction-conditioned compilation… · p. 6
  • On 144 minimal pairs, Phi-4 reaches 0.854 edge-correct and 0.799 clean, while both Qwen models are edge-correct on 0 of 36 pairs for placing position and topology in selection.That modifier is the hard case; the Qwen models place it in the semantic slot instead.Source: B. Instruction-conditioned compilation… · p. 6
  • For unseen Adjacent requests, widening repairs demoted signatures to 1.000 containment; repaired signatures reach 1.000 episode accuracy at 18.3 evaluations versus 30.4 for full recomputation.Widening keeps every primary-benchmark decision at 15.3 evaluations.Source: B. Instruction-conditioned compilation… · p. 6

C. Effects of missed dependencies and footprints

In execution and frozen-evaluator tests, inferred maintenance matches full recomputation while using fewer semantic evaluations. Missed dependencies or under-approximated footprints can compromise that fidelity, whereas conservative over-coverage preserves correctness at added evaluation cost.

C. Effects of missed dependencies and footprints

  • Under-approximated footprints degrade as evaluations fall to 8.2, while over-approximation retains 1.000 agreement, accuracy, and recall at up to 18.5 evaluations.When a changed variable is uncertain, invalidating the whole candidate set retains 1.000 accuracy at 27.5 evaluations versus 30.6.Source: C. Effects of missed dependencies and… · p. 6
  • Omitting semantic history changes 10 of 1,200 decisions under the reference-field interface; a K=2 union restores agreement on all decisions at 11.5 evaluations.Under the compiled-field interface, both policies lack history and agree on all decisions, but both score 0.329 episode accuracy.Source: C. Effects of missed dependencies and… · p. 6

D. Closed-loop and observation-derived robot updates

  • With frozen Phi-4 and Qwen3-14B evaluators, inferred maintenance matches all 1,200 post-event decisions at 11.2 evaluations versus 30.6, reproducing each evaluator's full-recomputation accuracy.Agreement measures maintenance fidelity, not semantic-model competence; the evaluator's underlying task accuracy is unchanged.Source: D. Closed-loop and observation-derived… · p. 6
  • In 32 closed-loop AI2-THOR episodes, inferred maintenance matches full recomputation in all final target decisions at 10.47 evaluations versus 26.00; event-wise relevance matches 30 of 32.The test uses exact-predicate qualification and metadata-derived footprints, not perception-driven grounding.Source: D. Closed-loop and observation-derived… · p. 7
  • For observation-derived removal footprints, the fixed operating point contains the affected room in all 54 held-out houses, with a mean footprint of 2.19 of 4.61 rooms.The test uses matched scans and a supplied event type; scan and detector costs are excluded.Source: D. Closed-loop and observation-derived… · p. 7, Figure 3 · p. 7

E. Physical-robot demonstration

The Unitree G1 demonstrations test reuse, reselection, refresh, and abstention across four scenarios; MIND preserves full-recomputation decisions while reducing semantic evaluations in selective-update cases. The conclusion conditions decision preservation on Proposition 1’s assumptions and notes the hardware study’s limited scope.

E. Physical-robot demonstration

  • The hardware study used ROS 2, lidar localization, Nav2, YOLO-World, and a 4-bit onboard judge across four scenarios and 26 scored runs.Eight runs were excluded with reasons; the setup used one four-zone laboratory layout.Source: E. Physical-robot demonstration · p. 7
  • In scenario 3, position affected qualification, so both policies re-judged every room and inherited the same evaluator error across all six runs.The evaluator read the inventory line but not the position line; navigation and perception passed.Source: E. Physical-robot demonstration · p. 7
  • MIND used 3 semantic evaluations versus 9 for full recomputation in scenarios 1 and 2, and 4 versus 15 to 18 before both policies abstained in scenario 4.In scenario 2, the target re-ranked from C to A after the robot moved, without another semantic evaluation.Source: E. Physical-robot demonstration · p. 7, Figure 4 · p. 8

VII. CONCLUSION

  • Under Proposition 1’s assumptions, selective maintenance preserves full-recomputation target decisions while reducing semantic re-evaluation.The conclusion says the evaluations examine spatial scope, request-specific variables, stage placement, and failures from missed dependencies and footprints.Source: §VII · pp. 7–8
  • The hardware evidence covers four scenarios on one layout, while broader instruction coverage and robustness to changing evaluator outputs remain open.Controlled benchmarks also assume populated semantic maps and a restricted state vocabulary, with dependency labels not independently annotated.Source: §VII · p. 8
Loading 2610.19262v1…