PaperlayerView exact revision on arXiv

Source-linked AI summary

CausGround: Causal Interaction Grounding for Adaptive Embodied Planning

Yijun Shen, Diqi He, Jinbang Guo, Wanyue Zhang, Jiaming Mi, Huan Yang, Zhongyuan Wang, Zhiwei Yu, Kai Zhang

arXiv:2610.29702v1cs.RO

TL;DR

Embodied MLLM planners need environment-specific knowledge of when actions are executable and how they change the world, but existing adaptation can be costly or difficult to reuse. CausGround grounds reusable action preconditions and effects in current observations, and reports gains across simulated planning, interaction reasoning, and real-robot manipulation.

  • Problem

    MLLMs show promising planning, but reliable interaction requires environment-dependent action preconditions and effects that existing costly training or unstructured experience often does not capture or reuse effectively.

    Source: §1 · pp. 1–2

  • Method

    CausGround distills interaction experience into reusable action-centered preconditions and effects, organizes state factors for objects and relations, and grounds task-relevant dependencies for backward and forward planning.

    Source: §1 · pp. 2–3

  • Results

    24.9% to 53.3% average task success on VirtualHome, 43.0% to 57.0% closed-loop planning success on RoboCasa, and 25.8% to 31.9% average accuracy on SWITCH accompany real-robot manipulation validation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Explicit causal interaction knowledge provides a reusable grounding substrate for long-horizon embodied reasoning across diverse environments.

    Source: §1 · p. 3

Abstract

from arXiv · show

Multimodal large language models (MLLMs) show promising reasoning capabilities, but reliable physical interaction requires planning that accounts for when actions are executable and how they change the world. These preconditions and effects are environment-dependent and often not captured by MLLMs, making adaptation challenging. Yet existing approaches either require costly environment-specific training or rely on unstructured interaction experience that is difficult to reuse. Therefore, we introduce CausGround, a non-parametric causal interaction grounding framework that distills interaction experience into reusable action-centered causal knowledge. CausGround identifies action preconditions and effects and organizes them in a factorized world-state representation capturing object attributes and inter-object relations. At planning time, task-relevant causal dependencies are grounded in current observations to support backward reasoning over preconditions and forward reasoning over effects for action selection and multi-step planning. We evaluate CausGround on simulated high-level planning, interaction reasoning, and real-world robot manipulation. On adapted VirtualHome and RoboCasa benchmarks, CausGround improves vision-based high-level planning across MLLM backbones, increasing average task success from 24.9% to 53.3% on VirtualHome and closed-loop planning success from 43.0% to 57.0% on RoboCasa. On SWITCH, average interaction reasoning accuracy increases from 25.8% to 31.9%, indicating improved action selection, recovery, and verification. Moreover, real-world experiments show that CausGround reasons backward from MLLM-interpreted goals to derive executable high-level action sequences for sequential manipulation. These results demonstrate that reusable action preconditions and effects support long-horizon embodied planning across diverse environments without parameter updates.

1 INTRODUCTION

CausGround addresses the challenge of adapting embodied planning to environment-dependent action conditions and effects by grounding reusable causal knowledge in current observations. It reports gains across simulated planning, interaction reasoning, and real-robot manipulation.

1 INTRODUCTION

  • Reliable interaction requires knowing when actions are executable and how they change the world; unmet preconditions can trigger invalid actions, repeated attempts, and costly trial-and-error.For example, placing an object in a fridge requires opening it first.Source: §1 · pp. 1–2
  • Existing adaptation either requires costly environment-specific training and repeated optimization or retains experience in unstructured forms that are difficult to reuse and ground.Representations may also fail to jointly capture object attributes and inter-object relations needed to assess action preconditions.Source: §1 · p. 2
  • CausGround distills interaction experience into reusable action-centered preconditions and effects, then grounds task-relevant dependencies in observations for backward and forward planning without parameter updates.A local causal graph structures these dependencies for multi-step planning.Source: §1 · pp. 2–3
  • 24.9% to 53.3% average task success on VirtualHome, 43.0% to 57.0% closed-loop planning success on RoboCasa, and 25.8% to 31.9% average accuracy on SWITCH accompany real-robot validation.The real-world manipulation experiments derive executable high-level action sequences from interpreted goals.Source: §1 · p. 3
  • The factorized world-state representation jointly captures object attributes and inter-object relations, providing a shared basis for grounding interaction knowledge in visual observations.CausGround uses this representation to organize action-centered causal knowledge.Source: §1 · pp. 2–3

2 RELATED WORKS

Related work covers adaptation through model updates or external experience, varied world-state representations, and planners that predict action consequences. CausGround distinguishes itself by explicitly representing reusable action conditions, effects, and cross-action dependencies.

2 RELATED WORKS

  • Environment adaptation has shifted from fine-tuning, imitation learning, or reinforcement learning toward inference-time use of feedback, trajectories, and external memory.These approaches still depend on how experience is accumulated and how the world state is represented.Source: §2 · p. 3
  • World-state representations range from free-form text to object-centric and factorized forms, while embodied interaction also requires explicit relations among objects.The paper argues for jointly representing fine-grained object attributes and inter-object relations.Source: §2 · p. 3
  • World-model planners predict action consequences and evaluate candidate trajectories, but action conditions may remain implicit and multi-step dependencies may require repeated rollouts.CausGround instead exposes action conditions, effects, and cross-action dependencies as reusable environment-specific knowledge.Source: §2 · pp. 3–4, Figure 2 · p. 4

3 METHODOLOGY

CausGround distills interaction experience into reusable action-centered causal knowledge and grounds it in current tasks and world states for planning with a frozen planner. Its shared factorized representation connects visual observations to action preconditions and effects.

3.1 FACTORIZED WORLD-STATE REPRESENTATION

  • A factorized world state represents objects through semantic categories and attribute-value pairs, alongside explicit inter-object relations that can serve as action preconditions or effects.Using the same schema for observations and causal knowledge allows direct matching of stored conditions and effects against the current state.Source: §3.1 · pp. 4–5

3.2 ACTION-CENTERED CAUSAL KNOWLEDGE CONSTRUCTION

  • CausGround builds reusable action-centered causal knowledge by retaining preconditions that distinguish successful executions from failures and state changes that consistently follow successful actions.When one action’s effect satisfies another’s precondition, the framework links them through a causal dependency; it also separates shared action-family regularities from instance-specific knowledge.Source: §3.2 · p. 5

3.3 TASK-GROUNDED PLANNING

  • For each task, CausGround grounds task-relevant causal knowledge by reasoning backward from the goal over action effects and preconditions and retrieving related knowledge from its library.The resulting knowledge is constructed once at task start; at each step the planner combines it with the current observation, structured world state, and task description to select an action.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

CausGround is evaluated for visual high-level planning, interaction reasoning, component contributions, backbone generalization, and real-world manipulation. Across these settings, results show improved planning and reasoning, while qualitative cases illustrate how causal knowledge guides executable actions.

4.2 HIGH-LEVEL PLANNING PERFORMANCE

  • VirtualHome average success rose from 24.9% with Vanilla to 53.3%, and average final progress rose from 38.4% to 71.3%; RoboCasa success rose from 26.9% to 38.9% in one-shot planning and 43.0% to 57.0% in replanning.CausGround maintained stronger VirtualHome success and progress from short to long horizons and improved both RoboCasa protocols.Source: Table 1 · p. 7, §4.2 · p. 7

4.1 EXPERIMENTAL SETUP

  • Experiments compare methods under matched planner backbones, observations, task descriptions, and action spaces across planning and interaction-reasoning benchmarks.The main setup uses three Qwen3.5 backbones, with additional families evaluated separately.Source: §4.1 · pp. 6–7

4.3 CAUSAL STATE-TRANSITION REASONING ON SWITCH

  • SWITCH average interaction reasoning accuracy increased from 25.8% with Vanilla to 31.9%, with improvements across action, recovery, and verification categories.The benchmark measures reasoning over action execution and state transitions in long-horizon interactions.Source: Table 2 · p. 8, §4.3 · p. 8

4.4 COMPONENT ABLATIONS

  • Both local world-state representation and causal interaction knowledge improve over Vanilla, and the full CausGround performs best overall.The shared factorized state lets the planner match observations to retrieved action preconditions and effects.Source: §4.4 · pp. 8–9, Table 3 · p. 9

4.5 CROSS-FAMILY GENERALIZATION

  • Across six Qwen3.5, InternVL3.5, and Gemma4 models sized E4B–38B, CausGround raises average success from 23.3% to 50.7% and final progress from 38.4% to 67.3%.Gains appear across all three families and both smaller and larger models.Source: §4.5 · p. 9, Figure 5 · p. 9

4.6 QUALITATIVE ANALYSIS

  • In the chair case, Vanilla repeatedly attempts SIT without satisfying NEAR, whereas CausGround identifies the missing prerequisite and completes the task.The case illustrates causal knowledge guiding action selection when the target is recognized but an action precondition is unmet.Source: Figure 4 · p. 8, §4.6 · p. 9
  • In real-world manipulation, CausGround reasons backward from an MLLM-interpreted goal to retrieve relevant causal knowledge and derive executable actions.The example sequence is Open(Microwave), Pick(Bread), and Place(Bread, Plate).Source: §4.6 · p. 9, Figure 6 · p. 9

5 CONCLUSION

CausGround distills interaction experience into reusable causal knowledge organized around action preconditions and effects. Grounded in current tasks and world states, it supports backward and forward reasoning for long-horizon planning across environments.

5 CONCLUSION

  • CausGround represents reusable interaction knowledge through action preconditions and effects in a shared factorized world-state schema.The schema captures object attributes and inter-object relations.Source: §5 · p. 10
  • At planning time, it grounds task-relevant causal knowledge in the current task and world state to support backward reasoning over preconditions and forward reasoning over effects.Source: §5 · p. 10
  • The conclusion presents explicit causal interaction knowledge as a reusable grounding mechanism for long-horizon embodied planning across diverse environments.Source: §5 · p. 10

A.1.1 BENCHMARK CONSTRUCTION AND TASK STATISTICS

The VirtualHome appendix describes a vision-based benchmark and its shared closed-loop planning interface, alongside observation repair and semantic evaluation procedures. The setup preserves alternative valid action sequences while assessing task outcomes.

A.1.1 BENCHMARK CONSTRUCTION AND TASK STATISTICS

  • The adapted VirtualHome benchmark contains 75 household tasks across three difficulty splits, six scenes, 33 action types, and 78 object classes.Short, Medium, and Long tasks have 3–6, 7–10, and 11–22 reference actions, respectively; the suite totals 643 reference actions.Source: §A.1.1 · p. 14, Table 4 · p. 14

A.1.2 VISUAL PLANNING INTERFACE AND EXECUTION PROTOCOL

  • VirtualHome evaluation uses a shared interface in which the planner observes RGB scenes, predicts one high-level action, receives execution feedback, and replans.The shared interface specifies observations, action space, object vocabulary, and execution feedback.Source: §A.1.2 · p. 14, Table 5 · p. 14

A.1.3 OBSERVATION REPAIR

  • Observation repair corrects rendered content inconsistent with simulator transitions while preserving viewpoint, layout, identities, and unrelated scene content.Repaired observations are checked against simulator state by deterministic rules and manual inspection; that privileged state is not exposed to the planner.Source: §A.1.3 · pp. 15–16, Figure 7 · p. 16

A.1.4 EVALUATION PROTOCOL

  • Success Rate measures episodes satisfying the complete semantic task specification, while Progress measures partial completion at episode end.Reference programs are not required execution sequences; alternative valid action sequences can earn full credit, and failed attempts consume the interaction budget.Source: §A.1.4 · p. 16

A.2.1 BENCHMARK CONSTRUCTION AND TASK SETUP

The RoboCasa adaptation converts annotated manipulation trajectories into high-level planning sequences and evaluates both one-shot and closed-loop planning. Its shared action interface exposes task-level skills while withholding hidden simulator states and reference sequences.

A.2.1 BENCHMARK CONSTRUCTION AND TASK SETUP

  • RoboCasa Target Composite tasks are adapted for visual high-level planning by abstracting low-level control into task-level skill stages and using simulator observations as intermediate states.Source: §A.2.1 · p. 17

A.2.2 PLANNING INTERFACE AND EXECUTION PROTOCOL

  • The planning interface provides eight task-level APIs and evaluates complete-sequence one-shot planning alongside action-by-action replanning with RGB observations and SUCCESS/FAILURE feedback.All methods receive the same task information and interface, while hidden simulator states and reference action sequences remain unavailable.Source: §A.2.2 · p. 17

A.2.3 EVALUATION PROTOCOL

The appendix specifies how RoboCasa plans are evaluated and illustrates the planning interface, then describes the selected SWITCH subtasks and the state and causal representations used by CausGround.

A.2.3 EVALUATION PROTOCOL

  • A RoboCasa task counts as successful only when replayed execution reaches the official goal condition; inapplicable actions are failures, while unresolved trajectory branches are marked UNKNOWN.Methods are compared on shared test cases with no UNKNOWN transition; one-shot plans allow up to 40 actions, and closed-loop replanning allows 25 interaction attempts.Source: §A.2.3 · p. 17

A.2.4 REPRESENTATIVE HIGH-LEVEL PLANNING TRAJECTORIES

  • Figure 8 presents five representative RoboCasa tasks through intermediate RGB observations and high-level actions, abstracting low-level control at task-level action boundaries.The representation preserves task progression while exposing the action granularity used for high-level planning.Source: §A.2.4 · p. 17, Figure 8 · p. 18

A.3 SWITCH

  • SWITCH evaluation uses four subtasks—Action, Recovery, Verification Action, and Verification State—and reports their accuracies and macro-average on the original benchmark examples.The categories probe action selection, recovery from failures, verification of outcomes, and reasoning about resulting states.Source: §A.3 · p. 19, Table 6 · p. 19, Figure 9 · p. 19

B.1 FACTORIZED WORLD-STATE REPRESENTATION

  • CausGround represents visual observations as separate object instances with semantic attributes and typed relations, preserving relation meanings rather than inferring unsupported ones.The same factorized representation describes current states, action preconditions, and effects; for example, CLOSE is not treated as ON or INSIDE without evidence.Source: §B.1 · p. 20, Figure 10 · p. 20

B.2 ACTION-CENTERED CAUSAL KNOWLEDGE CONSTRUCTION

  • CausGround retains pre-action factors associated with action executability, extracts repeatable effects from successful executions, and separates generic from instance-specific knowledge.Effects of one action can be linked to another action's preconditions to form multi-step causal structures.Source: §B.2 · pp. 20–21, Figure 11 · p. 21

B.3 TASK-GROUNDED PLANNING

CausGround retrieves task-relevant causal knowledge, grounds it in the current observed state, and uses it to guide high-level action selection. The appendix also details library construction, task-time grounding, and the prompt interfaces used by evaluated planners.

C.3.1 CAUSGROUND LIBRARY CONSTRUCTION

  • CausGround constructs reusable action rules from controlled VirtualHome interventions and consistency-filtered RoboCasa and SWITCH trajectories.VirtualHome's library has 5,614 normalized rules across 40 action families; RoboCasa's contains 98 grounded mechanisms across eight high-level action families.Source: §C.3.1 · p. 23

C.3.2 TASK-TIME RETRIEVAL AND GROUNDING

  • At episode start, CausGround retrieves a task-relevant causal subgraph containing goal-related conditions, effects, and producer dependencies, then grounds it repeatedly in the current observed state.The retrieved graph remains fixed during execution while observations and success/failure history update the local state.Source: §C.3.2 · pp. 23–24

C.4 PROMPT TEMPLATES

  • The prompt templates provide reproducible interfaces, with environment-specific action and object vocabularies inserted at runtime..Source: §C.4 · p. 24

C.4.1 VANILLA

  • The Vanilla prompt asks for one next household action from the image and instruction in a prescribed JSON format.It excludes plans, simulator IDs, and action sequences.Source: §C.4.1 · p. 24

C.4.2 SELFREVISION

  • The SelfReVision prompts support detailed visual grounding, critique and revision, plan comparison, and execution of only the selected plan's first action.The prompts also prohibit inferring hidden states or simulator-specific IDs.Source: §C.4.2 · pp. 24–25

C.4.3 LOOKPLANGRAPH

  • The LookPlanGraph prompts convert visually grounded scene descriptions into a public graph and select actions using that graph and execution history.Only confirmed execution success updates stored graph state; failed actions do not commit predicted effects.Source: §C.4.3 · p. 26

C.4.4 CAUSGROUND

  • The CausGround prompt combines the current observation and factorized state with retrieved causal knowledge to choose an action addressing unmet prerequisites.It directs the planner to use effects and producer dependencies without treating the knowledge as a fixed action sequence.Source: §C.4.4 · p. 27

C.1 COMMON EXPERIMENTAL SETUP

The experiments use matched settings within each benchmark and freeze planner parameters, so comparisons vary planning context and method-specific reasoning mechanisms.

C.1 COMMON EXPERIMENTAL SETUP

  • Within each benchmark, methods share the planner backbone, task descriptions, visual observations, high-level action space, and evaluation protocol, with all planner parameters frozen.The paper attributes performance differences to planning context and method-specific reasoning mechanisms rather than parameter updates.Source: §C.1 · p. 22

C.2 BASELINE IMPLEMENTATIONS

Across three Qwen3.5 backbones, combining factorized state with causal interaction knowledge achieves the highest success rate and final progress. The appendix also describes the Vanilla, SelfReVision, and LookPlanGraph baselines.

C.2 BASELINE IMPLEMENTATIONS

  • Vanilla predicts one semantic action directly from the current observation, instruction, vocabulary, and execution history without a dynamic local state or auxiliary planning module.It also does not use CausGround or self-revision.Source: §C.2 · pp. 22–23
  • SelfReVision generates, critiques, and revises a multi-step plan, executes its first action, and replans from the updated observation.Plan refinements are limited to three rounds.Source: §C.2 · pp. 22–23
  • LookPlanGraph merges the current visually constructed graph with memory for graph-based planning and verification, committing graph updates only after executor success.Only its first proposed action is executed at each step.Source: §C.2 · pp. 22–23

D.1 CAUSAL KNOWLEDGE VS. FACTORIZED STATE ACROSS BACKBONES

  • Across all three Qwen3.5 backbones, combining factorized state and causal interaction knowledge achieves the highest success rate and final progress, with both components improving over Vanilla.Causal interaction knowledge provides the larger individual gain; the table extends the main-paper component comparison to additional backbone scales.Source: Table 7 · p. 27, §D.1 · p. 27

D.2 CAUSAL VS. SEMANTIC RETRIEVAL

Causal retrieval contributes most of the retrieval gain, while combining action preconditions and effects yields the best reported ablation result. In real-world tasks, backward reasoning derives executable action sequences for microwave and multi-object manipulation.

D.2 CAUSAL VS. SEMANTIC RETRIEVAL

  • Combining semantic and causal retrieval reaches 44.0% success rate and 61.5% progress, versus 42.7% success rate and 60.4% progress with causal retrieval alone.Semantic retrieval alone improves success rate from 10.7% to 18.7% and progress from 23.8% to 37.6%.Source: §D.2 · p. 28, Table 8 · p. 28

D.3 ACTION PRECONDITIONS VS. EFFECTS

  • Combining action preconditions and effects reaches 44.0% success rate and 61.5% progress, with effects alone slightly outperforming preconditions alone.Both preconditions and effects independently provide substantial gains over Vanilla.Source: §D.3 · p. 28, Table 9 · p. 28

E REAL-WORLD DEPLOYMENT

  • In real-world deployment, backward reasoning derives executable high-level sequences for microwave retrieval and sequential placement of banana and bread.The MLLM interprets the instruction into a goal state; the robot’s existing low-level controller carries out the resulting semantic actions.Source: E REAL-WORLD DEPLOYMENT · p. 29, Figure 13 · p. 29, Figure 14 · p. 29

Related papers

Loading 2610.29702v1…