PaperlayerView exact revision on arXiv

Source-linked AI summary

Zeva: In-Context Causal Learning for Generalizable Embodied Manipulation

Fu Chen, Xin Ding, Bingjia Huang, Xiangyu Li, Mingju Wang, Jiawei He, Kun Li, Wei Sun, Yunxin Liu, Hao Wu, Ting Cao

arXiv:2608.30880v1cs.RO

TL;DR

Pretraining alone cannot cover the unseen physical conditions encountered in embodied manipulation. Zeva addresses this by extracting causal interaction signals into dual-timescale memory and injecting retrieved evidence into a frozen policy, achieving strong benchmark performance and improving success over repeated attempts.

  • Problem

    Pretraining alone struggles with unseen physical conditions because similar actions can produce different outcomes during real-world deployment.

  • Method

    Zeva extracts action–state-change causality, stores it in dual-timescale memory, and injects retrieved signals as context into a frozen policy without weight updates.

  • Results

    Zeva achieves the best success rate among compared frontier VLA and WAM models, reaching 76.8% on RoboCasa365-Atomic5 and improving across repeated attempts.

  • Takeaways & Limitations

    Zeva enables deployment-time self-evolution as interaction experience accumulates while the policy remains frozen.

  • Takeaways & Limitations

    Zeva learns causality only from task-execution attempts rather than actively collecting interactions for learning.

Abstract

from arXiv · show

Generalizable embodied manipulation remains difficult to achieve through pretraining alone, due to unseen physical conditions in the real world. We argue that robots need to learn from their own physical interactions on the fly during real-world deployment and use this knowledge to inform subsequent actions. We present Zeva, the first framework that enables in-context learning from a robot's own physical interaction experience while keeping the policy model frozen. Zeva employs a Causal Interaction Extractor to encode an executed action and its induced state change into a causal interaction signal, which is stored in a dual-timescale causal memory. For subsequent actions, relevant causal interaction signals are retrieved from memory and injected into the frozen policy model as context. Experiments in simulation and real-world manipulation demonstrate that Zeva achieves the best performance among the compared frontier VLAs and WAMs and, more importantly, enables self-evolution during deployment without gradient updates. Its success rate continues to improve as the robot accumulates interaction experience. Furthermore, the acquired interaction experience can generalize across tasks.

I. INTRODUCTION

Zeva addresses brittle deployment by letting a frozen policy learn causal relations from its own physical interactions and use them in subsequent actions. It combines causal extraction, dual-timescale memory, and in-context policy injection, improving success across simulation and real-world manipulation.

  • Motivation: Pretraining alone struggles with unseen object geometries, contact dynamics, visual variations, and embodiment-specific execution errors during deployment.These conditions can make similar actions produce significantly different outcomes.
  • Approach: Zeva enables a frozen robot policy to learn interaction causality from physical experience without test-time weight updates.It extracts relations between actions and induced state changes for subsequent action generation.
  • Results: 76.8% success rate was achieved on RoboCasa365-Atomic5, the best result among compared frontier VLA and WAM models.Zeva was also evaluated on the real-world ChemLab-Evo benchmark across three difficulty levels.
  • Results: Success rate increased from 26% at the first attempt to 73% within four repeated attempts on RoboCasa365-Atomic5.ChemLab-Evo showed the same scaling trend, and human teleoperation demonstrations improved performance by up to 15%.
  • Approach: Zeva combines Causal Interaction Extraction, Dual-timescale Causal Memory, and In-Context Policy Injection for gradient-free self-evolution across repeated attempts.The framework retrieves causal interaction signals as context for the frozen policy.

A. Vision-Language-Action and World-Action Foundation Models

VLA and WAM policies obtain generalizable priors through large-scale pretraining, but their fixed deployment representations and weights limit adaptation to changing physical conditions. Zeva instead adds causal interaction memory and prompt injection to a frozen foundation policy.

  • Foundation Models: WAMs combine action generation with future visual prediction to absorb physical priors through large-scale video pretraining.They may jointly denoise future frames and actions or interleave video and action tokens autoregressively.
  • Limitations: VLA and WAM weights remain fixed after deployment, so their capability ceiling is set by the pretraining corpus.This leaves them exposed to physical conditions absent during pretraining.
  • Representation Learning: Low-dimensional behavioral representations can outperform high-dimensional direct mapping, but their representation spaces are fixed before deployment.ALAM, LAPA, BehaviorVLA, UniVLA, and related methods cannot accumulate deployment interaction experience as distributions change.
  • Zeva: Zeva adds causal interaction intelligence to a frozen foundation policy through extraction, dual-timescale memory, and prompt injection.The architecture uses BIT for within-attempt dynamics and PIM for cross-attempt experience.
  • Online Memory: Online-memory methods address long-horizon dependencies by maintaining perceptual, cognitive, video, language, or event-based historical memories.Examples include MemoryVLA, MEM, DIM-WAM, and MemoryWAM.
  • Deployment Adaptation: Gradient-based rollout adaptation can be too slow between attempts and vulnerable to capability collapse, while test-time gradients still have excessive adaptation latency.Zeva targets deployment-time causal adaptation without updating policy weights.

A. Problem Formulation: In-Context Causal Learning

In-Context Causal Learning asks an embodied agent to infer action–state-change relations within an episode while keeping policy weights fixed. Zeva organizes interaction evidence into causal context that guides action generation and supports implicit system identification.

  • A. Problem Formulation: In-Context Causal Learning: The agent must generalize to novel tasks by learning interaction causality in context rather than through standard imitation learning.The target relation is between actions and resulting state changes.
  • A. Problem Formulation: In-Context Causal Learning: The policy performs ICCL within a single episode while enforcing ∇θπ = 0, so its weights remain unchanged.At attempt r, it uses causal context extracted from previous interactions.
  • A. Problem Formulation: In-Context Causal Learning: The objective is to maximize success on unseen tasks by minimizing discrepancy between predicted and actual causal effects.The causal context M guides the frozen foundation model toward new physical dynamics.
  • 1) Causal Context as Interaction Evidence:: An interaction experience is represented as triplets consisting of an action, its execution, and the resulting visual state change.The effect is defined as ∆sτ = sτ+k − sτ.
  • 1) Causal Context as Interaction Evidence:: Each causal interaction unit ξτ encodes an action and its induced state change for use as interaction evidence.These units form the organized causal context M.
  • 1) Causal Context as Interaction Evidence:: Conditioning the frozen policy on M performs implicit system identification from past interaction signals.The policy infers latent properties such as object mass or joint constraints and adapts actions to unseen configurations.
  • B. Overview of Zeva: Zeva transforms raw interactions into dual-timescale memory and injects retrieved causal prompts into the foundation model.Its memory supports adaptation within and across attempts.

C. Part 1: Causal Interaction Extraction

Causal Interaction Extraction converts visual observations, actions, and observed effects into causal interaction units. These units separate task progress from interaction dynamics and are organized in short- and long-term memories for retrieval.

  • Causal Interaction Extraction: The Causal Transition Encoder maps raw interactions into a latent causal space.It bridges raw pixels and causal reasoning.
  • Interaction Recurrence: At each timestep, the encoder integrates visual latents sτ, action encodings uτ, and observed effects dτ into a recurrent Causal Interaction State Bτ.A gated recurrence maintains the interaction representation over time.
  • Interaction Representation: The Causal Interaction State is projected into a Phase Token pτ for task progress and a Causal Interaction Signal eτ for interaction dynamics.Together they form the interaction unit ξτ = (pτ, eτ).
  • Learning Objectives: The encoder is optimized so eτ captures physical causality through effect prediction, task identity clustering, and phase progression objectives.These objectives predict visual state changes, cluster same-task episodes, and encode monotonic execution progress.
  • Dual-timescale Memory: Zeva stores interaction units in two memory streams supporting adaptation within and across attempts.The Brief Interaction Trace provides recent context, while Persistent Interaction Memory accumulates experience across attempts.
  • Brief Interaction Trace: The Brief Interaction Trace stores a sliding window of recent signals and resets at the end of each attempt.It supplies immediate local context about current-action efficacy.
  • Persistent Interaction Memory: Persistent Interaction Memory accumulates causal units across attempts of the same episode and is cleared before a new episode.Similarity-based merging keeps new experience only when task progress and physical dynamics both match an existing record.

E. Part 3: In-Context Policy Injection

Zeva converts stored interaction histories into a structured causal prompt for the frozen foundation policy. Retrieval is phase-conditioned and combines task, phase, brief, and persistent-memory context for subsequent action generation.

  • Causal Prompt Construction: The final stage transforms stored experiences into a structured causal prompt for the foundation model.
  • Phase-Conditioned Retrieval: Phase-conditioned retrieval queries persistent memory for interaction evidence whose associated phases match current progress.
  • Causal Prompt Construction: The memory context integrates task and phase tokens with short-term and long-term interaction histories through a shared projector and memory encoder.
  • Inference Procedure: Zeva freezes all neural parameters, retains persistent memory across attempts, resets brief memory per attempt, and updates memory after each interaction step.
  • Causal Inference: Conditioned on the causal prompt, the frozen policy adapts actions using success or failure from previous interactions to maximize task success.

F. Inference

During deployment, Zeva keeps all parameters frozen while updating causal memory online. The foundation model uses the evolving memory context for decision making, following the complete in-context inference procedure.

  • Inference: During deployment, all parameters remain frozen while Zeva updates its causal memory online.
  • Inference: The foundation model treats the evolving memory context as dynamic context for decision making.

IV. EXPERIMENTS

The experiments evaluate Zeva across simulated and real-world manipulation benchmarks, repeated-attempt adaptation, and causal-memory generalization. Evaluation controls distinguish episodes from attempts and retain persistent memory only within an episode.

  • Experimental Questions: The experiments test benchmark success, post-deployment scaling without parameter updates, and causal-memory contributions to cross-task generalization.
  • Evaluation Units: An episode is a task instance with an initialization, whereas an attempt is one continuous policy execution until termination or success.
  • Memory Protocol: BIT is cleared at each attempt, PIM persists across attempts within an episode, and both memories are cleared before the next episode.
  • Simulation Benchmark: RoboCasa365-Atomic5 evaluates five representative kitchen-manipulation tasks across 50 independently randomized episodes per task.
  • Real-World Benchmark: ChemLab-Evo uses an ARX manipulator to evaluate seven chemical-laboratory tasks spanning three levels of increasing compositional complexity.
  • Evaluation Metrics: Table I reports ChemLab-Evo success rates over 20 randomized episodes and ordered-stage process scores for two long-horizon tasks.
  • Evaluation Metrics: Table II reports task success rates and macro-averages on RoboCasa365-Atomic5 over 50 randomized episodes.
  • Evaluation Protocol: Each evaluated ChemLab-Evo task uses 20 independently randomized episodes.

4) Comparative Baselines:

Zeva is compared with frontier VLA and WAM baselines under matched interfaces, datasets, and evaluation protocols. Results include success, process completion, post-deployment scaling, and ablations of causal memory components.

  • Comparative Baselines: Zeva is compared with LingBot-VA, Xiaomi-Robotics-1, τ0-WM, Fast-WAM, Cosmos3-Nano, and π0.5 using matched training datasets, interfaces, and protocols.
  • Evaluation Metrics: Success rate is reported alongside macro-average success rate, while normalized process Score credits valid ordered-stage progress before failure.
  • Post-Deployment Scaling Metrics: CSR@K measures the fraction of fixed episodes completed within an attempt budget, quantifying capability scaling with accumulated interaction experience.
  • Real-World Evaluation: Zeva improves ChemLab-Evo average success over the strongest baseline by 6.6, 5.0, and 5.0 percentage points at atomic, short-sequence, and complex levels.
  • Simulation Benchmark Results: Zeva achieves a 76.8% average success rate on RoboCasa365-Atomic5, exceeding Fast-WAM by 4.4 percentage points.

D. Ablation Studies

Ablations show that both causal-memory timescales contribute to performance, while accumulated interaction experience and one-shot human warm-up improve frozen-policy manipulation across simulated and real-world settings.

  • Memory Ablation: The full model performs best on all five evaluated ChemLab-Evo tasks.Removing both BIT and PIM produces the weakest result on every task.
  • Memory Ablation: Removing PIM reduces success rate by 10–20 percentage points, whereas removing BIT causes larger decreases of 15–30 percentage points.The largest drops occur on Pour Water and Titration.
  • Post-Deployment Capability Scaling: 65% to 100% on Pick Up Test Tube, 25% to 70% on Place Beaker, and 30% to 80% on Pour Water across ChemLab-Evo evolution milestones.These curves use separately selected fixed episodes for the three atomic tasks.
  • One-Shot Human Warm-Up: Human warm-up improves or matches success rate at every evolution milestone across three atomic ChemLab-Evo tasks.The gain reaches 20 percentage points on Place Beaker and 15 points on Pour Water; Pick Up Test Tube converges to 100% in both conditions.

F. Cross-Task Generalization of Causal Interaction Signals

Cross-task causal interaction signals can replace task-local signals while preserving much of the original performance, and their retrieval aligns transitions by physical effect across tasks.

  • Cross-Task Transfer: Nearest cross-task retrieval preserves or nearly preserves task performance, changing SR from 100% to 95% on Pick Up Test Tube and retaining 80% on Pour Water.The frozen policy and all remaining inputs stay unchanged during replacement.
  • One-Shot Warm-Up: A one-shot human demonstration initializes PIM so Zeva can reproduce approach, grasp, transfer, and placement stages with the frozen policy.The Balance Weighing comparison includes the corresponding end-effector trajectories and grasp and placement events.
  • Cross-Task Transfer: Nearest cross-task retrieval exceeds random replacement by 40 percentage points on Pick Up Test Tube and 35 points on Pour Water.Random replacement reduces SR to 55% and 45%, respectively.
  • Cross-Task Retrieval: Interaction-signal neighbors align pouring, gripper-closure, and lifting transitions by physical effect despite different objects, viewpoints, and task instructions.Examples include pouring matched across Pour Water, Prepare Salt Solution, and Extraction, and grasp-related transitions across different vessels.

2) CTE Representation Analysis:

CTE’s full interaction representation forms compact, task-specific clusters, while removing the effect stream increases dispersion and inter-task mixing.

  • CTE Representation Analysis: Full CTE produces compact, task-specific clusters with clear separation across all seven ChemLab-Evo tasks.The comparison uses t-SNE visualization of CTE interaction embeddings.
  • CTE Representation Analysis: Removing the effect stream increases within-task dispersion and inter-task mixing.The comparison is consistent with observed state changes contributing to task-discriminative causal interaction memory.
Loading 2608.30880v1…