PaperlayerView exact revision on arXiv

Source-linked AI summary

Zeva: In-Context Causal Learning for Generalizable Embodied Manipulation

Fu Chen, Xin Ding, Bingjia Huang, Xiangyu Li, Mingju Wang, Jiawei He, Kun Li, Wei Sun, Yunxin Liu, Hao Wu, Ting Cao

arXiv:2608.30880v1cs.RO

TL;DR

Pretraining alone cannot cover the unseen physical conditions encountered in embodied manipulation. Zeva addresses this by extracting causal interaction signals into dual-timescale memory and injecting retrieved evidence into a frozen policy, achieving strong benchmark performance and improving success over repeated attempts.

  • Problem

    Pretraining alone struggles with unseen physical conditions because similar actions can produce different outcomes during real-world deployment.

    Source: §I · p. 2

  • Method

    Zeva extracts action–state-change causality, stores it in dual-timescale memory, and injects retrieved signals as context into a frozen policy without weight updates.

    Source: §I · p. 2, 1) Causal Context as Interaction… · p. 4

  • Results

    Zeva achieves the best success rate among compared frontier VLA and WAM models, reaching 76.8% on RoboCasa365-Atomic5 and improving across repeated attempts.

    Source: §I · p. 2, §V · p. 9

  • Takeaways & Limitations

    Zeva enables deployment-time self-evolution as interaction experience accumulates while the policy remains frozen.

    Source: §I · p. 2, §V · p. 9

    Zeva learns causality only from task-execution attempts rather than actively collecting interactions for learning.

    Source: §V · p. 9

Abstract

from arXiv · show

Generalizable embodied manipulation remains difficult to achieve through pretraining alone, due to unseen physical conditions in the real world. We argue that robots need to learn from their own physical interactions on the fly during real-world deployment and use this knowledge to inform subsequent actions. We present Zeva, the first framework that enables in-context learning from a robot's own physical interaction experience while keeping the policy model frozen. Zeva employs a Causal Interaction Extractor to encode an executed action and its induced state change into a causal interaction signal, which is stored in a dual-timescale causal memory. For subsequent actions, relevant causal interaction signals are retrieved from memory and injected into the frozen policy model as context. Experiments in simulation and real-world manipulation demonstrate that Zeva achieves the best performance among the compared frontier VLAs and WAMs and, more importantly, enables self-evolution during deployment without gradient updates. Its success rate continues to improve as the robot accumulates interaction experience. Furthermore, the acquired interaction experience can generalize across tasks.

I. INTRODUCTION

Zeva addresses brittle deployment by letting a frozen policy learn causal relations from its own physical interactions and use them in subsequent actions. It combines causal extraction, dual-timescale memory, and in-context policy injection, improving success across simulation and real-world manipulation.

Motivation

  • Pretraining alone struggles with unseen object geometries, contact dynamics, visual variations, and embodiment-specific execution errors during deployment.These conditions can make similar actions produce significantly different outcomes.Source: §I · p. 2

Approach

  • Zeva enables a frozen robot policy to learn interaction causality from physical experience without test-time weight updates.It extracts relations between actions and induced state changes for subsequent action generation.Source: §I · p. 2

Results

  • 76.8% success rate was achieved on RoboCasa365-Atomic5, the best result among compared frontier VLA and WAM models.Zeva was also evaluated on the real-world ChemLab-Evo benchmark across three difficulty levels.Source: §I · p. 2
  • Success rate increased from 26% at the first attempt to 73% within four repeated attempts on RoboCasa365-Atomic5.ChemLab-Evo showed the same scaling trend, and human teleoperation demonstrations improved performance by up to 15%.Source: §I · p. 2

Approach

  • Zeva combines Causal Interaction Extraction, Dual-timescale Causal Memory, and In-Context Policy Injection for gradient-free self-evolution across repeated attempts.The framework retrieves causal interaction signals as context for the frozen policy.Source: §I · p. 2

A. Vision-Language-Action and World-Action Foundation Models

VLA and WAM policies obtain generalizable priors through large-scale pretraining, but their fixed deployment representations and weights limit adaptation to changing physical conditions. Zeva instead adds causal interaction memory and prompt injection to a frozen foundation policy.

Foundation Models

  • WAMs combine action generation with future visual prediction to absorb physical priors through large-scale video pretraining.They may jointly denoise future frames and actions or interleave video and action tokens autoregressively.Source: A. Vision-Language-Action and… · p. 2

Limitations

  • VLA and WAM weights remain fixed after deployment, so their capability ceiling is set by the pretraining corpus.This leaves them exposed to physical conditions absent during pretraining.Source: A. Vision-Language-Action and… · p. 2

Representation Learning

  • Low-dimensional behavioral representations can outperform high-dimensional direct mapping, but their representation spaces are fixed before deployment.ALAM, LAPA, BehaviorVLA, UniVLA, and related methods cannot accumulate deployment interaction experience as distributions change.Source: B. Behavioral and Latent-Action… · pp. 2–3

Zeva

  • Zeva adds causal interaction intelligence to a frozen foundation policy through extraction, dual-timescale memory, and prompt injection.The architecture uses BIT for within-attempt dynamics and PIM for cross-attempt experience.Source: Figure 2 · p. 3

Online Memory

  • Online-memory methods address long-horizon dependencies by maintaining perceptual, cognitive, video, language, or event-based historical memories.Examples include MemoryVLA, MEM, DIM-WAM, and MemoryWAM.Source: C. Memory-Augmented Robot Policies · p. 3

Deployment Adaptation

  • Gradient-based rollout adaptation can be too slow between attempts and vulnerable to capability collapse, while test-time gradients still have excessive adaptation latency.Zeva targets deployment-time causal adaptation without updating policy weights.Source: D. Deployment-Time Adaptation and… · p. 3

A. Problem Formulation: In-Context Causal Learning

In-Context Causal Learning asks an embodied agent to infer action–state-change relations within an episode while keeping policy weights fixed. Zeva organizes interaction evidence into causal context that guides action generation and supports implicit system identification.

A. Problem Formulation: In-Context Causal Learning

  • The agent must generalize to novel tasks by learning interaction causality in context rather than through standard imitation learning.The target relation is between actions and resulting state changes.Source: A. Problem Formulation: In-Context… · p. 3
  • The policy performs ICCL within a single episode while enforcing ∇θπ = 0, so its weights remain unchanged.At attempt r, it uses causal context extracted from previous interactions.Source: A. Problem Formulation: In-Context… · p. 3
  • The objective is to maximize success on unseen tasks by minimizing discrepancy between predicted and actual causal effects.The causal context M guides the frozen foundation model toward new physical dynamics.Source: A. Problem Formulation: In-Context… · p. 4

1) Causal Context as Interaction Evidence:

  • An interaction experience is represented as triplets consisting of an action, its execution, and the resulting visual state change.The effect is defined as ∆sτ = sτ+k − sτ.Source: 1) Causal Context as Interaction… · p. 4
  • Each causal interaction unit ξτ encodes an action and its induced state change for use as interaction evidence.These units form the organized causal context M.Source: 1) Causal Context as Interaction… · p. 4
  • Conditioning the frozen policy on M performs implicit system identification from past interaction signals.The policy infers latent properties such as object mass or joint constraints and adapts actions to unseen configurations.Source: 1) Causal Context as Interaction… · p. 4

B. Overview of Zeva

  • Zeva transforms raw interactions into dual-timescale memory and injects retrieved causal prompts into the foundation model.Its memory supports adaptation within and across attempts.Source: B. Overview of Zeva · p. 4

C. Part 1: Causal Interaction Extraction

Causal Interaction Extraction converts visual observations, actions, and observed effects into causal interaction units. These units separate task progress from interaction dynamics and are organized in short- and long-term memories for retrieval.

Causal Interaction Extraction

Interaction Recurrence

  • At each timestep, the encoder integrates visual latents sτ, action encodings uτ, and observed effects dτ into a recurrent Causal Interaction State Bτ.A gated recurrence maintains the interaction representation over time.Source: C. Part 1: Causal Interaction Extraction · p. 4

Interaction Representation

  • The Causal Interaction State is projected into a Phase Token pτ for task progress and a Causal Interaction Signal eτ for interaction dynamics.Together they form the interaction unit ξτ = (pτ, eτ).Source: C. Part 1: Causal Interaction Extraction · p. 4

Learning Objectives

  • The encoder is optimized so eτ captures physical causality through effect prediction, task identity clustering, and phase progression objectives.These objectives predict visual state changes, cluster same-task episodes, and encode monotonic execution progress.Source: C. Part 1: Causal Interaction Extraction · p. 4

Dual-timescale Memory

  • Zeva stores interaction units in two memory streams supporting adaptation within and across attempts.The Brief Interaction Trace provides recent context, while Persistent Interaction Memory accumulates experience across attempts.Source: D. Part 2: Dual timescale Causal Memory · p. 4

Brief Interaction Trace

Persistent Interaction Memory

  • Persistent Interaction Memory accumulates causal units across attempts of the same episode and is cleared before a new episode.Similarity-based merging keeps new experience only when task progress and physical dynamics both match an existing record.Source: D. Part 2: Dual timescale Causal Memory · p. 4

E. Part 3: In-Context Policy Injection

Zeva converts stored interaction histories into a structured causal prompt for the frozen foundation policy. Retrieval is phase-conditioned and combines task, phase, brief, and persistent-memory context for subsequent action generation.

Causal Prompt Construction

Phase-Conditioned Retrieval

Causal Prompt Construction

Inference Procedure

Causal Inference

F. Inference

During deployment, Zeva keeps all parameters frozen while updating causal memory online. The foundation model uses the evolving memory context for decision making, following the complete in-context inference procedure.

Inference

  • During deployment, all parameters remain frozen while Zeva updates its causal memory online.Source: F. Inference · p. 5
  • The foundation model treats the evolving memory context as dynamic context for decision making.Source: F. Inference · p. 5

IV. EXPERIMENTS

The experiments evaluate Zeva across simulated and real-world manipulation benchmarks, repeated-attempt adaptation, and causal-memory generalization. Evaluation controls distinguish episodes from attempts and retain persistent memory only within an episode.

Experimental Questions

  • The experiments test benchmark success, post-deployment scaling without parameter updates, and causal-memory contributions to cross-task generalization.Source: §IV · p. 5

Evaluation Units

Memory Protocol

Simulation Benchmark

  • RoboCasa365-Atomic5 evaluates five representative kitchen-manipulation tasks across 50 independently randomized episodes per task.Source: 2) Simulation Benchmark · p. 5

Real-World Benchmark

Evaluation Metrics

  • Table I reports ChemLab-Evo success rates over 20 randomized episodes and ordered-stage process scores for two long-horizon tasks.Source: Table I · p. 6
  • Table II reports task success rates and macro-averages on RoboCasa365-Atomic5 over 50 randomized episodes.Source: Table II · p. 6

Evaluation Protocol

4) Comparative Baselines:

Zeva is compared with frontier VLA and WAM baselines under matched interfaces, datasets, and evaluation protocols. Results include success, process completion, post-deployment scaling, and ablations of causal memory components.

Comparative Baselines

  • Zeva is compared with LingBot-VA, Xiaomi-Robotics-1, τ0-WM, Fast-WAM, Cosmos3-Nano, and π0.5 using matched training datasets, interfaces, and protocols.Source: 4) Comparative Baselines · p. 6

Evaluation Metrics

  • Success rate is reported alongside macro-average success rate, while normalized process Score credits valid ordered-stage progress before failure.Source: B. Evaluation Metrics · p. 6

Post-Deployment Scaling Metrics

  • CSR@K measures the fraction of fixed episodes completed within an attempt budget, quantifying capability scaling with accumulated interaction experience.Source: B. Evaluation Metrics · p. 6

Real-World Evaluation

Simulation Benchmark Results

D. Ablation Studies

Ablations show that both causal-memory timescales contribute to performance, while accumulated interaction experience and one-shot human warm-up improve frozen-policy manipulation across simulated and real-world settings.

Memory Ablation

  • The full model performs best on all five evaluated ChemLab-Evo tasks.Removing both BIT and PIM produces the weakest result on every task.Source: D. Ablation Studies · pp. 6–7
  • Removing PIM reduces success rate by 10–20 percentage points, whereas removing BIT causes larger decreases of 15–30 percentage points.The largest drops occur on Pour Water and Titration.Source: D. Ablation Studies · p. 7

Post-Deployment Capability Scaling

One-Shot Human Warm-Up

  • Human warm-up improves or matches success rate at every evolution milestone across three atomic ChemLab-Evo tasks.The gain reaches 20 percentage points on Place Beaker and 15 points on Pour Water; Pick Up Test Tube converges to 100% in both conditions.Source: 2) One-Shot Warm-Up from a Human… · p. 7

F. Cross-Task Generalization of Causal Interaction Signals

Cross-task causal interaction signals can replace task-local signals while preserving much of the original performance, and their retrieval aligns transitions by physical effect across tasks.

Cross-Task Transfer

  • Nearest cross-task retrieval preserves or nearly preserves task performance, changing SR from 100% to 95% on Pick Up Test Tube and retaining 80% on Pour Water.The frozen policy and all remaining inputs stay unchanged during replacement.Source: 1) Cross-Task Transfer · p. 7

One-Shot Warm-Up

  • A one-shot human demonstration initializes PIM so Zeva can reproduce approach, grasp, transfer, and placement stages with the frozen policy.The Balance Weighing comparison includes the corresponding end-effector trajectories and grasp and placement events.Source: Figure 5 · p. 8

Cross-Task Transfer

  • Nearest cross-task retrieval exceeds random replacement by 40 percentage points on Pick Up Test Tube and 35 points on Pour Water.Random replacement reduces SR to 55% and 45%, respectively.Source: 1) Cross-Task Transfer · pp. 7–8

Cross-Task Retrieval

  • Interaction-signal neighbors align pouring, gripper-closure, and lifting transitions by physical effect despite different objects, viewpoints, and task instructions.Examples include pouring matched across Pour Water, Prepare Salt Solution, and Extraction, and grasp-related transitions across different vessels.Source: 2) Cross-Task Retrieval · p. 8, Figure 8 · p. 9

2) CTE Representation Analysis:

CTE’s full interaction representation forms compact, task-specific clusters, while removing the effect stream increases dispersion and inter-task mixing.

CTE Representation Analysis

Related papers

Loading 2608.30880v1…