PaperlayerView exact revision on arXiv

Source-linked AI summary

CoDA: Co-Evolution of Data Engines and Action Policies

Tianjiao Yu, Zhilin Ou, Yifan Shen, Xinzhuo Li, Yuanzhe Liu, Onkar Kishor Susladkar, Fan Wang, Ismini Lourentzou

arXiv:2610.22362v1cs.ROcs.AI

TL;DR

Generative simulation makes environments easier to produce, but choosing useful training environments and supervision remains difficult. CoDA couples generation and policy learning through shared memory and Text2potential; across four manipulation tasks it reaches 27.6% out-of-distribution success, versus 5.3% after supervised fine-tuning.

  • Problem

    Choosing environments that address a policy’s failures and supervision that helps it learn remains difficult because validity and current success do not measure learning progress.

    Source: §1 · p. 1

  • Method

    CoDA shares construction traces, rollout-grounded failure diagnoses, and learning progress between generation and policy training, compiling diagnoses into generation and training programs with Text2potential value targets.

    Source: §1 · p. 2, §3.2 · p. 4, §3.5 · p. 5

  • Results

    27.6% out-of-distribution success across four manipulation tasks, versus 5.3% after supervised fine-tuning and 18.1% for same-budget success-rate-guided generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Measured progress on recurring difficulties guides further generation, and shared memory can transfer accumulated experience across policy backbones.

    Source: §1 · p. 2, §5 · p. 8

    Foundation-model errors can affect scenes and training signals, while physical-check coverage depends on the checks and interventions used.

    Source: D. Limitations · p. 15

Abstract

from arXiv · show

Generative simulation can automate robot environment construction, but physically valid environments are not necessarily useful for training. The challenge is to identify environments that address a policy's failures and supervision that helps it overcome them. We introduce CoDA, a framework in which a simulation data engine and an action policy co-evolve through a shared heterogeneous memory of environment construction, policy failures, and learning progress. The memory compiles validated diagnoses from a vision-language critic into generation programs that recreate difficulties and training programs that help the policy overcome them. We further propose Text2potential to convert these diagnoses into staged potentials for action-chunk value learning, providing dense supervision for policy updates while preserving the task's optimal policy under standard shaping assumptions. Measured learning progress guides further generation and program refinement. Across four manipulation task families, CoDA improves out-of-distribution success from 5.3% after supervised fine-tuning to 27.6%, exceeding the strongest closed-loop baseline by 9.5 percentage points with the same generation budget. On LIBERO-Pro, 15 rounds of generation and training raise success from 25% to 84% under position perturbations and from 1% to 76% under instruction perturbations.

1. Introduction

Generative simulation makes environments easier to build, but choosing environments that address policy failures and supervision that helps the policy learn remains challenging. CoDA uses shared memory to connect environment generation, policy training, and measured learning progress.

1. Introduction

  • The remaining challenge is selecting environments and supervision that help the policy learn, since validity and current success do not measure learning progress.Without that feedback, a generator may spend its budget on environments the policy has already mastered or cannot yet solve.Source: §1 · p. 1
  • CoDA couples a simulation data engine and action policy through shared memory that records scene repairs, rollout-grounded failure diagnoses, and learning progress.Diagnoses are localized to a generative factor and trajectory phase, and progress on recurring difficulty modes guides subsequent generation.Source: §1 · p. 2
  • 27.6% out-of-distribution success across four manipulation tasks, versus 5.3% after supervised fine-tuning and 18.1% for same-budget success-rate-guided generation.On LIBERO-Pro, fifteen rounds raise π0.5’s success from 25% to 84% under position perturbations and from 1% to 76% under instruction perturbations.Source: §1 · p. 2
  • CoDA’s reported gains also include 42.1% out-of-distribution success with more rounds and memory reuse that raises a second backbone from 3.8% to 21.1%.The second-backbone result uses accumulated memory without additional environment generation.Source: §1 · p. 2
  • The same diagnosis is compiled into reusable programs that recreate a difficulty for generation and help the policy overcome it through training.Both programs are indexed by difficulty mode, supporting reuse across tasks.Source: §1 · p. 2
  • Text2potential converts failure diagnoses into potentials over simulator states, providing dense action-chunk value targets while preserving policies that maximize original task return.Shaping requires potential evaluations only at chunk endpoints.Source: §1 · p. 2

2. Related Work

Related work spans agentic scene and task synthesis, curriculum and adaptive-environment methods, and policy improvement from rollouts. CoDA focuses on using policy learning progress and localized failure diagnoses to guide generation.

2. Related Work

  • Agentic simulation methods synthesize assets, scenes, tasks, and supervision, while CoDA adapts environment generation to policy learning progress.Examples include RoboGen [38], RoboTwin 2.0 [4], SimFoundry [29], and systems for automated scene construction and refinement [37] [39].Source: §2 · p. 2
  • Curriculum methods prioritize environments using regret, learning potential, or parameter changes [27] [49], while POET coevolves agents with parameterized environments [35].Language models have also been used to synthesize rewards [41] and generate adaptive environment programs [10].Source: §2 · p. 2
  • VLA policies learn from rollouts with sparse success rewards [32], while Q-Planning improves action selection with the base policy frozen [7].CoDA adopts an action-chunk critic and advantage-weighted regression [24], linking scene generation and policy learning through localized failure diagnoses.Source: §2 · p. 2
  • Lifelong methods retain reusable prompts or skill libraries [18], whereas CoDA links scene factors and trajectory phases to generation and policy learning.Source: §2 · p. 2

3. Method

CoDA links scene generation and policy training through shared memory that turns rollout-grounded diagnoses into reusable interventions. Measured learning progress directs future generation and program refinement.

3. Method

  • CoDA co-evolves a scene-generating data engine and action policy through shared memory that compiles validated failure diagnoses into generation and training programs.The generation programs recreate diagnosed conditions, while training programs specify subgoals and learning tools.Source: §3 · pp. 2–3, Figure 2 · p. 3

3.1. Seeding and Scene Representation

  • Scenes are factored into task, spatial, and object-attribute components that the engine can vary independently, with a sparse verifier defining task success.Generated scenes must also pass collision, stability, and scale checks before policy rollouts.Source: §3.1 · p. 3

3.2. Shared Heterogeneous Memory

  • Shared memory retains construction traces and rollout diagnoses, clusters recurring failures into difficulty modes, and compiles validated modes into complementary generation and training programs.Hardness tracks success rate, while importance tracks learning progress; generation programs reproduce conditions and training programs specify subgoals, potentials, and learning tools.Source: §3.2 · p. 4

3.3. Memory-Guided Scene Generation

  • Generation allocates budget across modes using hardness and learning progress, tests targeted variants against controls, and compiles a mode when targeted variants reduce policy success.The engine also reserves budget for novel environments and replay, while planner-successful variants can provide demonstrations.Source: §3.3 · p. 4

3.4. Text2potential

  • Text2potential converts validated diagnoses into phase-based subgoals and staged state potentials that densify value targets while retaining the task verifier as the final completion condition.Subgoal completion predicates are cumulative; the diagnosed phase guides task decomposition and the diagnosed cause guides potential construction.Source: §3.4 · pp. 4–5

3.5. Policy Training

  • Policy training uses shaped chunk-level value targets to score candidate action chunks and advantage-weight supervised policy updates; under stated shaping conditions, optimal task-return policies are preserved.Both successful and failed rollouts train the critic, and demonstrations add imitation supervision when available; measured progress assesses and routes training programs.Source: §3.5 · pp. 5–6

4. Experiments

CoDA improves out-of-distribution success across four manipulation task families and on LIBERO-Pro. Further rounds, diagnosis-based shaping, and memory reuse provide additional gains across settings.

Main results

  • 27.6% OOD success, versus 18.1% for success-rate-guided generation and 19.2% for the critic without memory, across four task families.Uniform generation reaches 8.5%, and domain randomization reaches 7.9%.Source: §4 · p. 6

Co-evolution over rounds

  • Extending CoDA from five to fifteen rounds raises OOD success from 27.6% to 42.1%; later rounds leave fewer unresolved difficulty modes and yield smaller gains.Removing memory produces less consistent progress, including two temporary drops in success.Source: §4 · pp. 6–7, Figure 3 · p. 7

LIBERO-Pro

  • After fifteen rounds, LIBERO-Pro success reaches 84% under position perturbations and 76% under instruction perturbations, versus 25% and 1% initially.The fifteen-round model exceeds ASPIRE by 7 and 9 percentage points overall, while standard LIBERO success is 97.4% versus 97.1% before adaptation.Source: §4 · p. 7, Table 2 · p. 7

Critic sensitivity

  • At a fixed training budget, reported OOD success values are 26.1%, 25.7%, and 24.9%, after a reduction from 27.6%; all four tested models outperform generation guided by task success rates.Even Qwen3-VL-235B exceeds that baseline by 6.8 percentage points.Source: §4 · p. 7, Table 3 · p. 8

Text2potential

  • 27.6% OOD success with Text2potential, versus 22.3% with sparse rewards alone; reaching 25% OOD success requires about 32% fewer policy updates.Task-description potentials reach 24.6%, disabling refinement reaches 26.1%, and value-only chunk selection reaches 21.4%.Source: §4 · p. 7, Table 4 · p. 8

Transfer across tasks and policies

  • Reusing memory collected with π0.5 raises OpenVLA-OFT OOD success from 3.8% to 21.1% without further environment generation.Transferred generation programs also yield greater learning progress than programs learned from scratch with the same LLM budget.Source: §4 · p. 8

5. Conclusion

CoDA couples simulation environment generation and policy learning through shared heterogeneous memory. Experiments report improved out-of-distribution success, program transfer across task families, and memory reuse across policy backbones.

Conclusion

  • CoDA co-evolves a simulation data engine with a trainable action policy through shared heterogeneous memory.Failure diagnoses grounded in policy rollouts guide environment generation and policy training.Source: §5 · p. 8
  • Measured learning progress determines which difficulties to revisit during generation and training.Source: §5 · p. 8
  • Experiments report improved out-of-distribution success, program transfer across task families, and memory reuse across policy backbones.Source: §5 · p. 8

A. Transferring Generation Programs Across Task Families

Generation programs can transfer across task families when a new scene satisfies their object requirements and relational constraints. Symbolic roles and validation determine applicability and whether the transferred difficulty remains useful.

Transfer lifting

Role-based requirements

Matching and validation

  • Valid bindings assign distinct scene objects to roles; transferred programs are validated and retained only when controls support the attributed difficulty and variant success falls within the success band.At most three valid bindings are executed per program.Source: A. Transferring Generation Programs… · p. 12

B. Implementation Details

The experiments use simulated manipulation environments with held-out evaluation suites and LIBERO-Pro, training primarily a 3B-parameter π0.5 policy. The setup specifies the assets, policy and value-learning configuration, generation budgets, and memory-transfer table scopes.

Environments and evaluation

  • Evaluation covers four manipulation task families with four held-out suites of 100 environments per family, plus a LIBERO-Pro setting initialized from 30 LIBERO source tasks.Held-out environments test generalization to unseen object instances, layouts, tasks, and categories; all results are mean ± standard error over three seeds.Source: B. Implementation Details · p. 12

Assets, observations, and actions

  • The asset set contains 1,200 rigid objects, 300 articulated objects, and 500 seed layouts; the robot is a 7-DoF Franka Panda with two 224 × 224 RGB cameras.Control runs at 10 Hz with seven-dimensional actions and action chunks of h = 8 steps.Source: B. Implementation Details · p. 12

Policy backbones and value learning

  • π0.5 is the primary 3B-parameter policy; both evaluated backbones use LoRA fine-tuning, and the value critic combines frozen DINOv2-B and T5-base encoders with a four-layer transformer decoder.At each rollout decision, the policy samples j = 8 action chunks and combines them using softmax weights from Qϕ with temperature 0.5.Source: B. Implementation Details · p. 12

Generation budgets and rollout protocol

  • The five-round experiments budget 320 accepted environments, 1,280 planner-certified demonstrations, and 2.5M LLM tokens per task family; extended runs use 15 rounds.Each round generates 64 targeted and 64 control environments per family; the table reports memory-design effects at a fixed generation budget.Source: B. Implementation Details · p. 12, Table 5 · p. 13

Transfer evaluation design

  • Table 6 compares cross-family generation-program transfer by learned-family rows and evaluation-family columns, reporting success-band variants and learning progress relative to within-family programs.Its final row is the same-token-budget from-scratch target-family baseline.Source: Table 6 · p. 13

Round scheduling and randomization

  • A new round starts when fewer than eight difficulty modes remain in the success band or in-band success changes by under two percentage points over 2,000 policy-update steps.The generation budget reserves ε = 0.15 for novel environments and 0.10 for replay.Source: B. Implementation Details · p. 13

C. Additional Experiments

Additional experiments examine program complementarity and transfer, shaping-related learning dynamics, diagnosis quality, and computational costs. Results report gains from retaining both program types, reuse across task families and policy backbones, and adaptive scheduling.

Complementary generation and training programs

  • 27.6% OOD success with both memory program stores exceeds the gains from either store alone, and reaches 25% OOD success in five rounds rather than fourteen without memory.Scene-generation-only retention raises OOD success from 19.2% to 22.8%, while training-program-only retention raises it to 22.1%; the combined 8.4-point gain exceeds the summed individual gains of 3.6 and 2.9 points.Source: C. Additional Experiments · p. 13

Transfer across task families

  • Reused generation programs achieve 60.3% of within-family learning progress across task-family transfers, versus 35.8% for programs learned from scratch with the same LLM token budget.The transfer advantage holds in every cross-family comparison.Source: C. Additional Experiments · p. 13

Transfer across policy backbones

  • Transferred memory improves OOD success by an average of 18.9 percentage points over the SFT baseline without memory, with results within two points of full CoDA in both backbone-transfer directions.The transferred-memory evaluation uses frozen memory and no additional environment generation.Source: C. Additional Experiments · p. 13, Table 7 · p. 14

Learning speed and memory accumulation

  • Potential shaping reaches similar final in-distribution success about 20,000 policy-update steps earlier than sparse-only training, and the critics select the same action chunk in 97% of end-of-training comparisons.Figure 5 tracks OOD success across memory snapshots; retaining both program types achieves the highest final success, approximately ten percentage points above either type alone.Source: C. Additional Experiments · pp. 13–14, Figure 5 · p. 14

Generation quality and diagnosis validation

  • 88.9% of CoDA scene proposals pass physical-validity checks versus 71.3% with uniform sampling, while targeting precision rises from 24.6% to 66.7% and coverage entropy remains 3.18 versus 3.42 bits.Rejecting 18.2% of diagnoses improves agreement with human annotations from 73.6% to 82.4% in a sample of 100 diagnoses assessed by two annotators.Source: C. Additional Experiments · p. 14, Table 8 · p. 14

Computational cost and qualitative results

  • CoDA raises OOD success from 18.1% to 27.6% over scalar feedback and uses 7.4M LLM tokens and 66 GPU-hours per 1,000 accepted environments.Adaptive scheduling reduces token usage and GPU time by approximately 11% versus fixed intervals while increasing OOD success from 26.1% to 27.6%; Figure 6 shows validated scene variations across four task families.Source: C. Additional Experiments · p. 14, Table 9 · p. 15, Figure 6 · p. 15

D. Limitations

CoDA depends on foundation models for scene generation, diagnosis, and potential compilation, and its validity checks and control experiments cover only the errors they test. The reported evaluation is limited to simulated manipulation; transfer to physical robots remains unestablished.

Model dependence

  • Foundation-model errors about objects, physical interactions, or task requirements can affect scenes and training signals, while detection depends on the coverage of validity checks and interventions.Results may also vary when underlying models change, making model versions and configurations important for reproducibility.Source: D. Limitations · p. 15

Evaluation scope and computational cost

  • The evaluation focuses on simulated manipulation, so physical-robot transfer remains to be established, particularly for tasks sensitive to contact dynamics, sensing, and actuation.Simulation and model-inference costs also constrain how many rounds and task families can be explored under a fixed budget.Source: D. Limitations · p. 15

E. Broader Impact

The broader-impact discussion presents CoDA as a way to produce diverse, verifiable training data without large teleoperation budgets, while identifying misuse and inherited foundation-model biases as concerns. It frames responsible research use around machine-checkable success verification and control-experiment audits.

Broader impact

  • CoDA is designed for research use and audits automated diagnoses with control experiments, while the authors identify potential misuse and inherited foundation-model biases as concerns.They describe the approach as requiring no physical data collection and as offering controllable, verifiable data generation when applied responsibly.Source: E. Broader Impact · p. 16

Related papers

Loading 2610.22362v1…