PaperlayerView exact revision on arXiv

Source-linked AI summary

Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models

Junyi Hu, Zhewen He, Zhenhua Li, Yi Fang

arXiv:2610.02398v1cs.RO

TL;DR

Demonstrations specify a manipulation task but also record the particular arm’s motion, which can tie the task to one execution. PEWAM compiles the demonstrated effect into an actor-free, editable program and samples robot execution from it. The reported pattern is that this approach pays off when copying breaks, while copying is stronger when the demonstrated motion remains valid.

  • Problem

    Demonstrations record both what happened to objects and how one arm made it happen, leaving the task entangled with a body’s motion and not independently editable or transferable.

    Source: §1 · p. 1

  • Method

    PEWAM compiles object keypoints, contact points, and terminal configuration into an actor-free effect program, then samples execution from a four-stream world-action model.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    The program pays off after pushes, on another arm, and from a person’s hand, while copying is stronger when the demonstrator’s motion stays valid.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The effect program is an editable interface that can be handed to another body, and tracking robot-caused effects lets re-solving absorb pushes that defeat replay.

    Source: §1 · p. 2, §5 · p. 10

    Effects with little object motion leave little to specify; pointer and gate rules are hand-set, entities are mostly simulator-read rigid boxes, and single 30-episode cells can vary by as much as the discussed differences.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

A robot demonstration records two things in the same frames: what happened to the objects, and how one particular arm made it happen. We condition on the first. A demonstration is compiled into an effect program: the 3D keypoint trajectories of the objects that moved, two points marking where each was held, and the configuration the scene ends in, with the demonstrator removed. PEWAM, a 71.5M-parameter world-action model, generates effect, robot execution, action and terminal state as four streams with independent flow-matching times, so clamping a program and sampling the execution turns inference into programming, re-solved closed loop from the live scene. On held-out LIBERO-Goal tasks, one demonstration's program completes 40 of 90 episodes, where the same backbone given a goal image or language, and published demonstration-conditioned methods, complete at most 19; on three of Meta-World's held-out classes it exceeds the best published results, though not on the five-class mean. Because a program is a set of coordinates, a person can edit it: the placement follows a shifted terminal state and the grasp turns with rotated contact points. The same program runs on four robot arms without retraining, and after a push, re-solving completes 23 of 60 episodes where replaying the demonstration completes 6. On a Franka arm, fine-tuned on real demonstrations of other tasks, programs compiled from single human videos complete 36 of 40 trials, against 22 for the same backbone conditioned on the video's last frame as a goal image.

1 INTRODUCTION

The paper treats a demonstration’s object effects—not the demonstrator’s particular motion—as the task, and introduces an actor-free program that can be edited and executed by a multi-task model. It reports that the program pays off when copying breaks, while copying is stronger when the demonstrated motion remains valid.

1 INTRODUCTION

  • A demonstration’s task is the change in the world, but motion-conditioned methods entangle it with one body’s execution, limiting editing and transfer.Object-motion methods separate the actor but generate motion from language or video and use a planner, controller, or per-task policy to execute it.Source: §1 · pp. 1–2
  • An effect program records relevant objects’ keypoints over time, two contact points, and the terminal configuration, without referring to the robot.PEWAM represents effect, end-effector execution, native action, and terminal state as four streams with separate flow-matching times; clamping the program and sampling execution makes inference a programming query.Source: §1 · p. 2
  • The program pays off where copying breaks—after pushes, on another arm, and from a person’s hand—while copying is stronger when the demonstrator’s motion stays valid.The paper evaluates on LIBERO, Meta-World and RLBench, on four arms, under pushes, from pixels, and on a real Franka arm programmed by human videos.Source: §1 · p. 2
  • The program is an editable interface that can be handed to another body, and progress is gated on effects caused by the robot so disturbances become new observations.The terminal state can shift placement, contact points can turn the grasp, and one program runs on four simulated arms and, from a human video, on a real arm.Source: §1 · p. 2

2 RELATED WORK

Prior work either couples task specification to a demonstrator’s motion or separates object motion but uses a distinct generation and execution pipeline. PEWAM instead conditions one world-action model on an actor-free, editable effect program.

2.1 WORLD-ACTION MODELS AND IMITATION FROM ONE DEMONSTRATION

  • World-action models jointly learn video and action, while one-demonstration methods encode or retarget the demonstration and therefore carry the demonstrator’s motion.PEWAM retains independent timesteps but conditions on an actor-free program that a person can read and edit.Source: §2.1 · pp. 2–3

2.2 OBJECT MOTION AS THE TASK INTERFACE

  • Object-motion methods separate the actor but typically generate motion from other inputs and execute it with a policy, controller, or optimisation; Im2Flow2Act uses separate flow generation and execution.PEWAM reads the effect from the demonstration, adds contact points and a terminal state, and samples execution from one multi-task model that also answers forward queries of the same program.Source: §2.2 · p. 3

3 METHOD

PEWAM separates the object-level effect of a demonstration from the robot-specific execution, then uses a four-stream model to generate execution from a clamped effect program in a closed loop. The program also supports edits to placement and grasp and transfer across robot bodies.

3.1 FROM A DEMONSTRATION TO AN EFFECT PROGRAM

  • An effect program records each relevant object's world-frame keypoint trajectory, two grasp-contact points, and terminal state, while execution and native action describe the robot.The effect and terminal state contain no robot geometry; the program's contact points can be rotated to request another grasp of the same effect.Source: §3.1 · p. 3

3.2 ONE MODEL, FOUR STREAMS

  • One 71.5M-parameter DiT models effect, end-effector execution, native action, and terminal state as streams with separate flow-matching times.Observation tokens are not noised, and a clean stream contributes no loss, allowing the clean streams to define the query.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
  • Training includes multiple executions of one effect with rotated grasps, whose median execution difference is 59 mm while their median terminal-state difference is 1.0 mm.The solutions share a program, so contact points distinguish solutions of the same effect.Source: §3.2 · p. 4

3.3 INFERENCE IS PROGRAMMING

  • Inference clamps the effect and terminal state, samples execution and action, runs four actions, then observes the scene and repeats the closed loop.A pointer selects the current program window by matching live keypoints; one call takes 348 ms on one RTX A6000.Source: §3.3 · pp. 4–5
  • Before engagement, the loop re-anchors the program window to the live scene; at engagement, it aligns the pointer with the demonstration's engagement frame.Engagement is detected when the gripper is commanded closed and moved entities have travelled 1 cm; the pre-engagement offset decays to zero at the terminal state.Source: §3.3 · p. 5
  • Editing the terminal coordinates shifts placement, rotating contact points turns the grasp, and another arm's joint angles are re-expressed without changing weights.The trajectory is blended toward the placement shift from the first motion; only camera images and joint angles are arm-specific.Source: §3.3 · p. 5

4 EXPERIMENTS

Across held-out tasks, PEWAM's actor-free program succeeds particularly on LIBERO-Goal, while results vary across benchmarks. Experiments also test program edits, execution transfer, visual inputs, and real-arm use, with ablations showing that representation and training choices affect programmability.

4.1 ONE DEMONSTRATION PROGRAMS A HELD-OUT TASK

  • 40 of 90 held-out LIBERO-Goal episodes succeed with the full program, versus none for the same backbone given a goal image, language, or no specification; terminal state alone succeeds in 49.The three tasks contribute no robot data to training; replaying the demonstrations succeeds in 87 of 90 episodes.Source: §4.1 · p. 5, Table 1 · p. 6
  • PEWAM exceeds every published result on three held-out Meta-World classes but falls below DAMI on the five-class mean; on RLBench it completes 160 of 720 episodes, versus 108 without a program and 575 for Instant Policy.The Meta-World comparison spans ML10 and ML45; the RLBench evaluation holds out 24 tasks.Source: Table 1 · p. 6, §4.1 · p. 7

4.2 THE PROGRAM IS EDITABLE

  • Edited terminal states shift realised placements with a slope of 0.71, while rotated contact points turn grasps in the commanded direction and realise 0.68 of the commanded rotation on average.With edited contacts, the model completes 22 to 24 of 30 episodes at ±20° and ±45°, against 27 unedited.Source: §4.2 · p. 7

4.3 ONE EFFECT, MANY BODIES AND EXECUTIONS

  • One program transfers across four arms without retraining, and different grasps can execute the same effect successfully across three sampling seeds.Bowl-task successes are 27, 24, 22, and 16 of 30 on the Panda, KUKA IIWA, Kinova Gen3, and UR5e; the three seeds succeed in 27, 28, and 28 of 30.Source: Table 2 · p. 7, §4.3 · p. 8
  • 23 of 60 pushed episodes succeed with closed-loop re-solving, compared with 6 when replaying the demonstration’s actions.Without the engagement gate, re-solving falls to 7 pushed successes.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 FROM PIXELS TO A REAL ARM

  • Rendered RGB-D programs complete 26 of 30 bowl and 19 of 30 bottle episodes, while adding live segmented keypoints yields 40 of 60 successes versus 39 from simulator state.The bowl’s tracked centre has a median error of 0.61 cm.Source: §4.4 · pp. 8–9
  • On a real Franka arm, programs compiled from single human videos complete 36 of 40 trials, versus 22 for the same backbone conditioned on the video’s last frame as a goal image.The model was fine-tuned on 156 real demonstrations of other tasks; shifting the terminal state 10 cm to either side yields 9 of 10 successes each, and a 5 cm push yields 7 of 10.Source: §4.4 · p. 9, Table 4 · p. 9

4.5 WHAT MAKES A PROGRAM PROGRAMMABLE

  • Representation matters: nine affine contact-point weights completed 26/30 unedited bowl episodes and cross-arm bowl sum 85, while realized rotation offsets stayed negative regardless of commanded sign; paired training improved contact-edit following.At 50 episodes per rotation, all four trainings with the paired corpus followed edits more closely than all three without (0.41 to 0.70 against 0.31 to 0.34; one-sided exact permutation p=0.029). Displacements opened the ML10 drawer in 16 of 30 episodes, where absolute coordinates opened none, but completed 1 of 90 held-out LIBERO episodes.Source: §4.5 · p. 9

5 LIMITATIONS AND CONCLUSION

PEWAM represents a task through the effects on the world, enabling people to edit programs and reuse them across bodies. Its scope is limited by low-motion effects, hand-set grasp rules, rigid-box entities, and results that vary across trainings.

5 LIMITATIONS AND CONCLUSION

  • Low-motion effects such as knob turns leave the program little to specify; Meta-World’s joint-position channel is the stated extension for these effects.The pointer and gate for grasped objects are hand-set, and entities are rigid boxes, mostly read from simulator state.Source: §5 · p. 9
  • Where demonstrator motion remains valid, copying it is stronger, RLBench’s specialist remains far ahead, and many reported cells are single trainings with up to 30-episode replicate variation.The protocol uses one environment seed and 30 episodes per cell.Source: §5 · p. 10, REPRODUCIBILITY STATEMENT · p. 10
  • The effect program is an editable, actor-free task representation that carries across bodies, while progress can be read from caused effects to absorb a push.A shifted terminal state changes placement, rotated contact points turn the grasp, and one program runs on four simulated arms and a real arm from human video.Source: §5 · p. 10
  • The evaluation uses public benchmarks and lab Franka trials; the displayed human videos show lab members’ hands and forearms without faces or other personal data.The authors state no direct path to harm beyond robot manipulation research.Source: ETHICS STATEMENT · p. 10
  • The paper specifies its training and evaluation protocols, derives simulation table cells from per-episode records, and plans to release code, weights, corpus construction, scripts, and preregistration files upon acceptance.One training takes about 3.5 hours on three consumer GPUs.Source: REPRODUCIBILITY STATEMENT · p. 10

A IMPLEMENTATION DETAILS

PEWAM is trained as a shared-backbone model over observation-conditioned queries and four streams, with a paired corpus that separates grasp choice from effect. At inference, clamped program tokens support single-demonstration closed-loop rollouts.

A IMPLEMENTATION DETAILS

  • Observations combine two RGB views, 16-D proprioception, and entity keypoints; training samples inverse, goal, forward, and inverse-dynamics queries with probabilities 0.55, 0.25, 0.15, and 0.05.Conditions are dropped with probability 0.1.Source: A IMPLEMENTATION DETAILS · p. 13
  • The shared 71.5M-parameter DiT processes effect, execution, action, and terminal-state streams, with stream losses weighted E:R:A:G = 1:1:1:0.5.Its backbone has depth 8, width 512, and 8 heads; training uses fp32 after bf16 autocast diverged twice.Source: A IMPLEMENTATION DETAILS · p. 13
  • Training uses 1.16M windows from 107 LIBERO tasks, augmented by a paired corpus of 56 scenes and 273 executions.Those extra executions represent multiple solutions to the same effect.Source: A IMPLEMENTATION DETAILS · p. 13
  • Every test episode is a closed-loop rollout programmed by one demonstration, with no parameter updates at test time.Simulation uses benchmark success checks; the real arm uses the Appendix L criterion.Source: Table 6 · p. 14
  • Inference uses 20 Euler steps and keeps clamped tokens clean; the controller re-plans every four control steps using the inverse query.On one RTX A6000, a re-plan takes a median 348 ms with 20 steps or 117 ms with 5, excluding simulation.Source: A IMPLEMENTATION DETAILS · p. 14
  • Within each paired group, executions share a canonical program, leaving contact points as the program input that distinguishes grasp choices.The construction reduces within-group program differences from 28.3 mm to 0.47 mm after phase alignment, while contact points still differ by 39.9 mm.Source: A IMPLEMENTATION DETAILS · p. 14

B PRE-REGISTERED CRITERIA AND OUTCOMES

The preregistered outcomes test held-out execution, editability, and robustness under transfer or disturbance, with results varying by task and criterion. Scope remains strongest for effects with substantial object motion; articulated-object channels extend the representation to some low-motion tasks.

B PRE-REGISTERED CRITERIA AND OUTCOMES

  • The contact-as-affine-weights variant reaches a held-out sum of 44 and cross-arm bowl sum of 85, but only 13 successes at −45° against the criterion of 24, so the paper keeps two contact points.Dense surface support also falls short, with following 0.31, held-out sum 29, and −45° success 14.Source: B PRE-REGISTERED CRITERIA AND OUTCOMES · p. 14
  • For the axis-symmetric corpus test, the less-followed-to-other-orientation density ratio is 1.10 rather than the required maximum of half, so the corpus remains unchanged.Further trainings indicate that the training seed sets the direction of the asymmetry.Source: B PRE-REGISTERED CRITERIA AND OUTCOMES · p. 14
  • The 50k checkpoint reaches the required four-arm bowl sum of 76, but the headline model remains at 30k, where every paper experiment runs.The other judged checkpoints score 73 and 66.Source: B PRE-REGISTERED CRITERIA AND OUTCOMES · p. 14
  • 23 of 60 pushed episodes are completed by re-solving against 6 by replay, while the selected gate completes 40 undisturbed episodes with four outcomes changed.A stricter gate completed 29 pushed episodes but changed seven undisturbed outcomes, so the paper uses the first gate.Source: B PRE-REGISTERED CRITERIA AND OUTCOMES · p. 15
  • All four retrained ablations fall within both preregistered bands for held-out execution and ±45° following, so neither criterion marks a component as needed.The bands were fixed before training: a held-out sum of 25 or less and following of 0.30 or less.Source: B PRE-REGISTERED CRITERIA AND OUTCOMES · p. 15
  • On three held-out Meta-World classes, the reported per-class rule is passed by shelf-place, sweep-into, and ML45 bin-picking; the five-class ML45 and ML10 outcomes are reported per class after sums of 48 against 47.For the joint-state channel, ML45’s five-class sum is 34.0 against 26.7% under transfer, while ML10’s other row is 73 against 51 of 150.Source: B PRE-REGISTERED CRITERIA AND OUTCOMES · p. 15
  • 52 additional successes over the no-program recipe are reported for the RLBench effect-program variant under amendment G’s end-effector-increment configuration.With fallback, the reported margin is 42 for the effect program, versus 13 for the relative variant.Source: B PRE-REGISTERED CRITERIA AND OUTCOMES · p. 15

C SCOPE

  • The joint-position channel extends the program to small articulated effects such as knobs, levers, and drawer handles; on RLBench, the program adds 52 successes to the same recipe without a program.Source: C SCOPE · p. 16

D LIBERO: FURTHER ANALYSIS

Further analysis locates performance variation in task proximity, pickup acquisition, and training seed, while documenting the tested scope of effect programs and grasp editability.

D LIBERO: FURTHER ANALYSIS

  • For low-motion effects, the LIBERO knob moves 0.3 cm; the program completes it 10 of 10 times in three LIBERO-90 scenes but 1 of 30 in the held-out scene.With the object displaced at the start, success is 12 of 15 at 4 cm and 10 of 15 at 8 cm.Source: D LIBERO: FURTHER ANALYSIS · p. 16
  • The held-out bowl and bottle tasks have nearest training effects 4.2 cm and 10.7 cm away, compared with object travels of 25 cm and 33 cm; the knob has matching training effects to one decimal place.The bowl-on-stove result is therefore not a test of a wholly unseen effect magnitude.Source: D LIBERO: FURTHER ANALYSIS · p. 17
  • Pick-and-place failures are 400-step timeouts with objects still at their starts, 26 cm and 33 cm from destination medians, while successes end 2.1 cm and 1.6 cm away.Knob failures stop a median 4.8 cm from target, compared with 2.0 cm for successes.Source: D LIBERO: FURTHER ANALYSIS · p. 17
  • Across three trainings, bottle outcomes vary by seed in 17 episodes, while failures remain timeouts with the bottle 33 to 34 cm from its destination.On the bowl task, trainings agree on all 30 terminal-state-query episodes and 26 of 30 full-program episodes.Source: D LIBERO: FURTHER ANALYSIS · p. 17
  • Across four PEWAM and three W8 trainings, all 12 pairwise comparisons favor PEWAM on contact following, giving the reported one-sided permutation result p=1/35=0.029.The two additional trainings follow at 0.66 and 0.34.Source: D LIBERO: FURTHER ANALYSIS · p. 17

E ABLATION VARIANTS

The ablations show that training queries and test-time choices affect different aspects of PEWAM, while the contact channel and paired corpus contribute to held-out performance and contact editing.

Contact representation, corpus and forward query

  • 40 of 90 held-out episodes succeed with the full recipe, versus 51 without the corpus; on four arms, the main model’s bowl sum of 89 is highest among the five variants.The full recipe’s mean bowl sum across four arms is 83.3; the contact channel alone gives 82.0, neither ingredient 78.5, and the corpus alone 63.5.Source: E ABLATION VARIANTS · pp. 17–18

Which queries to train

  • 29 of 90 terminal-state-query successes follow adding the unconditioned joint query, versus 56 to 58 without it; omitting terminal-state-query training drops effect-program success to 22.Without a program, the unconditioned variant moves the bowl a median 14.5 cm and ends 36 cm from the commanded effect.Source: E ABLATION VARIANTS · p. 18

Test-time changes

  • 27 of 30 Panda episodes succeed with arm-specific joint angles, but the other three arms fall to 1, 0 and 0; advancing the program window by the demonstration clock gives 27.Collapsing the two contact points to their midpoint gives four-arm sums of 77 and 40, versus 89 and 43.Source: E ABLATION VARIANTS · p. 18

Retrained variants

  • 37 of 90 episodes succeed when all streams share one timestep, 35 without the execution stream, and 42 with an eight-step effect window.Training the inverse query alone completes 41 of 90; the displacement-based effect variant completes 1 of 90 held-out episodes, so absolute coordinates remain the headline choice on LIBERO.Source: E ABLATION VARIANTS · p. 18

F EDITING, SAMPLING AND SELECTION

This section tests how programs respond to edits, sampled executions, and perception or replanning choices. Terminal-state edits move placement, while contact-point edits and sampling probe other aspects of program control.

Rotation direction

  • 0.64 and 0.47 of positive rotations versus 0.79 and 0.82 of negative rotations are realised in one training; the other seeds show no asymmetry or reverse it.The authors report that the direction of the asymmetry is set by the training seed.Source: F EDITING, SAMPLING AND SELECTION · p. 18, Table 10 · p. 21

Terminal edits

  • 1.01 (r=0.99) is the realised-versus-commanded shift slope for the terminal-only query, compared with 0.71 (r=0.65) for the full program.At ±10 cm, the terminal-only query puts the bowl beside the stove, where the benchmark predicate fires in 1 and 5 of 30 episodes.Source: F EDITING, SAMPLING AND SELECTION · p. 18, Figure 7 · p. 19

Terminal edits across ingredients

  • 0.80 and 0.81 are the full-program shift slopes for variants trained without either ingredient and with the corpus alone, compared with 0.71 for PEWAM.For the terminal-state-only query, the corresponding slopes are 0.95, 0.90 and 1.01.Source: F EDITING, SAMPLING AND SELECTION · p. 19, Table 11 · p. 21

A reversed program

How much program error the interface absorbs

Live keypoints from rendered RGB-D

  • 16 bowl and 9 bottle episodes succeed when only live keypoints are segmented, versus 27 and 12 from state; extracting the program from pixels gives 25 and 15.On the bowl, median error stays below 1 cm, but the per-episode maximum exceeds 5 cm in 14 (P1) and 11 (P2) of 30 episodes.Source: F EDITING, SAMPLING AND SELECTION · p. 21, Table 13 · p. 22

Ranking sampled executions

  • 44 of 90 episodes succeed when eight execution draws are ranked, versus 47 with the first draw; across 6,142 replanning steps, each draw is chosen 716 to 851 times.The draws are interchangeable along the path being scored; the reported diversity lies in the grasp.Source: F EDITING, SAMPLING AND SELECTION · p. 22

Re-planning interval

  • 44, 40 and 31 of 90 episodes succeed when replanning every 4, 8 and 16 steps on the earlier backbone; the headline model gets 38 at intervals of 8 and 16.Four-step replanning is used for the tables.Source: F EDITING, SAMPLING AND SELECTION · p. 22

G DISTURBANCE

The disturbance experiments examine how pushes affect program tracking and whether the gate prevents false progress. Replanning intervals and the earlier backbone provide additional comparisons.

G DISTURBANCE

  • In four bowl episodes that failed when pushed, the pointer jumped from frame 24 to frames 40–49 and stayed there while the bowl was never lifted.The gate blocks this advance until the object is held, because after motion begins the push can be matched as progress.Source: G DISTURBANCE · pp. 22–23, Figure 8 · p. 23
  • 29 of 60 pushed episodes succeed with stricter gate G2, but it changes seven undisturbed outcomes relative to the main table, exceeding the five-outcome criterion.G2 removes nearest-neighbour matching before engagement; the first gate is retained.Source: G DISTURBANCE · p. 23
  • 20 of 30 pushed bowl episodes succeed with the earlier backbone’s effect program, versus 13 with its inverse-dynamics query; the corresponding bottle counts are 8 and 3.The inverse-dynamics query receives the demonstrator’s end-effector trajectory advanced by the clock.Source: G DISTURBANCE · p. 23

H PUBLISHED BASELINES ON THE HELD-OUT PROTOCOL

  • 18 and 21 of 30 pushed bowl episodes succeed with replanning every 8 and 16 steps, versus 9 every 4 steps; the push lands at step 32 for the longer intervals.The published-baseline appendix evaluates methods on the same held-out episodes and success test, with a 400-step limit.Source: G DISTURBANCE · p. 23, H PUBLISHED BASELINES ON THE HELD-OUT… · p. 23

H.1 UWM AND THE DEMONSTRATOR’S OWN TRAJECTORY

The comparisons span reading demonstrations directly, copying a demonstrator’s trajectory, and evaluating published methods. Additional splits test held-out tasks and objects.

H.1 UWM AND THE DEMONSTRATOR’S OWN TRAJECTORY

  • 19 of 30 is UWM’s success count on the knob task, its one task with an effect essentially represented in pretraining; PEWAM completes 28, 28 and 29 of 30 on the cited tasks.UWM succeeds 18 of 20 on two tasks it was trained on and has seen neither LIBERO-10 nor the seven LIBERO-Goal training tasks.Source: §H.1 · p. 23, Figure 9 · p. 24
  • 30 and 24 are the translated demonstrator-trajectory scores on bowl and bottle, versus 21 and 12 for the effect program.Two same-task LIBERO episodes differ by a median of only 1.7 cm.Source: §H.1 · pp. 23, 25

H.2 INSTANT POLICY

  • 0, 0 and 8 of 30 are Instant Policy’s held-out success counts even after tripling the step budget and providing two demonstrations; at 400 steps it gets 0, 0 and 7.The adapted policy allows only about fifteen queries per episode under the 400-step limit.Source: §H.2 · p. 25

H.3 ATM

  • Neither ATM variant completes a held-out episode, although the checkpoint completes 12 of 70 training-task episodes from language and 3 of 70 with demonstration tracks.The training-task successes support the authors’ conclusion that held-out failures arise on new tasks rather than from a broken checkpoint.Source: §H.3 · p. 25

H.4 ZERO-WAM

  • 0 of 30 episodes succeed on each task for Zero-WAM; no target object moves more than 0.06 cm.The setting differs from its training domain in simulator, robot, cameras and the use of a robot video prompt.Source: §H.4 · pp. 25–26

I FURTHER LIBERO SPLITS

  • 27 of 30 episodes succeed on bowl-on-plate in the second LIBERO-Goal split, where three other tasks are held out.The top-drawer task has median object movements of 9.0 and 14.5 cm under the two queries.Source: I FURTHER LIBERO SPLITS · p. 26, Table 15 · p. 26
  • 0.0 cm is the median object displacement across 90 episodes for the lower pair that never saw LIBERO-Object; the upper pair’s objects finish a median 5 cm from target, versus 17 cm for the lower pair.The split holds out a bottle, box and carton that the weights never manipulated.Source: Table 16 · p. 26, I FURTHER LIBERO SPLITS · p. 26

J META-WORLD

Across Meta-World and RoboCasa, PEWAM uses a demonstration-derived effect program to execute held-out tasks, with results varying by task and evaluation setting. Programs compiled from human videos are also tested on a real Franka setup.

J META-WORLD

  • On ML10, the selected relative variant completes 82 of 150 episodes, against 73 and 51 for the other variants.The absolute-coordinate variant reaches 50.0% on shelf-place, 93.3% on sweep-into and 100% on door-close; the ML45 variant completes drawer-open and lever-pull 30/30 each.Source: J META-WORLD · p. 27, Table 17 · p. 27
  • On ML45, the joint-state recipe scores 41.3% versus 32.0% in the own layout and 34.0% versus 26.7% under transfer.It exceeds every published row on hand-insert at 36.7; the reference also exceeds published rows on box-close and door-lock, while the absolute variant completes door-unlock 26/30.Source: J META-WORLD · pp. 27–28, Table 18 · p. 28

K ROBOCASA

  • RoboCasa tests three held-out tasks selected across skill families using 1,382 released human demonstrations and 120 procedurally generated kitchens.The program retains movable objects and task-named fixture joints, expresses coordinates in the mobile-base frame, and trains on the other 22 tasks.Source: K ROBOCASA · pp. 28–29
  • 69 of 90 RoboCasa episodes succeed with the effect program, versus 44 with the observation-only reference and 50 with terminal state alone.Per task, the program completes 21/30 stove, 24/30 door and 24/30 microwave episodes.Source: K ROBOCASA · p. 29, Table 20 · p. 29

L PROGRAMS FROM REAL HUMAN VIDEO

  • A single click propagates object masks through human-video hand occlusions, with only brief gaps reported in three recordings.The 14 tabletop demonstrations use fixed-camera RGB video and image-coordinate programs; four robot tasks use RGB-D to express programs in the robot base frame.Source: L PROGRAMS FROM REAL HUMAN VIDEO · pp. 29–30, Figure 10 · p. 30
  • The Franka evaluation uses one human demonstration per task and no robot demonstration of the evaluation task.The model is fine-tuned on 156 real-world demonstrations of other tasks; trials begin within 8 cm of the demonstrated object placements.Source: L PROGRAMS FROM REAL HUMAN VIDEO · p. 30

M RLBENCH

On RLBench’s 24 held-out tasks, the increment-action effect-program variant outperforms the observation-only reference, while results depend on the action head and executor behavior. A second-GPU evaluation reproduces the reported findings under its stated comparison rule.

M RLBENCH

  • The RLBench evaluation holds out all 24 tasks and uses one demonstration per episode; training filters leave 39 tasks and 1,136 episodes.The increment action is a 7-D end-effector increment, while replaying held-out demonstrations through that execution path completes 616/720 versus 636/720 for absolute-pose replay.Source: M RLBENCH · pp. 30–31
  • 160 of 720 held-out RLBench episodes succeed with the increment-head effect program, versus 108 with the reference and 123 with the relative program.The 52-episode program-reference margin exceeds the 48 fixed in advance; four tasks account for most of it.Source: M RLBENCH · p. 31, Table 21 · p. 31
  • With the absolute-pose head, the relative program leads at 134 successes, followed by the program at 109 and reference at 93.Instant Policy completes 575 episodes with one demonstration and 630 with two; on CloseJar and OpenJar, the absolute-pose program succeeds 11 versus 9 and 2 versus 0, respectively.Source: M RLBENCH · p. 31, Table 21 · p. 31
  • The increment program has fewer no-movement episodes than the absolute-pose program, 338 versus 398 of 720, and its unreachable-pose share is 0.283 versus 0.303.In absolute-pose variants, tasks with no successes request 6,800–9,400 unreachable poses; the program’s first frame is within a median 1 mm of the scene on 22 tasks.Source: M RLBENCH · p. 32
  • A fallback for rejected poses reduces executed-step holds from 0.283 to 0.185 for the program and from 0.380 to 0.236 for the reference.The reference rises from 108 to 121 successes, while the program rises from 160 to 163; the reported configuration in §4.1 omits this fallback.Source: M RLBENCH · p. 32, Table 22 · p. 32

N REPRODUCTION ON A SECOND GPU GENERATION

  • On the second GPU generation, the findings reproduce under the stated same-machine comparison rule, and the ordering remains the same at 50 episodes per task.At that ceiling, the effect program completes 43, 19 and 1 of 50, versus 50, 35 and 0 for terminal state; borderline episodes can differ between GPU generations.Source: N REPRODUCTION ON A SECOND GPU… · p. 32, Table 23 · p. 32

O QUALITATIVE RESULTS AND VIDEOS

The qualitative material visualizes program compilation, execution, edits and failures, alongside selected comparisons and trajectory predictions. These examples illustrate how programs behave in specific recorded episodes rather than adding aggregate evaluation results.

O QUALITATIVE RESULTS AND VIDEOS

  • The qualitative records include one program executed on four arms, with episode lengths of 100, 99, 103 and 162 steps.The compilation figure shows the recorded demonstration, a robot-removed frame with entity keypoints, and the resulting track, terminal state and contact points.Source: Figure 11 · p. 33, O QUALITATIVE RESULTS AND VIDEOS · p. 33, Figure 12 · p. 34
  • Across 67 forward-query windows, predicted and realised displacement magnitudes correlate at 0.94; moving-object windows have a median error of 1.5 cm.In windows where the object stays still, median error is 0.30 cm.Source: O QUALITATIVE RESULTS AND VIDEOS · p. 34, Figure 19 · p. 39
  • Reversing the program leaves the bowl in place, while the forward program puts it on the stove.Both runs start from the demonstration’s initial state.Source: Figure 16 · p. 37
  • A 5 cm push leaves the effect program successful in 66 steps, while replay after the push ends 8.0 cm from the programmed end after 400 steps.Undisturbed, the program succeeds in 95 steps and open-loop replay in 93.Source: Figure 18 · p. 39
  • Figure 20 compares pixel-derived masks, boxes and object centres with simulator references, then shows execution succeeding from pixels in 80 steps and from state in 318.Across 30 paired episodes, 26 succeed from pixels and 27 from state.Source: Figure 20 · p. 40
  • Figure 22 contrasts selected baseline failures with effect-program successes: the reference retreats from the block, while the relative variant lifts it; Instant Policy hovers over the lid, while the program places it.Each row shows five frames from one rollout, using that method’s own recording.Source: Figure 22 · p. 42

Related papers

Loading 2610.02398v1…