PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Actions: Learning Future Operation Targets for Vision-Language-Action Models

Min Lin, Bokai Zhou, Jianqi Lin, Kaiyuan Yang, Yicheng Chang, Junjie Yu, Xiaofan Liu, Jianhua Han, Rui Li, Shen Zhao, Xiaodan Liang

arXiv:2610.13212v1cs.ROcs.AI

TL;DR

Because action targets describe commanded motion rather than physical outcomes, this paper trains a VLA to predict a future-grounded, action-decodable Effect. On DexJoCo rand-obj, AFCE achieves 50.97% overall success, compared with 39.15% for π0.5 and 35.76% for evaluated UVT.

  • Problem

    Action targets specify commanded motion but leave resulting physical changes implicit, while future or compact targets may not retain both task-relevant transitions and details needed for precise action decoding.

    Source: §1 · p. 1

  • Method

    AFCE learns a temporal Effect by jointly reconstructing executed actions, realized robot responses, and sparse world changes, then predicts and decodes that Effect for control.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    On DexJoCo rand-obj, AFCE achieves 50.97% overall success, compared with 39.15% for π0.5 and 35.76% for evaluated UVT; gains are particularly evident on dual-arm tasks.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    AFCE uses future physical evidence for supervision while requiring only current observations at inference time.

    Source: §6 · p. 10

    AFCE requires temporally aligned actions, robot states, and future images for target construction, and its fixed-size Effect must balance control detail with predictability.

    Source: §5 · p. 9

Abstract

from arXiv · show

Actions specify what a robot should execute, but do not describe what those commands will achieve. Vision-language-action (VLA) policies nevertheless learn primarily by predicting action chunks from images, language, and state. We introduce Action--Future Coupled Effect Learning (AFCE), which makes a future-grounded physical Effect the policy's primary prediction target. An offline tokenizer constructs temporally ordered Effect tokens from demonstrated actions, realized robot responses, and sparse visual transitions. Reconstruction objectives encourage the representation to retain executable detail and observed physical change. A flow-matching VLA predicts Effects from current images, instruction, and robot state, while a state-conditioned decoder converts them into action chunks. Embodiment-specific interfaces allow single- and dual-arm tasks to share the Effect model. On DexJoCo, AFCE achieves $50.97\%$ overall success, improving over $π_{0.5}$ and UVT by 11.82 and 15.21 percentage points, respectively. It leads on all five bimanual tasks, highlighting the value of Effect supervision for coordinated dexterous manipulation. The project page is available at https://afceproject.github.io/AFCE-Web/.

1 INTRODUCTION

The introduction argues that action targets describe commanded motion but leave physical outcomes implicit, creating a challenge for dexterous control. AFCE addresses this by predicting a future-grounded, action-decodable Effect and evaluating whether it supports closed-loop control.

1 INTRODUCTION

  • Dexterous manipulation makes action-only supervision incomplete because contact, hand configuration, and object pose can make similar commands produce different outcomes.Different motions can also achieve the same task-relevant change.Source: §1 · p. 1
  • The target must be predictable before acting, retain task-relevant information about physical transitions, and support recovery of executable continuous actions.The paper frames these as three coupled requirements for a useful policy target.Source: §1 · pp. 1–2
  • AFCE learns a temporal Effect by jointly reconstructing executed actions, realized robot-state changes, and sparse world changes.The action objective retains control detail, while robot-response and world-change objectives ground that detail in observed consequences.Source: §1 · p. 2
  • Future observations construct Effect targets offline, while the deployed VLA predicts them from current images, language, and robot state before a decoder produces action chunks.The figure contrasts this interface with direct-action VLA supervision and world-action models that use future dynamics to support control.Source: §1 · p. 2, Figure 1 · p. 2
  • In mixed single- and dual-arm training, AFCE improves aggregate task success, particularly on dual-arm tasks, while some fine-control tasks remain challenging.The evaluation separately tests representation reconstruction, replay from ground-truth Effects, and predicted-Effect closed-loop control.Source: §1 · p. 2

2 RELATED WORK

Related work spans direct action prediction, future-informed control, and latent policy targets. AFCE builds on these directions with a temporal Effect tied to physical changes and continuous action decoding.

2.1 VLA POLICIES AND ACTION SUPERVISION

2.2 WORLD MODELS AND FUTURE-INFORMED CONTROL

2.3 LATENT ACTIONS AND TRANSFERABLE POLICY TARGETS

  • AFCE extends latent-action targets by combining demonstrated commands, realized robot responses, and sparse visual transitions into temporally ordered Effects.A state-conditioned decoder maps Effects to native actions, while embodiment-specific adapters accommodate different state and action dimensions.Source: §2.3 · p. 3

3 ACTION–FUTURE COUPLED EFFECT LEARNING

AFCE replaces direct action-chunk prediction with a temporal Effect target built from actions, realized robot responses, and visual transitions. A VLA predicts Effects from current observations, while a state-conditioned decoder recovers actions; embodiment-specific interfaces support a shared Effect model.

3 ACTION–FUTURE COUPLED EFFECT LEARNING

  • AFCE trains a VLA to predict a temporal Effect from current images, instruction, and robot state, then decodes that Effect into an action chunk.The Effect is constructed from demonstrated actions, realized robot responses, and sparse visual transitions; future evidence is used for target construction, not deployment.Source: §3 · pp. 3–4

3.1 STAGE I: LEARNING AN EXECUTABLE, FUTURE-GROUNDED EFFECT

  • Stage I fuses action, response, and visual-transition evidence before temporal modeling, then reconstructs actions, robot responses, and visual changes to retain executable detail and physical consequences.The action and response features are aligned with the window's steps; visual evidence may include robot as well as object motion.Source: §3.1 · pp. 4–5

3.2 STAGE II: PREDICTING AND EXECUTING EFFECTS

  • Stage II predicts normalized Effects with a conditional flow-matching VLA and trains the state-conditioned decoder jointly using action-grounded supervision.The clean-endpoint estimate is decoded for action supervision; action-loss gradients reach the policy, and the decoder is used in both supervision and deployment.Source: §3.2 · p. 6

3.3 CROSS-EMBODIMENT FACTORIZATION

  • Separate embodiment-specific state and action interfaces let single- and dual-arm demonstrations share the Effect representation and prediction backbone without requiring identical Effect codes or trajectories.Reconstruction losses are averaged over valid coordinates within each configuration, so larger native action spaces do not receive greater weight solely because of dimensionality.Source: §3.3 · p. 7

4 EXPERIMENTS

AFCE is evaluated on DexJoCo under randomized object placement and table height, with success measured across six single-arm and five bimanual tasks. It leads overall and across all five bimanual tasks, while single-arm results and sensitivity analyses reveal uneven gains and execution limits.

4.1 EXPERIMENTAL SETUP

  • The benchmark comprises six single-arm and five bimanual tasks under rand-obj; each method is evaluated for 50 episodes per task across three environment seeds.The overall score equally averages all 11 tasks; reported variation is across evaluation seeds, using one trained checkpoint per method.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • 50.97 ± 1.65% overall success exceeds π0.5 by 11.82 percentage points (39.15 ± 2.11%) and UVT by 15.21 points (35.76 ± 0.56%).AFCE obtains the highest mean on seven tasks.Source: Table 1 · p. 7, §4.2 · p. 8
  • Ours leads all five bimanual tasks, with a 37.33% mean versus 9.73% for π0.5 and 14.53% for UVT.Microwave reaches 81.33% versus 32.00% for π0.5, while Assembly and Unlock iPad remain low at 9.33% and 7.33%.Source: §4.2 · p. 8
  • The single-arm mean is 62.33%, below π0.5 (63.67%) but above UVT (53.44%), with Pinch Tongs falling to 5.33% versus 43.33% for π0.5.Fold Glasses improves over π0.5 (62.00% versus 38.00%), although UVT performs best there; Figure 4 also shows the seed-0 precision-task failure rate rising from 51.3% to 58.7%.Source: §4.2 · p. 8, Figure 4 · p. 8
  • Bimanual gains contribute approximately 12.55 points to the overall mean, while the 1.34-point single-arm deficit offsets approximately 0.73 points, yielding an 11.82-point net gain.On seed 0, failure rates fall from 76% to 26% in Bimanual and from 95% to 79% in Coordination, but rise in Precision (51.3% to 58.7%) and Tool use (11% to 13%).Source: §4.2 · p. 8

4.3 EFFECT DIMENSIONALITY SENSITIVITY

  • Reducing Effect dimension d from 256 to 64 increases all three decoding errors and lowers total ground-truth-Effect replay from 25/110 to 18/110.The authors retain d = 256, which has the lowest measured reconstruction errors and highest total replay success among these variants; replay remains modest overall.Source: Table 2 · p. 8, §4.3 · pp. 8–9

4.4 EFFECT-CONSTRUCTION ABLATION

  • The full A + S + V tokenizer configuration achieves the highest group means: 62.33% single-arm and 37.33% bimanual, exceeding A-only by 5.11 and 6.40 points.Task-level outcomes are nonmonotonic: full AFCE improves Hammer Nail and Photograph over A-only but reduces Click Mouse from 65.33% to 54.67%, so the ablation does not establish an independent benefit from each added input.Source: Table 3 · p. 9, §4.4 · p. 9

5 LIMITATIONS

AFCE depends on aligned action, robot-state, and future-image data, while its fixed-size Effect must balance executable detail with predictability from current observations.

5 LIMITATIONS

  • Training targets require temporally aligned actions, robot states, and future images.Offline target construction also increases training cost.Source: §5 · p. 9
  • Visual features can miss subtle contacts or be affected by occlusion and viewpoint changes.The Pinch Tongs result illustrates an unresolved precision-control limitation.Source: §5 · p. 9
  • A fixed-size Effect must preserve control detail while remaining predictable from current observations.Prediction errors can further degrade action readout.Source: §5 · p. 9
  • Shared parameters across single- and bimanual interfaces do not establish embodiment-invariant representations.Source: §5 · p. 9

6 CONCLUSION

AFCE predicts and decodes temporal Effects for closed-loop control, using demonstrated actions, realized robot responses, and sparse visual transitions. On DexJoCo under rand-obj, it improves overall success over the evaluated baselines, with stronger gains on bimanual tasks while precision-sensitive single-arm control remains challenging.

6 CONCLUSION

  • 50.97% overall success on DexJoCo under rand-obj exceeds π0.5 at 39.15% and UVT at 35.76%.Source: §6 · p. 10
  • AFCE's gains are strongest on bimanual tasks, while precision-sensitive single-arm control remains challenging.Source: §6 · p. 10
  • AFCE constructs a temporal Effect from demonstrated actions, realized robot responses, and sparse visual transitions, then predicts and decodes it for closed-loop control.Source: §6 · p. 10
  • Future physical evidence supervises AFCE, but inference requires only current observations.Source: §6 · p. 10

A METHOD AND OBJECTIVE DETAILS

AFCE separates construction of a future-grounded Effect target from its prediction and action readout. Demonstration futures shape training targets but are not inputs to the deployed policy.

A METHOD AND OBJECTIVE DETAILS

A.1 EFFECT EVIDENCE AND ARCHITECTURE

AFCE fuses action, robot-response, and visual evidence into a temporal Effect, trains it to reconstruct executable actions and observed changes, then predicts Effects for decoded action readout.

A.1 EFFECT EVIDENCE AND ARCHITECTURE

  • Action, response, and visual branches produce temporally aligned features that QueryFusion and a temporal Transformer combine across the window.The response encodes changes in palm pose, fingertip positions, and finger joints; visual descriptors come from sparse future transitions and may include robot motion.Source: §A.1 · p. 12
  • A state-conditioned decoder maps the Effect to action sequences in each embodiment's native coordinates, using separate output interfaces despite a common Effect shape.Source: §A.1 · p. 12

A.2 LEARNING AN EXECUTABLE EFFECT

  • Tokenizer training reconstructs actions and auxiliary response and visual targets; action losses include grouped Huber errors and an SO(3) orientation discrepancy.Auxiliary weights increase during training.Source: §A.2 · p. 12
  • After tokenizer training, demonstration windows become fixed Effect targets normalized using channel-wise statistics fitted on tokenizer training demonstrations.The tokenizer and normalization statistics remain fixed in the second stage.Source: §A.2 · p. 12

A.3 EFFECT PREDICTION AND ACTION READOUT

  • The policy learns a velocity field in normalized Effect space from current image-language-state context, with a flow-matching objective and a clean estimate used for action supervision.The ground-truth-action auxiliary loss compares decoded actions with standardized demonstrated actions.Source: §A.3 · p. 13
  • Finger-motion supervision penalizes decoded changes across valid adjacent steps, and both action losses update the policy and inference decoder under a flow-time gate and training ramp.At inference, the predicted Effect is denormalized and decoded into a native action chunk; the future-evidence tokenizer is not run online.Source: §A.3 · p. 13

B DATASETS AND EVALUATION PROTOCOLS

AFCE is evaluated on 11 DexJoCo manipulation tasks using demonstrations for training and randomized-object closed-loop rollouts for policy evaluation. It records 50.97% overall success across the benchmark.

B DATASETS AND EVALUATION PROTOCOLS

  • The benchmark covers 11 DexJoCo tasks: six single-arm and five bimanual Panda–Allegro manipulation tasks involving tool use, finger motions, ordered interactions, and coordination.Success is determined by each task’s simulator-defined completion condition.Source: B DATASETS AND EVALUATION PROTOCOLS · p. 13, Table B.1 · p. 14
  • Training uses all 100 recorded episodes per task—1,100 episodes and 523,763 retained frames—with offline Effect targets built from demonstrations.At rollout, the policy receives current images, language, and proprioception; future evidence and privileged simulator object states are unavailable.Source: B DATASETS AND EVALUATION PROTOCOLS · p. 13
  • Closed-loop evaluation randomizes object placement and table height and aggregates 150 rollouts per task, or 1,650 rollouts per method.Ten flow-denoising steps produce each 30-step Effect chunk before decoding and replanning.Source: B DATASETS AND EVALUATION PROTOCOLS · p. 13
  • 50.97% overall success across 1,650 rollouts, with 62.33% on single-arm tasks and 37.33% on bimanual tasks.The single-arm and bimanual aggregates comprise 900 and 750 rollouts, respectively; reconstruction and ground-truth-Effect replay are reported separately as training-data diagnostics.Source: B DATASETS AND EVALUATION PROTOCOLS · p. 14

C ADDITIONAL EXPERIMENTAL RESULTS

Additional experiments examine the structure and action readout of learned Effects, compare illustrative task progress, and document implementation choices. The analyses distinguish in-sample diagnostics from closed-loop policy evaluation.

C ADDITIONAL EXPERIMENTAL RESULTS

  • The reported diagnostics use 110 training-demonstration trajectories and are in-sample, not held-out validation or closed-loop policy evaluations.D0 is the frozen Stage-I decoder; D1 is the jointly adapted Stage-II decoder used at deployment.Source: C ADDITIONAL EXPERIMENTAL RESULTS · p. 14
  • 36.91 mm position error with predicted-E/D1, versus 8.00 mm with GT-E/D0 and 8.30 mm with GT-E/D1, on matched demonstration windows.Predicted-E/D1 also has 7.68 degrees rotation error and 2.97 degrees finger error; these are demonstration-action errors, not task success rates.Source: C ADDITIONAL EXPERIMENTAL RESULTS · p. 14, Figure C.3 · p. 16
  • The GT-E projection shows organization by task and trajectory progress, but its t-SNE layout changes with settings and inter-cluster distances are not quantitative physical or semantic distances.The paired prediction error is measured in the original normalized Effect space, not the two-dimensional projection.Source: C ADDITIONAL EXPERIMENTAL RESULTS · p. 14, Figure C.1 · p. 15, Figure C.2 · p. 15
  • AFCE completes the illustrated Fold Glasses, Hanoi, Microwave, and Photograph interactions, while corresponding π0.5 rollouts fail at placement, contact, or final actuation.Examples include folding and placing glasses in a tray, aligning a Hanoi piece with its peg, and completing microwave and camera interactions.Source: C ADDITIONAL EXPERIMENTAL RESULTS · pp. 14, 16

D.1 IMPLEMENTATION AND HYPERPARAMETERS

  • The evaluated configuration uses a 30 × 256 Effect and separate single-arm and bimanual decoder output interfaces.Stage I has three 64-dimensional evidence branches; the Stage-I decoder is transferred to the policy and trained from update zero for execution.Source: §D.1 · p. 16

D.2 TRAINING AND INFERENCE PROCEDURE

Training proceeds from an offline Effect tokenizer to cached, normalized Effect targets and then a jointly trained policy and decoder. At inference, the policy predicts Effects from noise and decodes them into actions for receding-horizon execution.

D.2 TRAINING AND INFERENCE PROCEDURE

  • The procedure trains the Effect tokenizer on synchronized action, response, and sparse visual evidence, then caches its encoded Effects and freezes the tokenizer and normalization statistics for Stage II.Tokenizer diagnostics use 110 trajectories from training demonstrations and are not held-out validation.Source: §D.2 · p. 16
  • Stage-II training samples noisy Effects for flow matching, decodes a clean Effect estimate, and applies action and finger-difference losses when the flow-time gate is active.The policy and embodiment decoder are updated jointly.Source: §D.2 · p. 18
  • At execution, ten flow steps predict a normalized Effect, which is denormalized and passed with current robot state to the jointly trained decoder for action generation and replanning.The resulting native action chunk is executed with receding-horizon replanning.Source: §D.2 · p. 18
  • Stage-II action-loss weighting is restricted to τ ∈ [0.2, 0.7], with the GT-action coefficient calibrated to an approximately 0.1 weighted-median gradient ratio relative to flow matching.Finger-difference scaling uses the batch RMS of demonstrated adjacent differences, and only pairs with both steps Effect-valid are included.Source: §D.2 · p. 18
  • The policy is evaluated after 60,000 training updates, using one checkpoint per method and 50 episodes per task for each of three environment seeds.Reported standard deviations reflect evaluation-seed variation, not independent training runs.Source: §D.2 · p. 18

Related papers

Loading 2610.13212v1…