PaperlayerView exact revision on arXiv

Source-linked AI summary

ESP: Energy-Score Policy for One-Step Multimodal Action Generation

Lilika Makabe, Heecheol Kim, Yasuyuki Matsushita

arXiv:2610.07696v1cs.RO

TL;DR

Diffusion and flow-matching policies represent diverse valid actions, but repeated sampling requires multiple network evaluations. ESP uses the energy score to map context and noise to multimodal actions in one evaluation, achieving competitive success with substantially lower action-generation latency than flow matching.

  • Problem

    Diffusion and flow-matching policies model diverse action sequences, but their standard sampling procedures require repeated network evaluations to generate each action chunk.

    Source: §I · p. 1

  • Method

    ESP trains a direct context-and-noise-to-action map with the strictly proper energy score, which targets the conditional action distribution rather than a single prediction.

    Source: §I · p. 1, B. Energy-Score Endpoint Objective · p. 3

  • Results

    Across simulated and real-world manipulation tasks, ESP achieves competitive task success with substantially lower action-generation latency than flow-matching policies.

    Source: §I · p. 2, §VI · p. 8

  • Takeaways & Limitations

    The results support energy-score training as an efficient alternative to iterative generative policies for robotic control.

    Source: §I · p. 2, §VI · p. 8

    3.81× faster per optimizer step at K=8, flow matching has lower per-step training cost; ESP’s sample efficiency offsets much of this overhead in the reported experiment.

    Source: §V · p. 8

Abstract

from arXiv · show

Generative action models based on diffusion and flow matching have been increasingly adopted in vision-language-action (VLA) policies for their ability to capture diverse behaviors, including multiple valid action sequences under the same observation and instruction. Their iterative sampling procedures, however, require repeated network evaluations to generate each action chunk, increasing inference latency in closed-loop control. We propose ESP (Energy-Score Policy), a teacher-free approach that maps policy context and noise directly to an action chunk in a single network evaluation. ESP trains the action head with the energy score rather than mean squared error. Whereas squared-error regression targets the conditional mean, the energy score is strictly proper: its expected value is uniquely minimized by the target distribution. This provides a principled objective for learning multimodal action distributions without iterative sampling, with exact recovery at the population optimum when the model can represent the target distribution. Experiments on both simulation and real-world manipulation tasks demonstrate competitive task success with substantially lower action-generation latency than the flow matching baseline. These results support direct distributional learning as an efficient alternative to iterative generative robot policies.

I. INTRODUCTION

Iterative diffusion and flow-matching policies represent diverse actions but require repeated network evaluations. ESP uses the energy score to learn multimodal actions for one-step generation, with competitive task success and lower action-generation latency than flow matching.

I. INTRODUCTION

  • Diffusion and flow-matching policies represent multiple valid action sequences for the same context, but their standard samplers require repeated network evaluations.Those repeated evaluations make action generation time-consuming and motivate one-step generation.Source: §I · p. 1
  • ESP casts conditional action generation as strictly proper distributional regression, using the energy score to learn multimodal actions in one network evaluation.In expectation over the data, the energy score is minimized exactly when generated actions match the demonstration distribution.Source: §I · pp. 1–2, Figure 1 · p. 1
  • Unlike distillation and consistency approaches, ESP trains directly on training data in one stage without a pretrained teacher, trajectory or velocity targets, or an auxiliary critic.Its objective retains a unique distribution-matching optimum while keeping single-evaluation inference.Source: §I · p. 1
  • ESP replaces an existing VLA action head while retaining the pretrained vision-language backbone and conditioning; inference maps one noise draw to one action chunk.The backbone can remain frozen, and inference requires neither multi-step numerical integration nor candidate ranking.Source: §I · p. 1
  • ESP achieves competitive task success on LIBERO with one-step generation, while larger action heads further improve mean success under the same inference-time budget as flow matching.The broader experiments also report competitive success and substantially lower action-generation latency than the flow-matching baseline.Source: §I · p. 2

II. RELATED WORK

Prior generative policies model multimodal actions but often rely on iterative sampling, while one-step approaches use varied objectives. ESP applies the strictly proper energy score to one-step action generation.

II. RELATED WORK

  • Diffusion and flow-matching policies can represent multimodal actions, but require multiple network evaluations to generate action chunks.Examples include Diffusion Policy and VLA policies such as π0 [2], π0.5 [3], GR00T N1 [20], and FLOWER [21].Source: §II · p. 2
  • Distillation, consistency, and drifting methods target shorter sampling, but ideal population drifting loss does not guarantee distribution matching without additional assumptions [10, 23].The described methods include distillation, consistency models, FlowPolicy [8], MeanFlow-VLA [9], and Let It Be Simple [22].Source: §II · p. 2
  • Forward XM updates from the lowest-loss sampled candidate, but its finite-budget hard-min endpoint-distance objective is not generally strictly proper [11].This candidate-selection approach is related to multiple-choice learning, multiple-hypothesis prediction, and IMLE [27].Source: §II · p. 2
  • The energy score combines distances to the observation and between generated samples, and its expected value uniquely favors the data distribution under finite first moments.Unlike Gaussian-kernel MMD objectives, it does not require selecting a kernel bandwidth.Source: §II · p. 2

III. METHOD

ESP directly maps context and noise to action chunks, with the goal of reproducing the conditional action distribution. The section explains why pointwise squared-error regression does not achieve that goal for multimodal actions.

A. Problem Setup and Pointwise Regression

  • ESP parameterizes a direct generator from context and noise to an action chunk, inducing a conditional predictive distribution intended to match the data distribution.The goal is for the induced distribution to reproduce the conditional data distribution for almost every context.Source: A. Problem Setup and Pointwise… · p. 3
  • Adding noise does not ensure distributional consistency: under pointwise squared error, the minimizing generator maps almost every noise sample to the conditional mean.The induced prediction is therefore a point mass, recovering the target only when each context has a unique action.Source: A. Problem Setup and Pointwise… · p. 3

III. METHOD

  • Because valid actions can vary across outputs for a context, learning their distribution requires scoring the predictive distribution as a whole.The method section frames direct generation as an alternative to integrating a learned sampling trajectory.Source: §III · p. 2, A. Problem Setup and Pointwise… · p. 3

B. Energy-Score Endpoint Objective

ESP trains an implicit conditional action distribution with the energy score, which uses generated samples rather than requiring likelihood evaluation. Its strict propriety makes the target conditional distribution the unique expected-score optimum.

B. Energy-Score Endpoint Objective

  • The energy score can train ESP’s implicit predictive distribution using samples, although its density cannot be evaluated.The generator supplies samples from qθ(· | c), so training does not require likelihood evaluation.Source: B. Energy-Score Endpoint Objective · p. 3
  • For 0 < β < 2, the score combines distances from predictive samples to the observed action with a term measuring separation between predictive samples.The samples x and x′ are independent draws from the predictive distribution.Source: B. Energy-Score Endpoint Objective · p. 3
  • Strict propriety makes the expected objective uniquely minimal when the generated conditional distribution matches the data distribution for almost every context.Training minimizes the expected energy score over context-action pairs; each pair supplies one demonstrated action while the model generates multiple predictive samples.Source: B. Energy-Score Endpoint Objective · p. 3

C. Training Loss

Training approximates the energy-score expectations with multiple generated actions for each context-action pair. The TwoBranch figure illustrates a one-evaluation model against a 10-step flow reference, using one seed.

C. Training Loss

  • Training draws K ≥ 2 independent noise samples per context-action pair and compares every generated action with the demonstrated action and with other generated actions.Experiments use β=1 and K=8; inference still draws only one sample.Source: C. Training Loss · p. 3
  • Fig. 3 defines Match↑ as predictions within 3× noise standard deviations of either branch center, comparing one-evaluation generation with the same flow model at 10 Euler steps.The figure uses one seed and is presented as illustrative rather than as a discriminating benchmark.Source: Figure 3 · p. 4

D. Direct One-Step Generation

ESP represents the action endpoint directly, so inference needs one network evaluation rather than numerical integration of a learned field. The approach retains multimodal action generation and can be applied to existing policy architectures.

D. Direct One-Step Generation

  • Unlike flow and diffusion sampling that numerically integrates a learned field over solver steps, ESP directly maps context and noise to the action endpoint in one evaluation.Its number of function evaluations at inference is one, regardless of the training sample count K.Source: D. Direct One-Step Generation · p. 4
  • Multimodality comes from energy-score training, not from producing actions through multiple integration steps.The paper states that this enables one-step generation while achieving performance comparable to iterative generative policies.Source: D. Direct One-Step Generation · p. 4
  • Any conditional network that accepts context and noise can serve as the generator, allowing application to existing policy architectures, including VLA policies.The paper describes replacing the pointwise loss with the energy-score loss without modifying those architectures.Source: D. Direct One-Step Generation · p. 4

IV. EXPERIMENTS

The experiments evaluate one-step multimodal generation on synthetic regression and manipulation tasks. ESP achieves competitive LIBERO success with lower latency than iterative flow, and approaches iterative flow’s distributional fidelity on the controlled regression task.

A. Evaluation Protocol

  • Evaluation spans synthetic known-distribution regression, LIBERO VLA rollouts, and real-robot manipulation, using distributional metrics for known targets and rollout success otherwise.For π0.5 on LIBERO, the study freezes the vision-language backbone and replaces the flow-matching action head with a one-step ESP or XM head.Source: A. Evaluation Protocol · p. 4, B. Controlled Multimodal Regression · p. 5

B. Controlled Multimodal Regression

  • ESP reaches a 96.2% Match rate, versus 91.9% for XM and 97.1% for ten-step flow; one-step MSE and flow reach 0.7% and 4.2%.TwoBranch produces symmetric targets centered at ±√c with equiprobable signs; Match counts predictions within 0.24 of either center.Source: B. Controlled Multimodal Regression · p. 4
  • 8.47× action-expert speedup over native flow matching and 1.40× end-to-end speedup accompany competitive LIBERO success against ten-step flow.The end-to-end comparison includes VLM computation; the action-expert comparison measures the expert alone.Source: B. Controlled Multimodal Regression · p. 5, Figure 4 · p. 5, Table I · p. 5

D. Action-Expert Capacity

The expanded ESP action expert improves or matches LIBERO success while retaining one-step generation. In fixed-initialization trials, ESP also produces diverse successful trajectories and succeeds in more trials than XM or flow.

D. Action-Expert Capacity

  • The expanded action expert increases FFN width from 4096 to 73728 while retaining the frozen backbone, action-expert hidden size, attention heads, KV cache structure, horizon, and output shape.The expanded model has 7.204B parameters, including 4.280B on the action side and an unchanged 2.923B vision-language backbone.Source: D. Action-Expert Capacity · p. 5
  • 0.9371 ± 0.0036 mean success rate for expanded ESP compares with 0.9179±0.0140 for baseline ESP, with higher or comparable success on all four suites and lower SD.Both models use NFE=1; the expansion widens the action-expert FFN while keeping the vision-language backbone frozen.Source: D. Action-Expert Capacity · pp. 5–6
  • 162 of 192 trials succeed with ESP, compared with 127 for XM and 155 for flow; mean successful-trajectory spread is 2.122 cm for ESP and 1.965 cm for XM.The reported spread difference is 0.157 cm; ESP’s successful trajectories are more diverse than XM’s and similar in diversity to flow matching.Source: D. Action-Expert Capacity · p. 6, Figure 5 · p. 6, Table III · p. 6

F. Real-World Manipulation with GR00T

The real-world evaluation uses GR00T as the backbone VLA for two manipulation tasks.

F. Real-World Manipulation with GR00T

Environment DRAWER CLOSING PICK & PLACE

The real-world experiments evaluate drawer closing and pick-and-place with camera, language, and robot-state inputs. ESP reports higher overall success across the three tasks, while the supplied training-progress comparison shows similar LIBERO-SPATIAL success with fewer steps.

Results

  • ESP achieves overall success-rate gains across all three tasks and outperforms the baseline in real-world experiments.Figure 6 includes successful drawer-closing and pick-and-place rollouts; its table reports successful trials out of total trials.Source: Figure 6 · p. 7, Environment DRAWER CLOSING PICK & PLACE · p. 7

Environment and data

  • Each real-world task uses approximately 30 to 40 human-teleoperated demonstrations, with RGB images, a language instruction, and robot state as policy inputs.The images come from base-mounted and wrist-mounted Intel RealSense D435 cameras.Source: Environment DRAWER CLOSING PICK & PLACE · p. 7

Policy and baselines

  • ESP and Flow start from GR00T-N1.7-3B, freeze the vision-language backbone, and fine-tune the pretrained action head to predict and execute eight-action chunks.Flow uses its native objective with the default four-step sampler (NFE=4).Source: Environment DRAWER CLOSING PICK & PLACE · p. 7

Evaluation

  • Drawer success requires closing within 5 mm; pick-and-place success requires stacking without toppling, and rollouts can fail for excessive contact force or exceeding 180 s.Pick-only success is separately counted when the target block is lifted from the table.Source: Environment DRAWER CLOSING PICK & PLACE · p. 7

Training progress

  • 93.0% success at 15k steps (4.78h) for ESP is comparable to 93.5% for flow matching at 60k steps (4.98h) on LIBERO-SPATIAL.Flow matching processes optimizer steps 3.81× faster; the comparison is shown for K=8.Source: Figure 7 · p. 7

V. LIMITATIONS

ESP reduces action-generation latency through single-evaluation inference, while its energy-score training incurs greater per-step computation and memory costs. In the conclusion, the paper argues that wider action heads may use a practical control-frequency budget more effectively than iterative generation.

V. LIMITATIONS

  • ESP training costs more computation and activation memory because each update evaluates K noise-conditioned candidates and O(K^2) pairwise sample distances.For K=8 in π0.5 training, each flow-matching optimizer step runs about 3.81× faster than an ESP step.Source: §V · p. 8
  • Sample efficiency partly offsets the per-step overhead: ESP reaches comparable LIBERO-SPATIAL success with about a quarter as many optimizer steps and expert actions, in nearly the same wall-clock time.The comparison is 93.0% at 15k steps and 4.78h for ESP versus 93.5% at 60k steps and 4.98h for flow matching; reducing training overhead remains future work.Source: §V · p. 8

VI. CONCLUSION

  • Across simulated and real-world manipulation tasks, ESP achieves competitive task success while substantially reducing action-generation latency versus iterative flow-matching policies.ESP generates an action chunk in one forward pass; flow matching requires iterative sampling.Source: §VI · p. 8
  • With practical control-frequency constraints, the paper argues that increasing one-step action-head capacity may use compute more effectively than repeated evaluations of a smaller model.The conclusion connects this resource-allocation view to the improved performance of wider one-step action heads.Source: §VI · p. 8

Related papers

Loading 2610.07696v1…