PaperlayerView exact revision on arXiv

Source-linked AI summary

Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition

Jiahang Cao, Yize Huang, Hanzhong Guo, Rui Zhang, Mu Nan, Weijian Mai, Jiaxu Wang, Hao Cheng, Jingkai Sun, Gang Han, Wen Zhao, Qiang Zhang, Yijie Guo, Qihao Zheng, Chunfeng Song, Xiao Li, Ping Luo, Andrew F. Luo

arXiv:2510.01068v2cs.ROcs.LG

TL;DR

Costly interaction data and model-capacity limits constrain advances in diffusion- and flow-based robot policies. GPC combines pretrained policy scores with convex weighting and test-time search, and the paper reports consistent gains across benchmarks and real-world settings.

  • Problem

    Diffusion- and flow-based robot policies face scaling constraints from model capacity and the costly interaction datasets needed to realize larger models' potential.

    Source: §1 · pp. 1–2

  • Method

    GPC combines distributional scores from pretrained policies by convex weighting and test-time search, without additional model training.

    Source: §1 · p. 2

  • Results

    Across diverse benchmarks and real-world setups, GPC confirms consistent performance gains over single-policy baselines.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    GPC offers a simple, versatile way to improve robotic control using existing policies without additional training resources.

    Source: §7 · p. 10

    Fixed discretization in test-time weight search may overlook optimal values.

    Source: §7 · p. 10

Abstract

from arXiv · show

Diffusion-based models for robotic control, including vision-language-action (VLA) and vision-action (VA) policies, have demonstrated significant capabilities. Yet their advancement is constrained by the high cost of acquiring large-scale interaction datasets. This work introduces an alternative paradigm for enhancing policy performance without additional model training. Perhaps surprisingly, we demonstrate that the composed policies can exceed the performance of either parent policy. Our contribution is threefold. First, we establish a theoretical foundation showing that the convex composition of distributional scores from multiple diffusion models can yield a superior one-step functional objective compared to any individual score. A Gr\"onwall-type bound is then used to show that this single-step improvement propagates through entire generation trajectories, leading to systemic performance gains. Second, motivated by these results, we propose General Policy Composition (GPC), a training-free method that enhances performance by combining the distributional scores of multiple pre-trained policies via a convex combination and test-time search. GPC is versatile, allowing for the plug-and-play composition of heterogeneous policies, including VA and VLA models, as well as those based on diffusion or flow-matching, irrespective of their input visual modalities. Third, we provide extensive empirical validation. Experiments on Robomimic, PushT, and RoboTwin benchmarks, alongside real-world robotic evaluations, confirm that GPC consistently improves performance and adaptability across a diverse set of tasks. Further analysis of alternative composition operators and weighting strategies offers insights into the mechanisms underlying the success of GPC. These results establish GPC as a simple yet effective method for improving control performance by leveraging existing policies.

1 INTRODUCTION

Diffusion- and flow-based robot policies face scaling limits from model capacity and costly interaction data, while fine-tuning and reinforcement learning bring additional costs. GPC instead composes pretrained policies at inference time, with task-dependent weights, and can outperform its parent policies.

1 INTRODUCTION

  • Policy scaling is constrained by representational limits and the costly interaction data needed to realize gains from larger architectures.Supervised fine-tuning requires expensive data collection, while reinforcement learning adds reward engineering and extensive online interaction.Source: §1 · pp. 1–2
  • GPC combines pretrained policy scores through a convex mix and test-time search, without retraining, and can exceed either parent policy.The framework can combine diffusion- or flow-based, VA or VLA policies and different sensory inputs.Source: §1 · p. 2
  • The best weighting is task-dependent, even for a fixed set of parent policies.Figure 1 also depicts task-varying optimal weights.Source: §1 · p. 2, Figure 1 · p. 2
  • Across simulation and real-world evaluations, GPC consistently outperforms single-policy baselines.The paper also analyzes composition operators and weighting choices.Source: §1 · p. 2

2 RELATED WORK

Prior work explores compositional generative models and diffusion policies in robotics, while GPC extends policy composition across heterogeneous architectures and modalities. The paper distinguishes its approach from PoCo by searching task-dependent policy weights and evaluating broadly adopted simulation environments.

2 RELATED WORK

  • Composability combines components or distributions into a unified representation while preserving properties of the individual elements.Prior approaches include energy-based composition for visual generation and superposition of diffusion models.Source: §2 · p. 2
  • Compositional methods also extend beyond visual generation, including multi-agent debate and test-time verification for language generation.These are examples cited in the paper's related work.Source: §2 · p. 3
  • Diffusion policies represent expressive action distributions, with robot-learning work spanning multimodal, point-cloud, and vision-language-action policies.The paper notes that GPC can also be applied to flow-based policies.Source: §2 · p. 3
  • Unlike PoCo, which does not explore policy weights or validate in widely adopted simulation environments, GPC composes VA and VLA models across visual modalities and searches task-dependent weights.Related compositional diffusion work also addresses planning, constraint satisfaction, and motion planning.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe diffusion sampling as iterative denoising driven by a learned score and connect score functions to energy-based models. They also note that sampler coefficients depend on the noise schedule and solver.

3 PRELIMINARIES

  • Diffusion sampling iteratively denoises random noise, using a learned score in an update whose coefficients depend on the noise schedule and solver.The stated update accommodates DDPM, DDIM, and ODE/SDE-based solvers through coefficient choices.Source: §3 · p. 3
  • In energy-based models, the gradient of an energy function plays a role analogous to a diffusion score, and compositional EBMs sum energy contributions.Source: §3 · p. 3

4 THEORETICAL ANALYSIS OF CONVEX SCORE COMPOSITION

The theory establishes that convexly combining score estimators can reduce functional-level error and that sampling stability carries this improvement into generated trajectories. Together, these results provide a theoretical basis for policy composition, while practical use requires test-time weight search.

4 THEORETICAL ANALYSIS OF CONVEX SCORE COMPOSITION

  • Convex combinations of pretrained policy scores can reduce functional-level score error, and sampling stability bounds trajectory error through score error.This analysis provides the mathematical basis for policy composition.Source: §4 · p. 4

4.1 FUNCTIONAL-LEVEL IMPROVEMENT

  • A convex combination of two score estimators can achieve error no greater than the better individual estimator, and usually achieves strictly lower error when their errors differ.The combination can cancel errors because the estimators may deviate from the true score in different ways.Source: §4.1 · p. 4

4.2 SYSTEM-LEVEL STABILITY

  • Sampling dynamics are stable: terminal trajectory error is bounded by integrated score error and grows at most exponentially with the Lipschitz constants.The stability result relates score-estimation error to the deviation between approximate and oracle trajectories.Source: §4.2 · p. 4

4.3 IMPLICATIONS FOR POLICY COMPOSITION

  • When a convex score mixture improves functional-level error, stability propagates that advantage along the trajectory and tightens the sampling-error bound.The theory supports system-level gains from composition.Source: §4.3 · p. 5
  • The theory establishes that effective weights exist, but finding them analytically is intractable, so GPC uses test-time search.This search identifies effective weighting strategies in practice.Source: §4.3 · p. 5

5 OUR METHOD: GENERAL POLICY COMPOSITION

General Policy Composition (GPC) combines pretrained policies through compositional diffusion models, convex score composition, and test-time search. The framework also considers logical OR and AND composition; the supplied table captions report improvements on Robomimic, PushT, and RoboTwin.

5.1 COMPOSITIONAL DIFFUSION MODELS

  • Under an independence assumption, compositional diffusion models represent a trajectory distribution conditioned on multiple concepts, such as different visual modalities or model architectures.The formulation connects a joint trajectory distribution to its component conditional distributions.Source: §5.1 · p. 5, §5.2 · p. 6
  • Figure 3 shows GPC succeeding when one diffusion-policy component fails and performing better when both components work.The figure caption describes visualizations of different diffusion policies and their composition.Source: Figure 3 · p. 6

5.2 GENERAL POLICY COMPOSITION

  • GPC composes score functions from conditional policies by convex combination, with weights assigned to the concepts contributing to the trajectory estimate.The method uses test-time search over weights and returns the trajectory associated with the best evaluated success rate.Source: §5.2 · p. 6, §5.1 · p. 6

5.3 GPC WITH SUPERPOSITION

  • The framework also instantiates logical OR as time-dependent softmax weighting and logical AND as weights chosen to enforce agreement between policies.The AND formulation solves a linear system for weights that make policy scores consistent.Source: §5.3 · p. 6
  • Up to 7% improvement in success rate is reported for GPC on six diverse RoboTwin bimanual manipulation tasks; Table 1 also reports average improvement over base policies on Robomimic and PushT.The supplied Table 1 caption gives no numerical size for its average improvement.Source: Table 2 · p. 7, Table 1 · p. 7

6 EXPERIMENT

The experiments evaluate GPC across simulation benchmarks, real-world tasks, policy types, and composition settings. Across these settings, composition improves performance, while results also show that gains depend on policy weights and come with added computation.

6.1 EXPERIMENT SETTINGS

  • Experiments cover Robomimic, PushT, RoboTwin, and four real-world tasks, using pre-trained policies and average success rates over 200 rollouts (100 for RoboTwin).The compared baselines include vision-action and vision-language-action policies; tested compositions search weights from 0.0 to 1.0 in 0.1 steps.Source: §6.1 · p. 7

6.2 MAIN RESULTS: GPC ACROSS ARCHITECTURES AND MODALITIES

  • Composing policies across architectures and modalities improves results, including +5.51% for Florence-D+DP and +32% over RDT for RDT+DPpcd.The passage also reports +2.22% for DP+MP and an average SR increase from 0.46/0.65 to 0.70 (+5%) for DPimg+DPpcd.Source: §6.2 · p. 8
  • Up to +7.55% on Robomimic & PushT, +7% on RoboTwin, and +10% in real-world tasks: GPC consistently improved performance.The reported increases are average gains across the named settings.Source: §6.2 · p. 8

6.3 INFLUENCE OF WEIGHT CONFIGURATIONS ON GPC PERFORMANCE

  • With moderate-accuracy parents, Empty Cup Place rises from 0.42 and 0.62 to 0.86 (+24%); when one parent is much weaker, GPC can miss the stronger policy’s score.For Pick Apple Messy, DPpcd scores 0.26, DPimg scores 0.05, and GPC peaks at 0.25; in Dual Bottles Pick (Easy), GPC reaches 0.85 with w1=0.8 versus DPimg’s 0.77.Source: §6.3 · p. 9

6.4 TIME AND COMPUTATIONAL EFFICIENCY OF GPC

  • Restricting weight search to wstrong in [0.6, 0.9] reduces the reported search from about 2.5 hours to about 1 hour, while per-chunk latency rises from 0.09 s to 0.13 s.GPC adds both weight-search evaluations and inference cost from querying multiple policies per action chunk.Source: §6.4 · p. 9

6.5 COMPREHENSIVE ANALYSIS OF GPC EFFECTIVENESS

  • Figures 4 and 5 show GPC combining modality-specific distributions and refining coverage across architectures, with more coherent, concentrated execution-time samples than baselines.The execution-time analysis describes the baseline samples as scattered or noisy, particularly in later stages, and says GPC mitigates error accumulation.Source: Figure 4 · p. 8, Figure 5 · p. 8, §6.5 · p. 10
  • Logical AND composition raises DP+MP success rate to 64.92 (+25.73%), while logical OR raises Florence-D+DP to 77.59 (+16.34%).The passage notes that superposition does not directly apply to flow-based models and requires recomputing weights at every step.Source: Table 4 · p. 9, §6.5 · p. 10
  • GPC also yields clear gains with mismatched action-chunk lengths and inference steps, and three-policy composition matches or improves on the best two-policy configuration.For mismatched chunks, scores are combined over the overlap and the longer policy supplies the remaining tail.Source: §6.5 · p. 10, Table 8 · p. 10, Table 9 · p. 10

7 DISCUSSION

The discussion concludes that GPC improves robotic control by composing pre-trained policy scores, with theoretical analysis and experiments supporting gains across benchmarks and real-world setups. It identifies fixed-granularity test-time weight search and the computation needed to scale beyond a few policies as limitations.

7 DISCUSSION

  • GPC’s test-time weight search uses fixed discretization that may miss optimal values, and scaling beyond the mainly studied dual/triple-policy setting increases computation.The authors suggest adaptive or automatic search, feature sharing, or compact representations as possible directions.Source: §7 · p. 10
  • The conclusion presents GPC as a training-free framework that combines pre-trained policy scores, with theory and experiments supporting step-wise, trajectory-level, and empirical gains.The stated scope includes diverse benchmarks and real-world setups.Source: §7 · p. 10

APPENDIX

  • The appendix includes proofs of the single-step convex-improvement and score-to-sample stability results, plus assumptions and preliminary facts for the analysis.Its listed material also includes the proof that GPC tightens the terminal bound and a section connecting the propositions and corollary.Source: APPENDIX · p. 23, A ASSUMPTIONS, NOTATION, AND… · p. 24

B PROOF OF PROPOSITION 4.1 (SINGLE-STEP CONVEX IMPROVEMENT)

The section shows that the mean-squared error of a convex blend of two score estimators is a convex quadratic in the mixing weight. An interior minimizer can outperform both estimators, but strict improvement is not guaranteed when the best weight lies at a boundary.

B PROOF OF PROPOSITION 4.1 (SINGLE-STEP CONVEX IMPROVEMENT)

  • The blended estimator’s mean-squared error is a convex quadratic in its weight, strictly convex unless the estimators share both biases and noises.The estimator errors are decomposed into deterministic bias and zero-mean random noise; the noise term represents diffusion randomness, while the deterministic case corresponds to flow matching.Source: B PROOF OF PROPOSITION 4.1… · pp. 25–26
  • With unbiased estimators, the optimal blend strictly improves on both endpoints whenever their noise terms are not perfectly correlated.The unbiased-case analysis gives a unique minimizing weight when the quadratic coefficient is positive.Source: B PROOF OF PROPOSITION 4.1… · p. 26
  • When the unconstrained optimum lies strictly inside [0, 1], the minimum error is below both endpoint errors unless the corresponding endpoint gap vanishes.If the optimum falls outside the interval, the constrained optimum is at an endpoint and cannot strictly improve on both single estimators.Source: B PROOF OF PROPOSITION 4.1… · p. 26
  • If one estimator significantly outperforms the other, the optimal weight can lie on the boundary, recovering the better single-estimator performance without strict improvement.In the deterministic biased case, identical biases also make the error constant across all weights.Source: B PROOF OF PROPOSITION 4.1… · pp. 26–27

C PROOF OF PROPOSITION 4.2 (SCORE-TO-SAMPLE STABILITY)

The proof relates score-estimation error to the deviation between generated trajectories. Under its stated assumptions, a Grönwall argument bounds trajectory error using the time-integrated score error.

C PROOF OF PROPOSITION 4.2 (SCORE-TO-SAMPLE STABILITY)

  • The stability result compares oracle-score and estimated-score trajectories from the same initial condition and bounds their deviation under assumptions (A1)–(A2).Sampling is represented as an ODE whose vector field depends on the score estimate.Source: C PROOF OF PROPOSITION 4.2… · p. 27

C. GR ¨ONWALL (INTEGRATING FACTOR) AND THE PATHWISE BOUND

A. ABSOLUTE CONTINUITY AND THE ERROR DIFFERENTIAL INEQUALITY

D. EXPECTATION AND A READABLE UPPER BOUND

  • Taking expectations gives a readable expected sampling-error bound; a uniform score-error bound then yields Proposition 4.2’s stated result.The proof uses Tonelli’s theorem, Cauchy–Schwarz, and Jensen’s inequality to obtain the cleaner bound.Source: D. EXPECTATION AND A READABLE UPPER… · pp. 28–29

D PROOF OF COROLLARY 4.1 (GPC TIGHTENS THE TERMINAL BOUND)

The corollary links lower integrated score error to a tighter theoretical bound on terminal sampling error. The surrounding analysis explains the conditions, stability argument, and policy-composition choices behind that link.

D PROOF OF COROLLARY 4.1 (GPC TIGHTENS THE TERMINAL BOUND)

  • A convex composite score has a strictly reduced theoretical terminal-error bound when its integrated mean-squared score error is below that of each individual estimator.The bound is strictly increasing in integrated score error.Source: D PROOF OF COROLLARY 4.1 (GPC TIGHTENS… · p. 29

E.1 NORM DERIVATIVE INEQUALITY; E.2 INTEGRATING FACTOR; E.3 TONELLI, CAUCHY–SCHWARZ, AND JENSEN

  • The stability proof turns a differential inequality for trajectory deviation into a pathwise bound, then derives an expected bound using classical inequalities.The argument assumes the score error is bounded and the trajectories share initial conditions.Source: §E.1 · p. 29, §E.2 · p. 30

F HOW PROPOSITIONS AND COROLLARY FIT TOGETHER

  • The proof strategy links single-step score improvement to trajectory-level guarantees: Proposition 4.1 lowers score error, and Proposition 4.2 transfers that reduction to the sampling bound.The corollary combines these results once the functional-level inequality is strict.Source: F HOW PROPOSITIONS AND COROLLARY FIT… · p. 30

G THE FLEXIBILITY OF GPC WITH ANY PREDICTION TYPES

  • GPC converts heterogeneous prediction outputs to a common score or noise representation before convex composition, allowing different training parameterizations to be combined.The resulting composed score can be used with a standard ODE or SDE solver.Source: G THE FLEXIBILITY OF GPC WITH ANY… · pp. 31–32

H.1 THEORETICAL PERSPECTIVE: WHY CONVEX SCORE COMPOSITION CAN BE BETTER THAN INDIVIDUAL POLICIES

  • The theoretical account says an appropriate convex weight can improve per-step score accuracy, with sampling stability carrying the advantage along the denoising trajectory.The claim concerns the existence of a suitable weight at each diffusion timestep.Source: §H.1 · p. 33

H.2 EMPIRICAL PERSPECTIVE: WHAT DRIVES LARGER GAINS FOR SOME COMBINATIONS

  • RGB and point-cloud policies offer complementary scene cues, while different architectures can supply distinct inductive biases and error patterns for composition.The analysis cites DP+DP3 for modality diversity and π0 with Flow Policy for architectural diversity.Source: §H.2 · p. 33
  • Composition gains depend on task-specific base-policy strengths; the analysis reports best performance when the stronger policy receives more weight.Naive averaging or an unbalanced weight can let a weaker policy reduce overall performance.Source: §H.2 · p. 33

H.2 EMPIRICAL PERSPECTIVE: WHAT DRIVES LARGER GAINS FOR SOME COMBINATIONS; I.1 ROBOMIMIC; I.2 PUSHT

  • Weights are selected through sweeps because the optimal weight is task-dependent, and improved strategies for approximating it remain future work.The cited Robomimic and PushT setup evaluates each benchmark across 200 rollouts.Source: §H.2 · p. 33, §I.1 · p. 34, §I.2 · p. 34

I.3 ROBOTWIN

The RoboTwin experiments use selected tasks from versions 1.0 and 2.0, with aligned policy training settings for composition. The section also describes real-world camera and robot setup and notes that GPC can accommodate heterogeneous prediction types.

I.3 ROBOTWIN

  • The evaluation uses seven selected RoboTwin 1.0 tasks and six RoboTwin 2.0 tasks, with success rates measured over 100 rollouts per task.RoboTwin is described as a dual-arm manipulation benchmark combining teleoperated demonstrations and synthetic data.Source: §I.3 · p. 34
  • For RoboTwin 1.0, RGB-image and point-cloud diffusion policies use unified DDPM settings; RoboTwin 2.0 policies align their prediction setting with RDT.The point-cloud policy omits point-cloud color in RoboTwin 1.0.Source: §I.3 · p. 34
  • During training, VLA prompts include paraphrases, and test-time prompts use unseen paraphrases with the same schema.The passage gives example prompt phrasings for the same task schema.Source: §I.3 · pp. 34–35

I.4 REAL-WORLD EXPERIMENTS

  • The real-world setup uses RGB and depth cameras with Piper in master–slave teleoperation; the text reports GPC outperforms base policies and produces better trajectories.The stated cameras are Intel RealSense D435 for RGB and L515 for depth; Fig. 8 is identified as tracking results for placing bottles.Source: §I.4 · pp. 35–36, Figure 8 · p. 35

I.5 NOTATION FOR GPC FLEXIBILITY

  • Diffusion prediction types can be combined in GPC, provided the denoising process is aligned for consistency.The section points to guidance on reconciling heterogeneous prediction types.Source: §I.5 · p. 36

J ADDITIONAL EXPERIMENTAL RESULTS

The appendix supplies detailed composition results across simulation and real-world tasks, with full comparisons of operators and weights. It highlights higher real-world success rates and performance sensitivity to composition choices.

Robomimic and PushT

RoboTwin

  • RoboTwin 2.0 tables provide base-policy and composition comparisons across six tasks and tested weight settings.The listed tasks include Open Laptop, Place Burger, Put Object Cabinet, Hanging Mug, Stack Bowls Three, and Turn Switch.Source: J ADDITIONAL EXPERIMENTAL RESULTS · p. 36

Real-world Experiments

Summary

  • The extended results compare composition operators and weightings, documenting both average improvements and sensitivity to those choices.The appendix presents the results as a reference for understanding GPC's empirical behavior.Source: J ADDITIONAL EXPERIMENTAL RESULTS · p. 36

K VISUALIZATION ON ROBOT TASKS

This appendix section illustrates robot tasks, expands the related-work review, and outlines future directions for policy composition.

K VISUALIZATION ON ROBOT TASKS

L.1 COMPOSITIONAL GENERATIVE MODELING

  • Compositional generative modeling is motivated by modularly combining simpler factors, but fixed composition structures and theoretical gaps remain challenges.The review connects this approach with data efficiency and generalization to novel tasks and distributions.Source: §L.1 · p. 50

L.2 DIFFUSION MODELS IN ROBOT LEARNING

  • The diffusion-policy literature spans compact task-specific visuomotor models and larger policies built on foundation models or broad robot datasets.The reviewed work also includes flow-matching action policies and methods combining language reasoning with action generation.Source: §L.2 · pp. 51, 53

L.3 NON-DIFFUSION-BASED MODELS IN ROBOT LEARNING

  • Non-diffusion robot policies use sequence modeling, spatial grounding, data scaling, and hierarchy for manipulation, while navigation examples decompose intent into linguistic subgoals and low-level control.The review includes both manipulation and navigation methods.Source: §L.3 · pp. 53–54

M FUTURE WORK

  • Future work identifies fixed test-time weight discretization, the cost of adding policies, and stronger composition operators as open methodological directions.The authors also suggest extending composition beyond diffusion policies and exploring navigation tasks, while noting experiments focus on robotic VA and VLA manipulation.Source: M FUTURE WORK · p. 55

Related papers

Loading 2510.01068v2…