Source-linked AI summary
Conditional Trajectory Peaks: Single-Pass Multimodal Policies over Action Chunks
Di Wu, Rongtian Shen, Ping Liu, Xuhua Chen, He Zheng, Lingfeng Zhang, Tao Zhang
TL;DR
Robotic imitation must represent multiple executable futures for the same observation, while iterative generation adds latency and candidate diversity alone does not ensure continuity across replanning. CTP predicts complete action-chunk candidates in one pass, using DAPS for peak specialization and ETBT for cross-chunk consistency. On D3IL, it reports 100.0% success on Avoiding, 79.72% on Aligning, and 84.44% on Sorting-2.
Problem
Similar observations can admit distinct feasible futures, but squared-error behavioral cloning favors a conditional mean that may be infeasible, and iterative multimodal generation adds replanning latency.
Source: §I · p. 1Method
CTP predicts trajectory-level peaks over complete action chunks in one pass, using DAPS for specialization and ETBT to propagate and revise beliefs across replanning cycles.
Source: §I · p. 2Results
CTP reports 100.0% success on D3IL Avoiding, 79.72% on Aligning, and 84.44% on Sorting-2, with Avoiding exceeding the prior best result of 95.0%.
Source: C. Results on Key D3IL Tasks · p. 5Takeaways & Limitations
Experiments on Push-T, D3IL, and LIBERO show competitive multimodal closed-loop performance with substantially reduced online inference overhead versus iterative action generation.
Source: §V · p. 8
Abstract
from arXiv · showhide
Multimodal imitation learning requires diverse executable futures under the same observation and consistent behavior across replanning cycles. We present Conditional Trajectory Peaks (CTP), a single-pass policy framework that jointly predicts complete action-chunk candidates, probability masses, and trajectory scales. Distribution-Aware Peak Specialization (DAPS) specializes trajectory peaks using trajectory-level posterior responsibilities and mass- and scale-modulated overlap constraints. Evidence-Gated Trajectory Belief Transport (ETBT) maintains cross-chunk consistency through geometric correspondence between exchangeable candidate sets, while allowing current policy evidence to override historical constraints. CTP achieves a coverage score of 91.40% on Push-T; success rates of 100.0%, 79.72%, and 84.44% on D3IL Avoiding, Aligning, and Sorting-2, respectively. On LIBERO, CTP achieves an average success rate of 97.25%. In real-world dual-arm experiments, CTP preserves both placement modes in a two-plate task, succeeding in all 50 trials. On bottle uprighting and pen placement into a holder, it maintains success rates comparable to $\pi_{0.5}$ while reducing policy inference latency from 218.24 ms to 75.80 ms. These results demonstrate that single-pass trajectory modeling can combine multimodal behavior, closed-loop consistency, and efficient inference.
I. INTRODUCTION
Multimodal policies must represent distinct feasible futures while avoiding iterative-generation latency, training-time mode collapse, and behavioral switching across replanning cycles. CTP addresses these challenges with single-pass trajectory prediction, DAPS peak specialization, and ETBT cross-chunk belief transport.
I. INTRODUCTION
- Diffusion and flow-matching policies model multimodal actions but typically require repeated network evaluations, increasing latency and computation during high-frequency replanning.The introduction asks whether trajectory-level multimodal actions can instead be generated with one network forward pass per replanning step.Source: §I · p. 1
- Existing batched multi-hypothesis policies, [7] do not guarantee effective multimodality, since permutation symmetry can collapse candidates onto one dominant region.Single-pass policies need data-driven specialization that allocates probability to distinct valid modes and reduces mass on unsupported redundant candidates.Source: §I · p. 1
- Cross-chunk mode stitching occurs when probability changes switch between incompatible routes, joining individually feasible chunks into discontinuous or unsuccessful behavior.Candidate indices cannot identify corresponding behaviors across replanning because finite mixture components are permutation-symmetric.Source: §I · p. 1
- CTP predicts multimodal trajectory-level peaks over complete action chunks with one shared network forward pass and NFE = 1.It jointly predicts multiple trajectory candidates, probabilities, and scales.Source: §I · p. 2
- DAPS mitigates mode collapse through posterior responsibilities and probability- and scale-aware inter-peak regularization, without clustering or predefined mode assignments.This specializes trajectory peaks while reducing redundant candidate probability mass.Source: §I · pp. 1–2
- ETBT propagates beliefs across replanning cycles and adaptively revises them using current policy evidence to improve cross-chunk behavioral consistency.This complements candidate diversity by addressing behavioral continuity across decisions.Source: §I · pp. 1–2
II. RELATED WORK
Related work covers multimodal imitation policies, single-pass multi-hypothesis generation, consistency across action chunks, and multimodal motion prediction. Across these areas, trade-offs remain in representation, inference overhead, and closed-loop candidate selection.
A. Multimodal Policies for Robotic Imitation Learning
- MDNs provide a classical probabilistic framework for continuous multimodal regression, while diffusion and BESO demonstrate iterative multimodal robot control,.Other approaches use energy functions, clustered actions, action chunking, or parallel sequence generation, but trade-offs remain among expressiveness, overhead, and closed-loop selection.Source: A. Multimodal Policies for Robotic… · p. 2
B. Single-Pass Multi-Hypothesis Generation
- IMLE Policy generates action sequences from random latents and selects the closest demonstration, while PRISM generates multiple sequences in one batched forward pass [7].Directly parameterizing a normalized complete action-chunk distribution with one latent component spanning the horizon remains insufficiently studied.Source: B. Single-Pass Multi-Hypothesis… · p. 2
C. Consistency and Feedback across Action Chunks
- BID, IntentVLA, and IMLE Policy use past action or visual history to maintain intent across chunks, [15],.Identifying the same physical route across permutable candidate indices and releasing constraints when current evidence contradicts history remain challenges.Source: C. Consistency and Feedback across… · p. 2
D. Multimodal Trajectory Modeling in Motion Prediction
- Motion-prediction methods model multiple futures using latent-variable Transformers, diffusion, winner-takes-all variants, and sequential mode relationships.Robotic closed-loop control additionally involves replanning, contact feedback, and temporal permutations of mixture components, so single-step diversity is insufficient to characterize performance.Source: D. Multimodal Trajectory Modeling in… · p. 2
A. Problem Formulation and Framework Overview
CTP models distinct executable futures as complete action chunks, using a shared single-pass policy to predict trajectory peaks, probability masses, and scales. DAPS specializes peaks during training, while ETBT transfers behavioral beliefs across replanning cycles.
A. Problem Formulation and Framework Overview
- CTP seeks a conditional policy over distinct executable futures given an observation history and target action chunk.The target chunk has horizon H and action dimension D; the observation history has length L.Source: A. Problem Formulation and Framework… · p. 2
- A task-specific normalized trajectory representation transforms action chunks using the current reference state, without predefining behavioral modes.The representation accommodates absolute positions, relative displacements, and other action parameterizations.Source: A. Problem Formulation and Framework… · p. 2
- CTP combines trajectory-level conditional peak modeling, DAPS, and ETBT in a single-pass framework for multimodal closed-loop control.DAPS specializes peaks during training, while ETBT transfers behavioral beliefs across replanning cycles.Source: A. Problem Formulation and Framework… · pp. 2–3
- One shared backbone and trajectory head predict all K trajectory peaks, probability logits, and scale parameters in a forward pass.The backbone encodes observation history into a context representation; the head predicts trajectory parameters for each peak.Source: B. Trajectory-Level Conditional Peak… · p. 3
- Each peak’s probability mass represents its conditional mode probability, while its scale describes residual dispersion in normalized action-chunk space.Neither quantity represents execution success probability or physical path width.Source: B. Trajectory-Level Conditional Peak… · p. 3
- A discrete mixture component selects one complete trajectory to explain the full action-chunk horizon.Mode identity belongs to the complete chunk rather than varying independently at each step.Source: B. Trajectory-Level Conditional Peak… · p. 3
- A trajectory peak is a component center, and its learned scale captures within-mode residual dispersion while likelihood penalizes unrestricted scale inflation.Larger residuals favor broader components.Source: B. Trajectory-Level Conditional Peak… · p. 3
- The trajectory basis supports either low-rank action-chunk parameterization or direct time-domain prediction, with NFE = 1 per replanning cycle.The low-rank setting uses R < H; direct prediction uses R = H and B = I_H.Source: B. Trajectory-Level Conditional Peak… · p. 3
C. DAPS
DAPS addresses mode collapse by assigning trajectory-level responsibilities and applying an overlap constraint that targets redundant, high-probability peaks. Its distribution-aware specialization does not force unsupported candidates into additional modes.
C. DAPS
- DAPS addresses mode collapse, where exchangeable candidates converge to a dominant high-density region and become redundant.More candidates alone do not ensure coverage of distinct data modes.Source: C. DAPS · pp. 3–4
- The overlap constraint discourages high-probability peaks from repeatedly explaining the same data region, with separation pressure modulated by probability mass and distributional scale.This selective pressure differs from uniform repulsion and allows unsupported redundant candidates to reduce their probability mass rather than forcing extra modes.Source: C. DAPS · p. 4
- The core DAPS objective combines mixture likelihood with an inter-peak overlap term.Responsibilities and the distribution-aware overlap constraint together promote data-driven peak specialization.Source: C. DAPS · p. 4
- Trajectory-level posterior responsibilities assign candidate roles according to their ability to explain the demonstration data.Unlike hard winner-takes-all assignment based only on trajectory distance, responsibilities consider fitting error, probability mass, and trajectory scale.Source: C. DAPS · p. 4
D. Single-Pass Decoding and Receding-Horizon Execution
CTP decodes a trajectory candidate in one forward pass, executes only the first part of its action chunk, then replans from new observations.
D. Single-Pass Decoding and Receding-Horizon Execution
- With no cross-cycle history, CTP selects the candidate with maximum probability mass from the current observation’s trajectory peaks.Candidate generation requires one forward pass.Source: D. Single-Pass Decoding and… · p. 4
- CTP executes the first E ≤ H actions of the selected chunk before replanning from new observations.The trajectory centers are mapped back to the task action space for execution.Source: D. Single-Pass Decoding and… · p. 4
- Candidate generation needs no additional sampling, iterative denoising, or vector-field integration, maintaining NFE = 1.Source: D. Single-Pass Decoding and… · p. 4
E. ETBT
ETBT geometrically matches trajectory candidates across replanning cycles to propagate behavioral beliefs, then combines that history with current policy evidence. It preserves behavioral continuity without fixing candidate indices, while allowing sufficiently strong current evidence or unreliable correspondences to release historical constraints.
E. ETBT
- ETBT matches behavior across candidate sets with permutable indices, treating history as a soft prior rather than a hard constraint.Conditional correspondence normalization can use rowwise normalization or Sinkhorn normalization with marginal constraints.Source: E. ETBT · p. 4
- ETBT compares the previous candidate’s unexecuted suffix with current candidates’ prefixes using normalized trajectory correspondence.A task-specific mapping supplies the trajectory representation; correspondence costs are normalized by a robust trajectory scale before similarity and conditional probabilities are computed.Source: E. ETBT · p. 4
- Correspondence probabilities propagate the previous behavioral belief into the current candidate set, where a smoothed historical prior is fused with current policy probabilities.The prior mixes the propagated belief with a weak fallback prior; if only the previously executed candidate is retained, its belief is one-hot.Source: E. ETBT · p. 4
- Historical constraints are released when current evidence strongly favors another candidate or trajectory correspondence is unreliable, without extra backbone forward passes.Otherwise, ETBT selects the candidate with the strongest fused behavioral belief.Source: E. ETBT · p. 5
A. Experimental Setup
The evaluation covers Push-T, three D3IL multimodal-control tasks, and 40 LIBERO tasks. Under a unified Push-T protocol, CTP achieves the highest closed-loop performance among compared methods, predicting multiple trajectories with one network evaluation.
A. Experimental Setup
- The setup evaluates Push-T on 200 fixed initial conditions, three D3IL tasks, and 40 LIBERO tasks with a frozen π0.5 backbone.D3IL tasks represent path planning, pose alignment, and object sorting; LIBERO uses 50-step chunks and replans every 10 steps.Source: A. Experimental Setup · p. 5
1) Comparison with Existing Multimodal Policies:
- 0.9140: CTP achieves the highest unified-protocol closed-loop performance on Push-T, ahead of IMLE Policy (0.8874) and Diffusion Policy-C (0.8790).CTP improves over IMLE Policy by 0.0266 and trajectory-level MDN by 0.1177.Source: 1) Comparison with Existing Multimodal… · p. 5
- CTP predicts multiple trajectories with NFE=1, without iterative sampling of multimodal actions.Push-T comparisons use a unified evaluation because training and evaluation settings vary across published studies.Source: 1) Comparison with Existing Multimodal… · p. 5
2) Inference Efficiency:
CTP reports competitive outcomes across D3IL and LIBERO while reducing policy-call latency through single-pass inference. The reported results include success rates across three D3IL tasks, LIBERO benchmark comparisons, and faster inference than iterative generation.
2) Inference Efficiency:
- On the same RTX 5070, CTP mean and P95 invocation latencies are 0.9507 ms and 0.9745 ms, versus Diffusion Policy’s 902.8522 ms and 904.8401 ms.CTP has 74.041M parameters versus Diffusion Policy’s 65.783M; the latency difference primarily reflects single-pass output versus iterative generation.Source: 2) Inference Efficiency · p. 5
C. Results on Key D3IL Tasks
- 100.0%, 79.72%, and 84.44% success on Avoiding, Aligning, and Sorting-2, respectively; Avoiding exceeds the best previously reported result of 95.0%.In the Fig. 3 example, the MLP collides with an obstacle while CTP reaches the goal from the same initial state.Source: C. Results on Key D3IL Tasks · p. 5, Figure 3 · p. 6
D. LIBERO: Transfer to Vision–Language–Action Models
- 97.25% average success on LIBERO is close to Motus (97.70%) and Fast-WAM (97.60%), below LingBot-VA (98.45%), and above local π0.5 (96.75%).Cross-model success-rate comparisons are not from strictly matched protocols.Source: Table IV · p. 6, D. LIBERO: Transfer to… · p. 6
- 218.24 ms to 75.80 ms: CTP reduces mean LIBERO policy-call latency by 65.27% (2.88× speedup) without reducing the 50-step prediction or 10-step execution horizon.Timing includes the vision-language backbone, data processing, and belief decoding; CTP and local π0.5 use the same evaluation protocol.Source: D. LIBERO: Transfer to… · p. 6, Figure 4 · p. 6
E. Ablations of Key Components
The ablations show successive gains from trajectory-level multimodal representation, DAPS, and ETBT, with ETBT producing the largest final Mean Score.
E. Ablations of Key Components
- 85.67% Mean Score after adding DAPS, a 5.37-point gain over the 80.30% score with K = 4.The paper attributes this gain to reduced candidate redundancy and greater differentiation among modes.Source: E. Ablations of Key Components · p. 7
- Increasing K from 1 to 4 raises Mean Score from 77.34% to 80.30%, a 2.96-point gain.The ablation successively adds trajectory-level multimodal representation, DAPS, and ETBT under the main experiments’ training and evaluation protocol.Source: E. Ablations of Key Components · p. 7
- 91.40% Mean Score: enabling ETBT with the same model weights adds 5.73 percentage points over the preceding configuration.The reported result supports the importance of consistent behavioral choices during receding-horizon replanning.Source: E. Ablations of Key Components · p. 7
F. Real-World Robot Experiments
Real-world dual-arm experiments test whether CTP preserves multiple valid placement modes and maintains task success while reducing inference latency. CTP succeeds in all 50 two-plate trials and achieves success rates comparable to π0.5 on bottle uprighting and pen placement.
F. Real-World Robot Experiments
- The experiments use a shared dual-arm robot and NVIDIA RTX 4090 platform across the two-plate, bottle-uprighting, and pen-placement tasks.The two-plate task tests preservation of spatially separated valid modes; the other tasks compare success and inference efficiency with π0.5.Source: F. Real-World Robot Experiments · p. 7
1) Placement into Either of Two Plates: Multimodal Behavior Preservation:
- All 50 two-plate placements succeed; CTP selects the left plate 31 times and the right plate 19 times, with no averaged trajectories between targets.The demonstration set contains 300 episodes, balanced at 150 per placement mode.Source: 1) Placement into Either of Two… · p. 8
2) Bottle Uprighting and Pen Placement: Task Success and Inference Efficiency:
- 218.24 ms to 75.80 ms: CTP reduces per-call inference latency by 2.88× while maintaining real-world success rates comparable to π0.5.Bottle-uprighting success is 95% for both methods; pen-placement success is 85% for π0.5 and 83% for CTP.Source: 2) Bottle Uprighting and Pen… · p. 8
V. CONCLUSION
- CTP combines trajectory-level multimodal distributions, DAPS peak specialization, and ETBT cross-chunk behavioral consistency with single-pass inference.The conclusion reports competitive performance on Push-T, D3IL, and LIBERO, alongside reduced online inference overhead.Source: §V · p. 8