Source-linked AI summary
Large Discrete Policy: Advancing Explicit Behavior Modeling with Stochastic Iterative Scoring
Zhenxin Li, Nadine Chang, Xinglong Sun, Jingde Chen, Wenhao Yao, Zi Wang, Maying Shen, Yu-Gang Jiang, Zuxuan Wu, Shiyi Lan, Jose M. Alvarez
TL;DR
Continuous denoising policies are expressive but difficult to interpret and can produce implausible actions, while discrete policies face expressivity and candidate-ranking challenges. LDiP iteratively scores and prunes plausible discrete actions with stochasticity in score space, and reports strong results across driving, manipulation, and vision-language-action settings.
Problem
Continuous denoising policies are difficult to interpret and can produce implausible actions, while discrete policies may struggle to express behavior through fixed vocabularies and rank near-optimal candidates.
Source: §1 · pp. 1–2Method
LDiP uses a fixed vocabulary of plausible actions and progressively prunes and re-scores candidates, perturbing scores rather than actions.
Source: §1 · p. 2Results
LDiP outperforms discrete and continuous baselines in autonomous driving, matches or exceeds diffusion policies in high-DoF manipulation, and LDiP-VLA outperforms diffusion- and flow-based VLAs.
Source: §1 · p. 3Takeaways & Limitations
The results suggest that LDiP offers a practical and interpretable alternative for behavior modeling while retaining physically plausible actions.
Source: §5 · p. 9LDiP falls behind Diffusion Policy on some manipulation tasks, leaves room for improvement under extreme closed-loop conditions, and needs further real-world validation.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Behavior policies are often formulated as continuous generative models, whose iterative denoising processes are expressive but difficult to interpret and prone to producing implausible actions. We propose the Large Discrete Policy (LDiP), a fully discrete behavior modeling framework that selects actions from a large vocabulary of physically plausible candidates. Rather than perturbing actions, LDiP improves expressivity through stochastic iterative scoring: it progressively re-scores and prunes candidates with score-space stochasticity, enabling fine-grained ranking and exploration among plausible actions while preserving an explicit decision process. Across end-to-end planning, closed-loop driving, robotic manipulation, and vision-language-action settings, LDiP consistently outperforms strong discrete and continuous baselines in autonomous driving, and exceeds or matches continuous generative policies in robotic manipulation. These results show that discrete policies, when equipped with effective scoring mechanisms, offer an expressive, plausible, and interpretable alternative for behavior modeling. Project website: https://zhenxinli.net/LargeDiscretePolicy/.
1 Introduction
Behavior learning conditions policies on visual observations, while continuous denoising is difficult to interpret and discrete policies face limits in expressivity and scoring. LDiP addresses this with stochastic iterative scoring over a fixed vocabulary of plausible actions.
1 Introduction
- Behavior learning trains policies conditioned on visual observations, commonly using offline demonstrations from expert agents.The paper situates this problem in applications including autonomous driving and robotic manipulation.Source: §1 · p. 1
- Continuous denoising policies model multimodal behavior but produce actions through an implicit refinement process that is hard to interpret and can yield implausible actions.One-shot discrete policies explicitly select from a vocabulary but rely on single-pass scoring.Source: §1 · p. 1, Figure 1 · p. 2
- The motivating gap is how to improve discrete-policy expressivity through better scoring while keeping the vocabulary fixed and physically plausible.The paper identifies ranking many near-optimal candidates—not necessarily a lack of feasible candidates—as a possible bottleneck.Source: §1 · p. 2
- LDiP iteratively prunes and re-scores a large action vocabulary, perturbing scores rather than actions to explore among plausible candidates.The process makes candidate comparisons, rankings, and final action selection explicit.Source: §1 · p. 2
- LDiP outperforms discrete and continuous baselines in autonomous driving, matches or exceeds diffusion policies in high-DoF manipulation, and LDiP-VLA outperforms diffusion- and flow-based VLAs.The reported evaluations span end-to-end planning, closed-loop driving, and robotic manipulation.Source: §1 · p. 3
2 Related Work
Related work contrasts implicit iterative denoising with explicit discrete action selection. LDiP builds on discrete scoring while differing from both candidate-refinement methods and action-token generation.
2.1 Iterative Denoising Policies for Behavior Modeling
- Diffusion and flow-matching policies generate actions through repeated refinement of noisy samples, making it difficult to inspect why behaviors are considered or how the final action is formed.Diffusion Policy demonstrated action generation for robotic manipulation, and similar approaches have since appeared in driving and vision-language robotics.Source: §2.1 · p. 3
2.2 Discrete Policies for Behavior Modeling
- Trajectory scorers explicitly rank candidate sequences, while hierarchical methods prune large sets before refinement but may inject noise into actions and compromise plausible trajectories.Behavior Transformers [43] also use clusters, but predict continuous offsets, so the final action remains continuous.Source: §2.2 · p. 3
- VLA methods such as RT-1 [5], RT-2, and OpenVLA [23] predict discretized actions as language tokens rather than explicitly scoring trajectory candidates.LDiP extends explicit trajectory scoring into a general behavior-modeling framework.Source: §2.2 · p. 3
- LDiP tokenizes a fixed action vocabulary, attends to visual observations and optional language, then scores, stochastically prunes, and re-scores candidates until one action remains.The decoder uses an MLP to score candidates.Source: Figure 2 · p. 4
3 Method
LDiP turns offline demonstrations into a fixed vocabulary of physically plausible action chunks, then scores and progressively prunes candidates while keeping action selection explicit. Its learning objectives, score perturbations, and VLA variants adapt this framework across behavior-modeling settings.
3.1 Model Architecture
- K-Means clusters fixed-horizon expert action chunks into a discrete vocabulary; an observation-conditioned decoder scores the physically plausible candidates.Action tokens query a Transformer decoder that attends to visual observations and, in VLA settings, language instructions.Source: §3.1 · p. 4
- LDiP scores the full vocabulary first, then repeatedly re-scores top-ranked candidates and outputs the highest-scoring action chunk.Progressive pruning keeps the decision process fully discrete while narrowing the action space to promising candidates.Source: §3.1 · p. 4
- Imitation learning uses distance-based soft labels and cross-entropy; driving additionally uses objective-scoring heads trained with binary cross-entropy.Driving selection uses a weighted average of multiple predicted scores.Source: §3.1 · p. 5
- Training perturbs candidate scores with Gumbel noise before pruning; inference uses deterministic scores for driving and noise-perturbed selection for manipulation exploration.Score-space perturbation explores among plausible candidates without changing the discrete action vocabulary.Source: §3.1 · p. 5
3.2 Vision-Language-Action (VLA) Architecture
- The VLA extension tests joint-attention action-group tokens, learned implicit action tokens, and a two-system design with a frozen VLM conditioning a separate action decoder.All three variants retain iterative action selection and scoring.Source: §3.2 · pp. 5–6
4 Experiments
The experiments evaluate LDiP across autonomous-driving planning and VLA models, closed-loop driving, and visual robotic manipulation. They also test its scoring components, VLA architectures, width and depth, and qualitative trajectory selections.
4.3 Main Results
- On HUGSIM and manipulation, the controller variant reaches 44.7 RC and 36.7 HD-Score, while LDiP beats Diffusion Policy on ToolHang and PushT.The reported manipulation success rates are 0.52 on ToolHang and 0.76 on PushT; HUGSIM performance is zero-shot, though Extreme scenarios remain challenging after controller tuning.Source: Table 3 · p. 7, Table 4 · p. 7, §4.3 · p. 8
4.1 Implementation Details / 4.2 Benchmarks and Metrics
- Driving benchmarks cover NAVSIM planning and VLA PDMS, plus zero-shot HUGSIM closed-loop driving; manipulation evaluation covers Lift, Can, ToolHang, and PushT.Implementation uses a 16,384-trajectory driving vocabulary, usually pruned to 512 and then 16 candidates; manipulation vocabulary sizes vary by task.Source: §4.1 · pp. 6–7, §4.2 · p. 7
4.3 Main Results
- LDiP achieves the best EPDMS across evaluated NAVSIM backbones and the best overall PDMS among the evaluated VLA baselines.The NAVSIM EPDMS scores are 84.7 with ResNet34, 88.1 with ViT-L, and 87.8 with V2-99; LDiP-VLA achieves 92.1 PDMS with Qwen3-VL-2B.Source: §4.3 · p. 7
4.4 Ablation Studies
- In the key-component ablation, iterative scoring raises EPDMS from 86.2 to 86.7, and stochastic scoring reaches 87.8 versus 87.2 with noise injected into top-ranked actions.The cross-attention-only decoder with a 1D convolutional tokenizer reaches 87.1 EPDMS while remaining efficient for large vocabularies; the stochastic-scoring result keeps the vocabulary physically plausible.Source: Table 5 · p. 8, §4.4 · p. 8
- Stochastic iterative scoring improves all three VLA designs, with the Two-System approach performing best; in manipulation ablations, it raises ToolHang to 0.60 and PushT to 0.78.Action Group Tokens require downsampling a vocabulary of up to 16K candidates, while Implicit Action Tokens reduce token cost but lose fine-grained geometric information.Source: Table 6 · p. 8, §4.4 · p. 8, Table 7 · p. 8
- Three scoring stages reach 87.8 EPDMS, while four stages slightly reduce performance; the best vocabulary setting uses 512 intermediate and 16 final candidates.Smaller candidate sets can discard useful actions too early, whereas oversized sets make final scoring more ambiguous.Source: §4.4 · p. 8, Figure 5 · p. 9
4.5 Qualitative Results
- Navtest visualizations show diverse plausible futures early in scoring, then progressively narrower selections that respond to traffic participants, occlusion, and uncertainty.Examples include shifting toward a safer trajectory under environmental uncertainty and selecting conservatively around potentially occluded distant agents.Source: §4.5 · p. 9, Figure 4 · p. 9
5 Conclusion
The conclusion presents LDiP as an expressive, explicit, and physically plausible discrete policy, with strong results across several behavior-modeling settings. It also identifies remaining gaps in manipulation, extreme closed-loop driving, and real-world validation.
5 Conclusion
- LDiP uses a large fixed action vocabulary and stochastic iterative scoring to compare and prune plausible candidates while keeping decisions interpretable.The conclusion characterizes the framework as improving discrete-policy expressivity without compromising its explicit and physically plausible formulation.Source: §5 · p. 9
- The authors report that LDiP falls behind Diffusion Policy on some manipulation tasks, leaves room for improvement in extreme closed-loop conditions, and needs real-world validation.These are stated as limitations on the current evidence and deployment scope.Source: §5 · p. 9
A Technical appendices and supplementary material
The supplementary material provides additional qualitative results and extended analyses of the approach.
A Technical appendices and supplementary material
- The appendices supplement the main paper with additional qualitative results and extended analyses.The passage describes the supplementary material's scope without specifying particular findings.Source: A Technical appendices and… · p. 14
A.1 Additional Qualitative Results (Autonomous Driving)
Supplementary Navtest examples depict smooth, conservative trajectories that align with lane geometry and adapt to nearby traffic. They illustrate the policy's behavior in both lane-following and interactive urban scenes.
A.1 Additional Qualitative Results (Autonomous Driving)
- Across curved and straight lanes, predictions remain aligned with the drivable corridor and maintain a larger safety buffer than the human trajectory.The examples include parked vehicles, dense traffic, and distant traffic signs; paired camera and Bird’s-Eye-Views show consistency with lane geometry and surrounding agents.Source: §A.1 · p. 14, Figure 6 · p. 14
- In interactive scenes with vehicles, pedestrians, cyclists, and intersections, actions are described as conservative and context-aware, avoiding abrupt deviations.The examples are characterized as maintaining smooth lane-following and reasonable interaction awareness in complex urban environments.Source: §A.1 · p. 14, Figure 6 · p. 14
A.2 Additional Qualitative Results (Robotic Manipulation)
The ToolHang visualization follows a full episode, showing top-8 candidate action chunks at each step alongside the action chunk selected by the policy. The candidates capture diverse feasible motions for assembling the frame and lifting the tool toward the hook.
A.2 Additional Qualitative Results (Robotic Manipulation)
- At each step, the sequence compares the pruned top-8 candidate chunks with the final action chunk executed by the policy.Source: §A.2 · p. 14, Figure 7 · p. 15
- The figure follows a full episode of ToolHang, showing key frames with the top-8 candidate action chunks and the selected action chunk.Source: §A.2 · p. 14, Figure 7 · p. 15
- Across time, the candidate chunks capture diverse feasible motions for assembling the frame and lifting the tool toward the hook.Source: §A.2 · p. 15
A.3 Efficiency
LDiP improves EPDMS over Diffusion Policy while avoiding its 100-step iterative denoising, and balances planning accuracy with runtime efficiency. It runs at 12.1 FPS on a single NVIDIA A100 GPU with PyTorch FP32.
A.3 Efficiency
- 12.1 FPS on a single NVIDIA A100 GPU with PyTorch FP32 accompanies LDiP’s best EPDMS score of 87.8.Source: Table 8 · p. 15, §A.3 · p. 16
- Compared with Diffusion Policy’s 100 DDPM denoising steps, LDiP runs substantially faster while improving EPDMS.Source: §A.3 · p. 15
- Among discrete policies, LDiP has comparable model size to the V2-99 backbone while using a larger action vocabulary and three lightweight selection steps.Source: §A.3 · p. 15
- The multi-step selection slightly reduces FPS relative to prior one-step methods, while the reported runtime remains 12.1 FPS.Source: §A.3 · p. 16
A.4 Broader Impact
The authors identify potential benefits for interpretable and physically plausible policies in autonomous driving and robotic manipulation, while warning of risks from real-world deployment. They call for rigorous closed-loop testing, safety monitoring, and human oversight.
A.4 Broader Impact
- The work may contribute to more interpretable and physically plausible policies for autonomous driving and robotic manipulation.Source: §A.4 · p. 16
- Real-world deployment can introduce safety risks if a policy fails in unseen conditions or inherits biases from demonstration data.Source: §A.4 · p. 16
- The authors state that practical use should require rigorous closed-loop testing, safety monitoring, and human oversight.Source: §A.4 · p. 16