Source-linked AI summary
SUN: Persistent Programs For Language-Grounded Control-to-Learning-to-Real Policies
Weiqi Wang, Zhi Li, Yudong Lei, David Martinez, Xiaofeng Gao, Yuxin Jiang, Chenfanfu Jiang, Yingnian Wu, Demetri Terzopoulos, Ran Gong
TL;DR
Long-horizon manipulation pipelines often lose the task semantics validated by control, leaving rewards and stage logic vulnerable to drift. Kuafu preserves typed SUN Programs through MPC screening, policy learning, and data generation; across nine tasks, it reports higher success than sparse-reward and alternative learners and transfers to physical robots. The paper concludes that control-validated semantics remain effective after behavior is amortized into learned policies.
Problem
Existing handoffs from model-based control to learned policies discard objectives, constraints, transitions, and satisfaction boundaries, allowing rewards to drift from control-validated specifications.
Method
Kuafu synthesizes typed SUN Programs from language and scene semantics, screens them with MPC, and retains their compiled interfaces while training stage-conditioned and residual policies.
Results
82.03% task-macro success across nine tasks, versus 35.67% for the sparse-reward baseline, while Kuafu-generated data supports downstream visual policies and zero-shot physical-robot transfer.
Takeaways & Limitations
Task semantics validated through control remain effective after behavior is amortized into learned policies, supporting robust multi-stage manipulation and sim-to-real transfer.
Takeaways & Limitations
SUN Programs depend on registered scene interfaces with informative object frames and axes and a finite typed-operator library; deformable objects and fluids require new operators.
Abstract
from arXiv · showhide
Bridging model-based control and learned policies in long-horizon manipulation has harbored a silent disagreement: control executes specified objectives, learning amortizes that behavior into a reactive policy, yet existing protocols discard task semantics, leaving rewards hand-crafted and behavior drifting from what control verified.We introduce Semantically UNified (SUN) Programs, typed executables where geometric and contact relations are defined once and compiled into aligned Model Predictive Control (MPC) costs, satisfaction predicates, RL rewards, transition guards, and diagnostics. Our system, Kuafu, driven by large vision language systems, automatically synthesizes SUN Programs from language and scene semantics, screens feasibility via MPC, and retains semantics while training stage-conditioned policies. Across nine tasks, Kuafu achieves 82.03% macro-success, outperforming sparse-reward (35.67%) and Stage-BC (24.75%) baselines. At 8192-way scale, it generates 10.57x the successful trajectory time per hour of human teleoperation. With 500 trajectories per task, Kuafu data trains DP3 policies to 46.0% simulation success (vs. 22.4% for alternatives) and 34.7% on physical Franka and Kinova robots. These results establish that simulation-screened task semantics can effectively amortize control into robust policies, without demonstrations or manual dense rewards, unifying symbolic planning and data-driven execution.
I. INTRODUCTION
Kuafu addresses semantic drift between model-based control and learned policies by preserving typed SUN Programs across language grounding, MPC screening, policy learning, and deployment. The framework improves multi-stage manipulation success while reducing manual reward engineering and supporting zero-shot sim-to-real transfer.
- Motivation: RL for multi-stage manipulation needs dense, structured rewards, but manually crafting them is laborious, brittle, and expensive to revise.Candidate reward effects are only evaluated after full policy training and rollout.
- Motivation: Model-based control verifies geometric objectives and transitions, yet handoffs to learning often discard the task semantics that justified the actions.This can cause policy rewards and stage logic to drift from control-validated specifications.
- Results: Across nine manipulation tasks, Kuafu achieves higher success with fewer language-model interactions and lower computational cost than competing methods, while enabling physical-robot sim-to-real transfer.The evaluation includes Franka and Kinova robots and reports significantly stronger downstream policy success from Kuafu-generated simulation data.
- SUN Programs: SUN Programs are typed executable artifacts that preserve task semantics across control, learning, and deployment.The same relations are compiled into aligned interfaces for MPC, monitoring, and reinforcement learning.
- Kuafu: Kuafu synthesizes SUN Programs from language and scene frames, screens feasibility with MPC, and trains stage-conditioned policies without manual reward engineering or seed demonstrations.Optimization rejects faulty objectives and transitions before repeated policy-training cycles, while Residual RL learns deployable contact-rich execution.
II. RELATED WORK
Prior work connects symbolic task structure, language grounding, optimization, and learning through executable costs, constraints, rewards, and trajectories. However, these approaches differ in how they expose task semantics to control and policy learning, and iterative reward refinement remains costly.
- Executable Task Representations: Executable task representations connect discrete task progression with continuous motion through symbolic planning, program generation, geometric feasibility, and scene grounding.Related approaches also encode temporal progress with finite-state rewards or signal-temporal-logic constraints.
- Language-Grounded Optimization: Language-grounded optimization converts task descriptions into composable 3D value maps or reward parameters evaluated by MPC.These methods expose spatial objectives to closed-loop control.
- Automated Reward Generation: Language models automate executable rewards and sim-to-real domain-randomization configurations, but optimized rewards can overfit observed behavior or encode invalid task specifications.A high return may therefore fail to translate into strong performance on the true task metric.
- Closest Precedents: ReKep, OmniManip, LATO, and MPC-Net respectively connect grounded costs, spatial constraints, learner-induced state distributions, or control objectives to manipulation learning.These precedents span closed-loop optimization, VLM checking, adaptive MPC teaching, and Hamiltonian-loss policy training.
III. METHODS
Kuafu constructs typed SUN Programs from language, scene frames, and operators, then preserves each grounded relation across control, monitoring, and learning. Stage-conditioned execution uses shared predicates and stage context to maintain task semantics.
- SUN Program construction: A SUN Program Γ is constructed from instruction I, scene frames F, and typed operator library L, then organized into ordered stages.The LLM selects existing typed components and binds their arguments to registered frames.
- Semantic Unification: Every grounded relation is compiled into an MPC objective, a stage-monitor predicate, and an RL learning term while retaining its identity, grounding, and stage membership.This shared correspondence is the paper’s Semantic Unification mechanism.
- Running example: For drawer opening, compiled relations measure displacement or angular deviation relative to the stage-entry snapshot for coordinated control and learning.
- Stage progression: The stage monitor evaluates current-stage predicates after each action and updates the stage only when the associated completion conditions are satisfied.For drawer opening, progression waits until predicates such as directional movement and fixed orientation are satisfied.
B. Program Validation and Acceptance
Kuafu screens candidate SUN Programs through closed-loop nonlinear MPC simulation, independently checks physical success, and repairs failures before freezing accepted semantics for policy learning.
- Independent task success: An independent evaluator gext provides a frozen sparse terminal outcome and is never exposed to program synthesis or learned policies.This separates external task success from the SUN monitor’s completion logic.
- Independent task success: For drawer opening, gext declares success only when the drawer joint reaches 12 cm displacement; otherwise geometric completion can be treated as false completion.
- MPC screening: Task-space nonlinear MPC consumes compiled SUN interfaces and screens each program in closed-loop simulation using direct multiple shooting.The controller replans from the updated runtime state after each observed action.
- Acceptance and repair: Each candidate is tested under 10 task-sampled initial conditions and accepted when at least one rollout achieves the physical outcome without any false completion.Failures can trigger up to five whole-program repair attempts based on measured violations.
- Acceptance and repair: After acceptance, Kuafu freezes grounded relations, tolerances, predicates, term definitions, and stage order before handing successful MPC trajectories to policy learning.
C. SUN-Guided Policy Amortization
Kuafu amortizes accepted MPC behavior into a reactive stage-conditioned policy, then adapts it with bounded residual reinforcement learning while retaining SUN-derived task supervision.
- Stage-conditioned behavioral cloning: 1,000 physically successful MPC trajectories train a deterministic Stage-BC policy that maps observations and active stages to 10-step action chunks.During learned execution, the reactive policy replaces the MPC optimization loop at each control step.
- Bounded residual reinforcement learning: The frozen Stage-BC policy anchors PPO-trained residual learning, while the retained SUN Program supplies stage context and program-derived signals.
- Bounded residual reinforcement learning: The residual policy is bounded coordinate-wise and combined with the Stage-BC prior under final environment action limits.
- Program-derived residual reward: Kuafu rewards monotonic within-stage progress using program-derived scores, stage transitions, physical success, false completion, and action terms.The score resets after stage transitions and terminates episodes at success, false completion, or timeout.
- Learning parameter calibration: An LLM tunes reward weights and residual parameters while preserving MPC-accepted predicates, relations, stage order, tolerances, and term definitions.
D. Data Production
For downstream visual-policy learning, Kuafu retains trajectories using a common independent physical-success criterion, carrying consistent outcome evaluation across data generators.
- Data retention: The downstream visual-policy study retains trajectories according to the common physical-success evaluator gext across all data generators.The retained SUN Program continues providing stage progression, completion monitoring, and false-completion detection during data generation.
IV. EXPERIMENTS
The experiments evaluate SUN Programs across nine multi-stage manipulation tasks, covering varied motion, contact, grasping, and precision capabilities under standardized simulation conditions.
- IV. EXPERIMENTS: Nine multi-stage tasks span pushing, pulling, twisting, reorienting, stacking, drawer manipulation, picking, placing, and precision contact behaviors.
- IV. EXPERIMENTS: Task-macro external-outcome success is the unweighted mean across tasks, so each task contributes equally regardless of rollout count.
A. Formulation Reliability
Kuafu reliably forms and validates executable programs through MPC screening and diagnostic recovery, while converting language interaction into a task-level rather than recurring execution cost.
- A. Formulation Reliability: 95.6% of 45 independent program-formation runs were accepted after diagnostic repair, up from 64.4% initially.Twenty-nine runs passed initially, and recovery accepted 14 of the remaining 16.
- A. Formulation Reliability: Kuafu uses 110.5k tokens per task, while VoxPoser’s cumulative interaction exceeds Kuafu’s after four rollouts.Persistent programs shift language interaction from a recurring execution cost to a task-level cost.
B. Performance and Ablation Analysis
Kuafu’s complete control-to-learning handoff substantially outperforms partial handoffs and competing reward-search or online-planning baselines, while retaining performance on longer task horizons at lower compute cost.
- B. Performance and Ablation Analysis: 82.03% task-macro success for Kuafu exceeds 35.67% for sparse reward, a 46.36-point gain.The three-run robustness aggregate reaches 79.43% success, with false-completion rates of 0.60%.
- B. Performance and Ablation Analysis: Removing the accepted Stage-BC prior yields 0% success, while Stage-BC alone achieves 24.75% and residual RL on stage-free BC reaches 18.56%.Removing or broadening the residual bound also collapses all tasks to 0% success.
- B. Performance and Ablation Analysis: Eureka’s best candidate reaches 18.24% success but its final policy falls to 12.85%, while VoxPoser achieves 10.00% and fails on five tasks.Eureka’s Task T0 performance drops from 45.14% to 0.21%.
- B. Performance and Ablation Analysis: Kuafu retains 91.4% of shorter-task performance across paired horizon tests, compared with a drop from 65.14% to 22.21% for sparse-reward handoff.Kuafu reaches 87.25% on T6 and 95.21% on T7, while non-Kuafu maxima are 13.81% and 38.80%.
- B. Performance and Ablation Analysis: 122.67 GPU-hours for Kuafu versus 479.60 for Eureka yields a 3.91× efficiency gain under one-GPU worker-hour accounting.Kuafu’s program formation costs 1.55 GPU-hours, while MPC validation and mentor collection dominate its compute.
A. Downstream DP3 success (%) (500 successful trajectories per task)
Under matched successful-trajectory budgets, Kuafu-generated data produces stronger downstream DP3 policies and more mentor-aligned, smooth trajectories, while substantially increasing successful-data throughput.
- A. Downstream DP3 success (%) (500 successful trajectories per task): During data collection, the learned controller replaces online MPC while the retained SUN Program supplies stage annotations, completion monitoring, diagnostics, and false-completion filtering.
- A. Downstream DP3 success (%) (500 successful trajectories per task): 46.02% task-macro success from Kuafu data exceeds SkillMimic’s 22.44% with 500 successful trajectories per task.The fixed DP3 learner and matched budget isolate downstream data utility; Kuafu has a 2.05× gap.
- A. Downstream DP3 success (%) (500 successful trajectories per task): 57.3% of Kuafu rollouts are Q95-qualified, versus 20.4% for MimicGen, a 2.81× gain.Among successful rollouts, Kuafu has normalized metrics of 1.04 completion time, 0.67 end-effector path, 0.52 action path, and 0.28 action second difference.
- A. Downstream DP3 success (%) (500 successful trajectories per task): 246.79 successful trajectory minutes per GPU-hour is 10.57× human teleoperation throughput.Human teleoperation produces 23.34 successful trajectory minutes per hour.
D. Visual Learning and Zero-Shot Transfer
Kuafu’s persistent SUN Programs support visual policies after privileged state and stage inputs are removed, achieving substantial simulation and physical transfer success. The evaluation also identifies grasp acquisition and limited generalization scope as important remaining boundaries.
- Visual Learning: 46.02% DP3 task-macro success was achieved after removing privileged state and stage inputs, exceeding the strongest RGB policy at 38.52%.ManiFlow-PC reached 46.46%, while DP3 reached 46.02% across the evaluated single-task policy families.
- Zero-Shot Transfer: 34.72% task-macro success was achieved across 106 physical trials and six tasks without real-world fine-tuning.Success was nonzero on every tested task, with calibrated RGBD-camera setups and constrained randomization.
- Limitations: Precise grasp acquisition remains the main observed physical failure mode, and the evaluation does not test held-out instructions, operators, scene semantics, or task compositions.SUN Programs also require informative registered scene interfaces and finite typed-operator libraries; deformable objects and fluids require new operators.
- End-to-End Results: The complete control-to-learning handoff reached 82.03% success, while fixed-volume DP3 training reached 46.02% success and enabled zero-shot transfer across 106 physical trials.The framework preserves an MPC-screened SUN Program alongside the learned policy.