Source-linked AI summary
PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation
Yuanzhe Liu, Jingyuan Zhu, Yuchen Mo, Gen Li, Xu Cao, Jin Jin, Yifan Shen, Zhengyuan Li, Tianjiao Yu, Wenzhen Yuan, Fangqiang Ding, Ismini Lourentzou
TL;DR
Long-horizon VLA manipulation remains difficult because existing methods lack task-relevant affordance reasoning and continuous subtask-progress tracking. PALM combines structured affordance prediction with progress-aware action generation, achieving state-of-the-art benchmark results and robust real-world performance. Its online recovery remains limited under execution drift, while affordance-oriented perception and annotation introduce overhead.
Problem
Existing VLA methods lack structured interaction cues and explicit progress tracking for reliably completing long-horizon, multi-step manipulation.
Method
PALM couples four structured affordance representations with continuous progress estimation and jointly decodes actions and progress in a closed-loop policy.
Results
PALM achieves state-of-the-art results, including 91.8% success on LIBERO-LONG and a 12.5% improvement on CALVIN ABC→D, with strong real-world robustness.
Takeaways & Limitations
Structured affordance reasoning and progress-aware action generation provide a unified approach for more consistent long-horizon robotic manipulation across simulation and real-world generalization settings.
Takeaways & Limitations
Online recovery remains limited under execution drift, and affordance-oriented segmentation, state grounding, and semantic interpretation add annotation, perception, and inference overhead.
Abstract
from arXiv · showhide
Recent advancements in vision-language-action (VLA) models have shown promise in robotic manipulation, yet they continue to struggle with long-horizon, multi-step tasks. Existing methods lack internal reasoning mechanisms that can identify task-relevant interaction cues or track progress within a subtask, leading to critical execution errors such as repeated actions, missed steps, and premature termination. To address these challenges, we introduce PALM, a VLA framework that structures policy learning around interaction-centric affordance reasoning and subtask progress cues. PALM distills complementary affordance representations that capture object relevance, contact geometry, spatial placements, and motion dynamics, and serve as task-relevant anchors for visuomotor control. To further stabilize long-horizon execution, PALM predicts continuous within-subtask progress, enabling seamless subtask transitions. Across extensive simulation and real-world experiments, PALM consistently outperforms baselines, achieving a 91.8% success rate on LIBERO-LONG, a 12.5% improvement in average length on CALVIN ABC->D, and a 2x improvement over real-world baselines across three long-horizon generalization settings.
1. Introduction
Current VLA policies handle short-horizon manipulation but struggle with long-horizon, multi-step execution because they lack structured interaction cues and explicit progress tracking. PALM addresses these gaps by combining structured affordance prediction with continuous subtask-progress estimation, achieving strong simulation and real-world results.
- Motivation: Current VLA methods struggle with long-horizon, multi-step manipulation in dynamic scenes despite progress on generalizable robot policies.Policies may succeed initially on tasks such as cleaning a cluttered table but fail to complete the full sequence.
- Motivation: Missing structured affordance cues and explicit state tracking leave object targets, interaction regions, placements, motions, and task stages ambiguous.This ambiguity contributes to repeated actions, skipped subtasks, premature termination, and incorrect success declarations.
- PALM: PALM predicts four affordance types—global, local, spatial, and dynamic—to represent object relevance, contact geometry, placement regions, and next-step motion.These affordance latents provide compact, task-relevant representations of the evolving scene state.
- PALM: PALM jointly predicts actions and continuous within-subtask progress in a closed loop to stabilize behavior and support smooth subtask transitions.The policy conditions action generation on affordance latents and multimodal context.
- Results: 91.8% success on LIBERO-LONG and a 12.5% improvement on CALVIN ABC→D demonstrate PALM’s gains over prior state-of-the-art baselines.Real-world tests across localization, lighting, and visual-distractor variations also show strong performance with limited fine-tuning data.
2. Related Work
Related work develops VLA policies from vision-language models and explores autoregressive or diffusion-based action generation. PALM extends these directions by integrating affordance reasoning and continuous progress supervision into a unified long-horizon control loop.
- Vision-Language-Action Models: VLA research repurposes pretrained vision-language models to map visual observations and language instructions to low-level robot actions.Large-scale robotics datasets are commonly used for fine-tuning.
- Vision-Language-Action Models: Existing action-generation paradigms include autoregressive prediction over tokenized sequences and diffusion-based denoising for control.These approaches represent complementary formulations of action generation within VLA policies.
- Imitation Learning with Progress Supervision: Progress-supervised imitation learning uses continuous progress representations, while PALM augments the action space with a scalar progress indicator for long-horizon consistency.PALM combines this progress reasoning with affordance representations in a unified VLA loop.
3. Method
PALM stabilizes long-horizon manipulation by combining future affordance prediction with continuous within-subtask progress tracking. Its affordance representations guide interaction, while progress-aware action decoding supports temporally consistent control and subtask transitions.
- Progress-aware policy: PALM predicts future affordance latents and continuous progress alongside actions to preserve temporal consistency across long-horizon subtasks.The affordance latent stabilizes perception, while progress provides a continuous within-subtask completion signal.
- Fine-grained affordances: Four affordance queries capture object relevance, contact geometry, placement regions, and motion dynamics at complementary scales.The query sets attend over language, vision, and state tokens to produce task-relevant affordance representations.
- Affordance representations: Global affordance identifies the instruction-relevant object and region, while local affordance predicts dense contact likelihoods for precise interaction.The global prior anchors subsequent cues, and local prediction analyzes fine-grained visual structure such as edges, textures, and part geometry.
- Progress-aware policy: Progress values pt ∈[0, 1] quantify within-stage completion and disambiguate visually similar observations that require different actions.PALM appends progress to the action output and encourages monotonic, stage-consistent evolution during execution.
- Action decoding: PALM conditions a denoising diffusion transformer on observations, instructions, robot state, and predicted affordances to generate n-step action–progress trajectories.The predicted action sequence and scalar progress sequence are decoded jointly for temporally consistent control.
4. Experiments
PALM is evaluated across simulation and real-world long-horizon manipulation settings, including benchmark comparisons, component ablations, data-composition studies, and generalization tests. Across these evaluations, PALM outperforms baselines, while ablations support the contributions of affordance prediction, progress prediction, inverse dynamics, and diverse training data.
- Simulation Experiments: PALM outperforms all baselines on CALVIN ABC→D, maintaining 82.0% success across five consecutive subtasks.The strongest prior baseline, Seer, reaches 64.3% at the five-task horizon.
- Simulation Experiments: PALM reaches 91.8% success on LIBERO-LONG, exceeding CoT-VLA’s 69.0% by 22.8%.Across all four LIBERO suites, PALM achieves a 94.5% average success rate.
- Ablation Studies: Global, Spatial, and Dynamic affordances improve performance across benchmarks, while Local affordance adds CALVIN gains but slightly reduces LIBERO-LONG performance.Spatial affordance restores and further improves performance, and the complete PALM configuration performs best.
- Ablation Studies: Removing affordance foresight reduces fine-tuning performance from 4.48 to 3.58, while removing progress prediction reduces pre-training performance from 4.48 to 3.73.Removing inverse dynamics prediction also reduces performance in both training stages.
- Ablation Studies: Excluding any single training-data type degrades performance, with the largest decreases occurring for In-the-Wild and Human-Annotated data.Removing pre-training data corresponding to the simulation benchmarks also causes a substantial performance decrease.
- Real-world Experiments: Real-world evaluation uses a six-subtask sequential pick-and-place task under visual distraction, random localization, and unseen-lighting conditions.The setup uses an xArm6 with a Gripper G2, two RealSense D455 cameras, and 200 fine-tuning demonstrations.
5. Conclusion
PALM couples structured future affordance prediction with continuous progress estimation in a closed loop for long-horizon robotic manipulation, achieving strong benchmark and real-world results.
- PALM couples structured future affordance prediction with continuous progress estimation in a closed loop.
- 12.5% improvement on CALVIN ABC→D and 91.8% success on LIBERO-LONG, with significant robustness across real-world long-horizon generalization settings.
Supplementary Material
PALM uses multimodal reasoning and specialized affordance queries to predict future interaction cues, then jointly generates temporally consistent actions and progress.
- Four sub-queries represent global object relevance, local contact geometry, spatial placement, and dynamic motion information.
- The model combines vision, language, state, affordance, and action-progress queries through a GPT-2 style multimodal transformer.
- Future affordance latents are decoded into object masks, placement points, contact heatmaps, and dynamic regions for training supervision.
- The action-progress decoder jointly predicts actions and scalar progress, aligning control with predicted future states for temporal consistency.
A.2. Training Details
PALM is pretrained on mixed robotic and long-horizon video data, then fine-tuned with affordance and continuous-progress annotations for efficient closed-loop control.
- Training data: Pre-training combines in-the-wild robot demonstrations with long-horizon videos containing fine-grained substeps and temporal annotations.
- Training data: Fine-tuning uses 942 robot trajectories annotated with affordance data and continuous progress labels through a semi-automated method.
- Implementation: The model uses a 68-million-parameter trainable architecture while keeping pretrained visual and text encoders frozen.
- Implementation: Inference uses 10 diffusion steps and achieves approximately 40 ms sampling time with a 10–15 Hz closed-loop frequency and cycles below 80 ms.
B.1. Task Setup and Success Criteria
The real-world task requires six sequential fruit-placement subtasks under randomized object positions, while perturbation and threshold studies evaluate progress-aware execution and transitions.
- Task setup: The task instructs the robot to place pineapple, grape, and orange into specified white and blue containers using two camera viewpoints.
- Success criteria: Success requires securely grasping each fruit and placing it onto or inside the correct target container.
- Success criteria: The setup tests irregular-object grasping, fine motor control, spatial targeting, language disambiguation, placement affordances, and color grounding.
- Robustness evaluation: Progress reliability is tested with random object relocations, unseen lighting changes, and visual distractions injected twice per episode.
- Progress threshold: The progress threshold controls sub-policy transitions: lower values risk premature termination, whereas excessively high values can cause execution stagnation.
- Progress threshold: 90% yields the highest success rates across chain lengths and an average sequence length of 4.48, while 70% and 100% reduce performance.
C.2. Ablation on Prediction vs. Reconstruction
PALM’s affordance foresight provides the strongest long-horizon control trade-off among tested prediction targets, outperforming future-image prediction and reconstruction while maintaining moderate latency.
- The ablation compares future affordance maps, future RGB observations, and current-input reconstruction using CALVIN ABC→D average length and inference latency.
- Affordance foresight achieves the highest average length at 4.48 with approximately 70 ms latency.
- Future RGB prediction reaches 4.17 average length at approximately 90 ms latency, below affordance foresight and slower than it.
- Auxiliary reconstruction is fastest at approximately 55 ms but produces the lowest average length of 3.58.
- PALM reduces repeat, skip, and premature-stop failures in real-robot six-subtask evaluations with 50 rollouts per method.
D. Qualitative Results and Visualization
PALM visualizes four complementary affordances across task progress, showing how structured predictions guide interaction, placement, motion, and robust long-horizon control.
- Global and local affordances identify what to interact with and where contact should occur during the drawer-sliding task.
- Spatial and dynamic affordances specify where to move and how the scene is expected to evolve.
- PALM predicts global, local, spatial, and dynamic affordances across sequential progress steps.These encode task-relevant objects and goals, contact points, placement regions, and motion trajectories.
- The visualization links structured affordance predictions to precise, robust control in a task where baselines often fail from ambiguity or limited spatial reasoning.
E. Broader Impacts
PALM frames structured affordance prediction and progress-aware state tracking as intermediate reasoning that can support coherent long-horizon manipulation and more transparent oversight.
- PALM combines future affordances with within-subtask progress estimation to maintain coherent behavior across complex multi-stage activities.
- Its affordances encode object identities, interaction points, spatial goal regions, and motion patterns rather than relying on direct sensorimotor mapping.
- Progress-aware affordance-based policies may help extend long-horizon manipulation toward homes, warehouses, and hospitals when paired with sufficient data and stronger inductive priors.
- Explicit affordance and progress signals integrate more naturally with safety monitors, constraint checks, and human-in-the-loop control while simplifying visualization and debugging.
- The broader-impact discussion calls for standardized evaluation, robust sim-to-real transfer, and safety and societal-impact assessments before wider deployment.
F. Limitations and Future Work
PALM’s online recovery remains limited when execution drift arises from partial observability, contact uncertainty, occlusions, or geometric deformations, while its structured perception pipeline adds overhead.
- Online recovery is limited under execution drift caused by partial observability, contact uncertainty, occlusions, and geometric deformations.
- Future work targets deviation detection, state-consistency checks, drift localization, replanning, and robust rollback.
- Affordance-oriented segmentation, state grounding, and VLM-based semantic interpretation create nontrivial annotation, perception, and inference overhead.
- The authors propose combining the task-conditioned controller with reinforcement learning, experience-based failure analysis, and 3D world perception.