Source-linked AI summary
Spatial-Language Attention Policies for Efficient Robot Learning
Priyam Parashar, Vidhi Jain, Xiaohan Zhang, Jay Vakil, Sam Powers, Yonatan Bisk, Chris Paxton
TL;DR
Mobile manipulation calls for spatial representations that handle changing views and robot embodiments, while existing attention policies can rely on fixed workspace and camera assumptions. SLAP uses point-cloud spatial tokens with interaction and relative-action modules, and outperforms PerAct on mounted and mobile setups, including a 4x task-success improvement in controlled mobile manipulation.
Problem
Mobile manipulation requires spatial representations that handle changing viewpoints and robot embodiments, while prior attention-based methods can assume fixed scene sizes and camera placements.
Source: §1 · pp. 1–2Method
SLAP uses multiresolution point-cloud tokens with an interaction module that predicts where to attend and a relative-action module that predicts continuous Cartesian actions.
Source: §3 · p. 4Results
Takeaways & Limitations
SLAP executes natural-language task instructions using ten demonstrations across five learned skills and three heuristic skills on a mobile manipulator.
Source: §1 · p. 2SLAP has high variance out of distribution and can fail completely if its interaction module misidentifies context; its relative-action policy also struggles with multimodal or noisy data.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Despite great strides in language-guided manipulation, existing work has been constrained to table-top settings. Table-tops allow for perfect and consistent camera angles, properties are that do not hold in mobile manipulation. Task plans that involve moving around the environment must be robust to egocentric views and changes in the plane and angle of grasp. A further challenge is ensuring this is all true while still being able to learn skills efficiently from limited data. We propose Spatial-Language Attention Policies (SLAP) as a solution. SLAP uses three-dimensional tokens as the input representation to train a single multi-task, language-conditioned action prediction policy. Our method shows an 80% success rate in the real world across eight tasks with a single model, and a 47.5% success rate when unseen clutter and unseen object configurations are introduced, even with only a handful of examples per task. This represents an improvement of 30% over prior work (20% given unseen distractors and configurations). We see a 4x improvement over baseline in mobile manipulation setting. In addition, we show how SLAPs robustness allows us to execute Task Plans from open-vocabulary instructions using a large language model for multi-step mobile manipulation. For videos, see the website: https://robotslap.github.io
1 Introduction
Mobile manipulation calls for spatial representations that handle changing viewpoints and robot embodiments, while prior attention-based policies can depend on fixed scene and camera assumptions. SLAP combines point-cloud tokens with interaction prediction and continuous action prediction, and is evaluated on stationary and mobile robots.
1 Introduction
- PointNet++-based tokens capture free-form spatial relations and support multiple moving-camera views and arbitrary-length input sequences.The representation is a varying-length stream of multiresolution spatial embeddings.Source: §1 · p. 1
- Prior attention-driven policies can assume fixed workspace size and camera placement and may take a long time to train, limiting rapid adaptation to new environments.The introduction frames spatially aware, embodiment-independent representations as important for generalizing to new positions, visual changes, and clutter.Source: §1 · pp. 1–2
- SLAP’s hybrid policy predicts relevant interaction locations, then predicts continuous relative actions for execution.The interaction module selects scene regions, and the action policy operates on predicted interaction points.Source: §1 · p. 2, Figure 1 · p. 2
- 80% success on 8 static real-world tasks and a 47.5% success rate with out-of-distribution objects; SLAP also achieves a 4x task-success improvement over PerAct in controlled mobile manipulation.The static-task comparison uses held-out scene configurations; the mobile comparison covers four skills.Source: §1 · p. 2
2 Related Work
Related work covers attention-based policies, point-cloud interaction methods, unknown-object manipulation, and language-conditioned skills. The approaches differ in their workspace assumptions, use of separate components, and support for planning or generalization.
Attention-Based Policies
- Attention-based policies often model the workspace as a 2D image or bounded 3D voxel cube, while CLIPort and PerAct use rigid grids and discrete action classification.These assumptions can constrain camera positioning, workspace location, and discretization size.Source: §2 · pp. 2–3
Language for Low-Level Skills
- SLAP’s two modules localize relevant scene features and use local context to predict executable actions, learning language-conditioned skills from few demonstrations in clutter.The interaction prediction module is distinct from the action prediction module.Source: Figure 2 · p. 3
Point-Cloud Methods
Manipulation of Unknown Objects
- Unknown-object manipulation commonly segments relevant objects before predicting grasps and movements, while object-detection systems such as Detic [14] and 3DETR [15] lack a manipulation component.Some approaches use separate purpose-built models for manipulation and task planning.Source: §2 · p. 3
Language and Robotics
Language for Low-Level Skills
3 Approach
SLAP represents manipulation as interaction-point selection followed by relative action prediction, with language instructions decomposed into atomic skills. It combines scanned point clouds with language and trains separate modules to select interaction points and predict actions.
3 Approach
- SLAP predicts an interaction point and relative motions for each atomic skill, then decomposes higher-level language tasks into sequences of those skills.The interaction point anchors the skill’s relative waypoints, and predicted actions are sent to a low-level controller.Source: §3 · p. 4
3.1 Scene Representation
- Scene input combines base-frame point clouds from scanning actions; πI uses 5 mm voxel tokens, while set-based processing preserves fine detail in larger scenes.The interaction module can receive 5,000–8,000 scene tokens.Source: §3.1 · p. 4
3.2 Interaction Prediction Module
- πI scores spatial features to select an interaction point; modified set-abstraction embeddings retain small features by combining original and evenly downsampled clouds.The layers encode local spatial structure at multiple resolutions.Source: §3.2 · pp. 4–5
3.3 Relative Action Module
- The relative-action module crops around the predicted interaction point and predicts gripper state, position offset, and orientation, using noisy centering to tolerate imperfect point predictions.Its LSTM-based architecture predicts variable-length skills, with three or four actions in the experiments.Source: §3.3 · pp. 5–6
3.4 Training SLAP
- Expert keyframes train the two modules separately: interaction selection uses cross-entropy plus locality loss, while relative actions use position, quaternion, and gripper losses.Operators record keyframes at crucial task stages.Source: Figure 4 · p. 6, §3.4 · p. 6
3.5 Task Planning
- For longer instructions, LLaMA maps generated language and code templates to sequences of atomic skills using in-context learning and adapter fine-tuning.For example, “put away the bottle on a table” is decomposed into navigation, picking, and placing skills.Source: §3 · p. 4, §3.5 · p. 6
4 Experiments
SLAP is evaluated against prior methods on real-robot manipulation, including unseen scenes and mobile task execution. The experiments also assess whether language models can plan with the learned skills.
4 Experiments
- 1.6× improvement over PerAct is reported across eight real-world Franka manipulation tasks, with SLAP notably better in both seen and unseen scenes.The unseen condition includes previously unseen object positions and distractors.Source: §4 · p. 7, Table 2 · p. 8
4.1 Longitudinal Task Execution on Stretch
- On Stretch, a multi-task model trained with 10 demonstrations per skill was tested in unseen settings, with the robot starting anywhere relative to the objects.The model covered five skills and was deployed in an end-to-end system using code-list plans.Source: §4.1 · p. 7
- Learned skills outperform heuristics in end-to-end performance except when Detic fails.The table caption gives no further performance values.Source: Table 1 · p. 7
- With Detic semantic features, IPM performance rises to 80% from 47.5% with unseen distractors, but significant relative-position perturbations still cause failures.When inferred plans produce language prompts mismatched with Detic’s object detections, missing semantic masks harm skill performance.Source: §4.1 · pp. 7–8
4.2 Task planning with in-context learning and fine-tuning LLaMa
- High Task Correctness from a small model supports compatibility between the defined skills and LLM task planning.The experiments compare in-context learning and adapter fine-tuning of LLaMA and report verb, noun, and combined accuracies, binary Task Correctness, and latency.Source: §4.2 · p. 8
5 Limitations
The system remains vulnerable to out-of-distribution context errors and noisy inputs, while its component policies and language-vision interfaces leave multiple failure points.
5 Limitations
- Fine-tuning outperforms in-context learning at the same latency, according to Table 4.The supplied passage does not state the table’s metric values.Source: Table 4 · p. 8
- Out-of-distribution context errors can cause complete failure when the interaction policy πI misidentifies the context.The relative policy πR also has difficulty with multimodal or noisy data.Source: §5 · p. 8
- Heuristic policies and unaligned language and vision models create multiple system failure points.The authors suggest end-to-end trainable architectures and cross-modal alignment could help.Source: §5 · p. 8
6 Conclusion
SLAP combines point-cloud structure with language and demonstrations to predict continuous end-effector actions, and the paper evaluates it on two hardware platforms, including a mobile manipulator.
6 Conclusion
- SLAP combines point-cloud input, language semantics, and demonstrations to predict continuous end-effector actions for manipulation.The paper reports that SLAP outperforms prior state of the art on both mounted and mobile robot setups.Source: §6 · p. 8
A.1 Data Collection and Annotation
- Franka data collection aggregates a point cloud from predefined scanning positions without assumptions about their number, arrangement, or resulting cloud size; Stretch data follows where the robot looks..Source: §A.1 · p. 13
- Demonstrations use kinesthetic teaching for Franka and controller teleoperation for Stretch, with keyframes recorded at salient task moments.The low-level controller connects the keyframes into the trajectory.Source: §A.1 · p. 13
A.2 Data Processing
The appendix describes how SLAP’s inputs are augmented and how its relative action module predicts actions from interaction-centered point-cloud crops. It also details loss weighting, training comparisons, and data collection choices.
A.2 Data Processing
- The training input scene is randomized with elliptical depth-channel dropout, multiplicative depth noise, additive point-cloud noise, whole-scene z-axis rotations within ±45°, and random cropping.Random cropping uses p = 0.75 around ˆpI + δ, with δ sampled from a Gaussian distribution and the crop radius randomly sampled in (1, 2) meters.Source: §A.2 · p. 14
A.3 Action Prediction Losses
- Gripper action is trained as a classification task using language and proprioceptive inputs; positional and orientation losses use separately weighted terms.The passage specifies the gripper inputs as task language, gripper state, finger distance, and time step; positional and orientation weights are 1 and 1e−2.Source: §A.3 · p. 14
B.1 MLP Implementation
- In the MLP implementation, cropped point-cloud features are combined with language and proprioception to predict position offset, absolute orientation, and gripper action.Training noise on the interaction point helps the policy recover when the predicted point is near, but not exactly at, the correct location.Source: Figure 6 · p. 15, §B.1 · p. 15
A.4 Skill Weighting
- Stretch skills use different empirically chosen position, orientation, and gripper weights for noisier tasks such as pouring and handover versus more consistent drawer tasks.The weights could be further optimized with hyperparameter tuning.Source: §A.4 · p. 15
A.5 Training for PerAct and SLAP
- PerAct and SLAP are trained on the same data volume, defined as augmented samples per collected sample, although their training-step counts differ.The reported difference reflects LSTM-based updates per trajectory versus PerAct updates per sample in a trajectory.Source: §A.5 · p. 15
B Relative Action Module
- The relative action module predicts end-effector poses relative to the interaction point from a local point-cloud crop around it.Its implementation varies between the Franka and Stretch platforms.Source: B Relative Action Module · p. 15
B.2 LSTM-based Implementation
The MLP struggled when action positions varied widely relative to interaction points across tasks. Adding an LSTM to condition on task and action improved learning over wider action distributions in initial Stretch experiments.
B.2 LSTM-based Implementation
- The MLP struggled with multimodal action-position distributions, prompting an LSTM that conditions each task and action on a hidden state.The LSTM showed better performance learning wider action distributions in initial Stretch experiments.Source: §B.2 · pp. 15–16
- The LSTM-based architecture showed greater stability when learning wider action distributions because of its conditioning effect.Figure 7 depicts the architecture built on the regression head and PointNet++ embeddings introduced in Figure 6.Source: Figure 7 · p. 16, §B.2 · p. 16
C.1 Dataset
The paper describes a procedurally generated language-to-skill dataset, task-planning evaluation, and a mobile manipulation pipeline. Experiments vary objects, clutter, and environments, while the execution system uses detection, heuristic policies, and SLAP.
C.1 Dataset
- More than 500k instruction-to-skill-sequence samples cover 16 task families, with 10% of each family held out for evaluation.Each family shares a sequence of atomic-skill verbs while instructions and interaction objects vary; the generated objects include more than 150 movable objects and surfaces.Source: §C.1 · p. 16
C.2 Models
- Close to 76% accuracy was achieved by in-context learning with five examples from the same task family.Fine-tuning was evaluated on held-out instruction phrasing and interaction objects, not unseen task families; subsequent results use the fine-tuned model.Source: §C.2 · p. 16
C.3 System Architecture and Plan Executionr
- The execution pipeline uses Detic-based search and heuristic top-grasp and surface-placement policies before SLAP predicts object interaction and action trajectories.SLAP is applied once the object is detected and in view; standoff orientations place the robot head-on before full action prediction.Source: §C.3 · p. 17
- Standoff orientations constrain the robot’s approach because action orientation is not robust to large rotational variation, although interaction-point prediction is robust to directionality.The robot uses predetermined standoff orientations and distances, with evaluation orientations of (1, 0, 0) or (0, 1, 0).Source: §C.3 · pp. 17–18, Figure 8 · p. 18
D Experimental Setup for Skills
- SLAP’s learned atomic skills are treated as tasks, with their actions defined as relative offsets from an interaction point.For example, pick and open-drawer skills use an interaction point and relative waypoints.Source: D Experimental Setup for Skills · p. 18
D.1 In vs. Out Of Distribution
- Experiments include in- and out-of-distribution objects, unseen clutter, and unseen environments, with test scenes varying from training scenes.The authors show examples of test-scene variation for both Stretch and Franka experiments.Source: §D.1 · p. 18
D.2 Skill Definitions and Success Conditions
The appendix defines manipulation tasks and their success conditions for Franka and Stretch, and describes language annotations used in experiments. Test scenes include varied object configurations, clutter, and environments.
D.2 Skill Definitions and Success Conditions
- Stretch placement tasks require the object to end inside the drawer, bowl, or basket, while pouring succeeds when at least 50% of candies reach the bowl.The candy threshold reflects comparison with liquid pouring, where the intended final configuration empties the liquid.Source: §D.2 · pp. 21–22
- Franka drawer-opening success is defined as 50% open because a fixed-base arm may be too close to its base to open the drawer fully.The test objects include within-distribution objects, with examples of Franka scene variation and Stretch training-versus-test variation.Source: Figure 9 · p. 19, Figure 10 · p. 19, Figure 11 · p. 19
- Stretch drawer-opening tasks use different success thresholds: 100% open for the top drawer and at least 50% for the bottom drawers.The bottom-drawer configurations use loop and cylindrical handles, respectively, and require different grasps; the unqualified closing task selects the open drawer from its state.Source: §D.2 · pp. 19–21
- Stretch pick-and-transfer tasks define success by stable object possession or delivery, including bottle pickup, lemon pickup, and handing an object to a person.Franka lemon pickup uses an open-loop criterion that also counts a lemon rolling out after contact following the second action as success.Source: §D.2 · pp. 21–22
D.3 Language Annotations
- Training language annotations are compared with held-out out-of-distribution language expressions, although language generalization is not the study’s focus.The paper reports robustness to different language expressions and lists the training and test language in separate tables.Source: §D.3 · p. 22
D.4 Out of distribution Results from SLAP
SLAP’s interaction predictor finds affordance points in cluttered scenes, but grasping an unseen wider bottle fails. The authors hypothesize reasons for mobile-manipulation performance and note that the non-collision-aware planner caused failures for both models.
D.4 Out of distribution Results from SLAP
- SLAP’s interaction predictor found placement and picking points amid heavy clutter, including a placement area in a cluttered drawer and a bottle interaction point beside a lemon.The predicted bottle interaction point differed from the expert’s and was lower on the bottle.Source: §D.4 · p. 22
- The unseen, wider bottle was detected and given a feasible trajectory, but execution failed because its wider girth required a more accurate grasp.The model could predict the interaction site and trajectory despite the bottle’s changed height, color, and girth.Source: §D.4 · p. 22, Figure 15 · p. 23
- The authors hypothesize SLAP’s better mobile-manipulation performance reflects semantic features, more efficient training, and higher resolution from non-grid point clouds.Source: §D.4 · p. 23
D.5 Motion Planning Failures
- The non-collision-aware planner caused failures for both models, more often for PerAct; the authors note collision-aware planning might prevent those PerAct failures.SLAP centers relative motions around an interaction point, whereas PerAct predicts each action as a separate trial.Source: §D.5 · p. 23
E.1 Ablations
The ablations compare SLAP with PerAct across tabletop and mobile settings, while later sections visualize learned attention, test held-out language, and situate SLAP among navigation representations.
E.1 Ablations
- SLAP outperforms PerAct on six of eight in-distribution and five of eight out-of-distribution Franka tasks, and on all four cluttered Stretch tasks.PerAct ties SLAP on two in-distribution and two hard-generalization cases, and outperforms it for bottle pickup in hard scenes.Source: §E.1 · p. 23
- Out-of-distribution success drops for both methods; SLAP errors include orientation prediction, while πI can be confused by dramatically different target poses and unseen distractors.The passage attributes stronger Stretch results to semantic features from Detic.Source: §E.1 · p. 23
E.2 Visualizing the Learned Attention
- Attention visualizations show PerAct failures from non-collision-aware drawer motion and disjoint, semantically wrong predictions, plus a SLAP failure on an extreme sideways drawer with unseen distractors.The SLAP attention mask ranks distractors among its top 5% and misses an optimal interaction point; the visualization selects points with the highest 5% of interaction scores for a language command.Source: §E.2 · p. 23, Figure 17 · p. 24, Figure 16 · p. 24
E.3 Language Generalization
- The language-generalization experiment tests in-distribution scenes on three representative tasks, using ten tests and two held-out language phrasings.The passage links this test to pretrained CLIP embeddings in πI, which can generalize to unseen language to some extent.Source: §E.3 · p. 25
F Additional Related Work
- Related navigation work uses voxel maps, 2D representations, object candidates, or implicit fields; USA-Net includes semantic features such as affordances and collision.The passage notes that CLIP-Fields does not address manipulation and that USA-Net representations could be incorporated for collision-aware action plans at prediction time.Source: F Additional Related Work · p. 25
- The appendix includes training-language examples and examples of out-of-distribution language annotations used for evaluation.The listed training examples include varied phrasings for object pickup, placement, and drawer skills.Source: F Additional Related Work · p. 26, Table 7 · p. 26, Table 6