PaperlayerView exact revision on arXiv

Source-linked AI summary

FOCI Policy: Focus on Object-Centric Interactions for Relational Manipulation Policies

Ze Fu, Pinhao Song, Yutong Hu, Renaud Detry

arXiv:2609.08743v1cs.RO

TL;DR

Action-centric policies often need many demonstrations across scene configurations, while existing object-centric representations can oversimplify or make interaction motion difficult to learn. FOCI POLICY automatically extracts interaction segments and models relative SE(3) motion between task-relevant objects. It reports strong performance across simulation and real-world tasks with substantially less training data, including 43.7% average success on RLBench-18 with one demonstration.

  • Problem

    Action-centric policies often require many demonstrations across diverse object poses and scene configurations, while object-centric methods can oversimplify interactions or use dense representations that are difficult to learn.

    Source: §1 · p. 2

  • Method

    FOCI POLICY automatically extracts interaction segments from demonstrations and predicts relative SE(3) motion between task-relevant objects from point clouds and language.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    Across RLBench, COLOSSEUM, and real-world tasks, FOCI POLICY reports strong one-demonstration performance, including 43.7% average success on RLBench-18.

    Source: §1 · p. 2, §5 · pp. 8–9

  • Takeaways & Limitations

    For rigid relational manipulation, the results suggest policies can be learned effectively without large-scale demonstrations by explicitly modeling interaction segments.

    Source: §5 · p. 9

    The framework assumes temporally separable, geometrically identifiable interactions, making it less applicable to continuous-contact tasks such as Push-T.

    Source: §5 · p. 9

Abstract

from arXiv · show

Object-centric manipulation policies improve generalization by modeling object motion instead of directly predicting robot actions. However, existing methods are often limited by representations which are either too simplistic to capture interaction dynamics or too dense to learn efficiently. We observe that many rigid relational manipulation tasks are governed by short interaction phases where the relative motion between task-relevant objects is tightly constrained. Based on this observation, we propose \textsc{Foci Policy}, an interaction-centric framework that achieves a two-fold abstraction: (1) temporally, by automatically extracting compact interaction segments from demonstrations;(2) spatially, by representing skills as relative $SE(3)$ motion between task-relevant objects, yielding invariance to scene configurations and robot embodiment. Experiments on RLBench, COLOSSEUM, and real-world tasks show that \textsc{Foci Policy} achieves strong performance with substantially less training data than prior object-centric and action-centric policies. These results suggest that modeling object-object interactions provides a simple and efficient inductive bias for rigid relational manipulation. Project page: \href{https://fitz0401.github.io/foci-page/}{fitz0401.github.io/foci-page/}.

1 Introduction

Action-centric policies learn observation-to-action mappings and often need many demonstrations across scene configurations, while existing object-centric representations can either miss interaction dynamics or be difficult to learn. FOCI POLICY instead extracts compact interaction segments and models relative object motion, reporting strong performance with one demonstration.

1 Introduction

  • Action-centric policies map observations directly to actions [3] [4] [5] [6] and typically require many demonstrations across object poses and scene configurations.Humans can infer object-relative motion from a single demonstration and reproduce it with different embodiments.Source: §1 · pp. 1–2
  • Static goal poses [7] [8] [9] can miss interaction dynamics, whereas dense motion flows [10] [11] are difficult to learn and prone to drift.These limitations motivate a compact representation of short intervals of constrained relative motion, rather than unconstrained transport.Source: §1 · p. 2
  • FOCI POLICY reports strong relational-manipulation performance with one demonstration, including 43.7% average success on RLBench-18.It also reports strong performance on RLBench, COLOSSEUM, and real-world tasks with a single demonstration.Source: §1 · p. 2
  • FOCI POLICY automatically extracts interaction segments and represents them as relative SE(3) motion between task-relevant objects, supporting invariance to scene configurations and robot embodiments.Its two-stage policy predicts these segments from object point clouds and language, then recovers executable actions.Source: §1 · p. 2

2 Related Work

Prior relational-manipulation policies use static object goals or dense motion flows, while temporal abstractions and task-and-motion planning address different parts of manipulation. FOCI POLICY represents skills as compact object-relative interaction segments.

Relational Manipulation Policies

  • Existing object-centric policies predict static object goal poses [7] [8] [9] or dense motion flows [10] [11], while FOCI POLICY models compact interaction segments between task-relevant objects.Object-centric representations can improve sample efficiency and task generalization, but static goals and dense flows have different limitations.Source: §2 · p. 2

Few-Demonstration Imitation Learning

  • FOCI POLICY exploits object-object interaction invariance across scene configurations for data-efficient generalization, contrasting with approaches relying on transferable priors, synthetic pretraining, or data augmentation [28] [30] [31] [12] [32] [33].Those alternatives can depend on domain alignment or augmentation quality.Source: §2 · p. 2

Relational Manipulation Policies

  • FOCI POLICY predicts object-relative motion first, then recovers end-effector trajectories by composing with the target-object pose for GOI tasks or the grasp offset for OOI tasks.The figure distinguishes these two action-recovery cases.Source: Figure 2 · p. 3

Temporal Abstraction for OOIs

  • Prior temporal abstractions extract sparse keyframes from end-effector motion or contact heuristics [3] [5] [34] [36], whereas FOCI POLICY abstracts object-centric interactions.The paper reports lower cross-demonstration trajectory variance than full trajectory imitation (Appendix A.1).Source: §2 · p. 3

Task and Motion Planning

  • Task-and-motion planning selects and composes skills, while FOCI POLICY learns the continuous object-relative interaction trajectory within a skill from demonstrations.This distinction matters for insertion or articulated-object manipulation not naturally expressed by fixed pick-and-place primitives.Source: §2 · p. 3

3 Method

FOCI POLICY learns short, constrained interaction segments in object-relative coordinates rather than full demonstrations. It automatically detects segment boundaries, predicts interaction goals and trajectories from object geometry and language, and recovers executable robot actions.

3 Method

  • The method isolates the interval where object relations are tightly constrained, excluding high-variance transport motion from the learning objective.This temporal abstraction targets the interaction segment τrel within [ts, tg] rather than the full episode [0, H].Source: §3 · p. 3
  • Representing interaction motion between task-relevant objects is intended to make skills independent of robot embodiment and scene configuration.The formulation reduces the state space to the relative motion that captures the task interaction.Source: §3 · p. 3
  • The predicted relative motion is converted to an end-effector trajectory using the estimated target pose for GOIs or the grasp offset for OOIs.For GOIs the gripper is the source and the offset is identity; for OOIs the offset maps the grasped object frame to the gripper, assuming a rigid grasp.Source: §3 · pp. 3–4

3.1 Problem Statement.

  • The policy learns a conditional distribution over relative interaction segments given source and target point clouds and a language instruction.At inference, the predicted segment is instantiated in the world frame to obtain executable actions.Source: §3.1 · p. 4

3.2 Identification of Interaction Segments

  • Change-point detection identifies the interaction interval from gripper linear and angular speeds and source-target distance, using a segment cost to isolate functional transitions without manual annotation.PELT finds change points; the segment cost uses an RBF kernel, and β controls the number of change points.Source: §3.2 · pp. 4–5

3.3 Object-Centric Interaction Prediction

  • The training data expresses source-object motion in the target frame and samples n waypoints within the detected interval, with both point clouds transformed into canonical frames.Each sample contains source and target point clouds, a language instruction, and the relative interaction segment.Source: Figure 3 · p. 4, §3.3 · p. 5
  • The model fuses source and target geometric features through cross-attention with language, then predicts an interaction goal and generates a trajectory conditioned on that goal.The goal stage uses a GMM to represent multiple valid goal modes; the trajectory decoder generates intermediate poses and is trained with goal-likelihood and trajectory-reconstruction losses.Source: §3.3 · pp. 5–6, Figure 4 · p. 5

4 Experiments

FOCI POLICY is evaluated across simulation benchmarks and real-robot tasks, with experiments testing data efficiency, robustness, interaction grounding, and deployment. The results show strong performance in low-data settings, while also exposing sensitivity to difficult interaction grounding and severe pose noise.

4 Experiments

  • The experiments test data efficiency on eight RLBench tasks, visual-shift robustness on COLOSSEUM, broader grounding difficulty on RLBench-18, and real-world performance across five tasks.The RLBench comparison uses one demonstration for FOCI POLICY versus five for all baselines; the real-robot evaluation uses one demonstration per task.Source: §4 · p. 6, §4.1 · p. 6, §4.2 · p. 8

4.1 Simulation Experiment

  • 25.3% vs. 20.1%/21.4% under All Perturbations: FOCI POLICY outperforms both VLA models on COLOSSEUM’s most severe visual-shift setting.The comparison is reported despite FOCI POLICY using only 1% of the data.Source: §4.1 · p. 7, Table 2 · p. 7
  • 79.6% success on easy-to-ground RLBench-18 tasks falls to 7.8% on hard-to-ground tasks; ground-truth poses raise the latter to 78.1%.The results identify reliable interaction grounding as a bottleneck under thin geometry, occlusion, or ambiguous interaction modes.Source: §4.1 · p. 7, Table 3 · p. 7
  • Ablations show that removing interaction segmentation substantially reduces performance, while the two-stage architecture consistently beats a one-stage trajectory predictor.The comparisons cover five RLBench tasks.Source: §4.1 · p. 7, Table 4 · p. 8

4.2 Real Robot Experiment

  • With one demonstration, FOCI POLICY outperforms replay on all five real-robot tasks, achieving over 53% task success across tasks and variations.The advantage is most pronounced on precision-critical Insert-Tube and Pour-Liquid tasks.Source: §4.2 · p. 8

5 Conclusion

FOCI POLICY represents relational manipulation as compact relative SE(3) motion between task-relevant objects, using automatically extracted interaction segments and a two-stage architecture. The paper reports one-shot generalization across simulation benchmarks and real-world tasks, while identifying boundaries in task structure, strategy coverage, and perception.

5 Conclusion

  • FOCI POLICY distills relational manipulation into compact relative SE(3) motion between task-relevant objects, with interaction segments extracted by change-point detection.The two-stage architecture models these interactions; the conclusion reports strong one-shot generalization across simulation benchmarks and real-world tasks.Source: §5 · p. 8
  • With one demonstration and a single-camera setup, FOCI POLICY achieves over 53% task success across all real-world tasks and variations.The conclusion’s table reports GOI, OOI, and full task success rates.Source: Table 5 · p. 9
  • The framework is less applicable to continuous-contact tasks, and one demonstration limits coverage when tasks admit multiple valid interaction strategies.It also relies on pretrained object segmentation and pose estimation; ambiguous or weakly observable geometry can degrade performance through unreliable canonicalization.Source: §5 · p. 9

A.1 Results of Interaction Segments

The appendix evaluates interaction-segment extraction by examining parameter selection, cross-demonstration variability, and whether detected intervals match meaningful task phases. The results indicate lower trajectory variance and task-dependent segment lengths, with a global penalty factor selected to balance compactness and consistency.

A.1 Results of Interaction Segments

  • Larger β yields longer intervals that preserve more interaction dynamics but raise trajectory variance; smaller β shortens intervals but risks omitting important motions.The parameter therefore balances interaction completeness against trajectory compactness.Source: §A.1 · p. 10
  • β = 0.6 provides the best cross-task trade-off between trajectory variance reduction and interaction-interval compactness, and is selected for experiments.The global choice is based on normalized translation variance, rotation variance, and interval length across RLBench-18 tasks.Source: §A.1 · p. 10, Figure 7 · p. 10
  • 78.6%/90.5% and 73.4%/92.3%: GOI and OOI segments reduce average translation/rotation variance relative to full RLBench-18 trajectories.The comparison uses 30 demonstrations per task.Source: §A.1 · p. 10, Table 6 · p. 11
  • After transformation into GOI and OOI object frames, interaction segments become more consistent across demonstrations than full world-frame trajectories.The visualization attributes the difference to isolating constrained manipulation phases while removing task-irrelevant transport variability.Source: §A.1 · p. 11, Figure 8 · p. 12
  • Detected intervals align with semantically meaningful phases, and the segmentation signals capture the transition from unconstrained transport to constrained interaction.The method also identifies reasonable intervals from noisier real-world signals.Source: §A.1 · p. 11
  • Interaction interval lengths are consistent within tasks but vary meaningfully across tasks, with precise alignment or sustained-contact tasks taking longer than simpler interactions.This pattern is reported as evidence that the method captures task-dependent interaction structure rather than fixed heuristics.Source: §A.1 · p. 11, Table 7 · p. 12

A.2 Robustness to noisy pose estimation on RLBench

The appendix stress-tests FOCI POLICY with added translation and rotation noise in estimated object poses. Performance degrades as noise increases, with task difficulty depending on geometric constraints and the reliability of pose estimation.

A.2 Robustness to noisy pose estimation on RLBench

  • Reliable pose estimation remains important for precision-critical tasks; improving geometric accuracy under challenging conditions is left to future work.The proposed direction includes stereo-based pose estimation.Source: §A.2 · pp. 12, 15
  • Under increasing pose noise, performance degrades gracefully overall, but severe perturbations cause over 50% relative drops on precision-critical Put-Umbrella and Put-Roll.Phone-on-base and stack-wine retain high success rates even under hard noise; the evaluation adds bounded translation and rotation perturbations at three levels.Source: §A.2 · p. 12, Table 8 · p. 15

A.3 Additional Results on RLBench-18 Tasks

On RLBench-18, reliable interaction grounding is the key qualification on FOCI POLICY’s low-data performance: estimated poses perform much worse on hard-to-ground tasks than ground-truth poses.

A.3 Additional Results on RLBench-18 Tasks

  • Ground-truth poses raise hard-to-ground performance from 7.8% to 78.1% and easy-to-ground performance from 77.9% to 98.2%.The authors attribute the gap primarily to reliable interaction grounding under imperfect object canonicalization.Source: §A.3 · p. 15
  • FOCI POLICY performs strongly with one demonstration on easy-to-ground tasks, but thin or symmetric objects and single-view occlusion make interaction grounding harder.Hard-to-ground examples include insert peg, sort shape, stack blocks, screw bulb, place cups, and stack cups; open drawer, put in drawer, and put groceries are affected by occlusion.Source: §A.3 · p. 15
  • The results identify reliable interaction grounding as a key bottleneck for data-efficient relational manipulation.The authors conclude that interaction-centric policies are highly data-efficient when task-relevant interactions can be reliably grounded.Source: §A.3 · p. 15

A.4 Additional Results with One Demonstration and One Camera

With one demonstration and one front-view camera, FOCI POLICY achieves a 63.0% average success rate across 10 RLBench tasks. The results highlight object-centric interaction modeling’s potential under extreme data constraints.

A.4 Additional Results with One Demonstration and One Camera

  • 63.0% average success across 10 RLBench tasks: FOCI POLICY significantly outperforms all baselines while requiring less training time.The evaluation uses one demonstration and a single front-view camera.Source: §A.4 · p. 16
  • Action-centric methods 3D Diffuser Actor and RVT-2 largely fail, with near-zero success on most tasks, while object-centric baselines perform better but inconsistently.The authors associate the object-centric variability with sensitivity to demonstration quality and consistency.Source: §A.4 · p. 16
  • These results highlight the importance of object-centric interaction modeling for robust manipulation under extreme data constraints.Source: §A.4 · p. 16

A.5 Additional Results on COLOSSEUM

With one demonstration, FOCI POLICY generalizes across diverse COLOSSEUM visual perturbations, while large geometric shifts remain challenging. Against VLA models trained with 100 demonstrations per task, it is strongest under the most challenging perturbation setting.

A.5 Additional Results on COLOSSEUM

  • COLOSSEUM tests 12 unseen distribution-shift categories, including changes to object appearance, lighting, background, distractors, and camera pose.The evaluation covers six representative tasks; each task–perturbation pair is tested over 25 trials and averaged across 3 runs.Source: §A.5 · p. 16
  • With one demonstration from the unperturbed environment, FOCI POLICY maintains robustness across visual variations, especially on tasks with well-defined geometric interactions.Generalization is non-trivial for unseen background, distractor, and camera-pose changes, but degrades more under large object-size shifts.Source: §A.5 · p. 16, Table 11 · p. 17
  • The results support interaction segments as an invariant, transferable representation for generalizing beyond training conditions without large-scale data.The authors also describe this representation as reducing reliance on large-scale demonstrations.Source: §A.5 · pp. 16–17
  • 50.8% average success with one demonstration—1% of the VLA data—and higher success than both VLA models under All Perturbations.VLA models achieve higher overall success rates, but use 100 demonstrations per task.Source: §A.5 · p. 17, Table 12 · p. 17

A.6 Ablation Study

The ablations favor automatic interaction segmentation and a two-stage policy that predicts a terminal interaction state before generating a trajectory. Special cases illustrate transfer across objects, conditions, and robot embodiments, alongside robot-free demonstrations.

A.6 Ablation Study

  • Automatic segmentation achieves the best or near-best results, while fixed intervals and full-trajectory learning degrade on precision-sensitive tasks.Full-trajectory learning nearly fails on put umbrella; fixed intervals can misalign with interaction phases, while full trajectories include task-irrelevant motion.Source: §A.6 · p. 17, Table 13 · p. 18
  • The two-stage model consistently outperforms the one-stage baseline across five tasks, with a larger gap on precision-sensitive put roll.It predicts a distribution over the terminal interaction state, then generates a trajectory conditioned on the sampled goal; the comparison uses five demonstrations and the same training iterations.Source: §A.6 · p. 18, Figure 11 · p. 18, Table 14 · p. 19
  • Pose features add comparable performance, while random masking improves robustness on most tasks, particularly put roll and put books.The authors omit pose features in the final model; removing masking can slightly improve put umbrella, where token dropping may discard critical geometry.Source: §A.6 · pp. 18–19

A.7 Special Case Study

  • The policy transfers zero-shot to visually distinct objects within a category, including a ceramic cup after training with a plastic cup.The authors attribute these cases to task-relevant geometric relationships rather than instance-specific appearance.Source: §A.7 · p. 19, Figure 12 · p. 20
  • The policy handles varied insertion configurations, clutter, and partial observations in the illustrated robustness cases.The reported examples include multiple valid insertion trajectories, distractor objects, and a target partly outside the camera view.Source: §A.7 · pp. 19–20, Figure 13 · p. 20
  • With one demonstration and no obstacle-avoidance training, the system completes scale-grape by planning collision-free paths between predicted interaction poses.The pipeline alternates motion planning with GOI and OOI inference.Source: §A.7 · p. 20, Figure 14 · p. 21
  • Using one Franka demonstration and a fixed gripper offset, FOCI POLICY transfers zero-shot to a Robotiq gripper across three tasks.The evaluated tasks are scale-grape, insert-tube, and open-drawer, with a changed table appearance.Source: §A.7 · p. 21, Figure 15 · p. 21
  • Robot-free human demonstrations yield stable manipulation despite hand-pose estimation noise, as interaction processing filters high-frequency errors.The pipeline estimates hand poses from RGB-D observations and approximates gripper poses from the thumb and index fingertips; the figure shows the processing and predicted trajectories.Source: §A.7 · pp. 21–22, Figure 16 · p. 22

A.8 Failure Cases

The failure analysis identifies pose estimation and motion planning as major bottlenecks, while the implementation details describe how interaction segments are detected and modeled from object data.

A.8 Failure Cases

  • Among 26 real-world failure trials, 54% stemmed from pose estimation, 27% from motion planning, and 19% from grasp instability.The authors focus subsequent analysis on pose estimation and motion planning as the primary system bottlenecks.Source: §A.8 · p. 22
  • Grasp instability occurs when an object shifts or slips during execution; the implementation estimates object pose only once, though tracking could mitigate this failure.The analysis therefore focuses on pose estimation and motion planning failures.Source: §A.8 · p. 22
  • Small or thin, symmetric, and occluded objects can defeat pose estimation, with resulting pose errors propagating into misaligned interaction trajectories and execution failures.Examples include limited geometric features, pose ambiguity, and partial visibility.Source: §A.8 · p. 23, Figure 18 · p. 23
  • With one demonstration, a semantically correct interaction can still yield an infeasible grasp, while a single observed interaction mode may miss placement variations.In Put-Plate, this can lead to an unfavorable grasp configuration or an incorrect placement; multiple demonstrations or language conditioning could mitigate the limitation.Source: §A.8 · p. 23, Figure 19 · p. 23
  • The authors characterize these limitations as challenges in interaction coverage and conditioning under minimal data, rather than consequences of the interaction-centric formulation itself.They identify multiple demonstrations or language conditioning as possible ways to mitigate limited interaction modes.Source: §A.8 · p. 24

B.1 Implementation Details

  • Training samples pair source and target object point clouds, relative motion, and language; estimated poses are converted to object-relative form, with gaps filled under a rigid-grasp assumption.Demonstrations yield multiple samples, and the highest-confidence estimated trajectory may be selected for supervision when multiple demonstrations are available.Source: §B.1 · p. 24
  • Interaction-segment detection combines gripper linear and angular speeds with object-centroid distance, then uses PELT change-point detection to locate the interaction interval.The start is the last detected change point before the goal time; the segmentation objective balances within-segment variation against a penalty on the number of change points.Source: §B.1 · p. 24
  • The model uses masked geometric encoding and multimodal fusion, then predicts a terminal pose distribution followed by intermediate trajectory poses.Point-cloud tokens are cross-attended, fused with language features, and processed by a Transformer encoder; the goal predictor uses a Gaussian mixture model.Source: §B.1 · pp. 24–25

B.2 Tasks and Baseline Details

The experiments span relational manipulation tasks in simulation and the real world, with comparisons against object-centric and action-centric policies. Real-world evaluation uses progressively harder pose and workspace variations, and the training pipeline is designed for rapid skill acquisition.

B.2 Tasks and Baseline Details

  • Eight RLBench tasks cover insertion, cluttered or occluded placement, and articulated or multistep manipulation, with test-time object poses randomly sampled.The random initial poses test generalization to unseen object configurations.Source: §B.2 · p. 25
  • Simulation comparisons include object-centric methods that predict object configurations or trajectories and action-centric methods that directly predict robot actions.The baselines are Imagination Policy, SPOT, 3D Diffuser Actor, and RVT-2.Source: §B.2 · p. 25
  • The RLBench-18 taxonomy groups tasks by perceptual grounding difficulty and interaction structural ambiguity, separating easy-to-ground from hard-to-ground tasks.The hard-to-ground group includes thin, symmetric, or heavily occluded objects.Source: §B.2 · p. 26
  • Real-world evaluation covers five relational manipulation tasks, including tube insertion, pouring, and opening a drawer.Pour-Liquid success is defined as at least three of ten small balls landing in the cup.Source: §B.2 · p. 26
  • The real-world Replay Policy transfers demonstrated gripper poses using estimated rigid transformations of the source and target objects, with both methods given one demonstration per task.Replay uses four key states, and FOCI uses two interaction waypoints for this comparison.Source: §B.2 · p. 27

B.3 Real-robot Experiment Details

  • Real-robot evaluation uses 15 trials per task, progressing from small pose perturbations to larger pose variations and then swapped object regions.The setup uses a Franka Emika Panda and a front-facing RGB-D camera, with separate goal-object and other-object regions.Source: §B.3 · p. 27, Figure 20 · p. 27
  • The real-world pipeline from demonstration to deployment takes under 10 minutes.The described stages include demonstration collection, mesh acquisition and processing, preprocessing, and model training.Source: §B.3 · pp. 27–28, Figure 21 · p. 28

Related papers

Loading 2609.08743v1…