PaperlayerView exact revision on arXiv

Source-linked AI summary

Phase-Conditioned Affordance Fields for SE(3)-Equivariant Visuomotor Policies

Xinzhe Chen, Liqi Huang, Sihua Ren, Haowen Sun, Mingyang Li, Zeyang Liu, Xingyu Chen, Xuguang Lan

arXiv:2610.17281v1cs.ROcs.AI

TL;DR

Global SE(3)-equivariant visuomotor policies use one transformation for the whole observation, although multi-object tasks shift the relevant geometry across phases. PAF conditions an equivariant policy on phase-conditioned point scores and their centroid anchor. It improves the underlying policy’s success across MimicGen settings and leads on real-robot tasks under both lighting conditions.

  • Problem

    Existing SE(3)-equivariant visuomotor policies assume one rigid transformation governs the whole observation, an assumption that does not capture changing task-relevant regions in multi-object manipulation.

    Source: §1 · p. 1

  • Method

    PAF scores observed points using recent robot history to infer phase, uses the score-weighted centroid to recenter the policy on the active region, and adds scores as point features.

    Source: §1 · p. 2

  • Results

    Across twelve MimicGen settings, PAF raises average SDP success from 48.8% to 56.3%, and it leads SDP and DP3 on six real-robot tasks under both lighting conditions.

    Source: §1 · p. 2, §5.3 · p. 9

  • Takeaways & Limitations

    PAF provides a phase-conditioned local geometric bias while preserving global SE(3) equivariance, with gains reported in simulation and real-robot evaluations.

    Source: §6 · p. 9

    PAF supplies a learned bias rather than strict local SE(3)-equivariance; its score invariance and anchor equivariance are provable only under a shared global transformation.

    Source: §7 · p. 9

Abstract

from arXiv · show

SE(3)-equivariance provides a useful geometric inductive bias for visuomotor policies, encoding that actions covary with rigid transformations of the observation. However, existing SE(3)-equivariant visuomotor policies build this bias on a global equivariance assumption that treats the entire observation as governed by a single rigid transformation, which breaks down in real-world manipulation tasks where the task-relevant region changes across different objects. We address this by learning a phase-conditioned affordance field (PAF) over the observed point cloud, a per-point score that the SE(3)-equivariant policy uses in two complementary ways. Its score-weighted centroid serves as a learned, phase-conditioned anchor that canonicalizes the observation and the action around the active region, replacing the fixed gripper-frame reference, and the score itself is appended as a per-point feature. This introduces a local geometric inductive bias, while preserving the policy's global SE(3) equivariance. We validate on multiple MimicGen tasks under varying spatial randomization, and PAF consistently improves the success rate of the underlying policy. On multi-object real-robot tasks, PAF outperforms baselines and keeps its lead under lighting variations.

1 Introduction

Global SE(3)-equivariant policies use one scene-wide reference, which cannot capture the changing active geometry of multi-object tasks. PAF uses phase-conditioned point scores to anchor the policy on that geometry, improving results in simulation and on real-robot tasks.

1 Introduction

  • Multi-object tasks shift the relevant geometry across phases, so one global scene transformation cannot select which object geometry should govern each action.Object-level formulations can be difficult to apply when object poses are unavailable or unreliable and phases involve inter-object interactions.Source: §1 · p. 1
  • Figure 1 contrasts SDP’s single global frame with PAF’s phase-varying affordance scores and centroid anchor across grasp, move, and hang.The proposed field supplies a local geometric inductive bias while preserving global SE(3) equivariance.Source: Figure 1 · p. 2, §1 · p. 2
  • PAF scores each observed point for phase relevance, uses the score-weighted centroid as an anchor, and appends the score as a point feature for the SDP backbone.It uses recent end-effector and gripper history to form phase context and self-supervised pseudo-labels from expert demonstrations, without segmentation labels.Source: §1 · p. 2
  • Across twelve MimicGen settings, PAF raises average SDP success from 48.8% to 56.3%, and on six real-robot tasks it leads under both lighting conditions.The largest simulation gain is at d3, from 48.5% to 60.3%; the real-robot average is 67.0% under normal lighting, 9.0 points above SDP, and its lead persists under out-of-distribution lighting.Source: §1 · p. 2

2 Related Work

Prior visuomotor policies use diffusion models, geometric equivariance, or local frames, but often rely on implicit geometric generalization, fixed references, or separate modules. PAF derives a phase-conditioned frame from affordance scores within one closed-loop policy.

2 Related Work

  • Diffusion policies capture multimodal actions, while affordance-prior methods such as AnchorDP3 [23] and AffordDP rely on external segmentation or foundation models.These methods leave the observation-to-action transformation largely implicit, so geometric generalization depends on demonstration pose coverage.Source: §2 · p. 2
  • Equivariant policies encode rigid-transformation relationships, but SDP and related methods tie actions to fixed or global reference frames despite phase-shifting task geometry.EquiDiff uses discrete SO(2)-equivariance; SDP uses continuous SO(3)-equivariance and gripper-frame canonicalization.Source: §2 · p. 2
  • Unlike local-frame methods that predict a goal pose or use hierarchical policies and external perception, PAF derives its frame origin from a phase-conditioned affordance field in one closed-loop policy.The field’s score-weighted centroid is the anchor, and its scores condition the policy while global SE(3) equivariance is retained.Source: §2 · p. 3

3 Global SE(3) Equivariance

The policy observes a point cloud together with recent robot history and predicts an action trajectory. SDP canonicalizes positions around the gripper and handles rotation equivariantly, but this global construction cannot accommodate independently moving objects.

3 Global SE(3) Equivariance

  • The policy state combines a point cloud with recent end-effector position, rotation, and gripper history, and its output is an action trajectory.Point-cloud points include coordinates and non-geometric features such as color; each action specifies translation, rotation, and a gripper command.Source: §3 · p. 3
  • SE(3)-equivariance means that transforming the observed state by a rigid transformation transforms the predicted action trajectory by the same transformation.The policy is required to satisfy this relationship for any rigid transformation.Source: §3 · p. 3
  • SDP removes translation by centering point positions and actions on the current end-effector, then uses an SO(3)-equivariant backbone for rotation.This canonicalization assumes one global rigid transformation acts on the entire observed state.Source: §3 · p. 3
  • With independently moving objects, gripper-frame canonicalization absorbs at most one object’s translation and leaves residual offsets for other objects.A shared scene rotation likewise cannot represent the combined motion of independently moving objects.Source: §3 · p. 3

4 Method

PAF uses recent robot motion to identify the manipulation phase and a phase-conditioned affordance field to focus the policy on the active region. It recenters and augments the observation for an equivariant backbone, while treating local equivariance as an empirical target rather than a guarantee.

4.1 Design Principle of PAF

  • PAF targets phase-local equivariance, but does not guarantee it because transforming only the active region changes cross-region geometric relationships.The design instead seeks to limit the effect of phase-irrelevant geometry on predictions.Source: §4.1 · pp. 3–4

4.2 Phase Context Encoding

  • The phase context encodes recent robot motion using invariant statistics, then conditions the affordance predictor and phase-to-slot router.The statistics include speed, gripper change, angular displacement, speed change, turning angle, and gripper state.Source: §4.2 · p. 4

4.3 Phase-Conditioned Affordance Field

  • The affordance field combines point scores with phase-selected slot memberships, and its score-weighted centroid provides the active-region anchor.The field predictor uses local geometry and end-effector relationships under the current phase context.Source: §4.3 · pp. 4–5
  • The anchor recenters positional inputs, while the affordance score is appended as a point feature before the observation reaches the SDP backbone.Predicted action positions are shifted back by the anchor at inference.Source: §4.1 · p. 4, §4.3 · p. 5

4.4 Training Objective

  • Training combines denoising, score, phase-classification, and slot-routing losses, with score pseudo-labels derived from future action positions rather than object annotations.Sub-phase and cycle labels are derived from gripper commands and observed gripper states.Source: §4.4 · p. 5

4.5 Equivariance Analysis of PAF

  • Under a shared global rigid transformation, scores remain invariant and the anchor transforms with the scene.For transformations restricted to the active region, cross-region KNN edges change, so the anchor does not track the transformation exactly.Source: §4.5 · p. 6, §A.4 · p. 14

5 Experiments

PAF improves simulation and real-robot success, while visualizations and analyses show phase-specific scoring and a learned anchor that focuses the policy on active geometry. The experiments also probe robustness across pose randomization, lighting changes, and local perturbations.

5.1 Simulation Experiments

  • 56.3% average success for PAF versus 48.8% for SDP across twelve MimicGen settings; at d3, the averages are 60.3% and 48.5%.The authors suggest PAF is most useful under full SE(3) randomization when the underlying policy can still exploit recovered geometric structure.Source: §5.1 · p. 7

5.2 In-Depth Analysis

  • In simulation visualizations, the field follows the object governing each sub-phase, including a shift from the nut to the peg on Square and across three objects on Coffee.Figure 4 marks the top-256 scored points and the anchor.Source: §5.2 · p. 7, Figure 4 · p. 7
  • The learned anchor is the dominant ablation component: removing it costs 17.3 points on average, while a directly regressed keypose anchor costs 16.3 points.Hard scope costs 9.3 points; removing phase, routing, or the score channel costs 6.5, 6.0, and 4.8 points, respectively.Source: §5.2 · pp. 7–8
  • Under inactive perturbations, the anchor moves at most 0.05 mm for 30 mm translations and 0.03 mm under rotation; active rotations change the affordance score by about 2 × 10^-4.PAF also has higher active-to-inactive action-change ratios than the w/o-anchor variant under translation and rotation on Square and Coffee; these tests characterize selectivity and stability, not exact local equivariance.Source: Table 3 · p. 7, §5.2 · pp. 8–9

5.3 Real-Robot Experiments

  • Across six real-robot tasks, policies trained on normal-lighting demonstrations were evaluated under normal and colored OOD lighting, which changes RGB but not geometry.Four tasks varied object poses across episodes; point clouds came from a single RGB-D image without multi-view fusion.Source: Figure 5 · p. 8, §5.3 · p. 9
  • 67.0% success for PAF versus 58.0% for SDP and 52.3% for DP3 under normal lighting; under OOD lighting, the rates are 59.8%, 49.2%, and 40.3%.PAF wins every task in both conditions, with a significant lead over SDP across 240 OOD-lighting trials (p = 0.02).Source: Table 4 · p. 8, §5.3 · p. 9
  • On real-robot tasks, the field shifts among phase-relevant objects—for example, from cup to drawer, back to cup, then to the drawer front during cup storage.Figure 6 pairs RGB keyframes with point clouds and highlights the top-256 predicted scores.Source: Figure 6 · p. 8, §5.3 · p. 9

6 Conclusion

PAF augments an SE(3)-equivariant visuomotor policy with a phase-conditioned affordance field that scores observed points and recenters the policy on the active region. It improves SDP's average success on MimicGen settings and leads SDP and DP3 on six real-robot tasks under both lighting conditions.

6 Conclusion

  • PAF augments an SE(3)-equivariant visuomotor policy with a local geometric inductive bias.Its point scores define a phase-conditioned anchor and are also fed back as a per-point feature; the reported results improve on MimicGen and lead baselines on six real-robot tasks under both lighting conditions.Source: §6 · p. 9

7 Limitations

PAF’s limitations concern the strength of its local equivariance guarantees, its affordance supervision, and the flexibility of its slot design.

7 Limitations

  • PAF guarantees score invariance and anchor equivariance only under a shared global transformation, not strict local SE(3)-equivariance.Its pseudo-labels come from future end-effector positions and may miss fine contact-rich geometry; slot count is tied to a task’s grasp-release structure.Source: §7 · p. 9

A.1 Multi-Object Equivariance Derivations

The derivation explains why a single scene reference is inadequate for independently moving objects, then details the phase-conditioned context and graph used to score points.

A.1 Multi-Object Equivariance Derivations

  • A fixed end-effector reference leaves residual offsets uj − ue under independent object translation, while independent rotations violate the network’s shared-rotation assumption.PAF instead learns a phase-conditioned anchor that recenters the region governing the current phase.Source: §A.1 · p. 13

A.2 Definitions of Phase Context Vector

  • The phase context comprises six rigid-motion-invariant statistics from recent robot history: speed, speed change, turning angle, angular displacement, gripper state, and gripper change.These statistics depend on displacement norms, angular differences, or scalar gripper values.Source: §A.2 · pp. 13–14

A.3 Graph Construction Details

  • The graph connects point and recent end-effector nodes by k-nearest neighbors and encodes distances with radial-basis-function edge features.The phase context is broadcast into node features before message passing.Source: §A.3 · p. 14
  • Independent MLP heads decode each point’s final message-passing feature into a base score logit and slot-membership logits.Source: §A.3 · p. 14

A.4 Proof of Score Invariance and Anchor Equivariance

The proof establishes global SE(3)-equivariance under a shared transformation, and the following section specifies how phase targets, slots, and affordance supervision are formed.

A.4 Proof of Score Invariance and Anchor Equivariance

  • Under a shared global SE(3) transformation, PAF’s scores remain invariant and its anchor transforms equivariantly, but the proof does not cover arbitrary object-wise transformations.Source: §A.4 · pp. 14–15

A.5 Phase Label and Slot Routing

  • The affordance pseudo-label targets the expert end-effector position at the first future gripper open-close transition, falling back to the horizon-end position when none occurs.The label is damped near the current end-effector and trained with binary cross-entropy.Source: §A.5 · p. 15
  • Phase labels distinguish open, opening-to-closed, closed, and closing-to-open horizons, while cycle count and gripper state determine the slot label.Source: §A.5 · p. 15
  • Per-point slot-membership logits receive no direct supervision; they are shaped through the anchor and score-feature inputs under the diffusion-policy loss.Source: §A.5 · p. 15

B.1 MimicGen Tasks

The section evaluates PAF across four multi-object tasks and spatial-randomization levels, compares it with several policy baselines, and checks performance against SDP on matched demonstrations.

B.1 MimicGen Tasks

  • The evaluation covers Square, Coffee, Threading, and Three-Piece Assembly at d2, d3, and d4 randomization levels, with increasing pose-randomization range.Source: §B.1 · p. 15

B.2 Baselines

  • Baselines span equivariant and non-equivariant policies using point-cloud, voxel, and RGB inputs; three methods are omitted because their settings or required inputs differ.The w/o phase ablation serves as an in-setting phase-agnostic affordance comparison.Source: §B.2 · p. 16

B.3 SDP Reproduction and Seed Variability

  • The SDP reproduction uses PAF’s training demonstrations and the same evaluation protocol, with regenerated demonstrations for d3 and d4 and published SDP results retained separately.Source: §B.3 · p. 16
  • 56.3% for PAF versus approximately 47.6% for reproduced SDP across twelve settings; PAF outperforms reproduced SDP in every matched-demonstration setting.The table reports mean success rates and standard deviations across three seeds.Source: Table 5 · p. 16, §B.3 · p. 16

B.4 Implementation Details

The implementation keeps the SDP backbone and uses task-dependent slot counts, while the real-robot evaluation covers six tasks with varying object layouts, phase labels, and lighting conditions.

B.4 Implementation Details

  • The diffusion backbone matches SDP, and the same-data comparison is designed to isolate the contribution of the phase-conditioned affordance field.The listed backbone parameters are inherited from SDP; optimization uses AdamW and EMA weight averaging.Source: §B.4 · p. 17

C.1 Setup and Rollouts

  • The six real-robot tasks span four pose-varying object layouts and two fixed layouts, including three tasks with non-upright object orientations.The setup uses a Franka Research 3 with a parallel-jaw gripper and a single third-person RealSense D435 camera; point clouds come from direct depth-image back-projection without multi-view fusion.Source: §C.1 · pp. 17–18, Figure 7 · p. 18
  • Real-robot policies use K = 2 slots, with supervision from four sub-phase classes derived from gripper commands and cycle index.Figure labels describe upcoming end-effector state changes and map to these classes; they are not additional supervision classes or fine-grained model predictions.Source: §C.1 · pp. 18–19
  • Figure 8 presents per-task PAF keyframes under normal and colored lighting, while Figure 9 marks representative failure moments.The simulation hyperparameter table uses task-dependent K = 2 × max_cycles; max_cycles is 1 for Square, Coffee, and Threading, and 2 for Three-Piece Assembly.Source: §B.4 · p. 16, Table 6 · p. 17, Figure 8 · p. 18

C.2 Failure Case Analysis

Representative failures distinguish earlier localization errors from later contact-rich interaction errors. The affordance-field module adds inference time, while the shared equivariant diffusion backbone remains the main bottleneck.

C.2 Failure Case Analysis

  • PAF rarely exhibits SDP’s earlier-phase misses across four tasks, but its representative failures concentrate in later contact-rich interactions.Examples include slipping during cup-loop hanging or flower-stem rotation, failed tool hooking, and a misaligned drawer-closing push; the authors identify fine-grained contact control as remaining headroom.Source: §C.2 · p. 19

C.3 Inference Overhead

  • Approximately 100 ms of added affordance-field prediction is a 13% overhead over SDP, whose shared equivariant backbone accounts for most of its 750 ms baseline inference time.DP3 is faster with a simpler non-equivariant backbone, at the cost of substantially lower success rates.Source: §C.3 · p. 19, Table 7 · p. 19

Related papers

Loading 2610.17281v1…