PaperlayerView exact revision on arXiv

Source-linked AI summary

ObstaDiff: Generalizable Diffusion Policy Learning via Obstacle-aware Representations

Jiawen Wang, Kevin Yao, Khalid Jawed

arXiv:2609.10918v1cs.ROcs.LG

TL;DR

Imitation learning remains difficult to generalize from clean settings to cluttered scenes with unstructured obstacles and variable visual conditions. ObstaDiff addresses this gap with a decomposed diffusion policy and a lightweight target–obstacle–background encoder, achieving higher success and lower obstacle collisions in greenhouse trials than representative imitation-learning baselines.

  • Problem

    Imitation-learning policies trained in clean settings struggle to generalize to cluttered scenes with unstructured obstacles and substantial visual variation.

  • Method

    ObstaDiff combines a lightweight obstacle-aware target–obstacle–background encoder with decomposed diffusion-based alignment and replay-based close-range interaction.

  • Results

    75.41% average task success and 8.20% average obstacle collision rate were achieved across 61 real-robot greenhouse trials per method, outperforming representative imitation-learning baselines.

  • Takeaways & Limitations

    The results support obstacle-aware approach behavior and improved generalization across target-pose, obstacle-layout, and target-appearance shifts in cluttered agricultural scenes.

  • Takeaways & Limitations

    Evaluation is confined to one indoor greenhouse testbed with three obstacle plant species and one target crop, while interaction remains replayed rather than learned.

Abstract

from arXiv · show

Imitation learning has achieved impressive results in robotic manipulation, yet most existing approaches assume clean backgrounds and lack explicit mechanisms for obstacle-aware motion generation. Extending such policies to cluttered, real-world scenes with unstructured obstacles remains a key generalization challenge. We present ObstaDiff, a decomposed diffusion-policy framework with a lightweight obstacle-aware visual encoder. ObstaDiff extracts a structured target-obstacle-background representation, enabling the downstream alignment policy to generate end-effector trajectories toward a target-centered bottleneck pose while reasoning about surrounding obstacles. We evaluate ObstaDiff on 61 real-robot greenhouse trials per method (366 executions in total). ObstaDiff achieves 75.41% average task success and 8.20% average obstacle collision rate, outperforming representative imitation-learning baselines and improving generalization in cluttered agricultural scenes.

1 Introduction

ObstaDiff addresses the difficulty of applying imitation learning to cluttered scenes with irregular obstacles and changing visual conditions. It combines structured obstacle-aware representations with a decomposed diffusion-policy framework and reports improved greenhouse generalization.

  • Cluttered agricultural scenes combine narrow, irregular passages with variation in viewpoints, illumination, temporal conditions, and target appearance.These conditions make fixed perception pipelines and policies trained on limited clean-scene demonstrations difficult to generalize.
  • ObstaDiff transforms RGB-D observations into an explicit target–obstacle–background representation for obstacle-aware diffusion-policy conditioning.The representation highlights target–obstacle spatial relationships and suppresses irrelevant background variation.
  • The method is designed to make both the target to reach and the obstacles to avoid explicit to the downstream policy.
  • The decomposed framework separates diffusion-based alignment from replay-based close-range interaction, allowing interaction objectives to change through the replay library.A data-driven bottleneck rule selects the interaction trajectory.
  • 75.41% average task success and 8.20% average obstacle collision rate were achieved across 61 real-robot greenhouse trials per method, outperforming representative imitation-learning baselines.The 366 executions covered target-pose, obstacle-layout, and target-appearance generalization.

2 Related Work

Prior work improves imitation learning through stronger visual representations, diffusion-based control, and obstacle-aware motion generation. ObstaDiff differs by explicitly structuring target–obstacle relationships rather than relying on generic image or point-cloud representations.

  • Visual imitation learning has progressed from spatial action prediction and multimodal behavior cloning to temporally coherent policies, while representation quality remains important for robust manipulation.
  • ObstaDiff uses a task-structured target–obstacle–background representation instead of a generic image or point-cloud backbone.This makes the spatial relation between the target and obstacles explicit.
  • Diffusion-policy research applies trajectory denoising and receding-horizon action generation to robotic planning and control, with later methods adding 3D conditioning, equivariance, or reusable sparse policy structure.
  • Cluttered agricultural motion methods can generate obstacle-aware trajectories, but are typically planners rather than closed-loop visuomotor imitation policies.Prior systems also use engineered perception, navigation, view planning, or shape completion.

3 Methodology

ObstaDiff combines a structured target–obstacle–background observation encoder with a decomposed policy that separates obstacle-aware alignment from close-range interaction. A conditional diffusion policy reaches a target-centered bottleneck pose, then switches online to replayed interaction actions.

  • 3.1 Problem Formulation: ObstaDiff separates visually guided alignment from close-range interaction around a target-centered bottleneck pose.The alignment endpoint is determined online by a gated nearest-neighbor criterion rather than a pre-specified pose.
  • 3.2 Structured Target–Obstacle–Background Observation: The four-channel TOB observation encodes background, obstacle, target, and normalized depth to preserve spatial relationships while suppressing background variation.An open-vocabulary detector produces semantic regions, which are converted into three masks and concatenated with aligned depth.
  • 3.3 Obstacle-aware Structured Observation Encoder: A lightweight StructuredObsEncoder uses separate semantic and depth branches with spatial softmax to extract compact target–obstacle geometry.The branch outputs are concatenated with end-effector pose; with two observation steps, the conditioning vector is 206-D.
  • 3.4 Alignment-stage Diffusion Policy: The alignment policy is a conditional diffusion model that predicts end-effector action sequences toward the bottleneck pose from encoded structured observations.Training corrupts clean alignment actions and trains a noise-prediction network; inference iteratively denoises 16-action sequences and executes the first 8 actions before replanning.
  • 3.5 Bottleneck Switching and Interaction: The system switches to the nearest replayed interaction trajectory when the current target state is sufficiently close to a stored initial target state.This decoupling lets one alignment policy support touching, harvesting, or close-up inspection by changing only the replay library, while avoiding unreliable close-range depth.

4 Experiments

Experiments evaluate ObstaDiff on real-robot greenhouse manipulation across target-pose, obstacle-layout, and target-appearance generalization. The study compares matched imitation-learning baselines, input ablations, and training behavior.

  • 4.1 Experimental Protocol: The protocol tests 61 distinct configurations: 20 target poses, 21 obstacle layouts, and 20 unseen target-appearance trials.OLG varies known plant species in unseen spatial arrangements, while TAG uses unseen green pepper varieties.
  • 4.2 Main Results: 75.41% mean TSR and 8.20% mean OCR make ObstaDiff the best method across the three 61-trial generalization settings.The evaluation comprises 366 executions total, with each method run on TPG, OLG, and TAG trials.
  • 4.2 Main Results: ObstaDiff outperforms ACT (RGB) and DP (RGB), with mean TSR of 75.41% versus 45.90% and 49.18%, respectively.Its mean OCR is 8.20%, compared with 14.75% for ACT (RGB) and 26.23% for DP (RGB).
  • 4.3 Ablation Studies: Depth or TOB inputs alone do not reproduce the full method: DP (RGB-D) reaches 50.82% TSR, while DP (TOB) reaches 45.90%.DP (TOB) has 27.87% mean OCR, whereas ObstaDiff uses the same TOB observation with a structured encoder.
  • 4.3 Ablation Studies: The structured encoder accounts for a 29.5-point TSR gap between DP (TOB) and ObstaDiff under otherwise matched observations, demonstrations, and batch size.Under obstacle-layout shift, DP (TOB) records 38.10% success and 38.10% collision rates.
  • 4.3 Ablation Studies: Adding TOB to ACT lowers mean OCR from 14.75% to 11.48% and improves smoothness from 1.16 to 0.77, but lowers mean TSR from 45.90% to 40.98%.TOB also reduces validation-loss fluctuation, showing that lower supervised loss alone does not guarantee higher real-robot success.

5 Limitations

ObstaDiff is evaluated within a constrained scope: human- or prompt-specified targets, demonstration-limited alignment, replayed interaction, and one indoor greenhouse setting.

  • 5 Limitations: The study does not address autonomous target selection or task planning because targets are human- or prompt-specified.The learned component focuses on low-level obstacle-aware alignment.
  • 5 Limitations: Results concern obstacle-aware approach behavior rather than contact-rich manipulation because the interaction stage replays demonstrated trajectories.The policy remains constrained by the demonstration distribution and has limited ability to explore behaviors outside collected data.
  • 5 Limitations: Evaluation is confined to one indoor greenhouse testbed with three obstacle plant species and one target crop.The comparison against point-cloud policies remains open in the moving eye-in-hand regime.

6 Conclusion

ObstaDiff uses a lightweight obstacle-aware encoder and decomposed execution scheme for cluttered greenhouse manipulation. Its structured representation and deployment design support obstacle-aware action generation and improved generalization beyond raw visual imitation.

  • 6 Conclusion: ObstaDiff converts eye-in-hand RGB-D observations into a structured target–obstacle–background representation for obstacle-aware action generation.The representation separates the target to reach, obstacles to avoid, and background to ignore.
  • 6 Conclusion: The decomposed execution scheme assigns visually guided alignment to the learned policy and close-range interaction to replayed actions.This avoids relying on unreliable eye-in-hand depth during contact-rich motions and supports different interaction objectives through the replay library.
  • 6 Conclusion: Real-robot experiments under target-pose, obstacle-layout, and target-appearance shifts show improved task success and obstacle avoidance over imitation-learning baselines.

A.1 Target–Obstacle–Background Preprocessing Details

The preprocessing pipeline converts eye-in-hand RGB-D observations into a four-channel target–obstacle–background input. It performs semantic parsing, mask refinement, depth alignment, normalization, and temporal batching for policy inference.

  • A.1 Target–Obstacle–Background Preprocessing Details: RGB and aligned depth streams are acquired from an eye-in-hand camera, with invalid depth values temporally hole-filled before processing.The RGB stream uses /camera/color/image_raw, while depth comes from /camera/aligned_depth_to_color/image_raw.
  • A.1 Target–Obstacle–Background Preprocessing Details: Emitter even–odd alternation reduces infrared-projector artifacts in RGB frames, while depth and RGB buffers are truncated to equal lengths.
  • A.1 Target–Obstacle–Background Preprocessing Details: YOLOv8s-Worldv2 detects the selected pepper fruit target and obstacles, whose boxes are refined into pixel-accurate masks with MobileSAM.The resulting semantic index map assigns per-pixel target, obstacle, and background labels.
  • A.1 Target–Obstacle–Background Preprocessing Details: The policy input resizes three semantic masks and aligned depth to 240 × 320, normalizes depth to [0, 1], and uses the channel order [bg_mask, obst_mask, plant_mask, depth_norm].Batched observations have shape X ∈ R^B×T_o×4×240×320 with T_o = 2.

A.2 Structured Observation Encoder

The StructuredObsEncoder separately processes semantic masks and depth, converts feature activations into keypoint-like coordinates, and combines them with end-effector pose for diffusion-policy conditioning.

  • A.2 Structured Observation Encoder: The encoder splits each four-channel TOB observation into three semantic-mask channels and one normalized-depth channel.
  • A.2 Structured Observation Encoder: Semantic and depth branches use lightweight CNNs with four convolutional blocks and channel-specific widths.The convolutional layers use kernels and strides (5×5, s=2), (3×3, s=2), (3×3, s=1), and (3×3, s=1).
  • A.2 Structured Observation Encoder: Spatial softmax converts branch feature activations into keypoint-like coordinates instead of flattening the feature maps.The semantic branch produces 32 coordinate pairs, yielding z^seg_t ∈ R^64.
  • A.2 Structured Observation Encoder: Branch outputs are concatenated with the 7-D end-effector pose to form the diffusion policy’s conditioning vector.With two observation steps, the final conditioning input to the diffusion U-Net has dimension 206.

A.3 Bottleneck Switching and Interaction Details

ObstaDiff transitions from learned alignment to replayed interaction through a target-centered nearest-neighbor rule. The switch uses depth readiness and image-plane proximity to select and execute a demonstrated interaction sequence.

  • A.3 Bottleneck Switching and Interaction Details: The replay library stores interaction samples and their initial target states, with N_lib = 26 in the implementation.Each replay entry includes an interaction-stage action sequence and a target state extracted from its first frame.
  • A.3 Bottleneck Switching and Interaction Details: The target state consists of the tracked target center and target depth, allowing current alignment observations to be compared with replay-entry initial states.
  • A.3 Bottleneck Switching and Interaction Details: Target depth is estimated from the p-th percentile of valid depth pixels in the central 40% of the target box, using p = 20.This biases the estimate toward the visible foreground surface of the target.
  • A.3 Bottleneck Switching and Interaction Details: The system selects the smallest-image-plane-error replay among depth-ready entries and switches when the selected replay satisfies the depth and image-plane gates.The experiments use a 5 mm depth gate and normalized image-plane error threshold ϵ_uv = 0.2.
  • A.3 Bottleneck Switching and Interaction Details: After switching, the system executes the interaction action sequence associated with the selected replay, avoiding reliance on eye-in-hand depth during close-range interaction.

B TOB Preprocessing Visualization

Figure 5 shows how raw RGB-D observations are transformed into structured TOB inputs for the policy. The pipeline separates background, obstacles, and target, then combines these masks with normalized depth.

  • B TOB Preprocessing Visualization: The TOB preprocessing pipeline converts raw RGB observations into background, obstacle, and target masks alongside normalized depth.The displayed outputs proceed from the raw image and semantic parsing to the three semantic masks and depth channel.
  • B TOB Preprocessing Visualization: The policy input stacks the three semantic masks with the normalized depth channel into a four-channel observation.
Loading 2609.10918v1…