ObstaDiff: Generalizable Diffusion Policy Learning via Obstacle-aware Representations

• Cluttered agricultural scenes combine narrow, irregular passages with variation in viewpoints, illumination, temporal conditions, and target appearance.These conditions make fixed perception pipelines and policies trained on limited clean-scene demonstrations difficult to generalize.Source: §1 · p. 2
• ObstaDiff transforms RGB-D observations into an explicit target–obstacle–background representation for obstacle-aware diffusion-policy conditioning.The representation highlights target–obstacle spatial relationships and suppresses irrelevant background variation.Source: §1 · p. 2
• The method is designed to make both the target to reach and the obstacles to avoid explicit to the downstream policy.Source: §1 · p. 2
• The decomposed framework separates diffusion-based alignment from replay-based close-range interaction, allowing interaction objectives to change through the replay library.A data-driven bottleneck rule selects the interaction trajectory.Source: §1 · p. 2
• 75.41% average task success and 8.20% average obstacle collision rate were achieved across 61 real-robot greenhouse trials per method, outperforming representative imitation-learning baselines.The 366 executions covered target-pose, obstacle-layout, and target-appearance generalization.Source: §1 · p. 2

• Visual imitation learning has progressed from spatial action prediction and multimodal behavior cloning to temporally coherent policies, while representation quality remains important for robust manipulation.Source: §2.1 · p. 2
• ObstaDiff uses a task-structured target–obstacle–background representation instead of a generic image or point-cloud backbone.This makes the spatial relation between the target and obstacles explicit.Source: §2.1 · p. 3
• Diffusion-policy research applies trajectory denoising and receding-horizon action generation to robotic planning and control, with later methods adding 3D conditioning, equivariance, or reusable sparse policy structure.Source: §2.2 · p. 3
• Cluttered agricultural motion methods can generate obstacle-aware trajectories, but are typically planners rather than closed-loop visuomotor imitation policies.Prior systems also use engineered perception, navigation, view planning, or shape completion.Source: §2.3 · p. 3

3.1 Problem Formulation

• ObstaDiff separates visually guided alignment from close-range interaction around a target-centered bottleneck pose.The alignment endpoint is determined online by a gated nearest-neighbor criterion rather than a pre-specified pose.Source: §3 · p. 3, §3.1 · p. 3

3.2 Structured Target–Obstacle–Background Observation

• The four-channel TOB observation encodes background, obstacle, target, and normalized depth to preserve spatial relationships while suppressing background variation.An open-vocabulary detector produces semantic regions, which are converted into three masks and concatenated with aligned depth.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Obstacle-aware Structured Observation Encoder

• A lightweight StructuredObsEncoder uses separate semantic and depth branches with spatial softmax to extract compact target–obstacle geometry.The branch outputs are concatenated with end-effector pose; with two observation steps, the conditioning vector is 206-D.Source: §3.3 · pp. 4–5

3.4 Alignment-stage Diffusion Policy

• The alignment policy is a conditional diffusion model that predicts end-effector action sequences toward the bottleneck pose from encoded structured observations.Training corrupts clean alignment actions and trains a noise-prediction network; inference iteratively denoises 16-action sequences and executes the first 8 actions before replanning.Source: §3.4 · p. 5

3.5 Bottleneck Switching and Interaction

• The system switches to the nearest replayed interaction trajectory when the current target state is sufficiently close to a stored initial target state.This decoupling lets one alignment policy support touching, harvesting, or close-up inspection by changing only the replay library, while avoiding unreliable close-range depth.Source: §3.5 · p. 5

4.1 Experimental Protocol

• The protocol tests 61 distinct configurations: 20 target poses, 21 obstacle layouts, and 20 unseen target-appearance trials.OLG varies known plant species in unseen spatial arrangements, while TAG uses unseen green pepper varieties.Source: §4.1 · p. 6

4.2 Main Results

• 75.41% mean TSR and 8.20% mean OCR make ObstaDiff the best method across the three 61-trial generalization settings.The evaluation comprises 366 executions total, with each method run on TPG, OLG, and TAG trials.Source: §4.2 · p. 8, Table 1 · p. 8
• ObstaDiff outperforms ACT (RGB) and DP (RGB), with mean TSR of 75.41% versus 45.90% and 49.18%, respectively.Its mean OCR is 8.20%, compared with 14.75% for ACT (RGB) and 26.23% for DP (RGB).Source: §4.2 · p. 8, Table 1 · p. 8

4.3 Ablation Studies

• Depth or TOB inputs alone do not reproduce the full method: DP (RGB-D) reaches 50.82% TSR, while DP (TOB) reaches 45.90%.DP (TOB) has 27.87% mean OCR, whereas ObstaDiff uses the same TOB observation with a structured encoder.Source: §4.1 · p. 7, §4.3 · p. 8
• The structured encoder accounts for a 29.5-point TSR gap between DP (TOB) and ObstaDiff under otherwise matched observations, demonstrations, and batch size.Under obstacle-layout shift, DP (TOB) records 38.10% success and 38.10% collision rates.Source: §4.3 · pp. 8–9
• Adding TOB to ACT lowers mean OCR from 14.75% to 11.48% and improves smoothness from 1.16 to 0.77, but lowers mean TSR from 45.90% to 40.98%.TOB also reduces validation-loss fluctuation, showing that lower supervised loss alone does not guarantee higher real-robot success.Source: Figure 4 · p. 8, §4.3 · p. 9

5 Limitations

• The study does not address autonomous target selection or task planning because targets are human- or prompt-specified.The learned component focuses on low-level obstacle-aware alignment.Source: §5 · p. 9
• Results concern obstacle-aware approach behavior rather than contact-rich manipulation because the interaction stage replays demonstrated trajectories.The policy remains constrained by the demonstration distribution and has limited ability to explore behaviors outside collected data.Source: §5 · p. 9
• Evaluation is confined to one indoor greenhouse testbed with three obstacle plant species and one target crop.The comparison against point-cloud policies remains open in the moving eye-in-hand regime.Source: §5 · p. 9

6 Conclusion

• ObstaDiff converts eye-in-hand RGB-D observations into a structured target–obstacle–background representation for obstacle-aware action generation.The representation separates the target to reach, obstacles to avoid, and background to ignore.Source: §6 · p. 9
• The decomposed execution scheme assigns visually guided alignment to the learned policy and close-range interaction to replayed actions.This avoids relying on unreliable eye-in-hand depth during contact-rich motions and supports different interaction objectives through the replay library.Source: §6 · p. 9
• Real-robot experiments under target-pose, obstacle-layout, and target-appearance shifts show improved task success and obstacle avoidance over imitation-learning baselines.Source: §6 · p. 9

A.1 Target–Obstacle–Background Preprocessing Details

• RGB and aligned depth streams are acquired from an eye-in-hand camera, with invalid depth values temporally hole-filled before processing.The RGB stream uses /camera/color/image_raw, while depth comes from /camera/aligned_depth_to_color/image_raw.Source: §A.1 · p. 13
• Emitter even–odd alternation reduces infrared-projector artifacts in RGB frames, while depth and RGB buffers are truncated to equal lengths.Source: §A.1 · p. 13
• YOLOv8s-Worldv2 detects the selected pepper fruit target and obstacles, whose boxes are refined into pixel-accurate masks with MobileSAM.The resulting semantic index map assigns per-pixel target, obstacle, and background labels.Source: §A.1 · p. 13
• The policy input resizes three semantic masks and aligned depth to 240 × 320, normalizes depth to [0, 1], and uses the channel order [bg_mask, obst_mask, plant_mask, depth_norm].Batched observations have shape X ∈ R^B×T_o×4×240×320 with T_o = 2.Source: §A.1 · p. 13

A.2 Structured Observation Encoder

• The encoder splits each four-channel TOB observation into three semantic-mask channels and one normalized-depth channel.Source: §A.2 · p. 13
• Semantic and depth branches use lightweight CNNs with four convolutional blocks and channel-specific widths.The convolutional layers use kernels and strides (5×5, s=2), (3×3, s=2), (3×3, s=1), and (3×3, s=1).Source: §A.2 · p. 14
• Spatial softmax converts branch feature activations into keypoint-like coordinates instead of flattening the feature maps.The semantic branch produces 32 coordinate pairs, yielding z^seg_t ∈ R^64.Source: §A.2 · p. 14
• Branch outputs are concatenated with the 7-D end-effector pose to form the diffusion policy’s conditioning vector.With two observation steps, the final conditioning input to the diffusion U-Net has dimension 206.Source: §A.2 · p. 14

A.3 Bottleneck Switching and Interaction Details

• The replay library stores interaction samples and their initial target states, with N_lib = 26 in the implementation.Each replay entry includes an interaction-stage action sequence and a target state extracted from its first frame.Source: §A.3 · p. 14
• The target state consists of the tracked target center and target depth, allowing current alignment observations to be compared with replay-entry initial states.Source: §A.3 · pp. 14–15
• Target depth is estimated from the p-th percentile of valid depth pixels in the central 40% of the target box, using p = 20.This biases the estimate toward the visible foreground surface of the target.Source: §A.3 · p. 15
• The system selects the smallest-image-plane-error replay among depth-ready entries and switches when the selected replay satisfies the depth and image-plane gates.The experiments use a 5 mm depth gate and normalized image-plane error threshold ϵ_uv = 0.2.Source: §A.3 · p. 15
• After switching, the system executes the interaction action sequence associated with the selected replay, avoiding reliance on eye-in-hand depth during close-range interaction.Source: §A.3 · p. 15

B TOB Preprocessing Visualization

• The TOB preprocessing pipeline converts raw RGB observations into background, obstacle, and target masks alongside normalized depth.The displayed outputs proceed from the raw image and semantic parsing to the three semantic masks and depth channel.Source: B TOB Preprocessing Visualization · p. 15, Figure 5 · p. 16
• The policy input stacks the three semantic masks with the normalized depth channel into a four-channel observation.Source: B TOB Preprocessing Visualization · p. 15, Figure 5 · p. 16