FetchBot: Learning Generalizable Object Fetching in Cluttered Scenes via Zero-Shot Sim2Real

• Dense clutter creates occlusions and obstacles that require policies to reason about geometry, minimize collisions, and generalize across objects, layouts, and materials.Source: §1 · p. 2
• Prior methods struggle with dense clutter, diverse real-world arrangements, noisy depth sensing, and incomplete voxel representations in occluded regions.These limitations affect heuristic, learning-based, point-cloud, and depth-based approaches.Source: §1 · p. 2
• 89.95% average real-world success rate demonstrates FetchBot’s strong performance in cluttered scenarios, including translucent, reflective, and irregular objects.The method significantly outperforms existing approaches in real-world experiments.Source: §1 · p. 2
• FetchBot uses foundation-model depth predictions to bridge sim-to-real and occupancy prediction to infer occluded information for more complete scene understanding.Source: §1 · pp. 2–3
• UniVoxGen generates 1M diverse cluttered scenes, while a dynamics-aware oracle policy supplies 500k representative demonstrations for training.Source: §1 · p. 2

• Fetching from clutter requires careful retrieval that minimizes disturbance, but simulated environments often fail to reproduce real-world density and unpredictability.Source: §2 · p. 3
• Point-cloud-based sim-to-real methods remain vulnerable to depth noise and inaccuracies at object edges and reflective surfaces.FetchBot instead uses predicted depth as an intermediate representation and unified 3D fusion for more complete perception.Source: §2 · p. 3

3.2 Voxel-based Cluttered Scene Generator

• Collision-free placement is difficult because simulator collision checks and object dropping are costly and can produce unrealistic dense configurations.Source: §3.2 · p. 3
• UniVoxGen efficiently generates realistic dense scenes through voxel union, intersection, difference, and transformation operations, producing 1 million training scenes.The scenes also provide dense occupancy ground truth.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 Dynamics-Aware Oracle Policy

• The oracle policy uses reinforcement learning to learn environmental dynamics and select low-impact fetching maneuvers.Its hierarchical scene encoder combines local object details with global scene context.Source: §3.3 · p. 4
• The hierarchical encoder processes object-level features locally and aggregates them into a global scene representation, preserving detail alongside scene context.Source: §3.3 · pp. 4–5, Figure 4 · p. 4

3.4 Vision-based Imitation Learning

• Predicted depth maps place simulation and real RGB inputs in a shared representation, avoiding direct reliance on noisy real depth sensors.Source: §3.4 · p. 5
• Multi-view features are projected into 3D query points and aggregated with deformable cross-attention before 3D refinement.The mechanism uses valid camera views and local projected features.Source: §3.4 · p. 5
• Semantic occupancy prediction within a local end-effector ROI distinguishes targets, obstacles, and robots while inferring unobserved structure from multi-view predicted depth.Complete UniVoxGen occupancy supervises this auxiliary task, improving robustness to occlusion and irregular partial observations.Source: §3.4 · p. 5
• The policy converts the 3D scene representation into actions using a transformer architecture and diffusion head, while encoder pretraining precedes imitation learning.Source: §3.4 · pp. 5–6

4 Experiments

• The experiments compare heuristic, motion-planning, and learning-based policies using environmental impact and success rate as the principal evaluation measures.Simulation uses 3000 densely constructed scenes and counts success only when fetching stays within a 3 cm obstacle-displacement threshold.Source: §4.1 · pp. 6–7, §4 · p. 6

4.1 Simulation Experiment

• 81.46% suction-cup and 91.02% parallel-gripper success rates demonstrate FetchBot’s strong simulated fetching performance under clutter and occlusion.Its occupancy representation infers occluded geometry and supports actions that minimize disturbance to nearby objects.Source: §4.1 · p. 7

4.2 Real World Experiment

• FetchBot generalizes to real-world tabletop and drawer tasks while handling transparent and reflective objects through predicted depth and occupancy-based scene completion.The method uses local occupancy and semantic cues to represent occluded regions and diversify local geometries during training.Source: Figure 6 · p. 7, Figure 7 · p. 7, §4.2 · p. 8

4.3 Ablation Study for Sim2Real

• Removing DepthAnything reduces real-world success from 86.60% to 60.00%, while removing occupancy reduces it to 73.33%.Using RGB alone produces the lowest success rate, 33.33%, showing that both modules contribute to sim-to-real fetching.Source: §4.3 · p. 8

Conclusion

• FetchBot demonstrates strong zero-shot transfer, effective clutter management, and handling of transparent, reflective, and irregular objects in real-world fetching.The conclusion describes these results as significantly outperforming previous methods and demonstrating practical value for complex tasks.Source: §5 · p. 8
• The framework combines a large-scale synthetic dataset with depth-conditioned action generation, RGB-to-depth prediction, and occupancy pretraining.These components are presented as the basis for bridging the sim-to-real gap and improving scene understanding under occlusion.Source: §5 · p. 8

Limitation

• Heavy occlusion can produce complex maneuvers that exceed robot-arm joint limits, while fully occluded targets may remain initially unreachable and large objects challenge single-arm grasping.The paper suggests dual-arm collaboration and reasoning for obstruction clearing and scene restoration as future directions.Source: §5 · p. 9

A.1 Scene Generation Method

• UniVoxGen accelerates realistic cluttered-scene generation through voxel-space collision checks and hand-designed placement rules, avoiding expensive simulator-based verification.It generates 1 million scenes in 12 hours on a workstation with 8 RTX 4090s.Source: §A.1 · pp. 15–16
• The generator uses voxel unions, intersections, differences, and SE(3) transformations to place objects while checking collisions and preserving object poses.Its rule set can also include unsolvable cases to better represent real-world clutter.Source: §A.1 · p. 15

A.2 Scene Generation Result

• UniVoxGen produces structured clutter for shelves and additional tabletop, drawer, and rack environments to support extension experiments.Shelf layouts preserve regularity rather than arbitrarily piling objects.Source: §A.2 · p. 16, Figure 10 · p. 16

B.1 Reward Function

• The oracle reward combines task completion, behavioral constraints, and environment-change penalties, with curricula that progressively tighten disturbance thresholds.The policy is trained with PPO and randomized object, camera, end-effector, and observation conditions to improve robustness.Source: §B.1 · pp. 16–17, §B.2 · p. 17

B.3 Curriculum Learning Methods

• Per-environment, unified, and policy-driven difficulty curricula all fail to produce satisfactory improvement.The tested approaches respectively adjust individual environments, advance all environments together, or focus training on failed scenarios.Source: §B.3 · p. 18
• The σ-based curriculum starts with a large σ for exploration and gradually reduces it to improve precision and stability.Unlike the tested scene-based curricula, it schedules training by task difficulty rather than manually defined or policy-derived scene difficulty.Source: §B.3 · p. 18
• Scene difficulty levels are homogeneous, so knowledge learned in easy scenarios transfers poorly to hard ones and provides weak learning gradients.The σ-based curriculum avoids specifying scene difficulty and directly schedules task difficulty.Source: §B.3 · p. 18

B.4 The Architecture of Oracle Policy

• The oracle policy observation concatenates a 64-dimensional scene representation, 12-dimensional target-relative proprioception, and a 6-dimensional previous action into o_t ∈ R^82.The decision module is a three-layer MLP implementing π: R^82 → R^6.Source: §B.4 · pp. 18–19

Scaling the Training Data for Occupancy Pre-Training

• 81.46% success is achieved with 500k scenes for occupancy pre-training, rising from 62.33% with 500 scenes.Policy-training data remains fixed at 500K, isolating the effect of occupancy pre-training scale.Source: §C.1 · p. 19, Figure 11 · p. 19

Scaling the Training Data for Policy Learning

• 81.46% success is achieved with 500k state-action pairs for policy learning, rising from 48.23% with 500 pairs.The 3D vision encoder pre-training volume remains fixed, isolating the effect of policy-learning data scale.Source: §C.1 · p. 19, Figure 12 · p. 19

Overall Findings

• Larger datasets improve both occupancy pre-training and policy learning, highlighting the importance of data scale for 3D vision policy performance.The ablation uses a suction-cup end effector.Source: §C.1 · p. 19

C.2 Ablation Study on Region of Interest (ROI) Size)

• 20 × 20 × 30 cm achieves the highest success rate of 81.46% with the lowest translation error of 2.78 cm and rotation error of 0.36 rad.This ROI encloses the approximately 5 × 10 × 12 cm targets while filtering distant distractions.Source: §C.2 · p. 20, Table 5 · p. 20

C.2 Ablation Study on Region of Interest (ROI) Size

• Larger ROIs reduce success and accuracy because irrelevant distant information weakens focus on the target region.The results favor a localized ROI scaled to the task rather than an arbitrarily small crop.Source: §C.2 · p. 20

C.4 The Details of 2D-3D Spatial Attention

• Deformable Cross-Attention aggregates projected multi-view features with visibility- and content-dependent weighting, improving robustness to occlusion and blur.3D queries project into valid camera views, where sparse learned offsets and attention weights sample 2D feature maps.Source: §C.4 · pp. 20–21

C.5 Processing of Relative Depth

• Relative depth is min-max normalized per image and processed as a three-channel input for a ResNet feature extractor.The normalization uses each map’s minimum and maximum values with a small stability constant.Source: §C.5 · p. 21

C.5.1 Mapping Relative Depth to Metric Voxels

• The method projects a predefined 5 mm 3D metric voxel grid into camera views rather than converting relative depth into metric depth or unprojecting features into a point cloud.Multi-view parallax and end-to-end supervision guide the learned mapping from relative-depth features to metric occupancy.Source: §C.5.1 · p. 21

C.6 Real-World Reconstruction Results

• Occupancy pre-training reconstructs occluded volume on simulation and real hardware, while intermediate encoder features—not final occupancy maps—feed the downstream policy.The reconstruction remains robust to transparent, reflective, and irregular objects.Source: §C.6 · p. 21, Figure 13 · p. 22

D Baseline Implementation Details

• RGB, point-cloud, RGB-D voxel, raw-depth, and predicted-depth baselines share the same DDPM action-generation method while differing in input representation.DDPM training uses 1000 denoising steps and inference uses 100 steps.Source: D Baseline Implementation Details · pp. 22–23

E.1 Setup

• The real-world platform uses a Flexiv Rizon 4S arm and two Intel D435i cameras providing wrist-mounted and third-person RGB views.The system uses a suction cup for boxed objects and a parallel gripper for bottled objects.Source: §E.1 · pp. 23–24
• Experiments primarily use a replicated retail environment, with additional cabinet and rack settings across 30 layouts.The object set includes boxed, bottled, transparent, reflective, and irregularly shaped items.Source: §E.1 · p. 24, Figure 17 · p. 24

E.2 Real-World System Design

• A 10 Hz policy layer sends target end-effector poses to a 100 Hz asynchronous inverse-kinematics controller for smooth, velocity-limited execution.The controller interpolates toward policy targets using small kinematically valid joint motions.Source: §E.2 · p. 24

E.3 Experiment objects

• The object set spans boxed, bottled, transparent, reflective, and irregularly shaped items.This diversity tests performance across different geometries and materials.Source: §E.3 · p. 24
• The failure illustrations include an above-approach joint-limit violation, a large object requiring dual-arm manipulation, and a completely occluded unreachable target.The last case requires moving obstacles, fetching the target, and restoring the obstacles.Source: Figure 18 · p. 25

F Failure Modes Analysis

• Excessively complex scenes can cause joint-limit violations and task failure when the robot approaches from above.The reported boundary also includes oversized objects needing dual-arm coordination and fully occluded targets needing obstacle rearrangement.Source: F Failure Modes Analysis · p. 25