Source-linked AI summary
Mobile Robot Manipulation using Pure Object Detection
Brent Griffin
TL;DR
Mobile robots can struggle when object detectors encounter new objects, tasks, or environments, and passive data does not use their ability to learn through interaction. ClickBot combines manipulation based on object detection with task-focused few-shot learning that gathers and targets new training examples. In robot experiments, it reports improved benchmark performance, visual-control learning, and grasping with few examples.
Problem
Off-the-shelf detectors are less reliable when objects, tasks, or environments change, while passive learning leaves robots’ interactive experience unused.
Source: §1 · p. 1Method
ClickBot performs mobile manipulation using object detection alone and uses TFOD to collect task data, identify when new few-shot examples are needed, and direct annotation.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The experiments show that detection-based control and task-focused data collection support visual control and grasping new objects in clutter with a single RGB camera.
Source: §1 · p. 1, §6 · p. 8The update loop depends on task-specific failure criteria and annotation of logged failure images before detection can be updated.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
This paper addresses the problem of mobile robot manipulation using object detection. Our approach uses detection and control as complimentary functions that learn from real-world interactions. We develop an end-to-end manipulation method based solely on detection and introduce Task-focused Few-shot Object Detection (TFOD) to learn new objects and settings. Our robot collects its own training data and automatically determines when to retrain detection to improve performance across various subtasks (e.g., grasping). Notably, detection training is low-cost, and our robot learns to manipulate new objects using as few as four clicks of annotation. In physical experiments, our robot learns visual control from a single click of annotation and a novel update formulation, manipulates new objects in clutter and other mobile settings, and achieves state-of-the-art results on an existing visual servo control and depth estimation benchmark. Finally, we develop a TFOD Benchmark to support future object detection research for robotics: https://github.com/griffbr/tfod.
1. Introduction
ClickBot addresses unreliable object detection when objects, tasks, or environments change by learning mobile manipulation directly from 2D bounding boxes. It combines detection-based control with task-focused few-shot learning and evaluates the approach across visual control, depth estimation, grasping, and cleanup.
1. Introduction
- Off-the-shelf detectors become less reliable when objects, tasks, or environments change, while passive learning leaves robots’ interactive experience unused.The paper frames this as a reason for robots to collect and learn from real-world interactions.Source: §1 · p. 1
- ClickBot performs mobile manipulation from object detection alone, using one RGB camera and no 3D object models.Its tasks include a visual-servo update formulation, motion-based depth estimation that improves as the robot approaches, and active multi-view grasp selection.Source: §1 · p. 1
- Task-Focused Few-Shot Object Detection (TFOD) lets ClickBot collect data during tasks, detect when new examples are needed, and target annotation to specific tasks.The paper reports that TFOD improves difficult or evolving tasks while reducing overall annotation costs.Source: §1 · p. 2
- 16.7% and 25.0%: ClickBot improves visual servo control and depth estimation, respectively, on the VOSVS Benchmark [16].Across robot experiments, it also learns visual control in less than 14 s, reduces learning variability by 65-85% versus prior visual servo approaches, and cleans scattered objects at 124.6 picks-per-hour.Source: §1 · p. 2
- The TFOD Benchmark provides configurable few-shot detection evaluation across diverse YCB objects using standard MS-COCO AP metrics [36].It is intended to support robotics-focused few-shot detection research where researchers may lack a robot or robotics data.Source: §1 · p. 2
2. Related Work
Prior object-detection and few-shot methods largely address static datasets or detection itself, while existing end-to-end visual-servo manipulation methods do not handle mobile settings. This work extends few-shot detection to task-focused learning in the wild and develops mobile visual servoing with lower annotation cost than a segmentation-based approach.
2. Related Work
- Static datasets do not account for new objects and settings encountered in the wild.Few-shot object detection addresses part of this limitation by detecting new objects from only a few annotated examples [10].Source: §2 · p. 2
- Few-shot detection work spans finetuning, meta-learning, incremental learning, one-shot matching, and other approaches, but few-shot data collection for new objects and application tasks has received scant attention.Prior application-oriented work either used substantially more than 30 examples and focused on detection [3], or required arbitrary 3D models and existing-dataset objects [63].Source: §2 · p. 2
- TFOD gathers few-shot examples in the wild and improves detection for specific tasks, letting task difficulty guide which examples receive annotation.Although the experiments use a finetuning method, the paper states that TFOD generalizes across few-shot detection methods.Source: §2 · p. 2
- Prior end-to-end visual-servo manipulation methods learn in fixed workspaces, leaving mobile challenges such as moving cameras, changing environments, and dynamic grasp positioning unaddressed.The paper’s mobile visual-servo approach uses features based on detection rather than pre-trained video-object segmentation [16].Source: §2 · pp. 2–3
- Mobile visual servoing requires less than 5% of the annotation cost of a prior segmentation-based approach while significantly improving performance.The method also learns visual servoing with a pseudoinverse feature Jacobian faster and more consistently than prior work.Source: §2 · p. 3
3. Task-Focused Few-Shot Object Detection
Task-Focused Few-Shot Object Detection (TFOD) uses task failures to guide detection updates for new objects and applications. It selects failure images for annotation and updates the detector with as few annotated examples as possible.
3.1. Task and Detection Model
- TFOD applies to tasks that observe images, detect labeled bounding boxes, define detection-based failure criteria, save failure images, and support detector updates from annotated examples.The failure criteria and image selection can vary by task.Source: §3.1 · p. 3
- TFOD updates the detector until a task completes without failure, aiming to minimize the number of annotated few-shot examples.Tasks use detections from an object detection model and task-specific failure criteria; failure images are saved for annotation and model updates.Source: §3.1 · p. 3
3.2. Task-Focused Data Collection
- When a detection-based task fails, the robot selects representative failure images, adds annotated examples to an aggregate set, and updates the detector.The process repeats across tasks; sharing examples across related tasks can reduce the total number needed, and updates are rarely needed unless objects or settings change.Source: §3.2 · p. 3
- Task-focused annotation selects new few-shot examples for tasks needing better detection, including Find, Grasp, and Placement.The figure illustrates the three tasks from left to right.Source: Figure 2 · p. 3
3.3. Few-Shot Annotation
- Users annotate saved failure images by drawing boxes around task-relevant objects; each bounding-box annotation takes about 7 seconds per object.The paper annotates one relevant image per update; object-free images may optionally be added as true negatives to reduce false positives.Source: §3.3 · p. 3
4. Detection-Based Manipulation
The manipulation pipeline uses object detection to find objects, position the robot, estimate depth, and plan grasps. Its visual servo controller learns from camera motion and detection-based error, while active perception informs depth and grasp selection.
4. Detection-Based Manipulation
- Detection-based manipulation uses a set of tasks and can also use standard detection and tracking algorithms for objects learned in advance.The Find Task locates task-relevant objects as the starting point for manipulation.Source: §4 · p. 3
4.1. Finding Objects for Manipulation
- The Find Task searches a sequence of camera poses and saves images for annotation if no object is detected; Sentry Mode later avoids treating object absence as failure.Once objects have been learned and manipulated, intermittent searches can find objects without generating examples when none are present.Source: §4.1 · p. 3
4.2. Learning Visual Servo Control from Detection
- The Servo Task uses target bounding-box center coordinates as image features and drives their error from desired coordinates below a task-specific pixel threshold.The thresholds are 10 pixels before depth estimation and 5 pixels before grasping.Source: §4.2 · p. 4
- A learned pseudoinverse feature Jacobian maps detection-based error to camera motion and is updated from observed motion and error changes, with a logical mask limiting which elements update.The update uses the difference between actual and predicted camera-position change; the mask prevents associations between unrelated velocity and error elements.Source: §4.2 · p. 4
4.3. Estimating Depth from Motion and Detection
- For depth estimation, the robot advances its camera and recalculates depth from detections and camera poses until the object is estimated to be within 0.2 m.For grasping, it uses the median of estimates sampled every 0.05 m of camera motion to mitigate proximity-based detection errors.Source: §4.3 · p. 4
4.4. Grasping with Active Perception and Detection
- For grasping, the robot rotates its camera to find the narrowest detection-based parallel grasp points, centers the grasp for balance, and closes with force.It rotates through π/2 radians, chooses the box with minimum height or width, and treats a lift with separated gripper fingers as success.Source: §4.4 · p. 5, Figure 3 · p. 5
5. Experimental Results
Robot experiments evaluate ClickBot’s visual-servo learning, benchmark performance, cluttered and dynamic pick-and-place, and TFOD. Results include improved benchmark performance, task-focused annotation, and successful manipulation across varied settings.
5.1. Experimental Setup
- Experiments use an HSR with a 640×480 RGB grasp camera streaming at 25 Hz and fine-tune the baseline detector for 1,000 iterations in under four minutes.The detector uses a 0.9 confidence threshold to reduce false positives at the cost of more false negatives.Source: §5.1 · p. 5
5.2. Learning Visual Servo Control from One Click
- 13.29 s and 13 Broyden updates yield the visual-servo model; relative to prior formulations, it uses 30–60% the updates and has 15–35% as much learned-parameter variation.The comparison uses a single consecutive set of 10 trials for each update formulation.Source: §5.2 · p. 6, Table 1 · p. 6
5.3. VOSVS Benchmark
- ClickBot achieves perfect visual-servo success and raises prior depth-estimation success from 42% to 67% on the VOSVS Benchmark.Depth estimation still has room to improve on the Tool and Kitchen sets.Source: §5.3 · p. 6, Table 2 · p. 6
- Task-focused annotation averages five updates per trial, directs more examples to Depth than Find and Servo combined, and reduces annotation time per object from 540 s to 26 s, a 95% reduction.Each bounding-box annotation takes 7 s; segmentation-mask annotation takes about 54 s.Source: §5.3 · pp. 6–7, Table 3 · p. 7
5.4. Pick-and-Place in Cluttered Environments
- 88% is the standard configuration’s best cluttered Grasp and pick-and-place rate, with less than a minute of annotation per object.The reported comparison attributes the advantage over ablations to having the most few-shot examples in clutter.Source: §5.4 · p. 7, Table 4 · p. 7
5.5. Pick-and-Place with Dynamic Locations
- ClickBot learns to grasp scattered cups and place them in bins with four few-shot examples, then places all nine cups in matching bins at 124.6 picks-per-hour despite bins moving after each placement.The paper attributes four-shot dynamic pick-and-place to replacing the Depth Task with a map made during the Find Task.Source: §5.5 · pp. 7–8
5.6. Task-Focused Few-Shot Detection Benchmark
- The TFOD Benchmark provides configurable k = 1, 2, 4 settings across 12 YCB classes, challenging clutter examples, and publicly available robot-collected data; baseline results show room for improvement, especially for small objects and one- or two-shot detection.The benchmark reports MS-COCO AP metrics.Source: §5.6 · p. 8, Table 5 · p. 8
6. Conclusions
ClickBot combines detection-based mobile manipulation with task-focused data collection and retraining to adapt to changing objects, tasks, and environments. Experiments show its RGB approach works when few-shot annotation is acceptable, while RGBD can supplement it or substitute when full 3D sensing is unavailable.
6. Conclusions
- ClickBot collects data during tasks and automatically selects new few-shot examples for annotation when it recognizes a detection error.The authors report that this helps the robot avoid many vision-based errors while adapting to changing objects, tasks, and environments.Source: §6 · p. 8
- ClickBot grasps objects in clutter with a single RGB camera using as few as four few-shot examples, achieving an 88% overall pick-and-place rate.The rate is on par or better than recent methods using RGBD cameras in fixed workspaces.Source: §6 · p. 8
- The RGB-based approach works for mobile manipulation when few-shot annotation is acceptable, and can supplement RGBD approaches or substitute when full 3D sensing is unavailable.Source: §6 · p. 8
- Future work will expand the approach to tasks such as manipulation across multiple cluttered rooms and add benchmark releases for new application areas.The authors release a detection benchmark to support evaluation and improvement in robotics settings.Source: §6 · p. 8
Supplementary Material:
The supplementary material provides additional benchmark findings, implementation details, individual annotation trials, and practical limitations. It also describes dynamic pick-and-place demonstrations and the released benchmark's relation to prior work.
Per-Object TFOD Benchmark Results
- For all objects except Gelatin, the k = 4 configuration performs best; Wood, Chips Can, and Box of Sugar are particularly inaccurate for k < 4.The figure reports baseline ClickBot TFOD test results and highlights opportunities for improvement, especially with one or two examples.Source: Supplementary Material · p. 12, Figure 7 · p. 13
Depth Estimate Convergence
- Active depth estimation processes observations as the robot approaches, using bounding-box expansion and camera movement in a least-squares formulation.The estimate generally improves with more data and becomes significantly more accurate as the object gets closer and optical expansion increases.Source: Supplementary Material · p. 12
Individual Trial Results for Task-Focused Annotation
- In individual trials, ClickBot learned all tasks for three cluttered objects without prior annotation using seven task-focused examples.Other trials learned Find, Move, and Depth for three objects with three examples, or transferred a prior grasp task with one Move example.Source: Supplementary Material · p. 12
Detection-Based Visual Servo Control
- The detection-based visual servo update learns control in under 14 seconds on average and reduces learning variability by 65–85% relative to prior approaches.Its detection-based features account for detection errors and multiple objects.Source: Supplementary Material · p. 14
Application Novelty
- The related-work comparison identifies this as the first work to use a real robot to learn few-shot mobile manipulation for novel objects.Source: Table 7 · p. 15
Challenges, Solutions, and Future Work
- The current grasping approach uses an overhead antipodal grasp at the object center, but some objects are too large for the gripper or cannot be grasped overhead.The authors propose alternative strategies such as lateral grasping for objects too large for overhead grasps.Source: Supplementary Material · p. 15
- The baseline detector uses Faster R-CNN, and the authors expect more advanced few-shot detection algorithms may improve performance and reduce annotation requirements.Source: Supplementary Material · pp. 12, 15
Cleaning Scattered Cups with Dynamic Pick-and-Place
- ClickBot cleaned up nine scattered cups in 260 seconds, or 124.6 mobile picks per hour, placing each in the correctly colored bin despite bin rearrangements.The dynamic pick-and-place setup uses RGBD detection to locate and grasp objects and find a placement location.Source: Figure 10 · p. 15, Figure 11 · p. 16