Source-linked AI summary
A Dataset for Developing and Benchmarking Active Vision
Phil Ammirato, Patrick Poirson, Eunbyung Park, Jana Kosecka, Alexander C. Berg
TL;DR
Robotic object recognition requires real, multiview perception data beyond isolated and biased image benchmarks. The paper introduces a dense RGB-D dataset, uses it for instance detection and reinforcement-learning-based next-best-move prediction, and finds that recognition remains sensitive to scale, occlusion, and viewing direction while active movement can improve classification.
Problem
Object-recognition benchmarks do not fully address robotics needs because robotic recognition involves biased real-world conditions, multiple images, and active sensing.
Method
The paper collects dense real RGB-D imagery with object labels, adapts a fast deep detector for instance detection, and trains a reinforcement-learning network for next-best-move prediction.
Results
Recognition performance is strongly affected by object scale, occlusion, and viewing direction, while the active system improves classification accuracy by moving to new image locations and outperforms two obvious baselines.
Takeaways & Limitations
Dense real-world imagery supports training and benchmarking active-vision systems without requiring a robot in the loop or synthetic computer-graphics models.
Takeaways & Limitations
The improved depth maps are not perfect, and the dataset leaves quantitative occlusion labeling for future work.
Abstract
from arXiv · showhide
We present a new public dataset with a focus on simulating robotic vision tasks in everyday indoor environments using real imagery. The dataset includes 20,000+ RGB-D images and 50,000+ 2D bounding boxes of object instances densely captured in 9 unique scenes. We train a fast object category detector for instance detection on our data. Using the dataset we show that, although increasingly accurate and fast, the state of the art for object detection is still severely impacted by object scale, occlusion, and viewing direction all of which matter for robotics applications. We next validate the dataset for simulating active vision, and use the dataset to develop and evaluate a deep-network-based system for next best move prediction for object classification using reinforcement learning. Our dataset is available for download at cs.unc.edu/~ammirato/active_vision_dataset_website/.
I. INTRODUCTION
The paper introduces dense real RGB-D imagery to study active vision for robotics, addressing the mismatch between isolated, biased recognition benchmarks and multiview robotic perception. It uses the dataset to evaluate instance detection and next-best-move prediction.
- Motivation: Dense scene imagery enables researchers to simulate a robot moving through indoor environments by sampling appropriate views.The dataset supports training and benchmarking active-vision systems without requiring a physical robot for every experiment.
- Dataset goals: The resource provides dense RGB-D data, 3D scene structure, object labels, and camera connectivity for active-vision experiments.Connectivity indicates what image would be seen after movements such as turning or translating from a camera position.
- Recognition baseline: A fast deep convolutional category detector is adapted to recognize specific object instances rather than only object categories.The paper distinguishes recognizing a category such as chair from recognizing a particular object such as a specific can.
- Recognition baseline: Detection performance falls under everyday variation in object scale, viewing direction, and occlusion, despite strong performance on large frontal views.This view-dependent behavior motivates controlling the sensing platform to acquire more informative imagery.
- Active vision: The authors validate dense sampling for active vision and train a reinforcement-learning deep network to select moves that improve object classification.The recognition component is pretrained externally, then combined recognition and movement selection are trained on densely sampled data.
II. RELATED WORK
Prior recognition datasets and active-vision studies leave a gap: they commonly use biased web imagery, limited views, tabletop scenes, CAD models, or costly robot-specific data. The paper positions dense real-scene imagery as a resource for broader active-vision evaluation.
- Object recognition datasets: Web benchmarks such as Pascal VOC, ImageNet, and MS COCO tend to contain medium-to-large, frontal, lightly occluded objects and emphasize categories.Their collection processes introduce biases from photographers, taggers, and search engines.
- RGB-D datasets: Existing RGB-D datasets include tabletop and single-object collections but lack a focus on simulating robot motion through environments.The paper notes that such data can still provide valuable training views for instance recognition.
- RGB-D datasets: Environment datasets generally provide only one or two paths, limiting the movement choices needed for realistic robotic navigation.An actual robot can choose among many possible routes through a scene.
- Active vision: Prior active-vision work includes view selection and theoretical approaches, while many next-best-view methods rely on CAD models or simplified motion assumptions.These constraints differ from active recognition in real, cluttered environments.
- Active vision: A related real-image system required separately collected tabletop data for active-vision evaluation, illustrating the difficulty of building large-scale datasets.That dataset contained about 30–60 images per scene.
III. DATA COLLECTION
The dataset captures multiple indoor scenes with densely sampled RGB-D views and structured camera motion, enabling virtual movement through real environments. Repeated scans and reconstruction support variation in object placement and scene coverage.
- Scene coverage: The collection contains 9 unique scenes, 17 scans, and 20,916 images across offices and homes that may include multiple rooms.Individual scenes contain 696–2,412 images.
- Scene coverage: Objects are labeled in 3D from dense scene reconstructions and projected into camera images to generate 2D bounding boxes.The reconstruction-based process avoids labeling every object separately in thousands of images.
- Camera sampling: The collection uses discrete camera positions and samples views every 30 degrees to balance overlap with manageable dataset size.This design provides multiple views at each visited point without collecting unnecessary intermediate video frames.
- Camera sampling: Camera positions form a rectangular grid with points 30 centimeters apart, yielding 58–201 positions per scene and many movement choices.The grid supports navigation through scenes using discrete, consistently sampled viewpoints.
- Repeated scans: Repeated scans relocate naturally movable objects while keeping fixed furnishings in place, increasing data and enabling learning from an initial scene scan.Chairs, books, and BigBIRD objects may move, whereas sofas and refrigerators remain fixed.
- Depth processing: Improved depth images help handle occlusion when projecting reconstructed object labels into RGB images.The paper compares initial and improved depth maps for this purpose.
A. Labels
The labeling pipeline combines camera geometry, dense reconstruction, depth improvement, and point-cloud projection to generate object-instance boxes and movement pointers. Manual inspection and size filtering address remaining label-quality problems.
- Movement labels: The dataset provides clockwise, counter-clockwise, forward, backward, left, and right movement pointers from each image.These pointers support navigation through the scene using natural robotic movements.
- Movement labels: COLMAP estimates camera position and orientation, which are used to calculate movement pointers for navigation.The reconstruction is sparse and relies on RGB structure from motion rather than depth.
- Object labels: Dense CMVS/PMVS reconstructions make objects recognizable for labeling, after which each instance point cloud is projected into images to obtain 2D boxes.Occlusion must be handled because cluttered scenes can otherwise cause projections through walls or other objects.
- Depth refinement: Improved depth maps combine nearby views, retain the smallest projected depth per pixel, and interpolate remaining holes.This reduces erroneous large depth values and fills missing values, including on specular surfaces.
- Quality control: The improved depths and reconstructions remain imperfect, so every bounding box is manually inspected for object identity and size quality.The dataset includes a size-based difficulty measure but leaves occlusion measurement for future work.
- Qualitative detection results: Detection scores vary consistently with camera position, indicating that some viewpoints are more reliable than others.The figure displays only cameras that see each instance and uses color to encode detection score.
IV. EXPERIMENTS
The experiments evaluate instance detection, robot-motion-specific system design, motion simulation, and an active-vision task using the dataset.
- The experiments establish a baseline for instance detection on the dataset.
- They examine why systems should be designed specifically for robot motion.
- They test dataset-based motion simulation and demonstrate an active-vision task.
A. Instance Detection
The study adapts a fast class-level detector for instance detection and evaluates it under two object-size regimes. The setup emphasizes real-time operation while testing the challenge posed by small boxes.
- 72 FPS motivates choosing SSD as a real-time baseline for instance detection.The detector is selected because it combines real-time detection performance with high-level accuracy.
- The evaluation first uses boxes at least 100x75 pixels, then includes all boxes at least 50x30.The second regime adds training data while creating a more difficult test scenario.
- SSD uses a VGG base network with additional feature maps produced through 1x1 and 3x3 convolutions.
- Mean Average Precision varies depending on the training and testing split.The same hyperparameter settings are used across all data splits.
B. Qualitative Results
Qualitative evaluation finds that detection reliability varies across camera positions and is strongly affected by occlusion and object pose. These view-dependent weaknesses motivate active movement toward better viewpoints.
- Detection is more reliable from some camera positions than from others.
- Occlusion and object pose can greatly impact detection despite training examples covering both cases.
- Poor detection outputs motivate moving the sensing platform to positions with improved performance.
C. Ability to Simulate Motion
The study uses paired video frames and camera displacement to assess whether dataset sampling can simulate motion. Detection-score changes increase with movement, while 30 cm offers a practical resolution with limited additional variation for most instances.
- Greater distance between cameras produces greater changes in detection score.Figure 5 plots camera-distance change on the x-axis and detection-score change on the y-axis for each instance.
- The motion-simulation procedure compares detection-score differences for instances across image pairs against camera displacement.The results come from four videos captured while driving a robot through scenes.
- 30 cm is selected as the sampling resolution to balance vision-system variation with data-collection practicality.
- For most instances, score changes at 30 cm are not much different from changes at 10 or 20 cm.
D. Active Vision
The active-vision system learns to select movements that improve object classification from changing views. It is trained with reinforcement learning and outperforms forward-only and random movement baselines.
- System and training: The instance classifier and action network share a fixed ResNet-18 feature extractor, while the overall system remains end-to-end trainable.The classifier is trained separately using composited BigBIRD objects and SUN397 backgrounds to reduce overfitting.
- System and training: The system receives an image and target bounding box, then selects among six movements: forward, backward, left, right, and two rotations.The action network predicts a score for each action at every timestep.
- System and training: REINFORCE trains the action network by rewarding action sequences that produce correct classification.The objective maximizes expected reward under the policy distribution represented by the action network.
- Evaluation: The action network is evaluated by measuring classifier accuracy as the maximum number of movements increases.Evaluation compares performance across timesteps and test-scene splits.
- Results: The learned policy greatly improves classification accuracy and outperforms always moving forward and choosing random actions.Figure 9 reports relative improvement as the number of virtual moves increases; example paths are shown qualitatively in Figure 7.
- Limitations and extensions: The authors identify multiview aggregation, recurrent action-history models, and removing the provided bounding box as potential extensions.Without a bounding box, the policy would need to consider hypotheses for both object location and class.
OUR SYSTEM IS TO MOVE IN THE SCENE TO INCREASE CLASSIFICATION
The section heading states that the system moves in the scene to increase classification accuracy.
- The proposed objective is to move in the scene to increase classification accuracy.
V. CONCLUSIONS
The paper introduces a labeled dataset for object recognition and active vision, establishes an instance-detection baseline, and demonstrates reinforcement-learning-based next-best-view selection on real indoor imagery.
- The dataset supports object-recognition development and active-vision benchmarking in challenging indoor environments.The dataset and processing toolbox are publicly available.
- The study establishes an object-instance-detection baseline and trains a next-best-view system with reinforcement learning.The recognition system selects movements intended to improve classification.
- Real densely sampled RGB-D imagery exposes variation in scale, viewpoint, and imaging conditions that may be absent from computer-graphics data.