Source-linked AI summary
CORe50: a New Dataset and Benchmark for Continuous Object Recognition
Vincenzo Lomonaco, Davide Maltoni
TL;DR
Continuous object recognition needs benchmarks that expose models to sequential, varied views while avoiding cumulative retraining and catastrophic forgetting. The paper introduces CORe50 with NI, NC, and NIC benchmarks and baseline approaches; naïve incremental tuning fails in complex NC and NIC settings, while CWR improves over the naïve solution but remains far from cumulative performance.
Problem
Continuous learning needs datasets with sequential batches and multiple unconstrained views across sessions, but most existing datasets were designed for static evaluation.
Method
The paper introduces CORe50, benchmarks for NI, NC, and NIC scenarios, and simple continual-learning approaches as baseline references.
Results
Naïve incremental tuning cannot avoid catastrophic forgetting in complex NC and NIC scenarios; CWR performs better than the naïve solution but remains substantially below cumulative performance.
Takeaways & Limitations
CORe50 supports comparative continuous-learning studies by emphasizing relative accuracy against the corresponding cumulative approach rather than absolute accuracy alone.
Takeaways & Limitations
The dataset remains subject to tracking imperfections because objects can extend beyond crop borders and receive no manual correction.
Abstract
from arXiv · showhide
Continuous/Lifelong learning of high-dimensional data streams is a challenging research problem. In fact, fully retraining models each time new data become available is infeasible, due to computational and storage issues, while naïve incremental strategies have been shown to suffer from catastrophic forgetting. In the context of real-world object recognition applications (e.g., robotic vision), where continuous learning is crucial, very few datasets and benchmarks are available to evaluate and compare emerging techniques. In this work we propose a new dataset and benchmark CORe50, specifically designed for continuous object recognition, and introduce baseline approaches for different continuous learning scenarios.
1 Introduction
CORe50 addresses the lack of datasets suited to continuous object recognition, where models must learn from sequential batches while retaining prior knowledge. It provides a benchmark and baseline approaches for New Instances, New Classes, and combined scenarios.
- Motivation: Static datasets usually separate training and testing, but continuous learning requires sequential batches and multiple unconstrained views of the same objects.Different sessions should vary backgrounds, lighting, poses, and occlusions.
- Continuous learning scenarios: New Instances introduces later batches containing new poses and conditions for known classes, requiring incremental consolidation without compromising earlier knowledge.
- Continuous learning scenarios: New Classes presents batches of previously unseen classes, requiring the model to learn them without losing accuracy on earlier classes.
- Continuous learning scenarios: New Instances and Classes combines new and known classes in later batches, requiring both consolidation of known classes and learning of new ones.
- Motivation: Updating models only with new data is more feasible than cumulative retraining, but preserving prior knowledge without old patterns remains difficult because of catastrophic forgetting.
- Contributions: CORe50 was introduced with benchmarks for NI, NC, and NIC scenarios and simple continual-learning approaches serving as baseline references.
2 Related Works
Prior continuous object-recognition datasets either lack realistic temporal sequences, impose controlled acquisition conditions, or have access limitations. CORe50 extends the closest robotic-vision alternatives with longer, more varied sessions and depth information.
- Continual-learning literature: Several continual-learning methods address forgetting, but most prior experiments used limited setups and primarily addressed the New Classes scenario.
- Dataset comparison: Datasets without temporal coherent sequences are excluded from the comparison because such sequences better support real-world applications such as robotic vision.
- Dataset comparison: Turntable datasets provide systematic pose and lighting control but lack complex backgrounds and occlusions, making generated exploration sequences unnatural.
- Dataset comparison: iCubWorld is closest to CORe50, while CORe50 offers more longer sessions, including outdoor sessions, more complex backgrounds, and depth information.
- CORe50 novelty: CORe50 provides 11 distinct acquisition sessions per object, enabling incremental strategies long enough to reveal learning trends.
3 CORe50
CORe50 contains 50 domestic objects across 10 categories, recorded as RGB-D videos across varied indoor and outdoor sessions. Automated cropping and tracking produce compact object-centered frames while retaining realistic motion, occlusion, and acquisition variability.
- Dataset composition: CORe50 contains 50 domestic objects in 10 categories and supports classification at either 50-object or 10-category level.Object-level classification is harder because same-category objects can be difficult to distinguish under some poses.
- Acquisition: Each object was recorded in 11 sessions—8 indoor and 3 outdoor—with 15-second videos at 20 fps captured by a Kinect 2.0 RGB-D sensor.Each session provides approximately 300 RGB-D frames per object.
- Acquisition: Operators smoothly moved and rotated hand-held objects at grab distance, with hand changes and self-occlusions occurring across sessions.
- Preprocessing: The acquisition interface first crops frames to 350 × 350 around a central region where the operator keeps the object.
- Preprocessing: A motion-based RGB tracker then extracts 128 × 128 boxes using temporal information, without requiring depth data.Objects may extend beyond crop borders when distance changes or tracking partially fails; no manual correction is applied.
- Dataset split: The final dataset has 164,866 128 × 128 RGB-D images, with three sessions reserved for testing and eight for training.Training and test sessions were balanced as much as possible across indoor/outdoor setting, holding hand, and background complexity.
4 Static Object Recognition Benchmark
CORe50 supports static object-recognition evaluation with medium-sized CNNs and temporally fused frame confidence. The benchmark uses challenging pose and acquisition variability while comparing training strategies.
- Benchmark scope: CORe50 remains a medium-size static benchmark despite being designed for continuous object recognition.High object-pose variability and complex acquisition settings make whole-training-data evaluation sufficiently difficult.
- Learning strategies: The static benchmark compares training from scratch, fixed pretrained features with a linear SVM, and fine-tuning pretrained models.These correspond to Mid-CNN from scratch, Mid-CNN + SVM, and Mid-CNN + FT.
- Learning strategies: Fine-tuning a pretrained model is often the most effective strategy when dataset size balances representative feature learning against overfitting.The passage presents this as a finding reported by many authors rather than a CORe50-specific numerical result.
- CNN evaluation: Table 2 evaluates CaffeNet and VGG models adapted to 128 × 128 inputs across different learning strategies.The test set uses sessions #3, #7, and #10, while the remaining eight sessions form the training set.
- Temporal fusion: Fusing confidence across temporally adjacent frames improves classification accuracy, but long windows without reset can mix classes and cause drift.Figure 5 reports object- and category-level Mid-VGG accuracy as the temporal window varies; 40–50 frames is described as a good compromise.
5 Continuous Object Recognition Benchmark
The continuous benchmark evaluates incremental learning under new instances, new classes, and mixed arrivals using fixed test data and repeated runs. Naïve learning can fail under new classes, CWR provides a simple baseline, and the mixed scenario remains difficult.
- Protocol: All continuous scenarios use sessions #3, #7, and #10 for testing, while the other eight sessions arrive sequentially in randomly shuffled batches.Accuracy is averaged over 10 runs, and cumulative learning uses the current batch together with all previous batches.
- NI: New Instances: In NI, successive batches add new poses and conditions for already known classes, and naïve SGD provides a baseline against cumulative training.The eight training sessions serve as sequential batches, enabling repeated memory refresh through new views and conditions.
- NI: New Instances: The accuracy gap between naïve and cumulative learning is quite modest in NI.Careful learning-rate and iteration tuning, including early stopping, can tame forgetting when new poses and conditions regularly refresh the model.
- NC: New Classes: NC introduces new object classes in sequential batches without memory refresh across batches, making preservation of earlier knowledge essential.The first batch contains 10 classes, while the remaining eight batches contain five classes each.
- NC: New Classes: The chosen NC Full Test Set keeps all classes in evaluation and provides a trade-off between comparative simplicity and usefulness.The same coherent test set is retained across NI, NC, and NIC; rejection-based evaluation is more realistic but harder to compare because accuracy depends on a threshold.
- NC: New Classes: Naïve NC learning completely forgets old tasks, whereas CWR performs fairly well as a simple baseline.CWR skips fc6 and fc7, keeps weights through pool5 fixed, and uses consolidated and temporary class weights.
- NC: New Classes: FW and CW perform significantly worse than CWR in the reported comparison.FW omits consolidated weights, while CW omits CWR’s re-initialization step.
- NIC: New Instances and Classes: NIC combines new classes and new instances across 79 batches, making it the closest scenario to many real-world applications described by the paper.The first batch contains 10 classes, later batches contain five classes, and each batch includes only one training sequence per class.
6 Conclusion
The paper introduces CORe50 and associated benchmarks for continuous object recognition, while showing that naïve incremental tuning remains inadequate and CWR leaves substantial room for improvement.
- CORe50 and its associated benchmarks support continuous learning studies for object recognition.
- Naïve incremental tuning cannot avoid catastrophic forgetting in complex NC and NIC scenarios.
- CWR outperforms the naïve solution, but its accuracy drop relative to cumulative learning remains large.
- Future work includes expanding classes and sessions, combining motion with depth, and using RGB-depth models.
A Using pre-trained CNN with different input size
The paper adapts pre-trained CNNs to CORe50's native 128 × 128 inputs by retraining fully connected layers, reducing inference cost while exposing an accuracy gap versus full-size models.
- The authors choose to reuse the network through the last convolutional layer and retrain fully connected layers for the new input size.They reshape inputs to 3 × 128 × 128 and reduce fc6 and fc7 from 4096 to 2048 units.
- The reduced units in fc6 and fc7 have no significant impact on accuracy according to the authors' observation.
- 3.4× for CaffeNet and 4.67× for VGG are the reported inference speedups from the adapted models.
- Mid-size networks lose about 10% accuracy relative to their original versions across SVM and fine-tuning experiments.The authors state that the reason for this accuracy drop is not totally clear.
- Experiments on NORB, COIL100, BigBrother, and iCub produced close results, suggesting zoomed images may enable more detailed feature extraction.The authors call for further investigation and competitive pre-trained mid-size networks for native inputs near 128 × 128.