Source-linked AI summary
Do We Need More Training Data?
Xiangxin Zhu, Carl Vondrick, Charless Fowlkes, Deva Ramanan
TL;DR
The paper asks whether continually expanding training datasets will keep improving object recognition or whether detectors saturate because of model and feature-space limits. It studies mixture models and compositional extensions on large datasets, finding that classic mixtures saturate quickly while richer compositional models better exploit additional data. The authors therefore point toward better models, representations, and learning algorithms as the route to further gains.
Problem
The paper investigates whether ever-larger training datasets can drive continued object-recognition progress without developing more complex detection models.
Method
The authors collect a dataset an order of magnitude larger than existing collections and evaluate mixture models, including compositional DPM extensions, as data and complexity increase.
Results
Classic mixtures show diminishing returns, while compositional representations with shared parts produce larger gains from additional training data.
Takeaways & Limitations
Better models with richer constraints and compositionality are needed to make better use of big data for detection.
Takeaways & Limitations
The analysis is constrained by the Bayes risk of the HOG feature space, which cannot be computed directly, and by possible suboptimal regularization in existing detectors.
Abstract
from arXiv · showhide
Datasets for training object recognition systems are steadily increasing in size. This paper investigates the question of whether existing detectors will continue to improve as data grows, or saturate in performance due to limited model complexity and the Bayes risk associated with the feature spaces in which they operate. We focus on the popular paradigm of discriminatively trained templates defined on oriented gradient features. We investigate the performance of mixtures of templates as the number of mixture components and the amount of training data grows. Surprisingly, even with proper treatment of regularization and "outliers", the performance of classic mixture models appears to saturate quickly ($\sim$10 templates and $\sim$100 positive training examples per template). This is not a limitation of the feature space as compositional mixtures that share template parameters via parts and that can synthesize new templates not encountered during training yield significantly better performance. Based on our analysis, we conjecture that the greatest gains in detection performance will continue to derive from improved representations and learning algorithms that can make efficient use of large datasets.
1 Introduction
The paper asks whether ever-larger training datasets can sustain object-recognition progress without more complex detectors. Experiments with large datasets and increasingly flexible mixture models indicate that classic mixtures saturate, while compositional representations can better exploit additional data.
- Continually increasing training data may not sustain object-recognition progress without more complex detection models.
- The study evaluates HOG templates trained with linear SVMs as training-data volume and model complexity vary.
- Off-the-shelf implementations sometimes decreased performance as additional data was added, contrary to expected improvement and eventual saturation.
- Even non-parametric models that grow with data showed diminishing returns after modest amounts of training data.
- Classic mixtures saturated despite regularization and outlier treatment, whereas compositional sharing across parts improved their ability to use additional data.
- The paper argues that future detection gains are more likely from improved representations and learning algorithms than from simple models trained on bigger datasets.
2 Big Detection Datasets
The paper constructs PASCAL-10X, a substantially enlarged training dataset for 11 PASCAL categories, and evaluates its collection quality and similarity to PASCAL 2010. It also examines face detection data and finds that sufficiently large negative sets make performance relatively insensitive to additional negative training images.
- PASCAL-10X: PASCAL-10X provides approximately 10 times as many training examples per category as standard PASCAL training data for 11 categories.The evaluation uses the 11 PASCAL 2010 trainval categories, containing more than 10,000 images.
- Additional experiments: Face models use up to 900 MultiPIE faces across 13 viewpoints spanning 180°, while experiments vary positive examples and hold negative training images fixed.Performance was relatively static with respect to negative training data once a sufficiently large negative set was used.
- Collection: The dataset was built from over one hundred thousand Flickr images using randomized category-keyword sampling designed to mimic PASCAL’s collection procedure.Images were filtered for category presence through MTurk classification tasks before bounding-box annotation.
- Annotation: MTurk workers annotated bounding boxes at an average rate of three images per second, but incomplete object labeling prevents PASCAL-10X from serving as a detection benchmark.Because not every object is labeled, false positives cannot be distinguished from true negatives.
- Quality validation: Quality analyses indicate that the collection and annotation pipeline produces training data similar to PASCAL, including comparable horse attribute distributions and near-chance user preferences.A user preferred PASCAL 2010 boxes 49% of the time and PASCAL-10X boxes 51% of the time.
3 Mixture models
The paper studies mixture models whose complexity grows with training data, comparing clustering strategies and compositional models that share part parameters. It presents DPMs and EDPMs as efficient high-capacity alternatives that synthesize or extrapolate templates beyond observed configurations.
- Independent mixtures: Mixture components capture additional object sub-category structure as training data increases.
- Independent mixtures: Partitioned sampling varies cluster count K and training size N while preserving nested data subsets and consistent cluster refinements.Repeated resampling supports confidence-interval estimation while reducing stochastic effects from inconsistent partitions.
- Independent mixtures: Supervised clustering uses viewpoint or manually grouped visual similarity, while unsupervised baselines use hierarchical k-means partitions.PASCAL-10X uses over-clustering followed by manual cluster merging because viewpoint labels are unavailable.
- Compositional mixtures: DPMs implicitly represent an exponentially large mixture of rigid templates by shifting and combining shared parts, then searching configurations efficiently.Each template corresponds to a part configuration, with its bias encoding the associated spatial deformation score.
- Compositional mixtures: Mixtures of DPMs share part-filter computations and dynamic-programming messages across components, greatly reducing mixture-dependent computation.The architecture is designed to increase expressive power while retaining efficient inference.
- Compositional mixtures: EDPMs extrapolate beyond observed exemplar shapes, whereas EPMs restrict test-time configurations to training exemplars.Their shape models differ through the allowed configurations, and EDPM inference can remain no more expensive than EPM inference on the same examples.
4 Experiments
Experiments show that scaling data and rigid mixture complexity produces diminishing returns, while compositional models with shared parts and extrapolation perform substantially better.
- 4.1 The importance of proper regularization: Cross-validation is essential because additional positive data can reduce performance under fixed regularization.The optimal regularization parameter depends on the number of positive examples.
- 4.2 The importance of clean training data: 900 versus 470 classification errors show that noisy hard examples can distort the decision boundary under hinge loss.The corresponding hinge objectives were 1.29 versus 3.48 for Frontal and All, respectively.
- 4.3 Performance of independent mixtures: ∼100 training examples per mixture and ∼10 mixtures mark the point where independent-mixture improvements quickly saturate.With fixed training data, splitting examples across too many mixtures eventually decreases performance through overfitting.
- 4.4 Performance of compositional mixtures: AP improves from 76% to 78.5% for faces and from 56% to 64% for buses with exemplar parameter sharing.For large training sets, these models approach latent DPM performance; EDPMs perform similarly to EPMs and DPMs.
- 4.4 Performance of compositional mixtures: 85% AP for supervised EPMs exceeds 76% for independent mixtures, while supervised DPMs reach 91% AP.The remaining gap is attributed to DPMs synthesizing configurations unseen during training.
- 4.4 Performance of compositional mixtures: Part models improve detection by sharing parameters across observed mixtures and extrapolating to implicit, unseen templates.This is motivated by the long-tail distribution of object shapes, including many rare configurations.
5 Related Work
The paper situates its analysis among recognition architectures, non-parametric methods, and scalable nearest-neighbor techniques, emphasizing parts as efficient computational indexing structures.
- Object detection: Face detection reaches 76% AP with mixtures, 85% with EPMs, and 91% with supervised DPMs.The comparison illustrates gains from parameter sharing and latent structure.
- Object detection: For buses, latent EPMs significantly outperform rigid mixtures and match standard latent DPMs.The comparison concerns mixture models with latent EPMs, EDPMs, and DPMs.
- Object detection: The analysis focuses on template-based recognition while noting that alternative architectures may have different dataset-size dependencies.The authors hypothesize that parameter sharing and extrapolation may generalize across architectures.
- Non-parametric models in vision: Object-shape histograms show long-tail distributions with common patterns and many rare cases.The figure bins examples by quantized part-location vectors for buses and faces.
- Scalable nearest-neighbors: Parts can support exact, efficient indexing into an exponentially large template set, not only semantic modeling.This offers a computational interpretation of compositional parts.
6 Conclusion
The conclusion argues that current HOG-template detectors cannot fully exploit growing datasets without careful regularization and richer compositional representations.
- 6 Conclusion: Cross-validating regularization is mundane but crucial when scaling detection systems to larger datasets.The conclusion also identifies sensitivity to noisy data as a central practical issue.
- 6 Conclusion: Accurate HOG templates can be trained with 100–200 positive examples rather than the thousands typically used.This indicates a relatively small effective capacity for HOG templates.
- 6 Conclusion: Current state-of-the-art detectors may require significant, perhaps exponentially larger, datasets for continued consistent improvement.The conclusion frames this as a limitation of relying on simple models backed only by more data.
- 6 Conclusion: Larger gains arise from richer model constraints and non-parametric compositional representations that make better use of additional data.The paper emphasizes parameter sharing, compositionality, and non-parametric encodings without changing HOG features or discriminant-function class.
- 6 Conclusion: Clean correspondence among training examples and compositional approaches are identified as promising directions.Examples include shared subcategories and corresponding spatial parts.