Source-linked AI summary
Material Recognition in the Wild with the Materials in Context Database
Sean Bell, Paul Upchurch, Noah Snavely, Kavita Bala
TL;DR
Recognizing materials in real-world images is difficult because appearance varies with texture, geometry, lighting, and context. The paper introduces the large, diverse MINC dataset and uses CNNs with a fully connected CRF for patch classification and dense segmentation. MINC supports reported results of 85.2% mean class accuracy for patch classification and 73.1% for per-pixel material prediction, while the paper concludes that large, well-sampled data is crucial.
Problem
Real-world material recognition is difficult because materials have rich, variable appearances and existing datasets include under-sampled categories.
Method
The paper introduces MINC and trains CNNs for patch classification, then combines fully convolutional predictions with a fully connected CRF for dense material segmentation.
Results
The study reports that its deep-learning evaluation surpasses prior material-recognition attempts across material classification and segmentation tasks.
Takeaways & Limitations
Large, well-sampled data is crucial for real-world material recognition and segmentation, while polygon-based CRF training improves boundaries over click-based training.
Takeaways & Limitations
OpenSurfaces has under-sampled categories and a single Flickr source, and expanding MINC to broader categories and cost-effective annotations remains future work.
Abstract
from arXiv · showhide
Recognizing materials in real-world images is a challenging task. Real-world materials have rich surface texture, geometry, lighting conditions, and clutter, which combine to make the problem particularly difficult. In this paper, we introduce a new, large-scale, open dataset of materials in the wild, the Materials in Context Database (MINC), and combine this dataset with deep learning to achieve material recognition and segmentation of images in the wild. MINC is an order of magnitude larger than previous material databases, while being more diverse and well-sampled across its 23 categories. Using MINC, we train convolutional neural networks (CNNs) for two tasks: classifying materials from patches, and simultaneous material recognition and segmentation in full images. For patch-based classification on MINC we found that the best performing CNN architectures can achieve 85.2% mean class accuracy. We convert these trained CNN classifiers into an efficient fully convolutional framework combined with a fully connected conditional random field (CRF) to predict the material at every pixel in an image, achieving 73.1% mean class accuracy. Our experiments demonstrate that having a large, well-sampled dataset such as MINC is crucial for real-world material recognition and segmentation.
1. Introduction
Material recognition in real-world images is difficult because materials vary in texture, appearance, lighting, shape, and context. The paper introduces MINC and combines large-scale data with CNNs and a fully connected CRF for material classification and segmentation.
- Motivation: Real-world material recognition is challenging because materials have diverse textures and appearances shaped by lighting, geometry, and context.Some materials are smooth and featureless, requiring subtle cues or contextual reasoning to distinguish them.
- Motivation: Existing datasets were limited by small category counts, few examples per category, hand-picked imagery, or under-sampled material categories.OpenSurfaces provided many real-world segmentations but still contained only tens of examples for some categories.
- MINC: MINC provides 3 million diverse, well-sampled material samples from Flickr scenes and staged Houzz interiors.The two image sources have different characteristics that expand the range of recognizable materials.
- Approach: The paper trains CNNs for patch classification and converts them into a fully convolutional model with a fully connected CRF for dense material prediction.Replacing fully connected CNN layers with convolutional layers reduces the computational burden relative to naive sliding-window classification.
- Contributions: The paper contributes both MINC with a three-stage crowdsourcing pipeline and a CRF-based method for simultaneous material recognition and segmentation.These contributions target scalable data collection and per-pixel material labeling.
2. Prior Work
Prior material-recognition datasets and methods progressed from controlled texture samples toward more diverse real-world images and learned representations. MINC extends this trajectory with substantially more real-world material labels.
- Material Databases: Early databases such as CUReT emphasized instance-level material classification under varied lighting and viewing conditions.CUReT contains 61 material samples captured under 205 lighting and viewing conditions.
- Material Databases: Later datasets such as KTH-TIPS explored generalization across different examples within the same material category.This work addressed transfer from one material sample to another, such as between different samples of wood.
- Material Databases: FMD introduced categorical material recognition with ten material categories and 100 Flickr samples per category.Its images were selected to illustrate varied material appearances.
- Material Databases: OpenSurfaces added over 20,000 real-world scenes labeled with materials and objects through crowdsourcing.Unlike earlier cropped stand-alone samples, its Flickr imagery preserved real-world context.
- Recognition Methods: MINC substantially expands OpenSurfaces with millions of new material labels.This positions MINC as a larger real-world categorical material database.
- Recognition Methods: Prior material-recognition work commonly classified image patches using hand-designed features, while CNNs recently achieved state-of-the-art results in recognition tasks.The paper builds on both feature-based material classification and modern CNN architectures.
3. The Materials in Context Database (MINC)
MINC was designed for large, balanced, diverse material recognition and built by augmenting OpenSurfaces with Houzz imagery, segments, and crowdsourced clicks. The resulting patches support both patch classification and full-image material segmentation.
- Database Requirements: The database requirements were sufficient size, balanced sampling of rare categories, diverse real-world appearances, and many material categories.These requirements were intended to support learning that generalizes beyond the training set.
- Sources of Data: OpenSurfaces supplied diverse Flickr scenes and many categories, but its annotation process left common materials overrepresented and rare categories underrepresented.Wood had nearly 20K samples, whereas water had only tens, and Flickr was the sole image source.
- Sources of Data: Houzz imagery was added to increase data for underrepresented categories and broaden material variety beyond Flickr scenes.The paper specifically notes that staged Houzz interiors contain varied types of polished stone.
- Segments, Clicks, and Patches: The three-stage click pipeline filters images, collects material clicks, and independently validates click labels.Sentinel answers are used in the first and third stages to block workers below specified accuracy thresholds.
- Segments, Clicks, and Patches: MINC uses material segments and clicks to support patch material classification and full-scene per-pixel segmentation.Clicks are cheaper and faster to collect than full material segmentations, while both annotation types are converted into square image patches.
- Segments, Clicks, and Patches: The crowdsourcing pipeline collected 2,341,473 annotations at an average cost of $0.00306 per annotation.Material clicks were collected for both OpenSurfaces and Houzz images.
- Segments, Clicks, and Patches: The final 23-category database contains 2,996,674 labeled patches from 436,749 images after Poisson-disk subsampling of segment interiors.Patch centers are separated by at least 9.1% of the smaller image dimension, producing an average of 9.05 patches per segment.
4. Material recognition in real-world images
The system converts patch-based CNN material classifiers into a multiscale dense predictor, then uses a fully connected CRF to produce per-pixel material labels. Patch scale balances contextual information against spatial resolution, while the evaluation covers CNN architectures and category-level accuracy.
- Dense CNN prediction: A patch-trained CNN is converted into a sliding-window detector that densely predicts materials across images.The converted network replaces fully connected layers with convolutional layers and produces predictions on a pixel grid.
- Dense CNN prediction: 32-pixel network strides are densified to 16-pixel prediction spacing by shifting the input image by half-strides.Convolution reuse limits the additional computation to later network layers for these shifts.
- Multiscale aggregation: Predictions from three resized image scales are upsampled and averaged into a probability map before CRF refinement.The input resizing makes larger context correspond to lower spatial resolution.
- CRF refinement: A fully connected CRF combines aggregated CNN probabilities with pairwise reasoning to assign a material label to every pixel.The unary energy is the negative log of aggregated softmax probabilities, while pairwise terms connect every pixel pair using color and position features.
- Evaluation: The experiments compare multiple GoogLeNet variants, AlexNet at multiple patch scales, ensembles, and category-level patch test accuracy.The supplied table captions identify mean class accuracy across CNNs and test accuracy by material category.
5. Experiments and Results
Experiments examine how architecture, context, data size, dataset balance, and supervision affect material classification and segmentation. Results show that balanced, larger-scale training data and carefully chosen CNN/CRF configurations improve performance across these tasks.
- Patch material classification: Scale 32% performs best overall, while 23.3% provides nearly the same accuracy with higher spatial resolution and is used in subsequent experiments.Mirror, wallpaper, and sky benefit from increasing context, unlike most categories.
- Patch material classification: More training data improves patch classification, with performance not yet saturated at 2.5 million training patches.The authors report diminishing returns as data increases, while noting that additional data may yield higher accuracy.
- Patch material classification: Balanced sampling is more effective than an equally sized unbalanced dataset, reaching performance only 1.7% below the full training set.The balanced setup uses all patches from the smallest category and randomly samples larger categories to match it.
- Full scene material segmentation: Training and testing on segments produces clean boundaries, whereas click-only optimization yields sloppy boundaries because clicks are not necessarily near object boundaries.Clicks are sufficient to train a CNN but not sufficient to train a CRF.
- Comparing datasets and prior methods: Cross-dataset results indicate that FMD-only training is insufficient for real-world classification, while fine-tuned CNNs outperform SIFT IFV+fc7 on MINC.On the reported MINC subset, fine-tuned AlexNet achieves 76.0 ± 0.2%, compared with 67.4 ± 0.5% for SIFT IFV+fc7.
6. Conclusion
MINC is a large, open material database used to evaluate deep-learning methods for simultaneous material classification and segmentation, surpassing prior attempts. The authors emphasize context-aware training and different annotation strategies, while identifying broader categories and richer attributes as future directions.
- MINC includes diverse everyday and staged-interior materials and is at least an order of magnitude larger than prior databases.The database supports evaluation of deep-learning methods for material classification and segmentation.
- Training with surrounding context is crucial for real-world material classification.
- Labeled clicks are cheap and sufficient for training a CNN, but polygon-based CRF training produces much better segmentation boundaries than click-based training.
- Expanding MINC to broader categories will require more varied image-mining methods and cost-effective annotation tasks.
- Future work includes identifying material attributes and jointly classifying and segmenting materials and objects.