Source-linked AI summary
A 4D Light-Field Dataset and CNN Architectures for Material Recognition
Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A. Efros, Ravi Ramamoorthi
TL;DR
Material recognition is difficult from 2D appearance because shape and lighting confound material cues, while prior light-field datasets are too limited for data-hungry CNNs. The paper introduces a 4D light-field material dataset and adapts CNN architectures to its inputs. The best architecture improves patch classification from 70% to 77% over single 2D images, providing baselines for future light-field research.
Problem
Material recognition is confounded by object shape and lighting, while prior light-field datasets are too small for deep learning and CNNs had not been adapted to 4D light-fields.
Method
The paper builds a 12-class light-field dataset with about 30,000 patches and adapts pretrained 2D CNNs with architectures and angular processing for 4D inputs.
Results
70% to 77% patch classification accuracy is achieved over single 2D image classification, with a reported absolute gain of about 7%.
Takeaways & Limitations
The results provide baselines and architectural insights for CNN-based light-field material recognition and potentially other light-field tasks.
Takeaways & Limitations
Direct 4D filters require many parameters and cannot be initialized from pretrained networks, while patch size trades accuracy against full-scene output resolution.
Abstract
from arXiv · showhide
We introduce a new light-field dataset of materials, and take advantage of the recent success of deep learning to perform material recognition on the 4D light-field. Our dataset contains 12 material categories, each with 100 images taken with a Lytro Illum, from which we extract about 30,000 patches in total. To the best of our knowledge, this is the first mid-size dataset for light-field images. Our main goal is to investigate whether the additional information in a light-field (such as multiple sub-aperture views and view-dependent reflectance effects) can aid material recognition. Since recognition networks have not been trained on 4D images before, we propose and compare several novel CNN architectures to train on light-field images. In our experiments, the best performing CNN architecture achieves a 7% boost compared with 2D image classification (70% to 77%). These results constitute important baselines that can spur further research in the use of CNNs for light-field applications. Upon publication, our dataset also enables other novel applications of light-fields, including object detection, image segmentation and view interpolation.
1 Introduction
The paper addresses material recognition from confounded 2D appearance by investigating whether 4D light-field information and CNNs adapted to it improve recognition. It introduces a mid-size dataset and reports higher accuracy than single-view 2D classification.
- Motivation: Material appearance is confounded by object shape and lighting, motivating reflectance-based cues for more robust recognition.Direct reflectance measurement is cumbersome and does not easily handle spatially varying BRDFs or BTFs.
- Motivation: Light-field cameras capture multiple viewpoints in one shot, exposing intensity variation across viewing angles for material recognition.The paper investigates whether this 4D information improves performance over 2D images.
- Contributions: The dataset contains 12 material classes with 100 labeled images per class and approximately 30,000 extracted patches.It is presented as the first mid-size light-field image dataset and includes per-pixel ground truth.
- Contributions: The authors implement CNN architectures for 4D inputs by adapting previous 2D models and adding angular processing.They reuse spatial filters from pretrained 2D models and add angular filters; patch training precedes full-image fine-tuning because direct FCN training was unstable.
- Results: 70% to 77% accuracy is achieved on extracted patches, a 7% boost over single 2D image classification.On full images, accuracy increases from 74% to 80%.
2 Related work
Prior light-field datasets are generally small or specialized, while material databases are larger but predominantly 2D. This paper positions its 1200-image 4D dataset as a mid-size resource for learning-based light-field research.
- Light-field datasets: Earlier light-field datasets include few synthetic or real scenes, camera-array captures, microscope data, or task-specific collections.The related-work discussion identifies a lack of large light-field datasets before this work.
- Material databases: Category-level material databases range from FMD’s 10 categories and 100 images per category to MINC’s 3 million patches across 23 materials.OpenSurfaces contains over 20,000 real-world scenes labeled with materials and objects.
- Material databases: Existing category-level material datasets are limited to 2D, preventing investigation of advantages from multiple views.The reflectance disk dataset captures viewing-angle intensities but lacks spatial information and is much smaller.
- Convolutional neural networks: CNNs have become successful in recognition, detection, and per-pixel segmentation, motivating their adaptation to light-field inputs.The discussion cites AlexNet, GoogLeNet, VGG, and fully convolutional approaches as relevant CNN developments.
3 The light-field material dataset
The dataset uses Lytro Illum captures spanning 12 material classes, varied acquisition conditions, and 4D angular views. Examples illustrate how reflectance variation can aid recognition beyond 2D appearance, while patch size balances accuracy against segmentation resolution.
- Dataset composition: The dataset contains 12 classes, including fabric, foliage, fur, glass, leather, metal, plastic, paper, sky, stone, water, and wood, with 100 images per class.The images were captured using a Lytro Illum camera.
- Acquisition: Images were collected by different authors across locations, viewpoints, lighting conditions, and camera parameters.The spatial resolution is 376 × 541 and the angular resolution is 14 × 14.
- Light-field structure: Each light-field image contains blocks representing different viewpoints of a 3D point.These angular views provide information beyond a conventional 2D image.
- Recognition benefits: Light-field information makes paper easier to distinguish from sky because paper changes reflectance across viewpoints while sky does not.A printed pillow photograph is classified as paper by the light-field model but as fabric by the 2D model.
- Patch size: Patch size trades recognition accuracy against full-scene segmentation resolution.Smaller patches improve spatial resolution but provide less context, while larger patches often improve accuracy.
- CNN architectures: The CNN architectures process 4D inputs through view pooling, channel stacking, EPI concatenation, or angular filtering.The figure describes angular coordinates ranging from (−3, −3) to (3, 3) and a 7 × 7 angular filter in one design.
4 CNN architectures for 4D light-fields
The paper adapts CNNs to 4D light-field inputs by testing representations and filters compatible with pretrained 2D networks. Its angular-filter and 4D-filter architectures achieve the highest classification accuracy among the compared methods.
- CNN formulation: The CNN maps a light-field image R to confidence scores for each material class and uses softmax loss for training.The network parameters are tuned during training, and softmax outputs are interpreted as predicted class probabilities at test time.
- EPI architecture: EPI representations sample horizontal and vertical slices through the light-field and concatenate them into a cube for network input.The central view is used as the reference, and only the first layer of the pretrained model is modified.
- Remap architecture: The remap representation replaces each image pixel with an angular block from multiple views, producing an input of size (ha × hs) × (wa × ws).Boundary viewpoints with very oblique viewing angles are excluded before processing.
- Angular filter: Angular filtering on the remap image reduces the representation to the original 2D spatial size before it enters the pretrained network.The intermediate layer applies an angular filter with stride ha, wa and output channel number C.
- 4D filter: 4D filters are decomposed into consecutive spatial and angular filters, while interleaving them mimics a 4D filter structure.This design reduces parameter and memory costs while allowing spatial filters to reuse pretrained 2D weights.
- Results: The angular filter and 4D filter methods obtain the highest classification accuracy among the evaluated architectures.The comparison reports average and variance across the architecture results, with 2D average only slightly above single-view classification and viewpool slightly below it.
5 Experimental results
The experiments compare CNN architectures and 2D versus light-field inputs, showing that suitable 4D representations improve material recognition, especially when object information is limited. Performance gains depend on material category, patch size, and architecture design, while results also transfer to a separate BTF dataset.
- 5.1 Comparison of different CNN architectures: 77% patch classification accuracy is achieved by the angular filter and 4D filter methods, about 7% higher than using 2D images alone.The angular filter is used as the primary comparison because it achieves similar performance to the 4D filter with substantially lower memory consumption.
- 5.1 Comparison of different CNN architectures: 76.6% and 73.7% accuracy result when angular filters are applied at layers 1 and 2, compared with 77.8% when applied directly to the input.The results support combining views earlier in the network; smaller decomposed filters perform worse than the full 7×7 angular filter.
- 5.2 Comparison between 2D and light-field results: Over 10% absolute gains occur for leather, paper, and wood, while glass is the only material whose performance drops with light-field inputs.The authors associate the gains with complex appearance effects such as subsurface scattering and inter-reflections, which multiple views can help disambiguate.
- 5.2 Comparison between 2D and light-field results: At patch size 128, the absolute gain is 7%; shrinking patches to size 32 raises the absolute gain to 10% and the relative gain to about 20%.Light-field benefits are strongest for small patches, where object identity is harder to recognize and local texture or reflectance becomes more important.
- 5 Experimental results: The experiments equalize input pixel counts and use patch-to-full-image fine-tuning because directly training a fully convolutional network is unstable.The full-image pipeline converts the patch model into a fully convolutional model before fine-tuning for material segmentation.
- 5.4 Results on other datasets: 65.2% classification accuracy is achieved when the model is directly applied to overlapping materials in the synthesized BTF database.With SVM features, light-field inputs reach 63.7% versus 59.8% for 2D inputs; after fine-tuning, they reach 73.0% versus 67.7%.
6 Conclusion
The paper introduces a substantially larger light-field dataset and demonstrates that exploiting 4D information improves material classification over single-view inputs. The resulting dataset and architectures provide foundations for further light-field research.
- The dataset contains 1200 images and is described as the first acquired with a Lytro Illum camera.
- Full-scene examples show more accurate material predictions with light-field inputs than with 2D inputs.
- The dataset supports material recognition while also providing a basis for other light-field applications.
- The experimental results report an absolute classification-accuracy gain of about 7% from using 4D light-field images instead of a single view.