Source-linked AI summary
Understanding deep features with computer-generated imagery
Mathieu Aubry, Bryan Russell
TL;DR
The paper addresses how CNN representations encode varying scene factors and how those effects differ across networks and layers. It renders controlled images from 3D CAD models, analyzes layer responses through factor decomposition and PCA, and finds distinct sensitivities to viewpoint, color, and combined factors, with the analysis relating to natural-image representations.
Problem
The paper asks how different CNN layers, architectures, and training tasks represent scene factors such as object style, viewpoint, color, and lighting.
Method
The approach renders images while controlling scene factors, feeds them to trained CNNs, decomposes layer responses by factor, and quantifies and visualizes variation with PCA.
Results
Across AlexNet, Places, and Oxford VGG, the study finds that viewpoint sensitivity decreases in later layers, color is relatively more important for Places, and style-viewpoint features are often approximately linearly separable in higher layers.
Takeaways & Limitations
The analysis highlights differences in factor sensitivity across CNN architectures and layers and relates rendered-image feature analysis to natural-image representations.
Takeaways & Limitations
The multiple-factor analysis assumes that the scene-factor parameters are sampled independently.
Abstract
from arXiv · showhide
We introduce an approach for analyzing the variation of features generated by convolutional neural networks (CNNs) with respect to scene factors that occur in natural images. Such factors may include object style, 3D viewpoint, color, and scene lighting configuration. Our approach analyzes CNN feature responses corresponding to different scene factors by controlling for them via rendering using a large database of 3D CAD models. The rendered images are presented to a trained CNN and responses for different layers are studied with respect to the input scene factors. We perform a decomposition of the responses based on knowledge of the input scene factors and analyze the resulting components. In particular, we quantify their relative importance in the CNN responses and visualize them using principal component analysis. We show qualitative and quantitative results of our study on three CNNs trained on large image datasets: AlexNet, Places, and Oxford VGG. We observe important differences across the networks and CNN layers for different scene factors and object categories. Finally, we demonstrate that our analysis based on computer-generated imagery translates to the network representation of natural images.
1. Introduction
The paper introduces a controlled, rendering-based framework for analyzing how CNN representations encode scene factors, addressing limitations of prior filter visualizations. It compares networks and layers, finding distinct sensitivities to viewpoint, color, and combined factors while relating rendered-image analysis to natural-image representations.
- 1. Introduction: The study systematically analyzes how trained CNNs represent intrinsic and extrinsic scene factors using data where those factors can be independently controlled.Factors include category, style, color, 3D viewpoint, and lighting configuration.
- 1. Introduction: Computer-generated imagery from large 3D model collections provides dense control over rendering parameters for studying CNN feature responses.The approach presents rendered images to trained CNNs, decomposes responses by known factors, quantifies their relative importance, and visualizes them with PCA.
- 1. Introduction: Features varying along style and viewpoint can often be approximated by a linear combination of factor-specific features, especially in higher CNN layers.
- 1. Introduction: Viewpoint sensitivity decreases in later CNN layers, with VGG fc7 less viewpoint-sensitive than the corresponding AlexNet and Places representations.
- 1. Introduction: Color is more important relative to object style for Places than for AlexNet and VGG, especially when the varying color belongs to the background.
- 1. Introduction: The rendered-image analysis is related to understanding CNN representations of natural images.
2. Approach for deep feature analysis
The approach uses controlled image collections, especially computer-generated scenes, to separate how CNN representations vary with individual scene factors. It applies PCA and factor-wise feature decomposition to quantify variation, relative importance, and residual entanglement.
- Approach for deep feature analysis: Controlled computer-generated images let the analysis separate the influence of independently sampled scene factors on CNN representations.The factors may include style, position, scale, rotation, lighting, and color.
- Approach for deep feature analysis: PCA identifies the principal modes of variation in high-dimensional CNN features and provides low-dimensional visualizations of the image embeddings.The retained components form an optimal linear approximation, and intrinsic dimensionality is measured by the dimensions explaining 95% of variance.
- Approach for deep feature analysis: Marginal features are computed for each factor by averaging CNN responses over all other independently sampled factors.PCA can then be applied separately to each factor’s marginal features.
- Approach for deep feature analysis: The feature representation is decomposed into the sum of factor-specific marginal features and a residual component.Computer-generated rendering provides the parameter combinations needed to compute the marginal terms directly.
- Approach for deep feature analysis: Relative variance measures each factor’s importance, while low residual relative variance indicates that the factors are largely separated in the CNN layer.The variance decomposition uses zero-mean, uncorrelated terms, and the residual measure does not depend on relative factor sampling.
3. Experimental setup
The experiments evaluate higher-layer features from AlexNet, Places, and Oxford VGG using abstract stimuli and rendered ModelNet objects under controlled visual factors. The setup combines quantitative factor sampling with PCA visualizations across object categories, colors, lighting, and positions.
- Experimental setup: The study compares AlexNet, Places, and Oxford VGG features from the higher layers pool5, fc6, and fc7.Places shares AlexNet’s architecture but was trained on scene images, whereas Oxford VGG uses a different architecture.
- Experimental setup: Inputs include constant-color images, rectangular patches, and rendered ModelNet CAD views spanning object styles, viewpoints, illumination, and uniform colors.The rendered stimuli are designed to vary intrinsic and extrinsic scene factors under controlled conditions.
- Experimental setup: For constant-color fc7 inputs, AlexNet and Places produce PCA embeddings resembling a hue color wheel, with greater variation for blue and green channels.The figure compares the embeddings across the different CNNs.
- Experimental setup: Quantitative sampling uses 36 azimuth angles, 36 positions, 40 scales, 36 light positions, and 125 colors, while visualizations use denser sampling.The authors report that changing the sampling density did not alter the quantitative results.
- Experimental setup: In the synthetic rectangle experiment, higher layers assign increasing relative variance to aspect ratio and decreasing relative variance to 2D position.This pattern indicates greater emphasis on shape and less emphasis on image location.
4. Results
Experiments on abstract stimuli, rendered 3D objects, and natural images show that CNN layers encode scene factors differently, with higher layers generally separating factors more cleanly and becoming less viewpoint-sensitive.
- 4.1. 2D abstract stimuli: Relative variance associated with position decreases across layers, while aspect-ratio variance increases from less than 10% in AlexNet pool5 to 37% in fc7.The residual also decreases in higher layers, indicating cleaner separation of the two factors.
- 4.2. Object categories: Decomposing rendered-object features separates style and viewpoint, revealing category-dependent rotation geometry and substantial viewpoint information loss between pool5 and fc6.Car and chair embeddings differ, and the fc6 rotation embedding is qualitatively different from pool5.
- 4.2. Object categories: Higher CNN layers become progressively less sensitive to viewpoint, with VGG fc7 less viewpoint-sensitive than AlexNet and Places.This pattern is reported across rendered-object experiments and is consistent with the abstract-stimulus analysis.
- 4.2. Object categories: Places features assign more variance to color than AlexNet and VGG, whereas AlexNet and VGG emphasize foreground color more than background color in fc7.For Places fc7, background-color variance exceeds foreground-color variance, with a reported 3.8× difference.
- 4.3. Natural images: The rendered-image analysis transfers to natural images: ETH-80 produces the same high-level conclusions as rendered objects, although differences are less pronounced.The smaller ETH-80 dataset limits the number of variations that can be explored.
5. Conclusion
The paper introduces a method using controlled variation of 3D-model stimuli to analyze CNN feature sensitivity to object style, viewpoint, and color across networks and layers.
- The method compares CNN sensitivity to object style, viewpoint, and color by varying network stimuli according to controlled scene factors.