Source-linked AI summary
Interpretable Convolutional Neural Networks via Feedforward Design
C. -C. Jay Kuo, Min Zhang, Siyang Li, Jiali Duan, Yueru Chen
TL;DR
CNN parameters are conventionally learned by BP, but deep-network optimization is difficult to analyze end to end. This paper proposes a data-centric FF alternative using Saab transforms and cascaded least-squared regressors, and compares it with BP on MNIST and CIFAR-10. The comparisons report classification and adversarial-robustness behavior while examining the relationship between the two designs.
Problem
End-to-end analysis of deep CNN working principles remains challenging, while BP determines parameters through mathematically intractable non-convex optimization.
Method
The paper derives layer parameters in one pass from preceding-layer data statistics, using Saab transforms for convolutional layers and cascaded linear least-squared regressors for fully connected layers.
Results
The study compares BP- and FF-designed CNNs on MNIST and CIFAR-10 for classification and adversarial robustness, reporting classification-accuracy gaps of 2.7% for MNIST and 6% for CIFAR-10 between FF and BP designs.
Takeaways & Limitations
The FF design provides a complementary, interpretable approach to CNN filter-weight selection and sheds light on the traditional BP design.
Takeaways & Limitations
FF performance remains under development, and overlapping pseudo-classes can cause leakage or interference in the LSR mapping.
Abstract
from arXiv · showhide
The model parameters of convolutional neural networks (CNNs) are determined by backpropagation (BP). In this work, we propose an interpretable feedforward (FF) design without any BP as a reference. The FF design adopts a data-centric approach. It derives network parameters of the current layer based on data statistics from the output of the previous layer in a one-pass manner. To construct convolutional layers, we develop a new signal transform, called the Saab (Subspace Approximation with Adjusted Bias) transform. It is a variant of the principal component analysis (PCA) with an added bias vector to annihilate activation's nonlinearity. Multiple Saab transforms in cascade yield multiple convolutional layers. As to fully-connected (FC) layers, we construct them using a cascade of multi-stage linear least squared regressors (LSRs). The classification and robustness (against adversarial attacks) performances of BP- and FF-designed CNNs applied to the MNIST and the CIFAR-10 datasets are compared. Finally, we comment on the relationship between BP and FF designs.
1. Introduction
The paper addresses the difficulty of end-to-end CNN analysis by proposing an interpretable feedforward design that derives parameters from data statistics without backpropagation. It uses Saab transforms for convolutional layers and cascaded least-squared regressors for fully connected layers, then compares BP and FF designs on MNIST and CIFAR-10.
- CNN interpretability remains challenging despite prior work on representations, routing paths, nonlinear activation, and convolutional filters as signal transforms.
- The proposed FF design avoids BP by deriving each layer’s parameters from output statistics of the preceding layer in one pass.The approach is data-centric and is presented as a reference methodology for interpreting CNN architectures.
- Convolutional layers use Saab transforms, PCA variants with an added bias vector intended to annihilate activation nonlinearity and reduce dimensions.The transforms support spatial-spectral filtering as convolutional representations become progressively coarser.
- Fully connected layers use cascaded linear least-squared regressors to map feature representations through pseudo-label and class spaces.This construction progressively reduces intermediate dimensions while increasing the discriminability of selected dimensions.
- The study compares classification and adversarial robustness of BP- and FF-designed CNNs on MNIST and CIFAR-10 and examines cross-layer cross-entropy values.
2. Background
The paper interprets CNN neurons as affine filters followed by nonlinear activation, then motivates feedforward convolutional design through data-derived subspaces. It addresses sign confusion with the Saab transform, which avoids the spectral doubling used by the earlier Saak transform.
- A CNN neuron performs affine computation with filter weights and a bias, followed by nonlinear activation such as ReLU.
- Anchor vectors can act as correlators that measure pattern similarity or as a set of vectors spanning a projection subspace.
- PCA provides a feedforward alternative to backpropagation by deriving convolutional anchor vectors from the covariance statistics of input data.
- ReLU resolves sign confusion by blocking one polarity, whereas the absence of ReLU can make positively and negatively correlated inputs indistinguishable downstream.
- The Saak transform resolves sign confusion by pairing each kernel with its negative, but this doubles spectral dimensions.
- The Saab transform is introduced as a PCA variant that addresses sign confusion without doubling spectral dimensions.
3. Feedforward design of convolutional layers
The feedforward convolutional design transforms local image neighborhoods into spatial-spectral representations using PCA-derived anchors, Saab bias adjustment, and cascaded filtering with pooling. These operations provide interpretable, data-driven layers without backpropagation.
- Spatial-spectral filtering: The convolutional layers perform cascaded spatial-spectral transformations that progressively reduce spatial resolution while projecting local neighborhoods onto PCA-based kernels.The design trades spatial resolution for spectral representations to enhance the discriminability of selected dimensions.
- Saab transform and bias selection: The Saab transform selects anchor vectors and bias terms so activation nonlinearity is neutralized, allowing the same output with or without ReLU.Equal bias terms place the bias vector in the DC subspace and simplify cascaded Saab analysis.
- Saab transform and bias selection: The input space is decomposed into a DC anchor and orthogonal AC subspace, with PCA applied to AC components to obtain additional anchor vectors.The DC anchor is the normalized constant vector, while AC anchors are selected from principal components of AC data.
- Spatial pooling: Maximum pooling is interpreted jointly with convolution as projecting an enlarged neighborhood onto a lower-dimensional subspace of filters.For the described 2 × 2 pooling block, the compound operation projects a 6 × 6 patch through smaller 5 × 5 neighborhoods at locations A–D and then onto anchor vectors.
- Multi-layer compound filtering: Cascaded spatial-spectral filtering and pooling capture visually similar patterns that are spatially displaced.The cascade generates compound filters representing image patterns at multiple scales; in the FF design, target patterns are linear combinations of orthogonal PCA-filter responses.
4. Feedforward design of FC layers
The feedforward fully connected design uses cascaded linear least-squared regressors whose outputs are one-hot vectors. Hidden-layer pseudo-labels are generated by clustering, while feature-space overlap limits how well regressors can separate pseudo-classes.
- The LeNet-5 fully connected portion contains three cascaded layers mapping dimensions 375→120, 120→84, and 84→10.
- Each feedforward fully connected layer is treated as a linear least-squared regressor, with one-hot outputs generalized from the final layer to hidden layers.
- Because hidden-layer inputs lack labels, k-means clustering groups them into as many clusters as the layer has output nodes.
- Removing the two hidden layers reduces the model to one-stage least-squares regression, whose performance is reported as not very good.
- Stages I and II create auxiliary labels by clustering training samples within each original class, producing pseudo-classes for regression targets.
- Pseudo-class overlap in the input feature space limits the ability of least-squared regressors to map samples cleanly to distinct orthogonal one-hot targets.
5. Experimental Results
Experiments compare BP-, FF-, and hybrid-designed LeNet-like CNNs on MNIST and CIFAR-10, including classification accuracy and robustness against adversarial attacks. The results show accuracy gaps between FF and BP designs, vulnerability to targeted attacks, and a proposed but out-of-scope ensemble direction for FF models.
- Classification performance: Experiments compare BP and FF CNN designs on MNIST and CIFAR-10, with a hybrid design using FF convolutional layers and BP-trained MLP layers.The modified LeNet-5 preserves two convolutional and three fully connected layers for CIFAR-10 color images.
- Classification performance: 2.7% and 6% are the performance gaps between FF and BP designs on MNIST and CIFAR-10, respectively.The reported drops are attributed separately to the decision subnet and feature-extraction subnet without BP optimization.
- Robustness against adversarial attacks: Three adversarial attacks—FGS, BIM, and Deepfool—are evaluated against BP- and FF-designed models individually.Deepfool produces the least visual distortion, while FGS- and BIM-attacked images have poorer visual quality.
- Robustness against adversarial attacks: Classification accuracy degrades significantly when BP and FF models face adversarial attacks targeting their respective designs.The experiments compare attacked-image accuracies in Tables 3 and 4; the text links high-quality attack vulnerability to CNN end-to-end interconnection rather than parameter-selection methodology.
- Robustness enhancement via ensemble methods: FF-1, FF-2, and FF-3 vary k-means initialization or cluster allocation, enabling multiple FF models that can be combined through ensemble methods.The authors state that ensemble robustness is beyond this work and that building multiple FF networks costs significantly less than building multiple BP networks.
6. Discussion
The discussion contrasts BP and FF designs in principle, interpretability, modularity, robustness, complexity, and representation. FF uses data statistics and sequential transformations, but remains less mature and vulnerable to adversarial attacks.
- Principle: BP optimizes a chosen architecture and cost function, whereas FF derives spatial-spectral transformations from data statistics and uses sequential LSR operations.FF convolutional layers do not require labels; FC layers provide multi-stage decision processing.
- Interpretability: FF is mathematically transparent, but its filter-weight selection differs from BP, so its explanations do not fully transfer to BP networks.The paper studies intermediate-layer cross-entropy to connect the two designs.
- Modularity: FF decouples feature extraction from classification or regression, allowing the decision module to connect extracted features with label-space outputs.This follows the traditional pattern-recognition decomposition into feature extraction and classification/regression.
- Robustness: Both BP and FF are vulnerable to adversarial attacks when the fixed, known network is targeted, while attacks cause catastrophic degradation mainly for that target network.The authors suggest fusing multiple FF networks as an ensemble, with low-cost model variation through k-means initialization.
- Training Complexity: FF can reduce training complexity through statistics because PCA and LSR may use small training subsets; hundreds of CIFAR-10 images can estimate a 75 x 75 covariance matrix.The covariance matrix converges quickly because each image supplies 784 patches, but the authors do not report a complexity number.
- Performance and Extensions: FF performance remains less established because the design is still in its infancy, while heterogeneous-region processing could use segmentation-guided PCA filter codebooks.The proposed extension assigns different filter sizes to segmented homogeneous regions and higher layers’ spatial-spectral cuboids.
- Signal Representation: BP representations are sparse and redundant with stronger correlations, whereas FF representations are orthogonal within each layer and broad-band rather than narrow-band.The comparison frames BP as using a learned multi-layer dictionary and FF as using principal-component projections.
7. Conclusion and Future Work
The paper presents FF CNNs as an interpretable complement to BP and uses comparisons to illuminate differences between the designs. It concludes with exploratory extensions involving ensembles, advanced architectures, statistical tools, and data-centric research questions.
- Conclusion: The paper proposes an interpretable FF CNN design as a complementary approach to selecting CNN filter weights and compares it extensively with BP.The authors state that the FF design sheds light on the traditional BP design.
- Future Work: Future work includes ensemble methods for improving classification and addressing adversarial attacks, plus interpretable designs for ResNet, DenseNet, and GAN architectures.The paper also calls for additional statistical tools and data-centric investigation of data collection, labeling, and sufficiency.
- Scope: The FF methodology is still in its infancy and is presented as exploratory basic CNN research with preliminary experimental results.The authors hope this new research direction will motivate follow-up work.
Appendix: Bias Selection
The bias-selection procedure augments PCA-derived responses with a constant displacement so Saab-transform outputs are non-negative. The required displacement is bounded using the input norm and a small positive margin.
- Anchor Vectors: PCA eigenvectors of the AC input define Saab anchor vectors, while a sufficiently large bias ensures non-negative response elements.The input and output are flattened AC random vectors defined on three-dimensional cuboids.
- Bias Addition: A constant displacement vector is added to the AC-subspace response, shifting every component by the same length d.The shifted vector is constructed from the response and a constant-element unit vector.
- Nonnegativity Constraint: The displacement d is chosen so every shifted response component is non-negative, using an inequality-based bound.The derivation combines the response bound with the displacement requirement to obtain a lower bound on the bias.
- Practical Selection: The lower-bound choice is relaxed by adding a small positive δ, which is the setting used in the experiment.The supplied formulation applies this choice for k = 0, 1, ..., K−1.