Source-linked AI summary
Tensor Regression Networks
Jean Kossaifi, Zachary C. Lipton, Arinbjorn Kolbeinsson, Aran Khanna, Tommaso Furlanello, Anima Anandkumar
TL;DR
Fully connected layers flatten activation tensors, discarding multilinear structure and requiring many parameters. The paper introduces end-to-end trainable Tensor Contraction and Tensor Regression Layers that preserve tensor structure through low-rank operations. Applied to ImageNet and UK Biobank MRI tasks, these layers reduce parameters while maintaining or improving performance.
Problem
Flattening activation tensors before fully connected layers discards multilinear structure and requires many parameters.
Method
The paper introduces end-to-end trainable TCLs and TRLs that preserve tensor structure through contraction and low-rank multilinear regression.
Results
Almost 65% fewer parameters were achieved on ImageNet while improving performance, with superior results across three UK Biobank MRI tasks.
Takeaways & Limitations
The layers provide compact networks that preserve multidimensional topological structure while achieving similar or better accuracy with fewer parameters.
Takeaways & Limitations
The paper notes that additional regularization terms for noisy inputs remain for future investigation.
Abstract
from arXiv · showhide
Convolutional neural networks typically consist of many convolutional layers followed by one or more fully connected layers. While convolutional layers map between high-order activation tensors, the fully connected layers operate on flattened activation vectors. Despite empirical success, this approach has notable drawbacks. Flattening followed by fully connected layers discards multilinear structure in the activations and requires many parameters. We address these problems by incorporating tensor algebraic operations that preserve multilinear structure at every layer. First, we introduce Tensor Contraction Layers (TCLs) that reduce the dimensionality of their input while preserving their multilinear structure using tensor contraction. Next, we introduce Tensor Regression Layers (TRLs), which express outputs through a low-rank multilinear mapping from a high-order activation tensor to an output tensor of arbitrary order. We learn the contraction and regression factors end-to-end, and produce accurate nets with fewer parameters. Additionally, our layers regularize networks by imposing low-rank constraints on the activations (TCL) and regression weights (TRL). Experiments on ImageNet show that, applied to VGG and ResNet architectures, TCLs and TRLs reduce the number of parameters compared to fully connected layers by more than 65% while maintaining or increasing accuracy. In addition to the space savings, our approach's ability to leverage topological structure can be crucial for structured data such as MRI. In particular, we demonstrate significant performance improvements over comparable architectures on three tasks associated with the UK Biobank dataset.
1. Introduction
Deep networks often flatten multilinear activation tensors before fully connected layers, losing multimodal structure and requiring many parameters. The paper introduces end-to-end trainable tensor layers that preserve this structure while reducing parameters and improving performance on ImageNet and MRI tasks.
- Multimodal data such as images, videos, and MRI scans naturally form higher-order tensors whose topology can be preserved by tensor methods.
- CNNs flatten high-order activation tensors before fully connected layers, losing multimodal information and requiring many parameters.
- The paper introduces Tensor Contraction Layers and Tensor Regression Layers as end-to-end trainable components that exploit multilinear structure in neural networks.
- Low-rank tensor operations reduce parameters with minimal accuracy impact while replacing fully connected layers and preserving multimodal structure.
- Almost 65% fewer parameters were achieved on ImageNet with VGG and ResNet while improving performance, alongside superior results on three UK Biobank MRI tasks.
- Prior tensor-regression methods typically rely on analytical solutions, large data tensors, downsampling, or compact features and do not scale to ImageNet.
- The paper identifies no prior work combining tensor contraction or tensor regression with deep learning in an end-to-end trainable fashion.
2. Mathematical background
The paper establishes notation and core tensor operations used throughout its models, including n-mode products, generalized inner products, Tucker decompositions, unfoldings, and vectorization.
- Tensors are multidimensional arrays, with vectors as first-order tensors and matrices as second-order tensors.
- Tensor unfolding: Tensor unfolding reshapes a tensor into a matrix by mapping one mode to rows and the remaining modes to columns.
- Tensor vectorization: Tensor vectorization flattens a tensor into a vector using the paper's row-wise element ordering.
- n-mode product: An n-mode product replaces one tensor mode with a new dimension determined by multiplication with a matrix.
- Generalized inner-product: A generalized inner product contracts matching modes between tensors to produce an output tensor.
- Tucker decomposition: Tucker decomposition represents a tensor using a lower-rank core projected along each mode by factor matrices.
3. Tensor Contraction Layer
Tensor Contraction Layers reduce activation tensors to compact cores while preserving multilinear structure, offering a lower-parameter alternative to flattening and fully connected layers. Their factors are learned end-to-end, and the resulting parameter count changes from a product to a sum across modes.
- Tensor contraction layers: TCLs apply differentiable tensor contraction to activation tensors, producing compact cores while preserving multilinear structure.The contraction projects each non-batch mode into a lower-dimensional rank.
- Tensor contraction layers: TCLs require fewer parameters and less computation than comparable fully connected rank reduction while retaining tensor structure.A fully connected layer discards the multilinear information during flattening.
- Tensor contraction layers: A TCL contracts each activation mode using learned factors while leaving the batch mode unchanged.The projection factors are learned with the rest of the network by gradient backpropagation.
- Model analysis: The proposed architecture can contract activations before tensor regression, replacing flattening and fully connected layers with a low-rank tensor mapping.For multiclass outputs, the regression weights extend to a fourth-order tensor.
- Model analysis: For a TCL, the parameter-count product across modes becomes a sum, substantially reducing parameters relative to an equivalent fully connected layer.The TCL uses Σ_k I_k × R_k parameters, whereas the comparable fully connected layer uses a product across modes.
4. Tensor Regression Layer
Tensor Regression Layers replace flattening and fully connected output layers with low-rank multilinear regression directly on activation tensors. They jointly model inputs and outputs while remaining trainable within neural networks.
- Tensor Regression Layer: TRLs map high-order activation tensors to outputs through a low-rank regression weight tensor instead of flattening activations.The regression weights are factorized into a core and mode-specific factors.
- Tensor Regression Layer: The TRL jointly models input and output structure by constraining the regression weights to fixed multilinear rank.The output includes a bias term and can represent batch labels with multiple output dimensions.
- Tensor regression as a layer: The TRL’s regression weights are represented by a core tensor and factor matrices across input and output modes.The factors have dimensions determined by the corresponding input or output mode and its rank.
- Tensor Regression Layer: Tensor regression is incorporated as a trainable neural-network layer, allowing feature learning and regression to occur jointly.This replaces traditional flattening plus fully connected output layers and learns the regression factors with neural features.
- Gradient backpropagation: Gradients for the regression weights and core are computed with respect to each factor and the core using tensor-product formulations.The section derives equivalent unfolded expressions for backpropagation.
5. Efficient implementation of tensor regression layers
An equivalent TRL formulation performs most computation in a low-rank subspace, improving efficiency while preserving the tensor-regression mapping. It can be implemented using learned pseudoinverse factors and is equivalent to applying a TCL followed by a TRL.
- Efficient implementation: The TRL can be rewritten as contractions of the input with transposed factor matrices followed by contraction with a transformed core and output factor.This separates input projection from low-rank regression computation.
- Efficient implementation: Most computation is performed in the low-rank subspace rather than directly over the dimensions of the activation tensor.The reformulation is used directly for implementation.
- Efficient implementation: The implementation learns the pseudoinverse of each input factor and contracts the activation tensor with these factors before regression.The resulting expression uses factors V(0) through V(N) and an output-side transformed core.
- Efficient implementation: This formulation is equivalent to first applying a TCL and then a TRL, with identity factors on the input modes.Under this equivalence, the low-rank constraint applies only to output modes.
- Efficient implementation: Hardware acceleration can reduce transposition overhead by extending BLAS primitives for tensor contractions.The proposed acceleration addresses transpositions required during contraction.
6. Experiments
Experiments evaluate tensor contraction and regression layers on ImageNet, synthetic regression, and UK Biobank MRI. The results show substantial parameter savings, competitive ImageNet accuracy, improved MRI prediction, and low-rank benefits for noisy or small-data settings.
- ImageNet: Up to 65% space savings did not impact ImageNet accuracy, while larger savings caused only small decreases.At about 25% savings, performance marginally improved; nearly 80% savings produced less than 1% decreases in Top-1 and Top-5 accuracy.
- MRI prediction: TRLs significantly outperformed fully connected 3D-ResNet baselines on all three UK Biobank MRI prediction tasks.For age regression, MAE improved from 2.96 to 2.70 years; gender classification reached 99.47% accuracy versus 0.79% baseline error.
- Ablation studies: Using a full-rank 6 × 7 × 6 activation tensor achieved a 2.71-year MAE, similar to lower-rank setups and significantly better than the baseline.This experiment examined whether MRI gains reflected preserved topology or only regularization.
- Ablation studies: On synthetic noisy regression, low-rank TRL weights better recovered ground-truth structure than fully connected regression and acted as an implicit regularizer.The study compares both models under Gaussian inputs and added Gaussian noise.
- Ablation studies: TRLs were easier to train on small datasets and less prone to overfitting than fully connected linear regression.The comparison varied the number of training samples and attributed the difference to low-rank regression weights.
- ImageNet: More than 65% space savings were achieved with almost no performance deterioration when reducing activation size with TCLs.The reported trade-off includes up to 80% space savings with negligible impact on performance in TRL rank studies.
7. Conclusions
The paper preserves multilinear activation structure with end-to-end trainable TCLs and TRLs, using low-rank constraints to reduce parameters while maintaining similar accuracy. These layers also regularize activations and regression weights and can yield more interpretable models requiring less data.
- TCLs reduce input dimensionality without discarding multilinear structure, while TRLs map input tensors to outputs using low-rank regression weights.
- Low-rank regression constraints regularize both activations and regression weight tensors.
- The resulting compact networks achieve similar accuracies with far fewer parameters.
- The regression-weight structure supports more interpretable models while requiring less data to train.