PaperlayerView exact revision on arXiv

Source-linked AI summary

Tensor Methods in Computer Vision and Deep Learning

Yannis Panagakis, Jean Kossaifi, Grigorios G. Chrysos, James Oldfield, Mihalis A. Nicolaou, Anima Anandkumar, Stefanos Zafeiriou

arXiv:2107.03436v1cs.CV

TL;DR

Computer vision needs representations that preserve multidimensional visual structure while supporting efficient and robust learning. This article reviews tensor methods for representation learning and deep learning, emphasizing their applications in vision and providing TensorLy-based notebooks; the reviewed developments include efficient network parametrization and improved robustness to noise and adversarial attacks.

  • Problem

    Existing surveys leave a gap concerning recent tensor-method advances for representation learning and deep learning in computer vision.

  • Method

    The article provides an in-depth practical review of tensor methods for visual representation learning and deep learning, supported by TensorLy-based companion notebooks.

  • Results

    The reviewed tensor methods support memory- and compute-efficient architectures and improved robustness to random noise and adversarial attacks.

  • Takeaways & Limitations

    Tensor methods provide a framework for preserving visual data structure while learning representations and designing deep architectures for computer vision.

Abstract

from arXiv · show

Tensors, or multidimensional arrays, are data structures that can naturally represent visual data of multiple dimensions. Inherently able to efficiently capture structured, latent semantic spaces and high-order interactions, tensors have a long history of applications in a wide span of computer vision problems. With the advent of the deep learning paradigm shift in computer vision, tensors have become even more fundamental. Indeed, essential ingredients in modern deep learning architectures, such as convolutions and attention mechanisms, can readily be considered as tensor mappings. In effect, tensor methods are increasingly finding significant applications in deep learning, including the design of memory and compute efficient network architectures, improving robustness to random noise and adversarial attacks, and aiding the theoretical understanding of deep networks. This article provides an in-depth and practical review of tensors and tensor methods in the context of representation learning and deep learning, with a particular focus on visual data analysis and computer vision applications. Concretely, besides fundamental work in tensor-based visual data analysis methods, we focus on recent developments that have brought on a gradual increase of tensor methods, especially in deep learning architectures, and their implications in computer vision applications. To further enable the newcomer to grasp such concepts quickly, we provide companion Python notebooks, covering key aspects of the paper and implementing them, step-by-step with TensorLy.

I. INTRODUCTION

This introduction presents tensors as natural representations of multidimensional visual data and reviews their role in representation learning and deep learning for computer vision. It also identifies a literature gap in recent tensor-method developments and outlines the paper’s tutorial and computational resources.

  • Tensor foundations: Tensors generalize matrices to multiple modes, providing a mathematical structure for multidimensional data.Their order is the number of indices needed to address elements.
  • Tensor foundations: Visual measurements naturally form tensors because spatial, color, temporal, and sample dimensions correspond to meaningful modes.Batches of grayscale images form third-order tensors, while temporal visual data can require fourth-order tensors.
  • Motivation: Flattening multidimensional visual data into vectors loses topological structure and dependencies among modes while increasing the data required for accurate model estimation.The introduction connects this increase to the curse of dimensionality.
  • Tensor methods for representation learning: Tensor decomposition and component-analysis methods learn structured representations while mitigating dimensionality issues, supporting visual tasks such as factor recovery, denoising, and inpainting.Applications include representations of local object parts, shape and motion, and surface normal and reflectance factors.
  • Tensor methods in deep learning: Deep networks remain highly over-parameterized, motivating tensor methods for architecture design and robustness to adversarial attacks and random noise.The cited discussion links decomposition of neural-network building blocks to robustification.
  • Scope and resources: The paper addresses a gap in surveys of recent tensor methods for representation learning and deep learning in computer vision, providing a broad review and tutorial-style Python notebooks.The notebooks are based on TensorLy and accompany the paper’s sections.

B. Transforming tensors into matrices and vectors

This section introduces tensor unfolding and related tensor operations that transform or combine multidimensional arrays while preserving their mode structure. It also defines products, contractions, and convolution used in tensor methods.

  • Tensor unfolding: Mode-n unfolding maps an Nth-order tensor to a matrix whose rows index mode n and whose columns collect the remaining modes.The column count is defined from the product of the other mode sizes.
  • Tensor unfolding: Tensor unfolding reorders tensor fibers as matrix columns, with definitions differing according to the underlying row-wise or column-wise element ordering.The section uses row-wise C-ordering, while another common convention uses column-wise Fortran ordering.
  • Tensor unfolding: A third-order tensor can be organized through fibers and unfoldings, with slices obtained by fixing one index.The accompanying illustration and table show these representations for a third-order tensor.
  • Tensor products: The Kronecker product expands each entry of one matrix into a scaled copy of the other matrix, while the Khatri–Rao product applies a column-wise Kronecker construction to matrices with matching column counts.Both products are defined as matrix operations used by surveyed tensor methods.
  • Tensor products: Outer products form rank-one tensors, and n-mode products multiply a tensor along one mode, changing that mode’s dimension.The n-mode vector product is the corresponding contraction with a vector.
  • Tensor products: Generalized inner products contract shared modes of tensors, while convolution applies a kernel along a selected mode.In deep-learning practice, the stated convolution operation uses cross-correlation, which differs by flipping the kernel but does not affect end-to-end learned results.

D. Tensor diagrams

Tensor diagrams provide an intuitive graph-based notation for tensor algebra, representing tensor order and contractions through vertices and edges. The section also relates these operations to matrix rank and standard matrix multiplication.

  • Tensor diagrams: Tensor diagrams make tensor operators easier to read and write when contractions involve multiple sums, modes, and indices.They are presented as a pictorial alternative to cumbersome index-based notation.
  • Tensor diagrams: Tensor diagrams represent tensors as vertices and their modes as edges, so a vertex’s degree equals the tensor’s order.Scalars, vectors, and matrices therefore have zero, one, and two incident edges, respectively.
  • Tensor contractions: Contracting two tensors over a common dimension is represented by connecting their corresponding edges.For matrices, this notation expresses ordinary matrix multiplication through a shared index.
  • Matrix rank: Matrix rank is the number of linearly independent columns or, equivalently, linearly independent rows.For an I × J matrix, rank is at most min(I, J), and equality defines full rank.

F. Norms

The paper introduces element-wise and Schatten-p norms, emphasizing how ℓ1 and nuclear norms provide practical convex surrogates for sparsity and rank. It also situates tensor methods within the hardware and software infrastructure needed for efficient computation.

  • F. Norms: The element-wise ℓ0 pseudo-norm counts nonzero tensor elements, while the ℓ1-norm sums absolute values and serves as a convex sparsity surrogate.The Frobenius norm generalizes the p=2 case to higher-order tensors.
  • F. Norms: Schatten-p norms operate on matrix singular values; the Schatten-1 nuclear norm is the tightest convex envelope of rank.The nuclear norm is defined as the sum of singular values.
  • A. Hardware: GPUs made large-scale deep learning feasible by executing tensor operations in parallel across thousands of cores and achieving hundreds of TFLOPS.GPUs prioritize throughput, whereas CPUs generally prioritize low latency.
  • B. Software: Efficient tensor computation depends on coordinated hardware support, memory ordering, optimized libraries, and frameworks that abstract distributed execution.TensorLy provides a high-level API for tensor operations and decompositions across NumPy, PyTorch, and TensorFlow backends.
  • B. Software: Sparse tensors require specialized representations such as COO because storing them densely wastes memory and becomes prohibitive at very large scales.General-purpose dense-array libraries do not directly support efficient sparse-tensor operations.

IV. REPRESENTATION LEARNING WITH TENSOR DECOMPOSITIONS

Tensor representation learning extends matrix-based approaches to multidimensional visual data, preserving structure while extracting low-dimensional representations. The section covers matrix decomposition, tensor decompositions, and their practical trade-offs.

  • A. Matrix decomposition and representation learning: Representation learning extracts useful low-dimensional structure from high-dimensional visual data affected by appearance, geometry, semantics, and noise.Visual data are often not exactly low-rank because of noise, outliers, and nonlinear underlying structure.
  • A. Matrix decomposition and representation learning: Matrix decomposition factorizes a data matrix into two factor matrices, with low-rank models expressing it as a sum of rank-one matrices.The low-rank condition is R < min{I, J}.
  • A. Matrix decomposition and representation learning: Sparse coding learns an input representation as a sparse linear combination of dictionary atoms, while tensor-structured dictionary learning extends this idea to higher-order tensors.The atoms need not be orthogonal and may form an over-complete spanning set.
  • A. Matrix decomposition and representation learning: Low-rank constraints in multivariate regression reduce unknown parameters and can facilitate reliable estimation and model interpretation, with analogous tensor-regression formulations.Tensor decomposition is presented as the higher-order generalization of matrix decomposition.
  • 1) Canonical-Polyadic (CP) decomposition: CP decomposition generalizes matrix rank by expressing a tensor as a sum of R rank-one tensors, although computing tensor rank is NP-hard.The CP model is also called PARAFAC or Kruskal format and may be unique under mild algebraic assumptions.

2) Tucker decomposition:

The Tucker decomposition represents a tensor through a core tensor multiplied by factor matrices along each mode. Its compressed form captures interactions among factor-matrix columns, while tensor-train models instead contract a sequence of third-order cores.

  • 2) Tucker decomposition: Tucker decomposition expresses X as a core tensor multiplied along each mode by a set of factor matrices.The model is non-unique and is written as X = G ×1 U(1) ×2 U(2) ×3 ··· ×N U(N).
  • 2) Tucker decomposition: When Rn ≪ In for every mode, the Tucker core provides a compressed representation of X and records interactions among factor-matrix columns.With orthonormal factor matrices, the model is called HOSVD.
  • 2) Tucker decomposition: Because observed tensors are noisy, Tucker and CP decompositions are typically approximated by minimizing a fitting loss, often using least squares.The associated optimization problem is non-convex when multilinear ranks are fixed.
  • 2) Tucker decomposition: ALS estimates Tucker or CP factors iteratively by fixing all but one factor and solving a linear least-squares update, without a guarantee of convergence to a minimum.ALS is identified as the most common optimization algorithm for these decompositions.
  • 3) Tensor-Train: Tensor-train decomposition represents X as a product of third-order cores with open-boundary conditions R1 = RN+1 = 1.Periodic boundary conditions yield the Tensor-Ring variant.
  • Tensor diagrams: Tensor diagrams represent higher-order CP, Tucker, and tensor-train decompositions more conveniently than direct visualizations limited to third-order tensors.The paper uses diagrams for a fifth-order tensor example.

C. Robust tensor decomposition

Robust tensor decomposition models separate low-rank structure from sparse, potentially large non-Gaussian corruption. The section describes convex relaxations, exact-recovery conditions, and alternatives to least-squares fitting for noisy visual data.

  • C. Robust tensor decomposition: Visual measurements can contain gross errors from pose, illumination, occlusion, acquisition artifacts, and missing data that are not well modeled as Gaussian noise.These corruptions motivate robust tensor methods.
  • C. Robust tensor decomposition: Robust tensor modeling seeks a decomposition into a low-rank tensor L and a sparse noise tensor S.The sparse-noise assumption reflects outliers affecting only a small fraction of measurements.
  • C. Robust tensor decomposition: Because tensor-rank optimization is intractable, some methods approximate rank using a convex combination of the mode-wise ranks of L.The weights αn are nonnegative and sum to one.
  • C. Robust tensor decomposition: Replacing discrete rank and ℓ0 terms with nuclear- and ℓ1-norm surrogates produces a tractable convex optimization problem extending principal components pursuit to tensors.Under certain conditions, the resulting formulation is guaranteed to exactly recover the low-rank component.
  • C. Robust tensor decomposition: Replacing least-squares loss with the ℓ1-norm yields more robust approximations of CP and Tucker decompositions under sparse, non-Gaussian noise.The standard least-squares formulations are sensitive to this noise regime.
  • Tensor component analysis: Tensor component analysis projects tensor samples into low-dimensional multilinear subspaces while optionally imposing reconstruction, scatter, orthogonality, sparsity, or non-negativity criteria.Supervised variants incorporate within-class and between-class affinity relationships.

E. Tensor-structured dictionary learning

Tensor-structured dictionary learning preserves multilinear structure that vectorization loses, while separable and robust formulations address computational, scalability, and outlier-related challenges.

  • Motivation: Flattening image patches into vectors loses multidimensional structure and can make dictionary-learning solvers computationally burdensome.These limitations hinder applicability to massive tensor datasets.
  • Separable dictionary learning: Separable dictionaries represent samples using pairs of bases whose Kronecker product forms the dictionary, with regularizers promoting sparse codes and low mutual coherence.The dictionary is constrained to have orthogonal columns.
  • Theory: Tensor-structured dictionary learning can have significantly lower sample complexity than unstructured vector-based learning.This theoretical result suggests improved performance may be achievable with separable dictionaries.
  • Robust formulation: Robust Kronecker Component Analysis combines separable dictionary learning with low-rank promotion and sparse representations to address scalability and gross-error limitations.It uses a tensor formulation that accounts for sparse noise and outliers.
  • Robust formulation: RKCA expresses a separable dictionary as a regularized tensor decomposition and provides global optimality guarantees.This contrasts with classic dictionary-learning models described in the passage.

F. Tensor regression

Tensor regression preserves activation-tensor structure by constraining regression weights to low-rank forms, while tensor contraction and regression layers provide compact alternatives to flattened fully connected layers.

  • Tensor regression: Tensor regression generalizes linear regression to higher-order observations by expressing outputs as inner products with same-dimension weight tensors.Low-rank assumptions on the weight tensor reduce parameters and can prevent overfitting.
  • Tensor regression: Low-rank tensor regression includes CP, Tucker, higher-order partial least squares, Tensor PCA, and quantile-regression formulations.These approaches impose different low-dimensional structures on regression weights or latent subspaces.
  • Tensor contraction layers: Tensor Contraction Layers project activation tensors along non-batch modes into smaller core tensors using learned factor matrices.The factors are learned end-to-end by gradient backpropagation.
  • Tensor contraction layers: TCLs preserve multilinear structure while using fewer parameters than corresponding unstructured fully connected layers.Their structured weights can be interpreted through an equivalent tensor-regression formulation.
  • Tensor regression layers: Tensor Regression Layers replace flattened-output regression with low-rank regression weights, commonly using Tucker structure and tensor contraction.CP and TT formats can also be used.
  • Tensor regression layers: TRLs are much more parsimonious than fully connected layers and can incorporate implicit regularization from low-rank weights.The formulation supports tensor-valued responses in addition to the scalar-output presentation.

D. Parametrizing convolutional layers

Tensor decomposition factorizes convolutional kernels into sequences of smaller operations, enabling separable convolutional blocks that reduce computational cost and parameter counts.

  • Convolutional factorization: Convolutional kernels are naturally tensors, so convolutional layers can be compressed directly through tensor decomposition.A 2D convolution is represented by a fourth-order tensor.
  • Convolutional factorization: A 1×1 convolution acts as a tensor contraction and is frequently used to create channel bottlenecks.In this setting, the spatial kernel dimensions equal one.
  • Kruskal convolutions: CP decomposition converts a convolutional kernel into a Kruskal form whose factors implement a sequence of efficient depthwise separable convolutions.The factorized sequence replaces the original convolution.

3) Tucker convolutions:

Tucker decomposition reformulates convolutions and networks through low-rank factors, improving parameter efficiency while supporting theoretical analysis, compression, and robustness-oriented tensor methods.

  • Tucker convolutions: Tucker convolution first transforms channel dimensions, applies a smaller convolution, and then restores the target channel dimension.The input and output channel factors parameterize 1×1 convolutions, while spatial factors are absorbed into the core.
  • Architecture design: Tensor decompositions can unify residual architectures and motivate architecture search over tensor-decomposition structures and parameters.A block-term formulation was used to unify architectures such as ResNet while improving parameter efficiency.
  • Theory: CP-based replacement of convolutional layers yields an upper bound on the compressed network’s generalization error related to layer rank.The bound improves previous results on network compressibility and generalizability.
  • Full-network parameterization: T-Net jointly parameterizes all convolutional layers of a stacked-hourglass network with one low-rank eighth-order tensor.Low-rank constraints provide large parameter savings with little to no performance impact, while small compression ratios can improve performance.
  • Robustness: Tensor methods also address robustness because deep networks are vulnerable to capture noise, corruption, adversarial attacks, and domain shift.Tensor dropout introduces training randomization as one reviewed robustness strategy.

1) Tensor Dropout:

Tensor dropout applies Bernoulli randomization to tensor decompositions, introducing stochasticity in CP rank or Tucker latent subspaces. Related tensorized architectures use decompositions to represent higher-order polynomial interactions and multiplicative model structure.

  • Tensor Dropout: Bernoulli tensor dropout keeps or discards CP rank-1 components, thereby inducing stochasticity in the decomposition rank.Each component is controlled by a Bernoulli variable sampled with probability θ of being retained.
  • Tensor Dropout: In Tucker decompositions, Bernoulli dropout acts in the latent subspace by contracting the core along each mode with diagonal sketching matrices.The sketching matrices are M^(k) = diag(λ^(k)).
  • Tensor Dropout: Tensor dropout applies sketching matrices in decomposition latent subspaces and improves robustness to random noise and adversarial attacks.The approach applies to both CP and Tucker decompositions.
  • Defensive Tensorization: Combining tensor dropout with neural-network reparametrization can drastically improve a DCNN’s robustness to adversarial attacks.The combined approach was reported successful against a wide range of attacks when used with adversarial training.
  • Polynomial Networks: Higher-order polynomial networks use tensor decompositions to reduce the exponentially growing parameter count of multivariate polynomial expansions.Low-rank assumptions and factor sharing make the representation more practical for high-dimensional inputs.
  • Polynomial Networks: Recursive polynomial expansions can be implemented as hierarchical neural networks whose depth is determined by the approximation order.The tensor decomposition and factor-sharing choices influence the resulting architecture.
  • Attention Mechanisms: Attention mechanisms and polynomial networks use tensor structures to capture higher-order multiplicative interactions among data or parameters.Multi-head attention can be represented by a third-order tensor and compressed with CP decomposition.

VI. APPLICATIONS IN COMPUTER VISION

Tensor methods support computer-vision applications by extracting and disentangling latent factors, learning multilinear representations, and exploiting tensor structure in downstream tasks. These approaches cover supervised and unsupervised analysis, deep autoencoders, feature extraction, compression, classification, and medical or hyperspectral imaging.

  • Latent Factor Analysis: Computer-vision applications often require disentangling factors such as shape, motion, pose, viewpoint, illumination, and identity.The resulting representations can support view- and pose-invariant classification and structure-from-motion.
  • Latent Factor Analysis: Tensor methods analyze multiple variation factors either by arranging vectorized data into factor-indexed tensors or by imposing tensor structure on deep autoencoder latent spaces.The latter approach represents latent factors through multilinear interactions.
  • TensorFaces: TensorFaces models visual data with factor matrices for people, views, illumination, expressions, and pixels, combined through a multilinear mixing tensor.These factors support invariant-feature extraction and related analyses of motion, gait, pose, tracking, and structure from motion.
  • Unsupervised Factor Analysis: Unsupervised tensor methods can recover multiple latent variation factors from vectorized images without labels when the number of variation sources is known.The recovered factors can be used for face editing, shape-from-shading, surface-normal estimation, and viewpoint- or illumination-invariant recognition.
  • Deep Latent Spaces: Deep autoencoder models impose multilinear structure on latent variables and decode their multiplicative interactions to reconstruct images.Manipulating the latent factors enables image editing such as face editing.
  • Feature Extraction and Applications: Tensor component analysis and decompositions provide features for downstream classification and analysis of faces, bodies, hyperspectral images, tumors, lesions, and brain activity.Applications include dimensionality reduction, feature extraction, image compression, hyperspectral classification, and coupled brain-region discovery.

C. Tensor methods in inverse problems

Tensor methods address inverse problems and large-scale visual prediction by using low-rank or robust tensor representations, while tensorized deep networks offer compactness and robustness advantages. Their adoption remains constrained by rank selection, decomposition choice, and numerical stability during training.

  • Inverse Problems: Low-rank or robust tensor decompositions provide bases for denoising images and videos and imputing missing visual data.These models represent visual observations as tensors and estimate a low-rank tensor for restoration or completion.
  • Inverse Problems: Multilinear component-analysis extensions have been applied to discover common and individual features across data views for MRI denoising and completion.The methods emphasize shared structure across multiple views.
  • Tensor Regression: Low-rank tensor regression supports classification and prediction for face, body, hyperspectral, medical, and video data while reducing parameter counts relative to vectorized models.Applications include face classification, pose estimation, pain estimation, crowd counting, land-cover classification, and fMRI analysis.
  • Deep Learning Applications: Randomized tensor regression layers have significantly improved age-prediction results from MRI scans compared with state-of-the-art algorithms.Tensor regression layers are also used for affect estimation from faces and land-cover classification.
  • Tensorized Deep Networks: Low-rank tensor factorization can improve robustness to noise, regularize models, accelerate computation, reduce parameters, and support domain adaptation or information sharing.Tucker factorization additionally yields a latent subspace represented by its core.
  • Challenges: Tensor methods often require rank estimation even though determining tensor rank is generally NP-hard, so practitioners typically rely on heuristics.Rank selection remains a significant challenge despite end-to-end training reducing the importance of the exact rank.
  • Challenges: There is no principled general method for choosing the most appropriate tensor decomposition or jointly optimizing decomposition and rank for a compressed network.The choice depends on tensor structure and is intertwined with efficient architecture design.
  • Challenges: Training tensorized deep networks is challenging because repeated tensor contractions can cause exploding or vanishing gradients, especially at reduced precision.Stable learning requires principled initialization and normalization methods that account for tensor structure.
Loading 2107.03436v1…