PaperlayerView exact revision on arXiv

Source-linked AI summary

Tensor Networks for Dimensionality Reduction and Large-Scale Optimizations. Part 2 Applications and Future Perspectives

A. Cichocki, A-H. Phan, Q. Zhao, N. Lee, I. V. Oseledets, M. Sugiyama, D. Mandic

arXiv:1708.09165v1math.NAcs.LG

TL;DR

The paper addresses the scalability limits of tensor-based models for high-dimensional data and optimization. It reviews low-rank tensor networks and contractions for large-scale applications, showing that they reduce problems to manageable lower-scale subproblems while supporting reported gains in several signal-processing tasks.

Abstract

from arXiv · show

Part 2 of this monograph builds on the introduction to tensor networks and their operations presented in Part 1. It focuses on tensor network models for super-compressed higher-order representation of data/parameters and related cost functions, while providing an outline of their applications in machine learning and data analytics. A particular emphasis is on the tensor train (TT) and Hierarchical Tucker (HT) decompositions, and their physically meaningful interpretations which reflect the scalability of the tensor network approach. Through a graphical approach, we also elucidate how, by virtue of the underlying low-rank tensor approximations and sophisticated contractions of core tensors, tensor networks have the ability to perform distributed computations on otherwise prohibitively large volumes of data/parameters, thereby alleviating or even eliminating the curse of dimensionality. The usefulness of this concept is illustrated over a number of applied areas, including generalized regression and classification (support tensor machines, canonical correlation analysis, higher order partial least squares), generalized eigenvalue decomposition, Riemannian optimization, and in the optimization of deep neural networks. Part 1 and Part 2 of this work can be used either as stand-alone separate texts, or indeed as a conjoint comprehensive review of the exciting field of low-rank tensor networks and tensor decompositions.

Tensorization and Structured Tensors

This section develops tensorization strategies that convert data and parameter sets into structured higher-order tensors for low-rank analysis and computation. It covers representations and applications spanning convolution, structured operators, signal processing, source separation, image and similarity analysis, regression, system identification, feature extraction, and sinusoid analysis.

Tensorization and Structured Tensors

  • Tensorization constructs higher-order tensors by rearranging, aligning, augmenting, transforming, or parameterizing data, and its form can affect subsequent decomposition performance.Examples include RGB image stacks, multichannel EEG tensors, time-frequency spectrograms, multiscale transforms, and low-rank network representations for large parameters.Source: Tensorization and Structured Tensors · pp. 4–5
  • Tensorization is task- and structure-dependent, with super-compression—especially in QTT formats—motivating applications in source separation, system identification, regression, and feature extraction.Low-rank tensor-network cores also enable operations such as multiplication, convolution, inner products, and Fourier transforms on huge parameter sets.Source: Tensorization and Structured Tensors · p. 5

1.1 Reshaping or Folding

  • Reshaping or folding preserves entries and sequential order without extra memory, while higher-order folding can expose low-rank structure for tensor-network-based blind source separation.Quantizing exponential sequences can yield rank-1 tensors, and broad trigonometric, polynomial, and rational function classes can produce approximate low-rank TT formats [Khoromskij, 2011a,b, Oseledets, 2012].Source: §1.1 · pp. 5–7

1.2.1 Toeplitz Folding

  • Toeplitz tensors generalize Toeplitz matrices into higher-order structured operators whose tensor-vector products represent convolutions of multiple vectors.They are generated from a vector, can be constructed recursively, and provide multilinear representations of convolution.Source: §1.2.1 · pp. 9–10

1.2.2 Hankel Folding

  • Hankel tensors similarly encode convolution and retain structured properties: fibers reconstruct the generating vector, two-index slices are Hankel matrices, and equal dimensions produce symmetry.Convolution is represented using reversed input vectors and, when needed, zero-padded Hankel tensors.Source: §1.2.2 · pp. 10–11

1.2.3 Quantized Tensorization

  • Toeplitz and Hankel quantized tensorizations enlarge sample counts—for dimensions I_n = 2, entries grow from L to 2^(L-1)—so they suit short signals, particularly multivariate autoregressive modeling.The enlargement is a material limitation despite the structured representation.Source: §1.2.3 · p. 12

1.2.4 Convolution Tensor

  • Convolution tensors encode convolution through multilinear products, with their structured slices producing Toeplitz or circulant matrices from vector contractions.The construction generalizes from two-vector convolution to N-way convolution using an (N + 1)th-order tensor.Source: §1.2.4 · pp. 12–13

1.2.5 QTT Representation of the Convolution Tensor

  • A convolution tensor for an Nth-order Toeplitz tensor has a QTT representation with rank no larger than the number of input vectors, enabling distributed Toeplitz generation from QTT-formatted vectors.For the third-order case, the convolution tensor has a rank-2 QTT representation, and Figure 1.3 illustrates Toeplitz-matrix generation in QTT format.Source: §1.2.5 · pp. 14–16, Figure 1.3 · p. 16

1.2.6 Low-rank Representation of Hankel and Toeplitz Matrices/Tensors

  • Hankel and Toeplitz foldings convert structured data into tensors whose low-rank representations are preserved under element-wise products, with CP/TT rank bounded by the product of the input ranks.For example, rank-3 and rank-2 Hankel tensors for sin(ωt) and t yield a Hankel tensor for t sin(ωt) with at most rank-6.Source: §1.2.6 · pp. 17–18
  • Low-rank tensorizations of Hankel and Toeplitz structures support rank-controlled representations, fast convolution, and applications including blind source separation and harmonic retrieval.The Toeplitz QTT rank is at most N times the generating vector’s QTT rank, while QTT-formatted convolutions can be computed through core contractions.Source: §1.2.6 · pp. 19–20

1.3 Tensorization by Means of L¨owner Matrix (L¨owner Folding)

  • L¨owner folding supports rational approximation f(x) = A(x)/B(x), allowing discontinuities and spiky data while making tensor-model complexity and accuracy governed by the L¨owner-matrix rank.The resulting model is amenable to higher-order tensor representation and has minimal computational complexity relative to the selected rational approximation order.Source: §1.3 · p. 20

1.4 Tensorization based on Cumulant and Derivatives of the Generalised Characteristic Functions

  • Cumulants or partial derivatives of generalized characteristic functions tensorize noisy observations for blind source separation, whose CP decomposition retrieves the mixing matrix H.Using derivative tensors at distinct processing points and estimating H as a joint factor matrix improves identification accuracy; the approach assumes statistically independent, non-Gaussian sources with known number and non-collinear mixing columns [Comon and Rajih, 2006, Yeredor, 2000].Source: §1.4 · pp. 21, 23, Figure 1.4 · p. 22
  • Prior TT decomposition followed by CP decomposition produced more stable estimates and approximately 2 dB higher mean SAE than CP alone for orders 7 and 8 with many sources.The two-stage method applies TT decomposition to high-order derivative tensors before CP decomposition in TT-format to retrieve the mixing matrix.Source: §1.4 · p. 24
  • Higher-order derivative tensors with CP decomposition estimated the mixing matrix well, with 7th- and 8th-order derivatives outperforming lower orders, although accuracy declined as sources increased.The results used three derivative tensors and averaged mean and best MSAEs over 100 runs; the accompanying figure reports mean SAE and cumulant-based MSAEs.Source: §1.4 · p. 24, Figure 1.5 · p. 25

1.4.1 Tensor Structures in Constant Modulus Signal Separation

  • For 32-QAM and 64-QAM separation at 15 dB SNR, coupled tensor decompositions perfectly reconstructed both estimated signals with zero bit error rates.The method constructs tensors from polynomial roots of squared constant moduli and forms coupled CP decompositions to estimate demixing and mixing-related matrices.Source: §1.4.1 · pp. 25, 28, Figure 1.6 · p. 29

1.5 Tensorization by Learning Local Structures

  • Local-patch tensorization learns hidden relations among neighboring image blocks, decomposes each block, and reconstructs pixels by averaging overlapping approximations.For RGB images, each constructed tensor contains a central h × w × 3 patch and (2d + 1)^2 nearby blocks; the approach was demonstrated on noisy peppers images.Source: §1.5 · pp. 28–30

1.6 Tensorization based on Divergences, Similarities or Information Exchange

  • Similarity-based tensorization forms an Nth-order nonnegative symmetric tensor whose entries encode N-way divergences or similarities, supporting interaction analysis, clustering, and classification.A further mode can represent changes across time or trials.Source: §1.6 · pp. 30–31

1.7 Tensor Structures in Multivariate Polynomial Regression

  • Multivariate polynomial regression represents nonlinear variable interactions as tensor–Vandermonde products, while TT and QTT formats reduce the exponential parameter burden for high-dimensional models.The raw weight tensor has exponentially many parameters in the number of variables; QTT representations can retain the full model or use a truncated model with fewer core tensors.Source: §1.7 · pp. 32–33

1.8 Tensor Structures in Vector-variate Regression

  • The multivariate polynomial regression formulation for scalar observations can be extended straightforwardly when observations are vectors or tensors.Source: §1.8 · p. 33
  • In generalized tensor-variate regression, the weight tensor’s curse of dimensionality can be addressed by representing it in canonical form, TT/MPS, or hierarchical Tucker format [Nguyen et al., 2015, Qi et al., 2016, Stoudenmire and Schwab, 2016, Cohen and Shashua, 2016].Higher-order inputs can be handled by replacing original vectors with vectorizations, since tensor inner products can be expressed through the vectorizations of both tensors.Source: §1.8 · pp. 35–36

1.9.1 Discrete Volterra Model

  • Raw discrete Volterra kernels require (M+n-1)!/[n!(M-1)!] coefficients at order n, making estimation feasible only for relatively small memory and low-dimensional inputs.Although the kernels are symmetric, higher-order terms still require many measurements.Source: §1.9.1 · p. 38

1.9.2 Separable Representation of Volterra Kernel

  • Separable Volterra kernels convert nonlinear system identification into estimating factor matrices, reducing the parameter count to M Σ_n R_n + 1, with R_1 = 1.The model supports CP kernels and alternative tensor-network formats, including TT representations evaluated through fast mode contractions.Source: §1.9.2 · pp. 38–39

1.9.3 Volterra-based Tensorization for Nonlinear Feature Extraction

  • Volterra tensorization turns nonlinear feature extraction into standard linear discriminant analysis, while CP or TT structures control exponentially growing eigenvector lengths.The transformed problem uses between- and within-scattering matrices and generalized principal eigenvectors.Source: §1.9.3 · p. 40

1.10 Low-rank Tensor Representations of Sinusoid Signals and their Applications to BSS and Harmonic Retrieval

  • Low-rank tensor structures of sinusoidal and modulated signals support blind separation and harmonic retrieval under multiple tensorization schemes.The section considers sinusoidal, polynomially weighted, exponentially damped, and polynomially weighted damped signals.Source: §1.10 · pp. 40–41

1.10.1 Folding - Reshaping of Sinusoid

  • A folded sinusoid has matrix rank 2, three-way multilinear rank (2,2,2) or CP rank 3, and N-way multilinear and TT rank (2,2,...,2).The three-way rank-3 decomposition is not unique, whereas the TT representation preserves rank-(2,2,...,2).Source: §1.10.1 · pp. 41–42
  • A signal shorter than L = 576 samples suffers about 12 dB worse estimation quality, while doubling signal length improves SAE by 2 dB on average.The section suggests augmenting short signals with multiway Toeplitz or Hankel tensors before source extraction.Source: §1.10.1 · p. 44

1.10.2 Toeplitz Matrix and Toeplitz Tensors of Sinusoidal Signals

  • Toeplitz tensorization represents each damped sinusoid with TT rank-(2, 2, . . . , 2), and TT-SEPA separates length-66 mixtures much better than the earlier example.The TT-decomposition results were slightly better than the Kumaresan-Tufts method, while TT-KT and TT-linear prediction can further exploit low-rank structure.Source: §1.10.2 · p. 47

1.10.3 Hankel Matrix and Hankel Tensor of Sinusoidal Signal

  • Hankel tensorization yields a TT-Tucker representation with factor matrices similar to Toeplitz tensors but different core tensors, including quantized forms with repeated cores.These representations are summarized alongside other quantized tensor formats in Figure 1.14.Source: §1.10.3 · pp. 47–48, Figure 1.14 · p. 48

1.11 Summary

  • Suitable tensorization is determined by whether the generated tensor reveals low-rank structure, enabling TT/QTT construction when the generating vector has a low-rank QTT representation.Toeplitz and Hankel foldings replicate signal segments to augment entries, making them suitable for short signals; tensorization also represents large model parameters in low-rank formats.Source: §1.11 · p. 49

Supervised Learning with Tensors

This section develops tensor-valued supervised learning for regression and classification, progressing from tensor regression, PLS, and SVM formulations to low-rank, kernelized, and discriminant-analysis methods. It emphasizes preserving structured inputs through multilinear projections, constrained tensor decompositions, tensor-specific kernels, and regularization.

Supervised Learning with Tensors

  • Tensor-based supervised learning preserves object structure, reducing unknown parameters and addressing information loss and overfitting when samples are limited or data are high-dimensional.Supervised learning models p(x, y) or p(y|x), with regression for continuous responses and classification for categorical responses.Source: Supervised Learning with Tensors · pp. 50–51

2.1 Tensor Regression

  • Tensor regression predicts continuous or categorical responses from vector-, matrix-, or tensor-valued inputs using a tensor weight model trained on input-output samples.The framework includes squared-loss and logistic-loss formulations, and general contractions support multi-response, vector autoregressive, and pair-wise interaction models.Source: §2.1 · pp. 52, 53, 55
  • CP, Tucker, and TT/HT representations express large regression weight tensors compactly, while Tucker regression offers greater parsimony, mode-specific ranks, and explicit inter-mode interactions than CP regression.CP and Tucker models can be optimized with alternating least squares; HT representations yield H-Tucker tensor regression for very high-order tensors.Source: §2.1 · pp. 53–54

2.2 Regularized Tensor Models

  • Regularized tensor models constrain weights through loss penalties, sparsity profiles, low rank, or tensor norms to reduce complexity in high-feature, small-sample regression and classification.Tensor sparsity can target whole tensors, slices, or fibers, while low-rank constraints produce factorized weight tensors and TT-rank-constrained models.Source: §2.2 · pp. 55–57
  • Scaled latent trace norms estimate rank more reliably across unequal mode sizes, but latent-norm methods require many variables and SVD-based approaches are infeasible for very large high-order tensors.Tensor-network decompositions are therefore needed to perform operations using lower-order core tensors in very large-scale applications.Source: §2.2 · p. 58

2.3 Higher-Order Low-Rank Regression (HOLRR)

  • HOLRR fits tensor-valued responses by minimizing ridge-regularized loss subject to a prescribed multilinear rank for the regression tensor W.The Tucker-form regression tensor uses orthogonal factor matrices, and its core and factors can be obtained through eigenvector-based subspace computations.Source: §2.3 · pp. 59–60

2.4 Kernelized HOLRR

  • KHOLRR extends HOLRR to feature spaces by replacing the input representation with a kernel matrix while retaining an orthogonal Tucker-form weight tensor.Its core uses K(K + γI)-based ridge regularization, and the required subspaces are computed from eigenvectors associated with the kernel matrix K = ΦΦT.Source: §2.4 · pp. 60–61

2.5.1 Standard Partial Least Squares

  • Standard PLS jointly decomposes X and Y into latent variables and loadings whose components maximize covariance, enabling prediction of new responses from X.The model assumes the extracted latent variables predict Y and have a linear inner relation with Gaussian i.i.d. residuals.Source: §2.5.1 · pp. 61–63

2.5.2 The N-way PLS Method

  • N-way PLS generalizes PLS by applying CP decomposition to tensor predictors and matrix decomposition to responses, sharing latent variables across both.For a third-order predictor, each component includes one latent vector and one loading vector per tensor mode, with normalized loading and response weights.Source: §2.5.2 · p. 63, Figure 2.3 · p. 64

2.5.3 HOPLS using Constrained Tucker Model

  • HOPLS jointly decomposes predictor and response tensors into constrained Tucker blocks sharing sample-mode latent vectors, with block ranks controlling model fit and complexity.HOPLS can model data better than N-way PLS through Tucker blocks and reduces to N-way PLS when all block ranks are one; HOOI obtains the loading subspaces.Source: §2.5.3 · pp. 64, 65, 67, Figure 2.4 · p. 66

2.6 Kernel HOPLS

  • Kernel HOPLS extends HOPLS to possibly infinite-dimensional nonlinear feature spaces by decomposing mapped predictor and response tensors with shared latent representations.The kernel solution finds latent vectors T and U sequentially to maximize pair-wise covariance, while mapped tensor components are represented through training observations.Source: §2.6 · pp. 67–68
  • KHOPLS computes tensor kernel matrices, extracts latent scores by solving KXKYtr = λtr, and predicts new outputs through kernelized latent-space regression.For strict linearity between latent vectors and outputs in the original spaces, the output kernel is generally restricted to a linear kernel.Source: §2.6 · pp. 68–69

2.7 Kernel Functions in Tensor Learning

  • Tensor kernels preserve multilinear structure through factor, subspace-based, and probabilistic constructions, with probabilistic product kernels handling missing values, variable lengths, and multiway relations.Chordal-distance tensor kernels provide rotation and reflection invariance on Grassmann manifolds, while probabilistic kernels use information divergence between mode-wise generative models.Source: §2.7 · pp. 70–71

2.8 Tensor Variate Gaussian Processes (TVGP)

  • Tensor-GP models place Gaussian-process priors on functions over tensor inputs, enabling predictive distributions for regression and multinomial-probit classification.For classification, tensor-variate kernels are required because commonly used Gaussian RBF kernels are infeasible; non-Gaussian likelihoods require approximate inference.Source: §2.8 · pp. 74–76
  • The TVGP framework also supports additive multiplicative nonparametric regression by decomposing both the function and input space through rank-one tensor components and local Gaussian processes.Applying a CP decomposition yields local functions with Gaussian-process priors whose posterior and predictive distributions are obtained by Gaussian-process regression.Source: §2.8 · pp. 76–77

2.9 Support Tensor Machines

  • The support tensor machine section introduces tensor generalizations of SVM, beginning with the classical soft-margin SVM that maximizes the margin between two classes under slack-variable constraints.The standard formulation learns a projection vector and bias by minimizing a cost function subject to ym(w^Txm + b) ≥ 1 − ξm and nonnegative slack variables.Source: §2.9 · p. 77, §2.9.1 · p. 77

2.9.1 Support Vector Machines

  • Soft-margin SVMs maximize class separation while permitting marginal errors, and least-squares SVMs replace inequality constraints and absolute slack loss with equality constraints and squared loss.These changes make least-squares SVM solutions more convenient to obtain; when separable, the slack variables can be set to zero.Source: §2.9.1 · pp. 77–78, Figure 2.7 · p. 78

2.9.2 Support Matrix Machines (SMM)

  • Support matrix machines incorporate row or column correlations through nuclear-norm regularization, whereas direct vectorization is equivalent to standard SVM and discards matrix structure.Setting τ = 0 recovers the standard SVM, and the solution uses singular-value thresholding within an augmented-Lagrangian formulation.Source: §2.9.2 · pp. 78–79

2.9.3 Support Tensor Machines (STM)

  • Support tensor machines extend SVMs to Nth-order tensor samples for classification or regression using multilinear mode projections and slack variables for nonseparable data.The least-squares STM uses alternating least squares, while test labels are predicted by the sign of the learned multilinear decision function; vector inputs recover standard SVMs.Source: §2.9.3 · pp. 80–81

2.10 Higher Rank Support Tensor Machines (HRSTM)

  • HRSTMs represent separating parameters as sums of rank-one tensors, retaining tensor structure and improving classifier discriminative ability through multiple mode projections.Optimization proceeds iteratively by estimating one mode’s parameters at a time while fixing the others, reducing each subproblem to a standard vector SVM.Source: §2.10 · pp. 83–85

2.11 Kernel Support Tensor Machines

  • Kernel support tensor regression extends tensor machines to nonlinear regression by mapping tensor rows into high-dimensional feature spaces and constructing a matrix-valued kernel.Unlike SVR, the resulting kernel outputs a matrix rather than a scalar; an RBF kernel provides one example.Source: §2.11 · pp. 85–86
  • KSTR offers stronger learning and generalization than STR and handles nonlinearly separable problems more efficiently, but incurs substantially higher computational load.Its ε-insensitive regression formulation uses quadratic programming, slack variables, Lagrange multipliers, and iterative updates of v, u, and b.Source: §2.11 · pp. 86–87

2.12 Tensor Fisher Discriminant Analysis (FDA)

  • TFDA extends Fisher discriminant analysis to tensors with a multilinear decision function, but requires alternating projection algorithms because it has no closed-form solution.It separates positive and negative tensor measurements through projection vectors and a bias obtained iteratively.Source: §2.12 · p. 88
  • Regularized MFDA decomposes the weight vector into mode-wise factors and adds λw^TRw to improve numerical stability when the sample size is small relative to tensor dimensionality.The resulting optimization can be solved alternately by fixing all mode factors except one.Source: §2.12 · p. 88
  • HODA and MDA extend LDA to tensor samples by learning mode-wise projection matrices that maximize between-class separation relative to within-class scatter in projected tensor subspaces.The mode-n between-class scatter characterizes class separation using mode-n tensor unfoldings, while the objective preserves the given class structure across modes.Source: §2.12 · pp. 89–90

Tensor Train Networks for Selected Huge-Scale Optimization Problems · 3.1 Tensor Train (TT/MPS) Splitting and Extraction of Cores · P RI1I2¨¨¨INˆRn´1InIn+1Rn+1 (3.9)

The chapter uses low-rank TT networks to make huge-scale dimensionality-reduction and optimization problems tractable by converting them into smaller optimization sub-problems. It develops core-splitting and extraction procedures for ALS and MALS, supported by orthogonalized interface and frame-matrix formulations.

Tensor Train Networks for Selected Huge-Scale Optimization Problems

  • Low-rank TT networks convert optimization problems involving structured objects with over a billion entries into manageable lower-scale sub-problems solved with standard methods.The approach targets EVD/PCA, GEVD, SVD, linear systems, pseudo-inverses, LASSO, orthogonal CCA, and HOPLS through constrained optimization, contractions, and tensor-network transformations.Source: Tensor Train Networks for Selected… · pp. 91–92

3.1.1 Extraction of a Single Core and a Single Slice for ALS Algorithms

  • TT optimization begins by dividing a tensor train into left and right sub-trains so ALS can extract a selected core or its slices as smaller sub-tasks.The construction uses interface matrices and assumes left orthogonalization of cores before the target core and right orthogonalization afterward.Source: §3.1.1 · pp. 92–93, Figure 3.1 · p. 94
  • Orthogonalizing TT cores reduces contraction complexity and improves algorithmic robustness, while orthogonalized sub-trains yield orthogonal interface matrices.For an n-orthogonal TT network, the interface matrices satisfy the corresponding orthogonality relation.Source: §3.1.1 · p. 93
  • The frame equation expresses vec(X) linearly in the vectorized target core, enabling ALS updates through a tall-and-skinny frame matrix.This linearity with respect to each core is the key algebraic property underlying the single-core extraction formulation.Source: §3.1.1 · p. 93

3.1.2 Extraction of Two Neighboring Cores for Modified ALS (MALS)

  • Modified ALS, also called two-site DMRG2, extracts two neighboring TT cores and represents their joint update with an analogous linear frame equation.The extracted block is treated as a super-core in the two-site formulation.Source: §3.1.2 · pp. 94–95
  • Splitting a TT network into three sub-networks supports extraction of an arbitrary group of cores and can be implemented for any split positions n and p with p > n.The reshaped tensor is represented as a third-order tensor and decomposed using a Tucker-2/PVD model.Source: Figure 3.2 · p. 95

P RI1I2¨¨¨INˆRn´1InIn+1Rn+1 (3.9)

  • For the two-core block, recursive formulas combine left and right sub-trains into a super-core of size Rn-1 × InIn+1 × Rn+1.The construction assumes orthonormal columns in the left sub-train and orthonormal rows in the right sub-train, with corresponding frame-matrix orthogonality.Source: P RI1I2¨¨¨INˆRn´1InIn+1Rn+1 (3.9) · pp. 96, 98
  • The recursive TT/MPS formulas also express the split network in matrix form by reshaping the data tensor and indexing frontal slices associated with the neighboring-core block.Table 3.1 summarizes the basic recursive formulas, while Figure 3.3 illustrates the corresponding two-core frame equation.Source: Table 3.1 · p. 97, P RI1I2¨¨¨INˆRn´1InIn+1Rn+1 (3.9) · pp. 97–98

3.2 Alternating Least Squares (ALS) and Modified ALS (MALS)

ALS makes TT optimization feasible by minimizing over one core at a time in alternating forward and backward sweeps, while MALS improves flexibility by jointly optimizing neighboring cores and repartitioning them. Orthogonalization supports better-conditioned local problems, although ALS may converge slowly and lacks global-minimum guarantees.

3.2 Alternating Least Squares (ALS) and Modified ALS (MALS)

  • ALS replaces the complex simultaneous nonlinear minimization over all TT cores with a sequence of single-core optimizations.The TT variable is represented by its cores, but simultaneous minimization is usually too complex and nonlinear.Source: §3.2 · pp. 98–99
  • ALS updates one TT core while fixing the others, cycling through forward and backward half-sweeps until a stopping criterion is satisfied.Each local update minimizes the cost with respect to the current core, and a full forward-backward cycle forms one global iteration.Source: §3.2 · p. 99
  • MALS merges two neighboring TT cores, optimizes the resulting super-core, and splits it again using truncated SVD to adapt the TT rank.This addresses the unknown solution rank and can alleviate slow convergence from unfavorable initial ranks.Source: §3.2 · pp. 99, 101
  • ALS decreases the cost function but does not guarantee a global minimum, and its convergence can be slow when the initial TT rank is unsuitable.The unknown desired TT rank is a central limitation of standard ALS.Source: §3.2 · p. 99
  • For quadratic global costs, ALS produces small local quadratic problems whose conditioning is usually improved by orthogonalizing the TT cores.The monograph generally assumes left- or right-orthogonalized cores relative to the core being optimized.Source: §3.2 · p. 101

3.3 Tensor Completion for Large-Scale Structured Data

Tensor completion reconstructs large, high-dimensional structured arrays with missing or noisy entries by fitting a low-rank tensor-train representation to observed entries. The resulting problem is solved through ALS-based core updates, least-squares subproblems, and adaptive TT-rank estimation.

Tensor completion objective

  • Tensor completion reconstructs high-dimensional structured arrays when many entries are missing or noisy [Grasedyck et al., 2015, Steinlechner, 2016a,b, Yokota et al., 2016, Zhao et al., 2015, 2016].The method assumes a suitable low-rank tensor-network approximation exists for the incomplete data tensor.Source: §3.3 · p. 101
  • The reconstruction is constrained to a tensor-train manifold with prescribed TT ranks while matching the known entries through projection onto the sampling set.The projection PΩ retains observed entries of Y and zeros the unobserved entries.Source: §3.3 · p. 101

ALS optimization

  • ALS decomposes the large-scale reconstruction into smaller core-slice optimization problems, iterating over all cores across multiple sweeps and orthogonalizing cores after each sweep.Each core update minimizes the vectorized cost function sequentially.Source: §3.3 · pp. 102–103
  • Each core update is a standard linear least-squares problem, enabling efficient optimization even for huge-scale datasets.The least-squares formulation operates on vectorized core slices and the surrounding TT factors.Source: §3.3 · p. 103

Rank adaptation

  • Adaptive TT-rank estimation starts from either a maximum rank and rounds downward or a minimum rank and increases ranks until the residual tolerance ε or Rmax is reached.The rank-increasing strategy begins with rTT = t1, 1, . . . , 1u and is often more efficient because TT rank controls ALS complexity [Grasedyck et al., 2015, Steinlechner, 2016a,b].Source: §3.3 · p. 103

3.4 Computing a Few Extremal Eigenvalues and Eigenvectors

This section develops TT-network methods for computing a few extremal eigenpairs of very large structured symmetric matrices through smaller contracted subproblems and adaptive-rank techniques. It also presents EVAMEn, which uses residual- or gradient-based enrichment in block TT format to address ALS/MALS convergence limitations while retaining comparable computational and storage complexity.

3.4.1 TT Network for Computing the Single Smallest Eigenvalue and the Corresponding Eigenvector

  • Low-rank TT approximations can address eigenvalue problems for matrices with I = 10^15 or higher by converting a large-scale EVD problem into smaller EVD subproblems when TT ranks are relatively small.Source: §3.4.1 · pp. 104, 106

3.4.2 TT Network for Computing Several Extremal Eigenvalues and Corresponding Eigenvectors for Symmetric EVD

  • For several extremal eigenpairs, block TT jointly represents K mutually orthogonal eigenvectors, while recursive contractions and alternating sweeps reduce the trace minimization to linked small problems.The global minimum equals the sum of the K smallest eigenvalues, and local orthogonality constraints enable a dramatic computational-complexity reduction.Source: §3.4.2 · pp. 108, 109, 112

3.4.3 Modified ALS (MALS/DMRG2) for Symmetric EVD

  • MALS/DMRG2 optimizes two neighboring TT cores, splits them with a truncated SVD, and adaptively determines TT ranks while quantifying low-rank approximation error.Merging and splitting neighboring cores produces larger local subproblems than ALS but enables dynamic rank and error control during iterations.Source: §3.4.3 · pp. 112–113

3.4.4 Alternating Minimal Energy Method for EVD (EVAMEn)

  • EVAMEn is motivated by ALS getting trapped in local minima and converging relatively slowly, while MALS improves convergence without guaranteeing a global minimum.The supplied passage frames these limitations as the rationale for the subsequent alternating minimal energy method.Source: §3.4.4 · p. 114
  • The enrichment supplies gradient or residual information to avoid non-global minima and can achieve global convergence while maintaining computational and storage complexities comparable to standard ALS [Dolgov and Savostyanov, 2014].Residuals may use a preconditioned form, and even rough residual or gradient approximations can accelerate convergence.Source: §3.4.4 · pp. 114, 116, 118
  • At each micro-iteration, EVAMEn updates one TT core and concatenates it with a residual-derived core, expanding the local subspace while neighboring cores receive compatible zero padding.This core enrichment is also described as basis expansion or local subspace correction [Dolgov and Savostyanov, 2014].Source: §3.4.4 · p. 114, Figure 3.11 · p. 115
  • EVAMEn targets a single or few extreme eigenvalues and eigenvectors of a very large symmetric matrix using the block Rayleigh quotient in block TT format.The objective is J(X) = tr(X^TAX), with X = [x_1, ..., x_K].Source: §3.4.4 · pp. 114, 116
  • For efficiency, EVAMEn can compute enrichment cores independently through sequential contractions, and Hubig et al. [2015] show that AX suffices instead of the exact residual because both preserve the relevant column space.The method avoids explicitly forming large two-core objects such as A^(n,n+1) or X^(n,n+1).Source: §3.4.4 · p. 118

3.5 TT Networks for Tracking a Few Extreme Singular Values and Singular Vectors in SVD

Block TT networks compute a few largest or smallest singular values and corresponding left and right singular vectors by optimizing trace-based objectives under orthogonality constraints [Cichocki, 2013, 2014, Lee and Cichocki, 2015]. TT core contractions reduce the huge-scale problem to small matrix SVD subproblems, while alternative networks are needed when many extreme singular values are sought.

TT Networks for Tracking a Few Extreme Singular Values and Singular Vectors in SVD

  • Block TT networks target K largest singular values and corresponding singular vectors by maximizing tr(U^TAV) under U^TU = I_K and V^TV = I_K [Cichocki, 2013, 2014, Lee and Cichocki, 2015].The factors satisfy U ∈ R^{I×K} and V ∈ R^{J×K}.Source: §3.5 · p. 119
  • The K smallest singular-value problem uses the Moore–Penrose pseudoinverse A^†, and tensorizing the matrices yields an asymmetric network for computing either minimal or maximal singular values.Figure 3.12 depicts the corresponding left- and right-singular-vector network.Source: §3.5 · p. 119, Figure 3.12 · p. 120
  • TT core contractions reduce the infeasible optimization to per-core trace maximizations with orthogonality constraints, ultimately requiring SVD of a relatively small contracted matrix.The reduced objectives are tr((U^(n))^T A^(n) V^(n)) and tr((V^(n))^T(A^(n):U^(n))), with (U^(n))^TU^(n) = I_K and (V^(n))^TV^(n) = I_K for n = 1, 2, ..., N.Source: §3.5 · pp. 119, 121
  • For very large K = K_1K_2···K_N, block-n TT formats may be inefficient, motivating looped MPS/MPO alternatives whose loops can be eliminated by vectorizing orthogonal matrices.These alternatives use fourth-order TT cores, although contracting many loops is computationally complex.Source: §3.5 · pp. 121–122, Figure 3.13 · p. 121

U P RIˆK Ñ u P RIK, V P RJˆK Ñ u P RJK (3.46)

This section develops TT-network formulations for generalized eigenvalue, canonical-correlation, low-rank factorization, and huge-scale linear-system problems. It focuses on recursive contractions and TT-based optimization methods that reduce these problems to smaller core-wise systems while using structured representations, residual-gradient enrichment, and orthonormalization.

3.6 GEVD and Related Problems using TT Networks

  • GEVD estimates leading or smallest eigenpairs through trace optimization with orthogonality constraints, assuming A, B, and V admit good low-rank TT approximations.A is symmetric, B is symmetric positive-definite, and V is represented in TT format; minimizing targets the smallest eigenvalues.Source: §3.6 · pp. 124–125
  • TT contractions reduce huge-scale trace-ratio optimization to smaller trace-ratio minimizations involving recursively contracted matrices A^(n) and B^(n).The TT networks represent tr(V^TAV) and tr(V^TBV) while optimizing one quantity against the other under orthogonality constraints.Source: §3.6 · pp. 124–125

3.6.1 TT Networks for Canonical Correlation Analysis (CCA)

  • TT contractions transform huge-scale orthogonal CCA into smaller optimization problems for multiple canonical components.Sparse CCA additionally imposes sparsity penalties on the projection matrices, with TT/MPO penalties expressible through core fibers and fiber normalization replacing core orthogonalization.Source: §3.6.1 · pp. 127–129

3.6.2 Tensor CCA for Multiview Data

  • Tensor CCA maximizes a multilinear correlation over covariance-normalized canonical vectors for multiview data, while large view counts require distributed tensor-network representations.After covariance regularization, the problem becomes a rank-one approximation; when N > 5, the core tensor can become too large and may be represented in tensor-chain format.Source: §3.6.2 · pp. 129–130

3.7 Two-Way Component Analysis in TT Formats

  • Constrained TT/TC low-rank matrix factorization represents huge structured matrices and supports truncated SVD or pseudo-inverse computation after imposing orthogonality constraints.Without constraints, the factorization is nonunique and its TT components lack physical meaning; left- and right-orthogonal cores yield the R largest singular values and corresponding vectors.Source: §3.7 · pp. 131, 133, Figure 3.17 · p. 132

3.8 Solving Huge-Scale Systems of Linear Equations

  • TT/HT methods solve huge linear systems approximately in compressed form while accommodating smoothness, sparsity, and nonnegativity constraints.The framework covers general systems beyond the symmetric square-matrix cases emphasized by earlier TT/HT solutions.Source: §3.8 · p. 133

3.8.1 Solutions for a Large-scale Linear Least Squares Problem using ALS

  • ALS recursively contracts TT representations of A, x, and b to convert large-scale Tikhonov-regularized least squares into much smaller node-wise minimizations.The regularization uses γx^TL^TLx, where L is typically a first- or second-order discrete derivative matrix.Source: §3.8.1 · pp. 133–134
  • ALS converts a huge TT-formatted linear system into smaller core-wise systems solvable by standard methods, with recursive contractions avoiding explicit matrix multiplication.The approach targets Tikhonov-regularized least squares for huge non-symmetric systems Ax ≈ b.Source: §3.8.1 · p. 134, Figure 3.18 · p. 135
  • Left- and right-orthonormalized TT cores produce a symmetric semipositive definite reduced matrix that is better conditioned than the original huge matrix.Efficient solution depends on low-rank TT/QTT approximations; weakly structured data may require large ranks, making computation difficult or impossible.Source: §3.8.1 · pp. 134, 136

3.8.2 Alternating Minimum Energy (AMEn) Algorithm for Solving a Large-scale Linear Least Squares Problems

  • AMEn updates each TT core by solving a reduced least-squares system, approximates the gradient through TT contractions, and enriches cores using a partially projected gradient.The algorithm alternates core updates, residual updates, enrichment, and orthogonalization for an approximate TT solution of Ax − b.Source: §3.8.2 · p. 136, P RRn´1Jn¨¨¨JN (3.79) · p. 137

P RRn´1Jn¨¨¨JN (3.79)

  • AMEn enrichment concatenates compatible solution and enrichment cores, then orthogonalization expands the frame matrix's column space.Partial gradient projection makes the TT core sizes consistent, enabling concatenation and subsequent subspace expansion.Source: P RRn´1Jn¨¨¨JN (3.79) · pp. 137–138

L P P RRn´1Jnˆ(Rn+Qn),

This section develops tensor-network approaches for large-scale regression and sparsity-constrained optimization, using TT representations, regularization, and iterative algorithms to reduce computational complexity. It also identifies open challenges in optimizing nonsmooth penalties and selecting data permutations for effective TT approximations.

L P P RRn´1Jnˆ(Rn+Qn),

3.8.3 Multivariate Linear Regression and Regularized Approximate Estimation of Moore-Penrose Pseudo-Inverse

  • Regularized multivariate regression minimizes a trace-based cost over a TT-form solution X, with Tikhonov constraints and a Moore-Penrose pseudo-inverse special case.The formulation uses A P RIˆJ, B P RIˆK, X P RJˆK, and L P RJˆJ; when B = II and K = I, it becomes TT-form pseudo-inverse computation.Source: §3.8.3 · p. 138
  • Recursive TT-core contractions with ALS or MALS convert infeasible large-scale optimization into smaller linked subproblems, while regularization can reduce TT ranks, parameters, and ill-posedness.Source: §3.8.3 · p. 140
  • Future work includes selecting an optimal data-tensor permutation to obtain an effective TT-approximation tree for available samples.Source: §3.8.3 · p. 141

3.8.4 Solving Huge-scale LASSO and Related Problems

  • LASSO and related sparse regression models can be tensorized through IRLS, group decompositions, rank-one weight approximations, and low-dimensional subproblems solved with methods such as ADMM.The section notes that nonsmooth ℓ1- and ℓq-penalties remain insufficiently investigated in tensor-network settings, while IRLS replaces them with reweighted ℓ2-norms.Source: §3.8.4 · pp. 142–144

3.9 Truncated Optimization Approach in TT Format

Truncated iteration methods perform optimization steps in low-rank tensor formats and then truncate the resulting ranks to control computational growth. In TT format, they support broad optimization schemes and applications but require low-rank auxiliary quantities and can incur substantial truncation costs.

TT implementation

  • TT representations enable distributed optimization of huge vectors and matrices, but additions and gradient computations increase TT ranks, requiring subsequent TT-rounding.A preconditioned gradient step represents both the solution and gradient in TT format; the gradient’s TT ranks are typically large because of the operations used to compute it.Source: §3.9 · pp. 145–146, Figure 3.22 · p. 146

Truncated iteration

  • Truncated iteration combines a preconditioned gradient step with rank truncation controlled by tolerance ϵ, using hard or soft thresholding to adapt ranks during optimization.Hard thresholding removes singular values below a threshold, whereas soft thresholding applies shrinkage and corresponds to nuclear-norm minimization.Source: §3.9 · pp. 146–147

Applications

  • Truncated iteration has been applied to large-scale linear systems, PDE discretization, extreme-eigenvalue computation, and low-rank matrix or tensor completion across CP, Tucker, TT, and HT formats.Examples include Richardson, CG, and GMRES solvers; Lanczos, inverse iteration, and LOBPCG eigensolvers; and iterative hard thresholding.Source: §3.9 · p. 147

Advantages

  • The approach avoids constructing tensor networks for a specific cost function, has global convergence guarantees under restricted conditions, and extends to large-scale compressed sensing.Its applicability follows from the analogy between low-rank truncation and sparse signal estimation.Source: §3.9 · p. 147

Limitations

  • The methods require several auxiliary vectors to admit low-rank representations, while intermediate residuals and Krylov vectors may have high ranks and make per-iteration core truncation expensive.In GMRES, this difficulty can occur even when the solution and right-hand side are well approximated in TT format.Source: §3.9 · pp. 147–148

3.10 Riemannian Optimization for Low-Rank Tensor Manifolds

This section develops Riemannian optimization methods for low-rank tensor manifolds, covering tangent-space geometry, retractions, vector transport, projector-splitting methods, and dynamical low-rank approximation across Tucker, TT, and HT formats. It also examines convergence limitations and applications to exponentially parameterized interaction models, highlighting the need for stronger theory and scalable implementations.

3.10 Riemannian Optimization for Low-Rank Tensor Manifolds

  • Low-rank tensor optimization minimizes a scalar cost J(X) over a smooth manifold Mr, whose geometry can convert constrained optimization into unconstrained optimization on a constrained space.The framework applies broadly when the cost function is defined on manifold points.Source: §3.10 · pp. 148–149

3.10.1 Simplest Riemannian Optimization Method: Riemannian Gradient Descent

  • Riemannian gradient descent projects the gradient onto the tangent space, then uses retraction to return to the manifold; SVD and TT-SVD provide retractions for fixed-rank matrices and TT tensors.Retraction avoids expensive geodesic computation while preserving feasible iterates, and tangent-space projections can be computed without forming the full gradient.Source: §3.10.1 · pp. 151–152, Figure 3.23 · p. 152

3.10.2 Advanced Riemannian Optimization Methods

  • Vector transport maps directions between tangent spaces, enabling Riemannian conjugate-gradient methods to combine transported previous directions with projected gradients.Retraction and vector transport are also critical for sophisticated CG, Newton, and quasi-Newton algorithms.Source: §3.10.2 · p. 153, Figure 3.24 · p. 153, §3.10.3 · p. 155

3.10.3 Riemannian Newton Method

  • Riemannian Newton methods incorporate manifold curvature but are technically complicated and potentially unstable near singular points; trust-region variants were developed to improve global convergence [Absil et al., 2007, Boumal and Absil, 2011, Ishteva et al., 2011].The framework applies to Tucker and TT formats but not CP decompositions, whose fixed canonical-rank sets are not manifolds.Source: §3.10.3 · pp. 154–155

3.10.4 Low-Rank TT-Manifold

  • TT-manifold tangent-space points have ranks bounded by (2R1, . . . , 2RN−1), enabling rounding to rank (R1, . . . , RN−1) at O(NIR3) cost and preserving matrix-method structure.This makes Riemannian optimization a promising tool for low-rank TT-manifolds.Source: §3.10.4 · p. 156

3.10.5 Dynamical Low-Rank Matrix/Tensor Approximation

  • Dynamical low-rank approximation constrains trajectory velocity to the tangent space, reducing computation through manifold parametrizations; projector-splitting offers a stable explicit second-order integration scheme [Koch and Lubich, 2007, Lubich and Oseledets, 2014].Retraction is needed for discrete integration, whereas continuous trajectories remain exactly on the manifold.Source: §3.10.5 · pp. 157–158
  • Projector-splitting schemes improve convergence through the PU, PUV, then PV ordering and are exact under manifold conditions, while their retraction is second-order [Lubich and Oseledets, 2014; Absil and Oseledets, 2015].The approach generalizes to TT networks through sweeping algorithms for dynamical-system approximation and optimization with non-quadratic functionals [Lubich et al., 2015].Source: §3.10.5 · pp. 158–159

3.10.6 Convergence of Riemannian Optimization Algorithms

  • Riemannian optimization has incomplete convergence theory because curvature, singular points, global convergence, and low-rank approximation of nonexact solutions remain difficult.Local convergence follows from general theory, but spurious local minima can occur, while experiments observed staircase convergence through gradually increasing ranks [Kolesnikov and Oseledets, 2016].Source: §3.10.6 · p. 159

3.10.7 Exponential Machines for Learning Multiple Interactions

  • A tensor-train Exponential Machine analyzed a tensor with 2160 entries and was applied to a MovieLens recommender-system dataset with 105 users [Novikov et al., 2016].The model represents exponentially many interaction parameters compactly in TT format and optimizes them using Riemannian gradient descent, including stochastic mini-batch variants for large datasets.Source: §3.10.7 · pp. 160–161

3.10.8 Future Perspectives of Riemannian Optimization

  • Riemannian optimization remains promising because it preserves low-parametric tensor representations, whereas convex nuclear-norm surrogates for TT manifolds lack good guarantees [Rauhut et al., 2016].Stochastic variants support many data points [Novikov et al., 2016], but automatic-differentiation tools and methods beyond stochastic Riemannian gradient descent remain underdeveloped.Source: §3.10.8 · p. 162

3.11 Software and Computer Simulation Experiments with TNs for Optimization Problems

This section surveys rapidly developing software libraries for tensor decompositions and networks, alongside computer experiments demonstrating performance advantages in diverse large-scale optimization applications. It covers TT, HT, standard tensor-decomposition, quantum-network, parallel, and deep-learning implementations.

TT software

  • The TT Toolbox is the most complete TT/MPS, QTT software and supports AMEn linear solves, symmetric EVD, and huge-matrix inverse or pseudoinverse computations [Oseledets et al., 2012].TTeMPS complements it with object-oriented TT/MPS algorithms and AMEn computation of multiple extremal eigenvalues and eigenvectors [Kressner et al., 2014a].Source: §3.11 · p. 163

Standard tensor-decomposition software

  • Tensor Toolbox, N-Way Toolbox, Tensorlab, TDALAB, and TENSORBOX provide general-purpose, constrained, sparse, incomplete, structured, and user-friendly support for standard tensor decompositions [Bader and Kolda, 2006, 2015].Tensorlab also supports CP, BTD, and constrained Tucker optimization, joint factorization, tensorization, large-scale datasets, and arbitrary-order visualization [Vervliet et al., 2016].Source: §3.11 · pp. 163–164

HT and quantum-network software

  • The Hierarchical Tucker toolbox targets HT networks and avoids explicit SVD computation when truncating tensors already represented in HT format.Quantum-network software includes iTensor for open-source C++ DMRG development, Uni10 for PEPS and MERA contractions, and mptensor for parallel supercomputer computation.Source: §3.11 · p. 164

Optimization experiments

  • Computer experiments have validated significant tensor-network performance advantages for symmetric EVD, PCA, CCA, SVD, huge linear systems, matrix inverses, completion, dynamics, and nonlinear-function approximation [Dolgov et al., 2014].These applications include the cited TN methods for large-scale structured datasets, including AMEn and TT/HT approaches [Bolten et al., 2016, Dolgov, 2013, Dolgov and Savostyanov, 2014].Source: §3.11 · pp. 164–165

Decomposition applications

  • The reviewed application paradigms also cover CP and Tucker decompositions, TT/HT scientific-computing methods, and TT-based classification and clustering [Choi and Vishwanathan, 2014].The section notes these as recent publications applying low-rank tensor networks to diverse structured-data problems [Benner et al., 2016, Stoudenmire and Schwab, 2016].Source: §3.11 · p. 165

Deep-learning implementation

  • TensorFlow is an open-source Google platform using hardware tensor processing units and offering particularly strong support for deep neural-network applications.Source: §3.11 · p. 165

3.12 Open Problems and Challenges in Applying TNs for Optimization and Related Problems

Applying tensor networks to real-world optimization assumes that datasets admit sufficiently good low-rank TN approximations and that approximate solutions are acceptable. Future work must extend TN methods to constrained and sparse problems while improving scalability, contraction, approximation control, theory, interpretability, and visualization.

Core assumptions

  • TN applications require datasets to admit sufficiently good low-rank approximations and users to accept approximate solutions.These assumptions underpin the use of tensor networks for very large-scale optimization, scientific computing, and dimensionality reduction.Source: §3.12 · pp. 165–166

Constrained and sparse optimization

  • Future research should extend low-rank TN methods to constrained optimization with multiple constraints, nonsmooth penalties, sparsity, or nonnegativity, including sparse NMF, LP/QP, SDPs, SILPs, inverse covariance selection, regressor selection, and sparse PCA variants.The chapter identifies these as problems with efficient small- to medium-scale algorithms but unresolved large-scale TN opportunities.Source: §3.12 · p. 166

Algorithms and contractions

  • Parallel algorithms for broader optimization classes and efficient contractions are needed because most described algorithms are sequential, while exact contraction of looped networks can scale exponentially with tensor order.Approximate contractions may suffice in some applications but complicate approximation-error control, especially for PEPS, PEPO, and MERA.Source: §3.12 · p. 168

Tensor-network structures

  • The best TN structure for each dataset or application should provide sufficiently low-order, low-complexity approximations that reduce storage and computation to affordable levels.This motivates more sophisticated and flexible networks integrated with complex systems and optimization problems.Source: §3.12 · p. 169

Approximation and convergence

  • TN implementations need semiautomatic accuracy and rank criteria, a priori error bounds, and convergence analyses to address rounding-error accumulation, TT-rank explosion, and uncertain convergence properties.These controls are especially important because current TT decompositions and contractions still require tuning parameters such as approximation accuracy and TN-rank estimates.Source: §3.12 · p. 169

Theory, interpretability, and visualization

  • Further challenges include principled constraints for meaningful latent-feature extraction, uniqueness and optimality analyses of TN decompositions, and visualization techniques for huge tensors in TN formats.Resolving these issues would support faster, more reliable algorithms and wider practical adoption of low-rank tensor approximations.Source: §3.12 · pp. 168–169

Tensor Networks for Deep Learning

This section develops tensor-network frameworks for representing and optimizing deep learning models, including restricted Boltzmann machines and deep convolutional networks, by compressing high-order parameterizations while preserving useful approximation, interaction, or computational properties. It covers CP, HT, unconstrained HT, variable-order TTNS, MERA, and two-dimensional tensor-network constructions, relating them to hierarchical models, nonlinear tensor products, convolutional rectifier networks, distributed score representations, and multiscale structures.

Tensor Networks for Deep Learning

  • Tensor networks address deep-learning challenges by reducing parameters without dramatic performance loss and by enabling new or optimized DNN architectures.The chapter targets parameter reduction and expressive-power analysis, linking DNNs and TNs to support more efficient architectures.Source: Tensor Networks for Deep Learning · p. 172, §4.1 · p. 173

4.1 A Perspective of Tensor Networks for Deep Learning

  • Deep architectures are modeled through tensor decompositions: shallow networks correspond to CP rank-1 models, whereas networks with log2 N hidden layers correspond to hierarchical Tucker decompositions.The section also relates TN expressive power to entanglement bounds, while proposing that successful machine-learning datasets often have relatively low entanglement entropy.Source: §4.1 · pp. 173, 175
  • A DNN can be transformed into a TN, canonicalized and rank-rounded, then mapped back to a DNN with similar universal approximation power and fewer parameters.Truncated SVD can reduce TN ranks to the minimum required for a desired approximation accuracy; low-order sparsely interconnected cores can also avoid the curse of dimensionality.Source: §4.1 · pp. 173–174, Figure 4.1 · p. 174

4.2 Restricted Boltzmann Machines (RBM)

  • RBMs are probabilistic models whose hidden units learn relevant features, and their formulas can be interpreted as multiplicative products of “experts” and converted into TNs with interconnected diagonal core tensors.Higher-order RBMs increase modeling power but suffer parameter counts scaling with products of variables, motivating low-rank TN dimensionality reduction.Source: §4.2 · pp. 176, 178, 179, 180

4.2.1 Matrix Variate Restricted Boltzmann Machines

  • Matrix-variate RBMs use bilinear connections and fewer parameters than standard RBMs, enabling faster training while retaining comparable performance; their fourth-order weights can be approximated by TT or HT networks.For a four-way weight tensor, truncated SVD represents the weights using two third-order core tensors, with rank-1 yielding matrix factors.Source: §4.2.1 · pp. 180–181
  • Matrix-variate RBM factorization reduces free parameters from IJMK to IJ + KM + JM + IK while approximating interactions between input and hidden matrices.The corresponding weight tensor can be represented as an outer product of two matrices, with TT or HT available for higher-order tensors.Source: §4.2.1 · p. 181, Figure 4.5 · p. 182

4.2.2 Tensor-Variate Restricted Boltzmann Machines

  • CP-factorized tensor-variate RBMs reduce mapping parameters to R(K + Σ_n I_n), scaling linearly rather than exponentially with the number of modes while preserving conditional independence and fast RBM sampling.The factorization moderates multiplicative interactions between modes through hidden units.Source: §4.2.2 · p. 183

4.3 Basic Features of Deep Convolutional Neural Networks

  • DCNNs combine locality, optional weight sharing, and pooling, and generally outperform shallow networks on compositional function approximation and multi-class classification.Pooling reduces layer sizes and can induce translation invariance, while locality supports hierarchical representations and an exponential representation advantage over shallow networks.Source: §4.3 · p. 184

4.4 Score Functions for Deep Convolutional Neural Networks

  • DCNN score functions represent input patches through nonlinear feature maps and tensor inner products, while HT-based distributed coefficients make the formulation tractable for high-order classification.The related Tensorial Mixture Model uses nonnegative tensor-network cores and is described as promising for classification with partially missing training data, subject to scaling and nonnegativity constraints.Source: §4.4 · pp. 185, 186, 188
  • HT tensor networks address high-order DCNN coefficient tensors whose exponential size makes direct storage and implementation intractable, whereas CP may require large ranks and unstable decompositions.The score-function formulation is directly suited to ConvACs, while HT offers an alternative intended to avoid the curse of dimensionality.Source: §4.4 · pp. 187–188

4.5 Convolutional Arithmetic Circuits and HT Networks

  • The simplified HT ConvAC uses sparse third-order core tensors across L = log2(N) hidden layers with pooling-2 windows to approximate score-function grid tensors.Equal ranks across nodes and diagonal frontal slices simplify the network and reproduce basic ConvAC features.Source: §4.5 · pp. 189, 191, Figure 4.7 · p. 190
  • Unconstrained HT networks remove the strong core-tensor constraints of simplified HT models, while variable-order TTNS cores enable adjustable pooling-window sizes across hidden layers.Fifth-order and third-order cores support pooling windows of four and two, respectively; TTNS generalizes HT by allowing variable core orders.Source: §4.5 · pp. 191–193, Figure 4.9 · p. 193

4.6 Convolutional Rectifier NN Using Nonlinear TNs

  • Nonlinear outer products with associative and commutative activation-pooling functions convert ConvAC tensor models into convolutional rectifier networks, while weight sharing restricts coefficient tensors to be symmetric.The framework includes generalized Kronecker and Khatri-Rao products and represents generalized CP models as shallow rectifier networks; symmetric coefficient tensors further limit function approximation.Source: §4.6 · pp. 193–194
  • TT/MPS, TT/MPO, and tensor-chain networks offer simpler, deeper alternatives to HT for modeling deep networks, potentially reaching N hidden layers, while QTT can use L = N ¨ log2(I) layers.Standard HT yields L = log2(N) hidden layers, whereas TT/TC architectures may employ N hidden layers; the text highlights QTT for relatively large depth.Source: §4.6 · p. 195

4.7 MERA and 2D TNs for a Next Generation of DCNNs

  • MERA uses disentanglers and isometries for multiscale disentangling and coarse-graining, capturing scale invariance, polynomially decaying correlations, stronger compression, logarithmic area-law corrections, and efficient contraction.The section presents MERA as potentially more expressive than HT/TT architectures and as a basis for improved deep-network families, but frames these deep-learning applications as an ongoing investigation and conjecture.Source: §4.7 · pp. 196, 198
  • MERA and two-dimensional tensor networks are proposed as frameworks for analytically constructing new DCNNs and rigorously examining tensor-network correspondences, expressive power, and novel network designs.The proposed extensions include honeycomb or brick-wall lattices and PEPS networks with fifth- through seventh-order cores, with potential applications to classification, clustering, and prediction.Source: §4.7 · pp. 198–199, Figure 4.11 · p. 199

Discussion and Conclusions

The monograph reviews tensor decompositions and tensor-network approaches for large-scale machine learning and data analytics, emphasizing low-rank representations, distributed computation, and applications from dimensionality reduction to deep learning.

Discussion and Conclusions

  • The work reviews tensor-based supervised and unsupervised learning, regression, support tensor machines, tensor canonical correlation analysis, higher-order and kernel partial least squares, and deep learning.It aims to provide both a state-of-the-art review and new directions for large-scale optimization problems involving multiway, multi-relational, and multi-modal data.Source: Discussion and Conclusions · p. 200
  • Graphical tensor-network representations clarify tensor algebra and support efficient higher-order representations and cost-function optimization using Tucker, Tensor Train, and Hierarchical Tucker formats.The presentation also covers tensorization and distributed representations of vectors, matrices, and low-order tensors.Source: Discussion and Conclusions · p. 200
  • Low-rank tensor networks and contraction algorithms combat dimensionality and convert otherwise intractable huge-scale linear and multilinear optimization problems into affordable linked or distributed subproblems.These methods can achieve linear or sub-linear storage and computational complexity while addressing couplings among variables and multimodal, incomplete, or noisy data.Source: Discussion and Conclusions · pp. 200–201
  • Tensor networks offer a platform for improved deep neural-network architectures, with potential applications in efficient data and image classification, clustering, and prediction.The discussion highlights links between tensor networks and deep learning as a source of insights into multilayer neural networks and optimized DNN families.Source: Discussion and Conclusions · p. 201

1 Estimation of Derivatives of GCF

The section describes estimating unknown GCF derivatives from the sample first GCF, representing nth-order partial derivatives as nth-order tensors. It derives second-GCF derivatives using partitioned outer products, multiplicity counts, and tensor symmetrization, with approximations shown for orders 2–7.

Derivative estimation

  • Unknown GCF derivatives are estimated from the sample first GCF, whose nth-order partial derivatives are nth-order tensors of size I ˆ I ˆ ¨ ¨ ¨ ˆ I.The derivative tensors are expressed using exp(uTxt) and repeated outer products of xt.Source: §1 · p. 202

Second-GCF derivatives

  • Derivatives of the second GCF are constructed from outer products of k tensors with orders n1,...,nk satisfying 1 ď n1 ď ¨ ¨ ¨ ď nk ď N and n1 + ¨ ¨ ¨ + nk = N.The component tensors ψtn1,n2,...,nku(u) are formed from the corresponding njth-order derivative tensors.Source: §1 · p. 202

Tensor symmetrization

  • The operator S(A) symmetrizes an Nth-order tensor by combining its tensor permutations, and this operation can be applied once to the resulting tensor Ψx(u).The simplification avoids symmetrizing each tensor ψn1,n2,...,nk(u) separately.Source: §1 · p. 203

Partition multiplicities and approximations

  • The coefficient mn1,n2,...,nk counts partitions of {1, 2, . . . , N} into k distinct parts with prescribed sizes, accounting for repeated part sizes through replication numbers.Approximations of Ψx(u) are provided for derivative orders N = 2, 3, . . . , 7.Source: §1 · pp. 203–204

2 Higher Order Cumulants

For zero-mean centered mixtures, the first derivative vanishes, eliminating singleton partitions and simplifying higher-order cumulant computation, including orders 2 through 7.

2 Higher Order Cumulants

  • Centering the mixtures to zero mean makes the first derivative ψ(1) vanish.Source: §2 · p. 204
  • The resulting zero terms for singleton partitions simplify cumulant computation by allowing those partitions to be ignored.Source: §2 · p. 204
  • Cumulants of orders 2 through 7 can then be computed explicitly.Source: §2 · p. 205

3 Elementary Core Tensor for the Convolution Tensor

The elementary core tensor S is assembled from 2^N sparse Nth-order tensors whose ones occupy locations satisfying a specified index condition. Its subtensors are organized by boundary assignments and recursive relations, yielding pairwise orthogonal components with only N+1 nonzero tensors.

Sparse-tensor construction

  • S is constructed from 2^N sparse tensors S_1, S_2, ..., S_{2^N}, each of size 2 × 2 × ⋯ × 2, with ones only at specified index locations.The location condition is expressed using complemented indices, where ī_n = 2 − i_n.Source: §3 · p. 205

Core-tensor structure

  • For n = 1, ..., N, the boundary subtensors of S select consecutive sparse components through S(1, :, ..., :, 1, n) = S_{2n−1} and S(1, :, ..., :, 2, n) = S_{2n}.These assignments define the initial organization of the core tensor from the sparse tensors.Source: §3 · p. 205

Recursive definition

  • The remaining subtensors are recursively defined by transferring boundary indices according to relations involving S(m−1, ..., 2, n), S(m−1, ..., 1, N), and S(m−1, ..., 1, n−1).The recursion covers m = 2, ..., N and n = 1, ..., N, with separate cases for n = 1, n = 2, ..., N, and the terminal boundary.Source: §3 · p. 205

Orthogonality and sparsity

  • The sparse tensors are pairwise orthogonal, and exactly N+1 are nonzero: S_1, ..., S_N plus S_{N+2} for even N or S_{N+1} for odd N.The remaining N−1 core tensors are zero tensors.Source: §3 · p. 205

Related papers

Loading 1708.09165v1…