PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-Rank Tensor Networks for Dimensionality Reduction and Large-Scale Optimization Problems: Perspectives and Challenges PART 1

A. Cichocki, N. Lee, I. V. Oseledets, A. -H. Phan, Q. Zhao, D. Mandic

arXiv:1609.00893v3math.NA

TL;DR

Modern datasets create scalability challenges because tensor sizes grow exponentially with order and exhibit high volume, variety, velocity, and veracity. This monograph reviews tensor decompositions and networks, emphasizing Tucker and TT models, and presents them as tools for large-scale analytics and optimization.

  • Problem

    Tensor data can become computationally prohibitive because an Nth-order tensor with mode size I contains I^N elements, while modern datasets also demand scalable, robust, real-time, and heterogeneous-data processing.

  • Method

    The monograph unifies tensor-network terminology, notation, representations, and algorithms while reviewing CP, Tucker, TT, and extensions for large-scale data analytics and optimization.

  • Results

    Tensor-network formats provide compressed representations and a framework for analyzing linked tensor blocks with millions or billions of nonzero entries; TT methods also offer stable, error-controlled rank reduction.

  • Takeaways & Limitations

    Tensor networks extend multilinear component analysis and related optimization methods to datasets whose raw tensor representations are prohibitively large.

  • Takeaways & Limitations

    TT and HT ranks can increase with tensor order or desired accuracy, while Tucker representations become impractical beyond tensor order five because core size scales exponentially.

Abstract

from arXiv · show

Machine learning and data mining algorithms are becoming increasingly important in analyzing large volume, multi-relational and multi--modal datasets, which are often conveniently represented as multiway arrays or tensors. It is therefore timely and valuable for the multidisciplinary research community to review tensor decompositions and tensor networks as emerging tools for large-scale data analysis and data mining. We provide the mathematical and graphical representations and interpretation of tensor networks, with the main focus on the Tucker and Tensor Train (TT) decompositions and their extensions or generalizations. Keywords: Tensor networks, Function-related tensors, CP decomposition, Tucker models, tensor train (TT) decompositions, matrix product states (MPS), matrix product operators (MPO), basic tensor operations, multiway component analysis, multilinear blind source separation, tensor completion, linear/multilinear dimensionality reduction, large-scale optimization problems, symmetric eigenvalue decomposition (EVD), PCA/SVD, huge systems of linear equations, pseudo-inverse of very large matrices, Lasso and Canonical Correlation Analysis (CCA) (This is Part 1)

Introduction and Motivation

The paper frames tensor networks as a response to massive, varied, and high-dimensional datasets whose storage and computation become prohibitive. It reviews low-rank tensor representations and algorithms for compression, dimensionality reduction, component analysis, and large-scale optimization.

  • 1.1 Challenges in Big Data Processing: Tensor networks address big-data challenges involving excessive volume, velocity, veracity, and variety through sparse, distributed representations.Low-rank tensor networks approximate huge data tensors with interconnected low-order core tensors.
  • 1.3.1 Curse of Dimensionality: The curse of dimensionality makes an Nth-order tensor with mode size I require I^N elements, causing storage and computation to grow exponentially.Tensor network representations can alleviate this growth by using smaller low-order core tensors.
  • 1.4 Separation of Variables and Tensor Formats: A rank-2 CP representation reduces a 5-variate function sampled at 100 points per axis from 10^10 values to 1000 parameters.The CP representation uses NIR parameters, which scales linearly with tensor order and mode size.
  • 1.4 Separation of Variables and Tensor Formats: TT formats provide numerically stable, quasi-optimal rank reduction with approximation-error control, unlike often ill-posed high-order CP algorithms.Truncated SVD and adaptive cross-approximation are cited as mechanisms for rank reduction in TT formats.
  • Introduction and Motivation: The monograph emphasizes TT approximations while also considering HT and TTNS formats for high-order tensor representations.Tensorization converts structured vectors and matrices into smaller interconnected optimization sub-problems handled by existing optimization methods.
  • Introduction and Motivation: Tensor networks support compressed multidimensional data, distributed computation, robustness to noise and missing data, constraints, coupled-data analysis, and intuitive graphical operations.These capabilities are presented as advantages of tensorization and low-rank tensor decompositions into factor matrices or core tensors.
  • 1.6 Scope and Objectives: The monograph extends prior reviews by linking tensor-network models to extremely large-scale analytics, optimization, and coupled tensor blocks with millions or billions of non-zero entries.It positions tensor networks as a framework for discovering hidden structure in prohibitively large-scale data.
  • Introduction and Motivation: The framework extends matrix component analysis to multiway component analysis for multidimensional and multimodal data while targeting linear or sub-linear dataset scaling.It also covers linked tensors, dimensionality reduction, EVD/PCA, GEVD, SVD, and large structured linear systems.

Tensor Operations and Tensor Network Diagrams

Tensor operations reshape, combine, and contract multidimensional data, while tensor-network diagrams represent high-order tensors through interconnected cores. These representations support compression and distributed computation, but network design trades storage efficiency against contraction complexity.

  • Reshaping and matricization: Tensorization reverses vectorization or matricization by creating higher-order tensors from vectors, matrices, or lower-order tensors.The monograph also illustrates tensorization as a way to form high-order representations from structured lower-order data.
  • Reshaping and matricization: Matricization reorders tensor elements into matrices, including mode-n and canonical forms with dimensions determined by selected mode groups.Mode-n matricization uses mode-n fibers as columns; canonical matricization groups the first n modes against the remaining modes.
  • Distributed and cyclic networks: Distributed and cyclic tensor networks use smaller cores to improve storage and reduce parameters, but loops make contraction more difficult and increase computational cost.MERA formats can further reduce core order and free parameters while modeling more complex variable interactions.
  • Tensor-network diagrams: Tensor-network diagrams encode tensor modes as edges and cores as nodes, simplifying mathematical operations and allowing topology changes while preserving physical modes.Modified topologies can yield similar structures across diverse tensors, supporting joint analysis of interconnected datasets.
  • Generalized tensor products: Generalized nonlinear tensor networks replace standard outer products with associative and commutative nonlinear functions, enabling flexible approximations of high-order tensors.The HOPTA model can also generalize Hadamard, Kronecker, and Khatri–Rao products.

Constrained Tensor Decompositions: From Two-way to Multiway Component Analysis

Component analysis applies constrained factorization to mixed signals to extract components with desired properties or estimate the mixing matrix. Prior knowledge about latent-source statistics and diversity supports well-posedness and uniqueness.

  • Component-analysis framework: Component analysis factorizes observed mixed signals under constraints to extract components with specific properties or estimate the mixing matrix.The framework is presented as a constrained matrix-factorization approach.
  • Component-analysis framework: Prior knowledge about latent-variable statistics and diversity is used to improve the well-posedness and uniqueness of component-analysis problems.

3.1 Constrained Low-Rank Matrix Factorizations

Constrained low-rank matrix factorizations provide a foundation for component analysis, group data analysis, and dimensionality-reduction methods such as PCA and SVD.

  • Two-way component analysis: Two-way component analysis factorizes linked data matrices under constraints on interrelated component or factor matrices.Orthogonality and statistical-independence constraints connect the formulation to group PCA and ICA workflows.
  • Two-way component analysis: Possible factor-matrix constraints include orthogonality, sparsity, statistical independence, nonnegativity, and smoothness.
  • Established methods: Established matrix component-analysis methods include PCA, RPCA, ICA, NMF, SCA, and SmCA for blind source separation, feature extraction, and classification.
  • SVD: SVD is a special factorization with orthogonal factor matrices and a diagonal matrix of nonnegative singular values ordered monotonically.
  • SVD: Truncated SVD gives the optimal least-squares low-rank matrix approximation and underpins low-rank tensor approximations.The cited optimality also extends to arbitrary unitarily invariant norms.
  • Linked-data analysis: Simultaneous matrix factorizations support analysis of several linked datasets within tolerable computational time.

3.2 The CP Format

The CP decomposition represents an Nth-order tensor as a sum of rank-1 terms, with factor matrices and optional scaling coefficients. The section covers its estimation, computational bottlenecks, scalable alternatives, constraints, and applications.

  • Representation: CP decomposes an Nth-order tensor into a linear combination of rank-1 outer products represented by factor matrices and scaling coefficients.The tensor rank is the smallest number of rank-1 terms needed for an exact decomposition.
  • Representation: The decomposition can be written in matricized and vectorized forms using Khatri–Rao products of the factor matrices.These forms support sequential factor-matrix updates in optimization algorithms.
  • Algorithms: CP estimation commonly minimizes Frobenius-norm least-squares or least-absolute-error objectives, with ALS updating one factor matrix at a time.ALS alternates component optimization while holding the remaining factor matrices fixed.
  • Algorithms: The main ALS and gradient-descent bottlenecks are tensor–Khatri–Rao multiplication and pseudo-inverse computation for R × R matrices.ALS is most attractive for well-conditioned, high-SNR problems with well-separated, non-collinear components.
  • Scalable methods: O(log R) random projections can preserve tensor spectral information while avoiding eigen-gap dependence, and may outperform standard CP-ALS under uniform sampling.Tensor sketching and distributed systems such as DFacTo and GigaTensor are additional alternatives for huge tensors.
  • Properties and applications: CP is generally unique under mild conditions without additional factor constraints and is used in blind signal separation, exploratory analysis, and wireless communications.Prior properties such as nonnegativity, orthogonality, independence, or sparsity can nevertheless be incorporated into algorithms.

3.3 The Tucker Tensor Format

The Tucker format represents a tensor through a smaller core tensor and mode-specific factor matrices, generalizing CP and supporting multilinear-rank reduction. Its orthonormal form preserves norms and enables efficient computations such as fast Fourier transforms.

  • Representation: Tucker decomposes an Nth-order tensor into a relatively small core tensor and factor matrices that model interactions across modes.The core is typically much smaller than the original tensor dimensions.
  • Rank and formats: The multilinear rank is the tuple of ranks of the tensor’s mode unfoldings, and Tucker ranks can be selected using information criteria or dominant eigenvalues.The independent Tucker format requires all factor matrices to have full column rank.
  • Properties: For an independent Tucker format, the tensor and core have equal CP rank, while rank bounds satisfy R1 ≤ rankCP(X) ≤ R2R3⋯RN under ordered multilinear ranks.For symmetric tensors, an independent Tucker core is also symmetric.
  • Properties: In the orthonormal Tucker format, Frobenius and Schatten p-norms of the tensor and core are equal, and norm computation can have O(R^N) rather than O(I^N) complexity.Here R is the maximum multilinear rank and is typically much smaller than I.
  • Rank and formats: CP is a special Tucker case with a diagonal core, whereas unconstrained Tucker is not unique unless additional constraints reduce its indeterminacies.The Tucker-2 model is equivalent to the TT model for third-order tensors.
  • Extensions: Generalized Tucker replaces factor matrices with higher-order tensors and can lead to hierarchical Tree Tensor Networks State models.Kronecker-product identities provide equivalent matrix and vector representations.
  • Applications: When a tensor has low multilinear rank, applying FFTs to its smaller factor matrices instead of the full tensor considerably reduces computational complexity.This procedure is called the super fast Fourier transform in Tucker format.

3.4 Higher Order SVD (HOSVD) for Large-Scale Problems

HOSVD is an orthogonal Tucker model for large-scale tensors, computed from mode unfoldings using standard or randomized SVD procedures. Sequential, partitioned, and iterative variants address memory and approximation challenges.

  • Definition and structure: HOSVD uses orthogonal factor matrices and an all-orthogonal core tensor, extending the constrained Tucker decomposition to higher-order data.Its core slices are mutually orthogonal, and multilinear singular values guide truncation.
  • Truncated HOSVD: Truncated HOSVD discards components associated with small multilinear singular values and applies optimized matrix SVD procedures to obtain an approximation.The approximation is formed from a truncated core and orthogonal factor matrices.
  • Large-scale computation: Large unfolding matrices may exceed computer memory, so sequential divide-and-conquer SVD partitions them into slices sized for available memory.Sequential access avoids loading the entire unfolding into memory.
  • Large-scale computation: Randomized SVD reduces large low-rank matrices through random sketching, while subsampled random Fourier transforms can achieve O(IJ log(R)) execution complexity.The approach may be insufficiently accurate when singular values decay slowly.
  • Iterative refinement: HOOI improves Tucker approximations through an ALS procedure initialized by HOSVD and updated using leading eigenvectors of projected covariance matrices.The resulting factor matrices remain orthogonal.
  • Approximation limits: Low multilinear-rank approximation is well-posed, but truncated HOSVD is quasi-best rather than the best multilinear-rank approximation.The best approximation can instead be pursued with ALS-type HOOI algorithms.
  • Applications: HOSVD can jointly perform subspace selection and K-means clustering, and it provides a framework for anomaly detection through projected lower-dimensional structure.For very large datasets, anomaly analysis can use decompositions of sampled blocks and changes in factor matrices.

3.5 Tensor Sketching Using Tucker Model

Tensor sketching replaces huge tensors with substantially smaller approximating tensors to make storage and computation feasible. The reviewed approaches include sparsification, random projections, fiber selection, iterative methods, and distributed sketching.

  • Sketching concept: A tensor sketch is a substantially smaller matrix or tensor that approximates the original while reducing memory and processing requirements.Sketch computation may be distributed across multiple processors when the full dataset is infeasible to handle on one system.
  • Approaches: Reviewed tensor sketching approaches include sparsification, random projections, fiber subset selection, iterative sketching, and distributed sketching.These methods target efficient analysis of very large tensors.
  • Approaches: Sparsification stores tensors more efficiently and speeds factor-matrix multiplication by reducing nonzero entries through thresholding, quantization, or rounding.Element-wise sparsification removes sufficiently small tensor elements while retaining sufficiently large ones.

3.6 Tensor Sketching via Multiple Random Projections

Tensor sketching uses lower-dimensional random projections along selected modes to construct compact sketches of large tensors. For sufficiently low multilinear-rank tensors, these sketches can support perfect reconstruction, while projection distributions trade computational, storage, stability, and accuracy properties.

  • Sketch construction: Random projections compute structured low-rank tensor approximations from linear projections much smaller than the original data tensor.The framework targets large-scale numerical multilinear algebra and optimization problems.
  • Sketch construction: Full and mode-wise sketches are formed by multiplying the tensor with independent test matrices along all modes or all modes except one.The test matrices Ω_n have dimensions R_n × I_n, with R_n < I_n.
  • Design trade-offs: Projection distributions create tradeoffs in randomization, computation, communication, storage, arithmetic cost, numerical stability, and a priori error bounds.Gaussian and orthonormal-column projections are highlighted for practical performance and stability, while sparse Rademacher variants reduce storage and computational costs.
  • Sketch construction: A third-order tensor yields subtensor sketches such as I_1R_2R_3 or R_1R_2R_3, depending on which modes are projected.Projecting all three modes produces the smallest listed sketch, R_1R_2R_3.
  • Reconstruction: Sufficiently low-multilinear-rank tensors can be perfectly reconstructed from the sketch tensors.The reconstruction uses the full sketch Z and mode-specific factor matrices B^(n).

3.7 Matrix/Tensor Cross-Approximation (MCA/TCA)

Matrix and tensor cross-approximation reduce large data objects using selected columns, rows, fibers, and a small intersection core. Under low-rank conditions, these constructions can be exact or close to optimal, while selecting maximum-volume submatrices is computationally difficult.

  • MCA: Matrix Cross-Approximation factorizes a huge matrix using selected columns and rows together with a small core matrix.The method is also known as pseudo-skeleton or CUR decomposition.
  • MCA: The core matrix can be absorbed into either the selected-column or selected-row factor, yielding CR or column/row subset factorizations.The resulting bases are selected columns or rows of the original matrix.
  • MCA: For dimensionality reduction, columns and rows are selected for high statistical leverage, good low-rank fit, and low reconstruction error.The exact optimal core requires access to all matrix entries, which is impractical for large-scale data.
  • Limitations: Maximum-volume intersection submatrices make matrix cross-approximation close to optimal SVD, but finding them has exponential complexity.Fast greedy algorithms can find suboptimal matrices instead.
  • TCA: Tensor Cross-Approximation generalizes MCA through approaches including matricization and direct operation on tensor fibers.Fiber Sampling Tucker Decomposition uses a relatively small number of data-tensor fibers.
  • TCA: For an exact rank-(R_1,R_2,R_3) tensor, Fiber Sampling Tucker Decomposition selects P_n ≥ R_n indices per mode and forms an exact Tucker representation from fibers and an intersection subtensor.The core is computed from the selected intersection subtensor W.
  • TCA: A low multilinear-rank Nth-order tensor can be fully reconstructed from N fiber-based factor matrices and a core when the selected subtensor has matching multilinear rank.The selected dimensions satisfy P_n ≥ R_n for every mode.
  • TCA: Random projections provide an alternative way to estimate the subtensors and fibers used by tensor cross-approximation.The projection dimensions P_n are chosen at least as large as the target multilinear ranks R_n.

3.8 Multiway Component Analysis (MWCA)

Multiway Component Analysis extends component analysis and blind source separation to tensors by imposing mode-specific constraints. Tucker models accommodate orthogonality, independence, sparsity, smoothness, and nonnegativity across different modes.

  • Motivation: Tensor factorization requires suitable component constraints to obtain physically meaningful representations.Different constraints can be imposed on matrix factorizations of each matricized tensor mode.
  • Model: MWCA uses Tucker representations to combine different constraints across modes, including orthogonality, statistical independence, sparsity, smoothness, and nonnegativity.This flexibility accommodates distinct component properties in different modes.
  • Procedure: A Tucker-N MWCA procedure reduces and matricizes each mode, applies suitable two-way component-analysis or blind-source-separation algorithms, and optionally fine-tunes the factors and core.The core tensor models complex links among components in different modes.

3.9 Analysis of Coupled Multi-block Matrix/Tensors – Linked Multiway Component Analysis (LMWCA)

Linked Multiway Component Analysis jointly analyzes coupled multi-block tensors by separating components shared across blocks from block-specific components. It extends linked component-analysis approaches and can be generalized to tensor-network cores.

  • PVD: Population Value Decomposition is a special same-order, same-size case that can use common factor matrices and reduced-size block cores, and is equivalent to Tucker-2.With square diagonal cores, the model becomes equivalent to CP decomposition.
  • Generalization: Higher-order tensors can be reshaped and permuted into skinny third-order tensors so that PVD or Tucker-2 algorithms can be applied.The resulting third-order representation groups higher modes into one dimension.
  • Constraints: Constraints such as sparsity, nonnegativity, smoothness, and orthogonality can be imposed on common factor matrices to obtain meaningful and unique components.An SVD/QR-based orthogonal PVD algorithm is presented, but it is not optimal in the stated sense.
  • Motivation: Linked MWCA is designed for multi-modal, high-dimensional blocks that share latent components while retaining independent block-specific features.This setting motivates analyzing the blocks jointly rather than independently.
  • Model: The LMWCA model uses simultaneous Tucker-(1,N) decompositions with common mode-1 structure across coupled data tensors.The coupled tensors may differ in every mode except the shared first-mode dimension.
  • Components: LMWCA separates common components present across blocks from individual components representing block-specific processes.The number of common components C is constrained by the block ranks and is generally unknown.
  • Extensions: LMWCA complements group component analysis, PVD, and CCA/PLS, and its concept generalizes to linked tensor-train and hierarchical Tucker networks.The generalized tensor-network objective is to identify as many strongly correlated cores as possible.

3.10 Nonlinear Tensor Decompositions – Infinite Tucker

Infinite Tucker models extend Tucker decomposition to infinite-dimensional feature spaces through kernel and Bayesian approaches, while distributed variants improve scalability by partitioning observed tensors.

  • Infinite Tucker generalizes standard Tucker decomposition to infinitely dimensional feature spaces using kernel and Bayesian approaches.
  • The tensor model uses factor matrices and a core tensor, with vectorization expressed through Kronecker products of factor matrices and the core.
  • Tensor-variate Gaussian processes replace standard covariance matrices with nonlinear kernel covariance structures to model latent-factor interactions.
  • Distributed Infinite Tucker splits noisy observations into subtensors, assigns each a Gaussian-process model, and merges factor matrices through a similarity-controlling prior.
  • Sparse Gaussian-process variational inference is introduced because exact inference can remain intractable for tensors with many nonzero and zero entries.
  • For noisy tensors, Bayesian formulations place priors on the core and model observations with additive Gaussian noise.

Tensor Train Decompositions: Graphical Interpretations and Algorithms

Tensor Train representations express high-order tensors as chains of low-order cores in multiple equivalent scalar, slice, tensor, vector, and graphical forms. Quantization can yield small ranks and logarithmic storage complexity, while TT/MPO extends the framework to structured matrices.

  • Tensor Train representations: TT decomposition represents an Nth-order tensor through sparsely connected cores, with equivalent scalar, slice, tensor, vector, and graphical formulations.
  • Tensor Train representations: The TT tensor form contracts 3rd-order cores G(n) with boundary ranks R0 = RN = 1.
  • Tensor Train representations: TT vector forms use Kronecker or strong Kronecker products of core fibers and block matrices to represent vec(X).
  • Tensor Chain: Tensor Chain provides analogous representations to TT but uses matrix product states with periodic boundary conditions.
  • Matrix Product Operators: TT/MPO represents a structured matrix as a 2N-th-order tensor and a chain of 4th-order cores, equivalently merging each index pair into one vector-TT index.
  • Matrix Product Operators: Strong Kronecker products of block matrices support TT matrix construction and efficient operations on relatively small blocks.
  • Quantized Tensor Train: QTT ranks are usually below 20 and nearly independent of data size, reducing storage from O(I^N) to O(NR^2 log_q(I)).
  • Quantized Tensor Train: QTT introduces virtual dimensions that improve separability for structured data, enabling highly compressible approximations and efficient multilinear algebra.

4.5 Basic Operations in TT Formats

TT formats implement core tensor operations through small block or slice matrices, replacing raw-tensor computations with procedures whose costs depend on tensor order, mode size, and TT ranks.

  • 4.5 Basic Operations in TT Formats: Basic TT operations, including addition, inner products, norms, Hadamard products, Kronecker products, and matrix multiplications, operate on small core tensors.
  • 4.5 Basic Operations in TT Formats: TT addition concatenates corresponding cores and yields ranks bounded by the sums of the input ranks.
  • 4.5 Basic Operations in TT Formats: Hadamard products increase TT ranks to at most R_n times the corresponding input ranks, while reducing complexity from O(I^N) to O(IN(RR̃)^2).
  • 4.5 Basic Operations in TT Formats: Fourier, wavelet, cosine, and related linear transformations can be applied to individual cores when the tensor admits a low-rank TT approximation.
  • 4.5 Basic Operations in TT Formats: Inner products are evaluated by sequentially contracting cores and reduce complexity from O(I^N) to O(NI(R^2R̃ + RR̃^2)).
  • 4.5 Basic Operations in TT Formats: TT matrix-by-vector and matrix-by-matrix products use the C product of block matrices to construct output cores from corresponding input cores.
  • 4.5 Basic Operations in TT Formats: TT matrix and tensor operations are illustrated through network and tabular representations for equations, cost functions, and core-level algorithms.
  • 4.5 Basic Operations in TT Formats: A huge quadratic form z = x^T A x can be computed by first forming y = Ax in TT format and then evaluating x^T y.

4.6 Algorithms for TT Decompositions

TT decomposition algorithms construct low-rank cores sequentially using truncated or alternative matrix factorizations, while rounding controls rank growth after operations. These procedures provide efficient approximate representations with prescribed accuracy.

  • 4.6 Algorithms for TT Decompositions: TT algorithms represent high-order tensors and their low-rank approximations through efficient core-level operations with prescribed accuracy.
  • 4.6 Algorithms for TT Decompositions: Alternative low-rank matrix factorizations, including randomized SVD, CUR, RPCA, and related methods, can replace standard SVD in TT construction.
  • 4.6 Algorithms for TT Decompositions: TT-SVD sequentially applies truncated SVD to unfolding matrices, reshaping factor matrices into TT cores until the tensor train is complete.
  • 4.6 Algorithms for TT Decompositions: TT-Cross uses adaptive cross-approximation instead of computationally expensive SVD, with cross-approximation complexity scaling linearly in tensor order N.
  • 4.6 Algorithms for TT Decompositions: A Tucker-2/PVD approach extracts the first and last cores, then repeats the procedure on reshaped intermediate tensors.
  • 4.6 Algorithms for TT Decompositions: TT rounding recompresses overestimated ranks using QR/SVD while enforcing an error tolerance or maximum-rank bound.
  • 4.6 Algorithms for TT Decompositions: TT-rounding has complexity O(NIR^3) because SVD is applied only to relatively small matricized core tensors.
  • 4.6 Algorithms for TT Decompositions: TT sub-trains and interface matrices divide large-scale tasks into smaller subproblems for extracting individual cores or their slices.

L P RRm´1ImˆRm

The ASCU algorithm improves TT decomposition by sequentially optimizing individual cores while adjusting adjacent cores and maintaining orthogonality. Its updates use contractions and Tucker-2 or truncated-SVD subproblems to control TT ranks and approximation accuracy.

  • ASCU algorithm: ASCU sequentially optimizes one TT core while keeping the other cores fixed, similarly to modified alternating least squares.The method is described as an efficient TT decomposition algorithm.
  • ASCU algorithm: The algorithm contracts all modes except the updated mode, using left contractions before the core and right contractions after it.These contractions provide the local tensor used for core optimization.
  • Rank adjustment: Each update can adjust both adjacent TT ranks through a Tucker-2 decomposition that minimizes the local approximation error under rank constraints.The optimized core is embedded back into the Nth-order TT representation through factor matrices.
  • Algorithmic procedure: ASCU alternates left-to-right and right-to-left sweeps, applying orthogonalization and updating contracted tensors until a stopping criterion is met.The resulting TT cores satisfy left- and right-orthogonality conditions.
  • Rank adjustment: One-sided rank adjustment instead reduces the local problem to a truncated SVD of a matricized contracted tensor.The selected singular vectors determine the updated rank and reshaped TT core.

Discussion and Conclusions

Part 1 presents tensor networks as a systematic framework for compressing and analyzing extreme-scale multidimensional data. The authors position tensor networks as a bridge from multilinear representations to scalable optimization, with distributed optimization developed in Part 2.

  • Discussion and Conclusions: Tensor networks provide intrinsic compression through distributed data representation and information processing for extreme-scale multidimensional data.The conclusion frames this compression ability as central to their promise for dimensionality reduction and analysis.
  • Discussion and Conclusions: Unlike flat matrix representations, tensor networks model strong and weak couplings among multiple variables and accommodate multimodal, incomplete, and noisy data.Their mathematical foundation is multilinear algebra.
  • Discussion and Conclusions: Part 2 introduces a distributed framework that transforms huge-scale optimization into linked small-scale subproblems of the same type.This framework is intended to connect standard numerical algorithms with very large-scale optimization.
  • Discussion and Conclusions: The monograph is presented as a foundation for further machine-learning studies involving data with otherwise prohibitively large volume, variety, or veracity.The authors also describe a transition from numerical linear algebra to numerical multilinear algebra.
Loading 1609.00893v3…