Source-linked AI summary
Tensor Decompositions for Signal Processing Applications From Two-way to Multiway Component Analysis
A. Cichocki, D. Mandic, A-H. Phan, C. Caiafa, G. Zhou, Q. Zhao, L. De Lathauwer
TL;DR
The paper addresses the limitations of flattened matrix models for multi-sensor and large multiway datasets. It develops tensor decompositions and multilinear-algebra tools, showing relaxed uniqueness conditions, flexible modeling, and computational strategies for large tensors. The authors also identify probabilistic tensor models as an important extension beyond their independently valued-entry models.
Problem
Flat matrix models do not adequately represent multiway data with multiple interactions, while high-order tensors create storage and computation bottlenecks.
Method
The paper develops Canonical Polyadic, Tucker, and tensor-network decompositions alongside multilinear generalizations of signal-processing methods.
Results
Tensor decompositions provide relaxed uniqueness conditions, model interactions among complex latent components, and can reduce computation while preserving reconstruction quality.
Takeaways & Limitations
Tensor decompositions offer compact, flexible tools for component analysis of multi-sensor, multimodal, and tensorized vector or matrix data.
Takeaways & Limitations
The discussion is limited to tensor models whose entries vary independently, excluding probabilistic models for prior knowledge, data alphabets, or noise distributions.
Abstract
from arXiv · showhide
The widespread use of multi-sensor technology and the emergence of big datasets has highlighted the limitations of standard flat-view matrix models and the necessity to move towards more versatile data analysis tools. We show that higher-order tensors (i.e., multiway arrays) enable such a fundamental paradigm shift towards models that are essentially polynomial and whose uniqueness, unlike the matrix methods, is guaranteed under verymild and natural conditions. Benefiting fromthe power ofmultilinear algebra as theirmathematical backbone, data analysis techniques using tensor decompositions are shown to have great flexibility in the choice of constraints that match data properties, and to find more general latent components in the data than matrix-based methods. A comprehensive introduction to tensor decompositions is provided from a signal processing perspective, starting from the algebraic foundations, via basic Canonical Polyadic and Tucker models, through to advanced cause-effect and multi-view data analysis schemes. We show that tensor decompositions enable natural generalizations of some commonly used signal processing paradigms, such as canonical correlation and subspace techniques, signal separation, linear regression, feature extraction and classification. We also cover computational aspects, and point out how ideas from compressed sensing and scientific computing may be used for addressing the otherwise unmanageable storage and manipulation problems associated with big datasets. The concepts are supported by illustrative real world case studies illuminating the benefits of the tensor framework, as efficient and promising tools for modern signal processing, data analysis and machine learning applications; these benefits also extend to vector/matrix data through tensorization. Keywords: ICA, NMF, CPD, Tucker decomposition, HOSVD, tensor networks, Tensor Train.
L. De Lathauwer
The paper positions tensor decompositions as a richer alternative to flattened matrix analysis for multiway data, extending component-analysis tools through multilinear algebra.
- The article develops tensor-based generalizations of signal-processing methods including ICA, NMF, multilinear SVD, and PLS.
- Its perspective connects tensor analysis to signal separation, feature extraction, classification, and heterogeneous sensor modalities.
- Multiway data often require tensor representations because flattened two-way models cannot capture multiple interactions and couplings.
- Tensor decompositions differ structurally from matrix factorizations, with subtler rank, more relaxed uniqueness conditions, and distinct geometric properties.
2 Notations and Conventions
This section introduces tensors as multi-index arrays and establishes notation for their modes, entries, subtensors, fibers, slices, and reshaping operations.
- A tensor is a multi-index numerical array whose order counts modes such as space, time, frequency, trials, classes, or dictionaries.
- Vectors and matrices are tensors of order one and two, while fibers and slices are vector- and matrix-valued subtensors.
- Tensor manipulation includes reshaping, matrix unfolding, and mode-n multiplication of all vector fibers by a matrix.
- The section provides basic notation and product definitions used throughout the paper.
3 Interpretable Components in Two-Way Data Analysis
Two-way latent-variable analysis decomposes a data matrix into interpretable factors, loadings, scaling, and noise, but requires constraints to resolve ambiguities.
- Matrix factorization represents data as scaled rank-1 components formed from factor matrices A and B plus an error term E.
- The columns of B represent source signals or latent variables, while columns of A represent mixing vectors or factor loadings.
- Matrix models have arbitrary scaling and permutation indeterminacies, and unconstrained factorizations admit infinitely many factor combinations.
- Standard QR, EVD, and SVD factorizations obtain uniqueness through restrictive conditions such as triangularity and orthogonality.
4 Tensorization — Blessing of Dimensionality
Tensorization converts lower-dimensional, mathematically constructed, experimentally organized, or naturally multiway data into tensors whose dimensionality can provide compact representations and broader component models.
- Tensorization is the procedure of creating a data tensor from lower-dimensional original data.
- Large vectors or matrices can be rearranged into higher-order tensors and compressed when they admit low-rank tensor approximations.
- Mathematical constructions include moment tensors, covariance slices, and time-frequency or wavelet transformations of multichannel data.
- Experiment design can stack multifaceted observations, while tensor structures also arise naturally in images, videos, light fields, and discretized multivariate functions.
- Tensorization supports compact representations, decomposition uniqueness, flexible constraints, and more general identifiable components.
5 Canonical Polyadic Decomposition
The Canonical Polyadic Decomposition (CPD) expresses a tensor as a sum of rank-1 terms or, equivalently, a multilinear product with a diagonal core. Its uniqueness under relatively mild conditions supports signal separation, while computation and modeling require attention to noise, constraints, and rank-related pathologies.
- Definition and form: CPD represents an Nth-order tensor as a linear combination of rank-1 tensors, with the minimum exact number of terms defining the canonical decomposition.It is also called CANDECOMP/PARAFAC decomposition and is desired in signal separation.
- Definition and form: Equivalently, CPD writes the tensor as a multilinear product of factor matrices and a diagonal core containing the component weights.The diagonal-core view generalizes the matrix factorization interpretation to higher-order tensors.
- Rank and estimation: Tensor rank differs fundamentally from matrix rank: determining it is generally NP-hard, and complex-valued and real-valued rank-1 representations can require different numbers of terms.Practical rank assessment often targets the number of components recoverable with sufficient accuracy rather than exact rank determination.
- Uniqueness: Kruskal’s condition kB(1) + kB(2) + kB(3) ≥ 2R + 2 guarantees third-order CPD uniqueness up to scaling and permutation ambiguities.The Kruskal rank is the largest number of columns guaranteed to be linearly independent in every subset of that size.
- Uniqueness: Compared with matrix decompositions, CPD can be unique under relaxed conditions requiring components to be sufficiently different and their number not unreasonably large.These conditions are central to tensor-based signal separation and can be further relaxed when a factor matrix has full column rank.
- Computation: CPD computation uses explicit algebraic methods under suitable conditions or numerical fitting, commonly through alternating least squares and all-at-once optimization.ALS is simple and effective for a few well-separated components at sufficiently high SNR, but lacks guaranteed convergence; all-at-once methods are costlier and often more robust for ill-conditioned cases.
- Modeling limitations: A best rank-R tensor approximation is not always guaranteed to exist, so optimization may approach an unattained infimum with diverging, mutually canceling rank-1 terms.This behavior can indicate an underestimated component count, non-rank-1 tensor components, or excessive noise.
- Constraints and applications: Prior constraints such as statistical independence, sparsity, and orthogonality can improve noisy-data accuracy and robustness, aid interpretation, relax uniqueness conditions, or simplify computation.CPD is used for signal separation across audio, speech, biomedical, chemometric, and machine-learning applications, including algebraic ICA.
6 Tucker Decomposition
Tucker decomposition represents a tensor through mode-specific factor matrices and a typically smaller core tensor, enabling subspace extraction, compression, and flexible mode-wise constraints. Its factor bases are generally non-unique, although the represented subspaces are unique.
- Tucker representation: Tucker decomposition expresses X as a multilinear product of a core tensor G and factor matrices B^(n).The equivalent matricized form uses Kronecker products of the factor matrices.
- Multilinear rank: The multilinear rank (R1, R2, ..., RN) records the dimensions of the mode-wise subspaces and can differ across modes.This differs from matrix rank because higher-order tensors may have distinct ranks for each mode.
- Uniqueness: Unconstrained Tucker factors are rotation invariant, so the factor bases are non-unique while their column spaces remain unique.Post-multiplying factors by invertible matrices can be compensated by transforming the core with their inverses.
- Multilinear SVD and approximation: MLSVD obtains orthonormal mode-wise bases through SVDs of matricized tensors, with core-slice norms serving as multilinear singular values.Discarding components associated with small multilinear singular values yields a well-posed low multilinear rank approximation, though truncation is not always least-squares optimal.
- Multiway Component Analysis: MWCA combines different constraints across modes, such as orthogonality, nonnegativity, sparsity, and statistical independence, to match heterogeneous component properties.The illustrated example uses principal and orthogonal components in one mode, nonnegative and sparse components in another, and statistically independent components in the third.
- Applications: Tucker decomposition generalizes PCA-like subspace methods and can support classification, feature extraction, and harmonic retrieval.A low multilinear rank approximation may produce a higher SNR than the original raw tensor.
7 Block Term Decompositions
Block Term Decompositions extend CPD by modeling components with low multilinear rank rather than rank-1 structure. In correlated-source separation, matching the component structure enabled recovery with few samples where PCA, ICA, CPD, and Tucker methods had specific limitations.
- Definition and model structure: BTDs replace CPD’s rank-1 terms with components having low multilinear rank, allowing structurally more complex signal components.The illustrated forms include multilinear ranks (Lr, Lr, 1) and (Lr, Mr, Nr).
- Comparison with CPD: Rank-2 CPD estimated and inverted a mixing matrix, but its accuracy was compromised because the tensor components were not rank-1 representable.The experiment used five mixtures of two correlated, short-duration sources contaminated by white Gaussian noise.
- Comparison with PCA, ICA, and Tucker: Tucker approximation with multilinear rank (4, 4, 2) recovered the mixing-matrix column space but not individual mixing vectors because Tucker decomposition is non-unique.PCA failed under nonorthogonal mixing and correlated sources, while ICA failed because the signals were not statistically independent.
- Source separation: BTD models terms with multilinear rank (2, 2, 1) and matched the structure of the correlated-source separation data.The sources were recovered using as few as 6 samples in the noise-free case.
- Source separation: 6 samples sufficed for source recovery with multilinear-rank-(2, 2, 1) BTD in the noise-free case.This result was reported for the short-duration correlated-source experiment.
8 Higher-Order Compressed Sensing
Higher-order compressed sensing preserves tensor structure while reducing acquisition, storage, and computation burdens. Kronecker-structured dictionaries and sparse Tucker cores support recovery and completion, with a hyperspectral example achieving the same PSNR in substantially less computation than conventional Kronecker-OMP.
- Compressed sensing: Compressed sensing reconstructs a signal from M measurements with M < I by seeking a sparse representation in a known dictionary.Greedy OMP and ℓ1-norm methods are practical alternatives to direct ℓ0-norm minimization, which is not practical.
- Motivation: Vectorizing tensors can lose spatial and contextual relationships while making the associated dictionary prohibitively large.These are the two stated obstacles when extending compressed sensing to tensor data.
- Kronecker-CS: Kronecker compressed sensing replaces one large global dictionary with smaller mode-wise dictionaries in a Tucker representation with a sparse core.The tensor model seeks a sparsest core G subject to mode-wise dictionaries and sensing matrices.
- Computational benefits: Kronecker structure reduces storage requirements and permits efficient matrix-vector operations through products involving much smaller factor matrices.The text gives a storage reduction factor involving the mode dimensions and sensing dimensions.
- Structured sparsity: A sparse core localized in a small subtensor can provide additional computational advantages for higher-order OMP.This structure is identified as relevant to hyperspectral imaging and 3D astrophysical signals.
9 Large-Scale Data and Curse of Dimensionality
High-order tensors create severe storage and computation bottlenecks, so tensor decompositions use compressed representations and scalable computation strategies to manage large datasets.
- The curse of dimensionality: The curse of dimensionality makes an Nth-order tensor with equal mode size require I^N values, causing storage and computation to grow exponentially with order N.Tensor decompositions can also serve as efficient representation tools for these otherwise unmanageable datasets.
- Compressed representations: A CP representation reduces storage from I^N raw entries to NIR parameters, which scales linearly with tensor order N.The rank R can be selected according to the desired approximation accuracy, although CP approximation may involve numerical problems.
- Compressed representations: Tucker decomposition compresses an Nth-order tensor from I^N entries to factor and core parameters, with matrix SVD providing a reliable approximation when N is low.The resulting compression ratio is approximately (I/R)^N under the stated equal-dimension setting.
- Tensor networks: Tensor networks represent high-order tensors through sparsely interconnected low-order cores and matrices, providing highly compressed representations with numerically reliable approximation algorithms.Hierarchical Tucker and Tensor Train are examples of tensor networks.
- Tensor networks: Tensor networks can control approximation error to achieve desired accuracy and represent discretized multivariate functions with more values than the number of atoms in the universe.This illustrates the scale of functions that can be represented without explicitly storing every value.
- Computing decompositions: Large-scale computation can use blockwise parallel decomposition, compressed sensing, or sampled tensor fibers to reduce acquisition, manipulation, storage, and computation costs.Block decompositions are recombined into global components, while fiber methods select factor-matrix columns directly from tensor fibers.
10 Multiway Regression — Higher Order PLS (HOPLS)
Higher Order PLS extends partial least squares from matrices to multiway predictors and responses, preserving shared predictive structure while modeling interactions that unfolding can obscure.
- Partial Least Squares: PLS estimates shared latent scores between predictors X and responses Y, balancing X-data fit against Y-prediction when selecting the number of components.The score vectors fit X while their correlation with response scores supports prediction of Y.
- Partial Least Squares: Sequential orthogonal projections and deflation continue until the X-block rank is exhausted, balancing prediction accuracy against model order.The rank of Y need not decrease with each newly extracted score.
- Tensor generalizations of PLS: Flattening tensors enables matrix PLS but obscures multiway structure and compromises interpretation of latent components.N-PLS instead uses low-rank tensor approximations with rank-one tensor terms.
- Higher Order PLS: HOPLS models independent and dependent tensors through sequential approximate decompositions that maximize similarity between their score vectors.Its block-diagonal core tensors provide a BTD-type structure for components correlated in the first mode.
- Higher Order PLS: HOPLS predicts responses for new tensors through multilinear projections using newly computed scores and existing model parameters.The framework generalizes regression and similarity-analysis paradigms including CCA, ANOVA, ARMA, and LDA.
- Example: arm-movement prediction: In ECoG arm-movement decoding, tensor-based HOPLS outperformed standard PLS because it modeled interactions between complex latent components of predictors and responses.Standard PLS predicted individual marker movements only crudely because it did not adequately account for mutual information among the four markers.
11 Linked Multiway Component Analysis and Tensor Data Fusion
Linked multiway component analysis jointly decomposes multiple data views into shared and dataset-specific factors, enabling constrained fusion and classification based on common features.
- Data fusion: Data fusion jointly analyzes multiple views of a phenomenon, including visual and thermal images or electrophysiological recordings from different subjects and trials.Matrix and tensor factorizations provide tools for analyzing these linked datasets together.
- Linked multiway component analysis: LMWCA performs approximate joint Tucker decompositions of multiple tensors while separating shared factors from tensor-specific factors.The model estimates common components and dataset-specific components in each mode.
- Linked multiway component analysis: LMWCA captures mutual interactions among common components through dataset-specific core tensors and permits constraints such as independence, sparsity, orthogonality, and non-negativity.These constraints can be chosen to match the properties of the data.
- Linked multiway component analysis: Compared with group ICA and Independent Vector Analysis, LMWCA avoids assuming that only common components exist and that corresponding latent variables are statistically independent.The paper characterizes both assumptions as stringent and limiting.
- Example: object classification: In ETH-80 object classification, LMWCA uses common features for discrimination across 3280 images in 8 categories, each containing 10 objects with 41 views per object.The associated figure emphasizes classification from common features when the training set is small.
12 Software
Several software toolboxes support tensor decompositions, constrained factorization, multiway analysis, and tensor-network computation across signal processing and scientific-computing applications.
- Available software: The Tensor Toolbox supports basic operations on sparse and dense tensors, including CPD and Tucker formats.It is presented as a versatile framework for tensor operations.
- Available software: TDALAB and TENSORBOX provide user-friendly interfaces and advanced algorithms for CPD, nonnegative Tucker decomposition, and MWCA.These tools target both accessibility and advanced multiway-analysis workflows.
- Available software: Tensorlab offers numerical algorithms for CPD, BTD, and Tucker decompositions, with constraints and joint factorization of dense, sparse, and incomplete tensors.Its constraint library includes nonnegativity and orthogonality.
- Available software: The N-Way Toolbox includes constrained CPD, Tucker decomposition, and PLS for chemometrics, with support for constraints and missing elements.Examples of supported constraints include nonnegativity and orthogonality.
- Available software: The TT Toolbox, Hierarchical Tucker Toolbox, and Tensor Calculus library provide tools for tensor scientific computing.These resources complement decomposition toolboxes focused on multiway data analysis.
13 Conclusions and Future Directions
The paper presents tensor decompositions as tools for exploratory analysis of multifaceted, multi-sensor, and multi-modal data, offering possibilities beyond flat two-way methods. It identifies unresolved computational, theoretical, modeling, automation, and scalability challenges for future work.
- Tensor decompositions match multifaceted datasets and support exploratory analysis in multi-sensor and multi-modal signal processing.
- Overdetermined tensor representations provide flexibility for hidden-component discovery, robustness to noise, tolerance to missing samples and faulty sensors, and deterministic analysis of short signals.
- Uniqueness conditions and efficient algorithms exist for standard tensor models, but several advanced applications still require deeper theoretical support.
- Coupled decompositions for multiview datasets need theory covering existence, uniqueness, and numerical properties.
- Advanced tensor models require efficient iterative algorithms beyond the ALS class, while component-number and dimensionality estimation need automation under noise and outliers.
- Future work should extend constraint flexibility, develop efficient ultra-large-tensor storage and processing, and improve performance analysis and probabilistic modeling.Ultra-large tensors already occupy terabytes and are expected to require petabytes of memory; probabilistic decompositions can incorporate variable interactions, data alphabets, or noise distributions.