Source-linked AI summary
Learning Relevant Features of Data with Multi-scale Tensor Networks
E. M. Stoudenmire
TL;DR
The paper asks how physics-inspired coarse graining can produce useful, scalable representations for high-dimensional learning while avoiding costly direct feature-space optimization. It constructs layered tree tensor networks from unsupervised data statistics, then optimizes only a top tensor for supervised tasks, achieving strong MNIST and fashion-MNIST performance.
Problem
Direct optimization in feature space can be prohibitive because standard coefficient-based approaches scale quadratically or worse with training-set size.
Method
The paper uses unsupervised, iterative coarse graining to build a truncated tree tensor network, then optimizes only its top tensor for supervised learning.
Results
The approach achieves 97.44% test accuracy on MNIST and 88.97% test accuracy on fashion-MNIST in the reported experiments.
Takeaways & Limitations
Coarse-grained tensor-network representations can support supervised learning while reducing the feature space and retaining competitive performance on challenging data.
Abstract
from arXiv · showhide
Inspired by coarse-graining approaches used in physics, we show how similar algorithms can be adapted for data. The resulting algorithms are based on layered tree tensor networks and scale linearly with both the dimension of the input and the training set size. Computing most of the layers with an unsupervised algorithm, then optimizing just the top layer for supervised classification of the MNIST and fashion-MNIST data sets gives very good results. We also discuss mixing a prior guess for supervised weights together with an unsupervised representation of the data, yielding a smaller number of features nevertheless able to give good performance.
I. INTRODUCTION
The paper adapts physics-inspired coarse graining to machine learning through tensor-network representations of data, combining scalable computation with task-specific optimization at the network's top layer.
- Tensor networks: Tensor networks factor high-order tensors into contracted low-order tensors, enabling polynomial-cost operations that would otherwise face dimensionality barriers.This framework supports applications including compression, feature parameterization, and machine learning.
- Coarse graining: Coarse graining repeatedly removes small-scale structure while preserving larger-scale properties, motivating hierarchical representations for data.The paper connects this idea to renormalization-group methods, wavelet multiresolution analysis, and progressively coarser neural-network features.
- Model construction: The models combine an isometric tree tensor layer U, obtained unsupervised or from a mixed procedure, with a task-specific top tensor wℓ.In multitask settings, the top tensor includes an index running over labels.
- Relation to prior frameworks: The framework can be viewed as kernel learning with tensor-network weights or as a linear-activation, product-pooling neural architecture.The paper uses the kernel-learning perspective while constructing models resembling deep neural networks.
- Proposed approach: The proposed unsupervised procedure compresses high-dimensional data into a layered tree tensor network with controlled accuracy and scalable computation.The resulting representation can support learning tasks while leaving only the top tensor to be optimized for a specific task.
- Scalability: Training cost scales linearly with both training-set size and input dimension when the parameter count is fixed, while test evaluation is independent of training-set size.Stochastic optimization could reduce training-set scaling below linear.
II. MOTIVATION AND BACKGROUND
The paper motivates tensor-network learning by combining kernel methods' feature-space formulation with structured decompositions that make high-order representations more manageable.
- Representer theorem: The optimal kernel-learning weights lie in the span of training-set feature vectors, reducing the problem to coefficients associated with the training examples.This representer-theorem structure applies to many supervised tasks and some unsupervised tasks, including kernel PCA.
- Tensor-network background: Tensor-network methods provide established structures for compressing high-order tensors, including tensor trains, PEPS, tree tensor networks, and MERA.These networks differ in their representational trade-offs and computational properties.
A. Alternate Basis for Weight Parameters
The paper replaces direct feature-space parameterization with a truncated basis derived from the training data's covariance structure and represented efficiently as a tensor network.
- Motivation: Quadratic-or-worse optimization in training-set size can make coefficient-based kernel methods prohibitive for datasets containing millions of examples.Representer-theorem parameterization is useful but still scales with the training-set size.
- Reduced basis: A basis U can be obtained from the training-feature matrix through singular value decomposition, with small or zero singular-value directions discarded after truncation.The remaining parameters βn describe the weights in a reduced space, while U† maps the full feature space into it.
- Covariance diagonalization: The transformation U diagonalizes the feature-space covariance matrix ρ, whose eigenvectors associated with its largest eigenvalues define the retained directions.The paper notes that U can be computed efficiently in truncated form despite direct manipulation of the full matrix being poorly scalable.
- Truncation: Iterative truncation projects out directions where ρ has small or zero eigenvalues, rapidly reducing the space required for learning tasks.Computing only some tensor-network layers can still remove these low-significance directions.
- Task adaptation: Using matrices other than ρ can produce task-adapted tensor-network bases with fewer latent parameters without reducing model performance.The paper identifies mixed or alternative covariance choices as a route to task-specific representations.
B. Tensor Product Feature Maps
Tensor-product feature maps turn inputs into high-order feature tensors, making model weights high-order tensors that can be approximated by structured tensor networks.
- Feature construction: The full feature map is built from local feature maps and maps an N-dimensional input into a d^N-dimensional tensor-product feature space.Each local feature map has an index of dimension d.
- Weight structure: With tensor-product features, the model weights carry N indices of dimension d and therefore form an order-N tensor with d^N parameters.The setting of interest has small d and potentially hundreds or thousands of input dimensions N.
- Approximation: Directly storing or manipulating d^N parameters becomes infeasible as N grows, motivating tensor-network approximations of the optimal weights.This assumption makes the representation practical while retaining a structured approximation to the full tensor.
- Tensor decompositions: Tensor networks factor an order-N tensor into contracted low-order tensors, with known efficient algorithms for several network geometries.The paper lists MPS, PEPS, tree tensor, and MERA networks as examples.
- Design trade-offs: Different tensor-network geometries trade off manipulation complexity against the ability to represent higher-dimensional interactions and slowly decaying correlations.The paper frames network choice as a balance between computational tractability and representational capacity.
III. UNSUPERVISED COARSE GRAINING
The method unsupervisedly coarse-grains high-dimensional feature vectors by recursively learning local isometries that discard low-eigenvalue directions. Repeating this process yields a truncated tree tensor network that approximately diagonalizes the feature-space covariance matrix and provides reduced representations.
- Local coarse graining: Local isometries combine pairs of feature-space indices while projecting out subspaces associated with eigenvectors below a cutoff ϵ.The isometric constraint permits a unitary rotation followed by projection, with the output dimension no larger than the product of the input dimensions.
- Optimizing isometries: The optimal first isometry is obtained by diagonalizing the reduced covariance matrix ρ12 and retaining eigenvectors corresponding to its D largest eigenvalues.D can be selected so that the truncation error remains below a prescribed threshold ϵ.
- Layer construction: Each learned isometry coarse-grains the training feature vectors, after which new reduced covariance matrices are computed to construct successive layers.Calculations for different reduced covariance matrices and training-data summations can be performed in parallel.
- Tree tensor network: After log2(N) coarse-graining steps, the covariance matrix is approximately diagonalized with U represented as a tree tensor network.Without truncation, the procedure is equivalent to kernel PCA; with truncation, accuracy and efficiency depend on the data set.
- Downstream use: The resulting reduced features can be used for supervised learning, while the layered structure remains explicit unlike typical kernel PCA representations.The lower layers are computed from statistical properties of the data rather than supervised labels.
IV. SUPERVISED OPTIMIZATION OF THE TOP TENSOR
The supervised model keeps the unsupervised tree tensor network fixed and optimizes only a top tensor on coarse-grained features. On MNIST experiments, this approach achieved high training and test accuracy while reducing the feature space.
- Model setup: The supervised model combines fixed unsupervised isometry layers with a trainable top tensor operating on reduced feature representations.The coarse-grained inputs and top tensor occupy the same space, and the experiments use a quadratic cost function.
- Model setup: A shared tree network can support multiple supervised tasks by using a separate top tensor for each label or task.The authors report that one shared U was sufficient in the multi-task setting.
- Hyperparameters: The cutoff ϵ strongly affects bond dimensions, while model performance increases rapidly as the cutoff is lowered.Top-tensor optimization used conjugate gradients until convergence, typically requiring a few hundred iterations.
- Results: 97.44% test accuracy was obtained with cutoff ϵ = 10^-3, while optimizing only the top tensor produced C = 0.078.The corresponding training accuracy was 98.75%, with adaptive bond dimensions reaching 15 and top-index dimensions 107 and 151.
- Results: 98.08% test accuracy was obtained with cutoff ϵ = 6 × 10^-4, with C = 0.042 after optimizing the top tensor.The resulting top indices had sizes 328 and 444, reducing the original feature space of size 2784.
- Generalization: Good test-set generalization occurred without explicit regularization, although the authors state that more challenging data sets require further study.The reported interpretation concerns training only the top layer while deriving lower layers from data statistics.
V. MIXED TASK-SPECIFIC / UNSUPERVISED ALGORITHM
The mixed algorithm combines data covariance with covariance from prior supervised weights to adapt the tree tensors toward a task while retaining unsupervised information. On MNIST, this produced smaller representations without diminishing performance.
- Mixed covariance: The mixed method combines the feature-space covariance with a task-specific covariance derived from prior supervised weights.The resulting covariance is normalized and weighted by an empirical mixing parameter µ before determining the tree tensors.
- Mixed covariance: For µ = 0, the method recovers the unsupervised covariance, whereas µ > 0 adapts the tree tensors to both data and prior weights.At µ = 1, sufficiently small truncation can make the network reproduce the provided weights regardless of the data.
- Algorithm: Each mixed-algorithm layer coarse grains both the training data and the prior weights before computing the next reduced covariance.For MPS weights, this is implemented by contracting each tree isometry with pairs of MPS tensors.
- Algorithm: In multi-class tasks, the supervised covariance is formed by summing the covariance matrices associated with the prior weight MPS for each label.The labels may be represented as separate MPS weights or as one MPS with an external label index.
- MNIST result: Mixing supervised weights with data covariance yielded a task-adapted representation that could be compressed further without diminishing performance.The resulting models could also outperform the prior guess while retaining a partially unsupervised character.
VI. PARTIAL COARSE GRAINING: TREE CURTAIN MODEL
Partial coarse graining balances compression against expressiveness by leaving a high-order tensor for the top layer. The resulting tree-curtain model uses an MPS top tensor and achieved 88.97% test accuracy on fashion MNIST.
- Partial coarse graining: Computing fewer tree layers balances compressed representation against expressiveness lost through accumulated truncations.After R rescalings, the data tensors have order N_top = N/2^R.
- Partial coarse graining: The remaining high-order data tensor requires a top tensor of order N_top, or N_top + 1 for vector-valued multi-task output.This preserves a scalar or vector-valued final output after partial coarse graining.
- Tree-curtain architecture: An MPS top tensor produces a tree-curtain architecture that can be optimized after the training data pass through the tree layers.The top MPS can use established supervised or unsupervised learning methods.
- Fashion MNIST experiment: Fashion MNIST contains 28×28 grayscale images, ten labels, 60,000 training examples, and 10,000 test examples, with clothing photographs making classification harder than MNIST.The study used this dataset to test partial coarse graining.
- Fashion MNIST result: 88.97% testing accuracy was achieved by the optimized model, compared with 95.38% training accuracy.The reported test result was comparable to several standard approaches without preprocessing, though below the cited 93.7% GoogLeNet result.
VII. RELATED WORK
The paper connects its feature-space tensor-network method to kernel learning, tensor-network optimization, and neural architectures built from tree tensor networks. Its distinctive emphasis is constructing a low-rank approximation directly in feature space while preserving favorable scaling.
- Kernel learning: The feature-space covariance and kernel matrix have the same spectrum, linking the method to low-rank kernel approximations.The present work differs by constructing the low-rank approximation directly in feature space rather than in the dual formulation.
- Kernel learning: Unlike related kernel approximations, the method scales linearly with training-set size and input dimension while retaining a structured tensor-network model.The tree structure also permits experiments that optimize only the top layer.
- Tensor-network precedents: The layer-wise tree-network algorithm has precedents in tensor-network methods for density matrices, multigrid DMRG, quantum state tomography, and function reconstruction.These connections place the algorithm within a broader tensor-network optimization literature.
- Neural architectures: The approach can be viewed as an unsupervised adaptive training algorithm for convolutional arithmetic circuits, including variants with MPS upper layers.ConvACs are described as neural networks constructed from tree tensor networks.
VIII. DISCUSSION
The discussion frames tensor-network coarse graining as a physics-inspired approach to real-world data and highlights task-adapted representations as a central outcome. It also identifies architectural and analytical directions for future work.
- Interpretation: The method adapts coarse-graining ideas from statistical physics to data and resembles kernel PCA computed directly in feature space.The resulting representation can serve as a starting point for supervised or unsupervised tasks.
- Interpretation: The mixed covariance produced an improved representation, while alternative quantities besides covariance could provide other task-adapted bases.The discussion presents this as an open direction rather than a settled choice.
- Future architectures: Tree tensor networks have known coarse-graining deficiencies, motivating future investigation of MERA tensor networks.The proposed replacement is intended to address the limitations of the tree architecture as a coarse-graining scheme.
- Future improvements: The algorithms still have engineering room for improvement, including tree tensors with two-dimensional index groupings and feedback through the mixed algorithm.These suggestions concern architectural design and further adaptation of the tree tensors.
- Future analysis: A broader goal is to use renormalization-group analysis to identify data classes and architectures with suitable inductive biases.The discussion presents this as a desirable future outcome of analyzing datasets from that perspective.
Appendix A: Tensor Diagram Notation
Tensor diagram notation represents tensor networks graphically, encoding tensor order and index contractions through shapes and connecting lines. Its conventions emphasize connectivity and linear transformations while allowing complex operations to be expressed compactly.
- Tensor diagram notation represents networks of contractions of high-order tensors graphically.
- A tensor with n indices is drawn with n lines, each representing a specific index.
- Connecting lines between tensors indicate that the corresponding indices are contracted or summed over.
- The notation generally assigns meaning to network connectivity rather than line orientation, ordering, or assumed tensor symmetries.
- Every tensor-diagram element represents a linear transformation, unlike similar-looking neural-network diagrams.
- Diagrammatic notation compactly expresses operations such as traces, outer products, transposes, and matricization without additional symbols.
Appendix B: Representing a Linear Classifier as an MPS
An MPS can represent a linear classifier by choosing its tensors to reproduce the classifier under the local feature map φ(x) = [1, x]. The construction extends to general local feature maps, with bond dimension d + 1.
- An MPS-form model is guaranteed to equal a linear classifier with weights V under the local feature map φ(x) = [1, x].
- The construction maps linear-classifier parameters V into MPS weights by defining tensors for each fixed local feature value.
- A constant shift is included by adding a fictitious input component x_0 = 1 or adjusting the first A tensor.
- The mapping extends to d-dimensional local feature maps through an extended linear classifier representation.
- The resulting MPS has bond dimension d + 1.