Source-linked AI summary
Tree Tensor Networks for Generative Modeling
Song Cheng, Lei Wang, Tao Xiang, Pan Zhang
TL;DR
MPS generative models are limited by exponentially decaying correlations when modeling complex natural images. This paper develops a Tree Tensor Network Born machine with two-dimensional structure and sweeping learning and sampling algorithms, and reports stronger correlation modeling and likelihood performance than MPS.
Problem
MPS generative models have limited representation power for complex natural images because their correlations decay exponentially.
Method
TTN extends MPS generative modeling with a two-dimensional image structure, sweeping training, and sampling algorithms.
Results
TTN performs better than MPS on random-pattern training NLL and captures longer-range MNIST dependencies with better test-image NLL.
Takeaways & Limitations
TTN retains tractable normalization, canonical forms, and direct sampling while improving long-range-correlation modeling over MPS.
Takeaways & Limitations
Current tensor-network generative models lag state-of-the-art neural models, and better two-dimensional priors may require approaches such as PEPS with intractable exact partition functions.
Abstract
from arXiv · showhide
Matrix product states (MPS), a tensor network designed for one-dimensional quantum systems, has been recently proposed for generative modeling of natural data (such as images) in terms of `Born machine'. However, the exponential decay of correlation in MPS restricts its representation power heavily for modeling complex data such as natural images. In this work, we push forward the effort of applying tensor networks to machine learning by employing the Tree Tensor Network (TTN) which exhibits balanced performance in expressibility and efficient training and sampling. We design the tree tensor network to utilize the 2-dimensional prior of the natural images and develop sweeping learning and sampling algorithms which can be efficiently implemented utilizing Graphical Processing Units (GPU). We apply our model to random binary patterns and the binary MNIST datasets of handwritten digits. We show that TTN is superior to MPS for generative modeling in keeping correlation of pixels in natural images, as well as giving better log-likelihood scores in standard datasets of handwritten digits. We also compare its performance with state-of-the-art generative models such as the Variational AutoEncoders, Restricted Boltzmann machines, and PixelCNN. Finally, we discuss the future development of Tensor Network States in machine learning problems.
I. INTRODUCTION
Generative modeling learns a joint distribution from data and generates samples, but representing complex dependencies efficiently remains difficult. This paper applies Tree Tensor Networks (TTNs) to generative modeling, motivated by their suitability for two-dimensional data and longer-range pixel correlations than MPS.
- Generative modeling learns a joint probability distribution from training data and generates new samples, making it harder than discriminative modeling.
- Tensor Networks use low-rank decompositions to represent high-order objects with fewer parameters, including MPS and TTNs.
- Natural images may have sparse long-range correlations, motivating tensor-network representations that use relatively few parameters.
- TTN extends tensor-network generative modeling beyond MPS by naturally representing two-dimensional data and supporting more favorable pixel-correlation growth.
- The paper develops TTN learning and unbiased sampling algorithms, then evaluates the model on random binary patterns and binary MNIST.
- Experiments report better performance than tree-structured factor graphs and MPS while quantifying a gap with state-of-the-art neural generative models.
A. The data distribution and maximum likelihood learning
The learning objective fits a model distribution to the empirical distribution of binary training images. Negative log-likelihood is used as the cost, and minimizing it is equivalent to minimizing KL divergence up to a data-dependent constant.
- The training set contains |T| binary images, each represented as a length-n vector, defining an empirical data distribution.
- Generative learning seeks a parameter-efficient distribution p(x) that is as close as possible to the empirical distribution π{x}.
- Negative log-likelihood is introduced as the model-learning cost function.
- The NLL is bounded below by the dataset Shannon entropy because KL divergence is nonnegative.
- Minimizing NLL is equivalent to minimizing KL divergence because the Shannon-entropy term is independent of the model.
B. tree-structure Factor Graph as a generative model
The paper presents tree-structured factor graphs as tractable generative models whose hidden variables are summed out to obtain probabilities over visible pixels. Their tensor-network representation generalizes this construction through tensor contraction.
- A tree-structured factor graph models joint probabilities with visible pixel variables, hidden variables, and edge factor functions.
- Combining edge factors and summing over hidden states produces the unnormalized probability of a pixel configuration.
- For acyclic graphs, the sum-product algorithm computes normalization exactly without brute-force summation over 2^n configurations.
- Mapping factors to matrices and hidden nodes to identity tensors converts the tree factor graph into a tensor network.
- The TTN form uses general tensors rather than the restricted tensors of the original factor graph, extending the factor-graph model.
- Tensor contraction is more general than sum-product and can remain approximately applicable when sum-product no longer applies.
C. Tree Tensor Network Generative Model
The TTN generative model represents image probabilities through a tree of tensors connected by virtual bonds and physical pixel indices. Its Born-machine formulation and canonical form support tractable normalization and efficient contraction.
- The TTN extends the tree-structured factor graph using a first matrix, three-index tensors, virtual bonds, and physical bonds connected to image pixels.
- As a Born machine, TTN represents a quantum state Ψ(x), with p(x) induced by the squared wavefunction amplitude.
- The Born-machine representation is chosen for its expected expressivity and for the canonical form that simplifies and stabilizes contractions.
- TTNs have upper-, left-, and right-canonical forms depending on which tensor index remains open.
- QR-based canonicalization pushes non-canonical components toward a central tensor, producing a mixed canonical form.
- In canonical form, the normalization factor reduces to the squared norm of the central tensor.
D. Data representations
TTN supports both one-dimensional and image-aware two-dimensional data representations. The 2-D arrangement assigns tensors to local pixel regions, reducing artificial long-range correlations, while padding handles non-power-of-two image sizes.
- Data representations: The 1-D representation vectorizes image pixels, whereas the 2-D representation reshuffles indices to preserve the image lattice structure.Both arrangements use binary physical bonds with local dimension 2; the 2-D version changes the data order rather than the underlying TTN model.
- Data representations: Each tensor in the 2-D TTN models a local pixel area, reducing artificial long-range correlations in natural images.
- Data representations: When the number of input pixels is not a power of 2, zero-valued pixels are added at the chain ends or lattice boundary.This padding is analogous to padding in convolutional networks.
E. Training algorithm of the TTN
The TTN is trained by minimizing negative log likelihood, equivalently KL divergence up to a constant, through canonicalized sweeping SGD updates. Single-site and two-site updates offer different complexity–convergence trade-offs.
- Training objective: Training minimizes negative log likelihood, which equals the KL divergence between empirical and model distributions up to a constant.
- Sweeping updates: Sweeping SGD updates one TTN tensor at a time while canonicalization turns the network into an equivalent single-tensor computation.The algorithm alternates traversal directions and maintains canonical form with QR decompositions.
- Algorithm: The sweeping procedure begins with a canonicalized TTN, updates tensors using gradients, and propagates the canonical form through QR-based transformations.
- Update schemes: Single-site updates cost O(D^3), compared with O(D^5) for two-site updates, while their final convergence results are not significantly different in the authors’ experience.Two-site updates merge neighboring tensors and use SVD to restore the original tensor structure.
F. Direct sampling of the TTN generative modeling
TTN sampling is direct because the model can calculate its partition function exactly. Marginal and conditional probabilities are computed through tensor contractions, allowing image pixels to be sampled sequentially without Gibbs sampling.
- Direct sampling: Exact partition-function calculation lets TTN sample configurations directly without a Markov chain such as Gibbs sampling.
- Probability computation: The sampler computes a pixel’s marginal probability by leaving its bond open and squaring the resulting two-component vector.
- Evaluation: Figure 3 varies training NLL against data size with N = 16 and against system size with |T| = 50.
- Sequential sampling: Conditional probabilities are obtained after fixing sampled pixel bonds, enabling pixels to be sampled one by one.
A. Random dataset
Random-pattern experiments test whether TTN can memorize training samples and preserve their long-range dependencies. TTN remains effective as system size grows, whereas MPS becomes limited by exponentially decaying correlations.
- Evaluation setup: Random-pattern memorization evaluates expressive power using exact partition functions, normalized probabilities, and training NLL.Lower NLL indicates that more information from the training dataset has been captured.
- Capacity limits: When Dmax < |T|, training NLL does not reach the theoretical limit because bond capacity cannot encode all training-pattern information.The maximum information entropy of a bond is stated as ln(D).
- Scaling with system size: When Dmax ≥ m, TTN training NLL converges to the theoretical limit regardless of picture size because its pixel correlation length grows logarithmically.
- TTN versus MPS: With 10 training patterns, TTN reaches the theoretical NLL minimum at large N when Dmax = 10, while MPS fails even at Dmax = 12.The paper attributes the MPS difficulty to exponentially decaying correlation length and saturation of information in the middle bond.
B. Binary MNIST dataset
On binary MNIST, TTN models were evaluated through training and test NLL, generated samples, pixel correlations, and comparisons with MPS and other generative models. The 2-D TTN captured both short- and long-range correlations and achieved lower test NLL than the compared tensor-network models, though RBM and PixelCNN performed better.
- The binary MNIST benchmark uses 50,000 training images, 10,000 validation images, and 10,000 test images of 28 × 28 binary pixels.
- As Dmax increases toward |T| on 100 training images, training NLL approaches its theoretical minimum and samples reproduce training images exactly.
- TTN represents both short-range and long-range pixel correlations, whereas MPS faithfully represents short-range correlations but not the long-range structure.
- TTN, MPS, and PixelCNN provide exact NLL values, while RBM and VAE evaluate the partition function approximately.
- 2-D TTN achieved test NLL 94.25, compared with 1-D TTN 96.88, MPS 101.45, and the tree-structure factor graph 175.8.
- RBM reached approximate test NLL 86.3 and PixelCNN reached around 81.3, outperforming the tensor-network models on this benchmark.
IV. CONCLUSIONS AND DISCUSSIONS
The paper presents TTN as a generative model that preserves tractable MPS advantages while improving long-range correlation modeling and image performance. Experiments show stronger results than MPS on random patterns and MNIST, while future work targets richer image priors and tensor-network development.
- TTN model and training: TTN preserves MPS advantages, including tractable normalization, canonical form, and direct sampling, while addressing MPS’s exponential correlation decay.The authors also developed efficient single-site and two-site sweeping updates for minimizing the NLL loss.
- Experimental results: TTN achieves better training NLL than MPS with the same bond dimension when memorizing large random patterns.
- Experimental results: On handwritten digits, TTN captures longer-range dependencies better than MPS and achieves much better test-image NLL, indicating better generalization power.
- Limitations and future work: Future tensor-network models could use MERA or PEPS to preserve longer-range dependencies and encode stronger two-dimensional image priors.PEPS may improve image priors but sacrifices exact partition-function computation, requiring approximate contraction methods.
- Limitations and future work: The authors identify a performance gap between tensor-network generative models and state-of-the-art neural models such as PixelCNN.They attribute a key neural-model advantage to convolution-based image priors.
- Broader significance: Tensor-network generative learning is motivated partly by quantum machine learning and may support future migration of machine learning to quantum computers.
- Broader significance: Comparing tensor-network and traditional machine-learning generative models may help improve tensor-network algorithms by clarifying why neural models perform better.