Topology of deep neural networks

Gregory Naitzat; Andrey Zhitnikov; Lek-Heng Lim

Source-linked AI summary

Topology of deep neural networks

Gregory Naitzat, Andrey Zhitnikov, Lek-Heng Lim

arXiv:2004.06093v1Apr 13, 2020cs.LGmath.ATstat.ML

TL;DR

The paper asks how neural networks transform the topology of two class manifolds and why ReLU and depth are effective. It tracks Betti numbers through realistic, well-trained networks using persistent homology, finding progressive topological simplification, faster reduction with ReLU, and different layerwise patterns for shallow and deep networks. The study’s scope is limited by the computational difficulty of persistent-homology analysis on large, high-dimensional data.

Problem
The paper asks how neural networks transform the topology of two class manifolds and why ReLU and many layers are effective despite shallow universal approximation.
Method
The study tracks Betti numbers and persistent homology of class point clouds as they pass through layers of well-trained neural networks on simulated and real data.
Results
Neural networks progressively simplify class topology toward β_k(M_i) = 0 for k ≥ 1 and β_0(M_i) = 1; ReLU simplifies faster, while shallow networks concentrate changes in later layers.
Takeaways & Limitations
The findings support viewing deep networks as topology-changing maps whose depth and rectified activations aid progressive simplification toward linearly separable outputs.
Takeaways & Limitations
Persistent homology cannot be applied directly to many modern deep-learning data sets because of computational and memory complexity.

Abstract

from arXiv · show

We study how the topology of a data set $M = M_a \cup M_b \subseteq \mathbb{R}^d$, representing two classes $a$ and $b$ in a binary classification problem, changes as it passes through the layers of a well-trained neural network, i.e., with perfect accuracy on training set and near-zero generalization error ($\approx 0.01\%$). The goal is to shed light on two mysteries in deep neural networks: (i) a nonsmooth activation function like ReLU outperforms a smooth one like hyperbolic tangent; (ii) successful neural network architectures rely on having many layers, even though a shallow network can approximate any function arbitrary well. We performed extensive experiments on the persistent homology of a wide range of point cloud data sets, both real and simulated. The results consistently demonstrate the following: (1) Neural networks operate by changing topology, transforming a topologically complicated data set into a topologically simple one as it passes through the layers. No matter how complicated the topology of $M$ we begin with, when passed through a well-trained neural network $f : \mathbb{R}^d \to \mathbb{R}^p$, there is a vast reduction in the Betti numbers of both components $M_a$ and $M_b$; in fact they nearly always reduce to their lowest possible values: $β_k\bigl(f(M_i)\bigr) = 0$ for $k \ge 1$ and $β_0\bigl(f(M_i)\bigr) = 1$, $i =a, b$. Furthermore, (2) the reduction in Betti numbers is significantly faster for ReLU activation than hyperbolic tangent activation as the former defines nonhomeomorphic maps that change topology, whereas the latter defines homeomorphic maps that preserve topology. Lastly, (3) shallow and deep networks transform data sets differently -- a shallow network operates mainly through changing geometry and changes topology only in its final layers, a deep one spreads topological changes more evenly across all layers.

1. Overview

The paper studies neural networks as topology-changing maps that progressively simplify class manifolds, using Betti numbers and persistent homology to examine realistic architectures. It links faster simplification to ReLU and distributes topological changes across depth, while shallow networks concentrate them near the output.

1.1. Key findings.: ReLU reduces topology faster than smooth activations because ReLU is nonhomeomorphic and can collapse topologically complicated regions, whereas sigmoid-like maps preserve topology mathematically.The paper further reports that ReLU and Leaky ReLU differ despite both avoiding vanishing or exploding gradients.
1. Overview: The study examines realistic networks acting on entire classes or manifolds, contrasting with approaches focused on asymptotic architectures or single objects such as one image.Its stated goal is to explain layer-by-layer mechanics involving depth, width, and activation rather than treating the network solely as a black box.
1. Overview: Neural networks progressively simplify complicated class manifolds by reducing their Betti numbers toward β_k(M_i) = 0 for k ≥ 1 and β_0(M_i) = 1.The study follows entire class manifolds through network layers rather than tracking a single input object.
1.1. Key findings.: Shallow networks mainly alter geometry and concentrate topological changes in later layers, whereas deeper networks distribute those changes more evenly across layers.As depth is reduced, the final layers bear a greater share of topological simplification.
1. Overview: The analysis uses persistent homology and Betti numbers to track how point-cloud topology changes as data passes through network layers.This frames neural networks as writing or transforming data shape, in contrast to topological data analysis, which primarily reads shape.
1. Overview: The experiments use Betti-number evolution to connect topological simplification with final class separation in the network output.The reported output arrangement maps the two classes into separate clusters, corresponding to a simplified decision boundary.

2. Quantifying topology

The paper quantifies topology using Betti numbers and tracks how each class component’s shape changes through every neural-network layer. Persistent homology estimates these quantities from sampled point clouds, while topological complexity summarizes their total size.

Betti numbers: A contractible space has β0 = 1 and βk = 0 for all k ≥ 1, providing the simplest possible Betti-number pattern.The paper contrasts this pattern with circles, tori, and other spaces having nonzero higher-dimensional Betti numbers.
Topological complexity: Topological complexity is defined as the sum of all Betti numbers, so larger values correspond to more holes and greater topological complexity.This quantity is also connected to classical topology and gives a lower bound on stationary points for suitable functions.
Layerwise analysis: The study tracks the Betti numbers of the input, output, and every intermediate-layer image of each data-manifold component.This focuses on how network layers transform class-specific shapes rather than only analyzing the decision boundary.
Persistent homology: Persistent homology estimates Betti numbers from finite, possibly noisy point clouds sampled from the underlying manifold.The method introduces a geometric scale ε and uses the resulting simplicial complexes to estimate topology reliably.

3. Algebraic topology and persistent homology background

This section introduces simplicial complexes, homology, Betti numbers, and persistent homology as tools for estimating topology from point clouds. It also explains the computational pipeline and why large neural-network data sets require simplifications and restricted persistence calculations.

Simplicial complexes: A simplicial complex is a collection of simplices glued along faces and containing every face of each simplex.Abstract complexes encode the same combinatorial structure, enabling algebraic treatment of simplices.
Simplicial homology: Homology represents topology through chain vector spaces, boundary operators, cycles, and boundaries, with H_k defined as ker(∂_k)/im(∂_{k+1}).The construction is performed over F2, where arithmetic is modulo 2 and boundary signs are unnecessary.
Betti numbers: Betti numbers are dimensions of homology groups and count k-dimensional holes, while β0 counts connected components.Over F2, computing Betti numbers reduces to linear algebra on boundary matrices.
Persistent homology: Persistent homology varies the scale ε across a filtration, tracking how homology classes appear and disappear through nested simplicial complexes.At ε = 0 the Vietoris–Rips complex is discrete, while as ε →∞ it becomes contractible; persistence barcodes summarize the intermediate evolution.
Practical computation: Point-cloud homology computation smooths data, selects ε, constructs a Vietoris–Rips complex, and simplifies it without altering topology.Witness complexes and related methods reduce computational size while aiming to preserve homological information.
Practical computation: Full persistence is computationally challenging for high-dimensional neural-network data, so the experiments compute homology at every layer but persistence barcodes only once.The main constraints are computation and memory complexity, compounded by repeated analyses across randomly trained networks.

4. Overview of problem and methodology

The paper studies binary classification of two disjoint manifolds and tracks how well-trained neural networks transform their topology across layers. Experiments compare data complexity, activations, architectures, and real versus simulated settings.

Binary classification uses two disjoint manifolds Ma and Mb whose separation can be arbitrarily small but nonzero.
Training represents each class with labeled point clouds sampled densely from Ma and Mb.
The network is a composition of affine maps and activations, with intermediate maps νj used to inspect topology layer by layer.
A linear score function makes the final decision boundary a hyperplane in the output space.
The experiments begin with classes having multiple components, holes, and complicated entanglement, then track whether well-trained networks simplify them.
Both simulated and real-world data are used, with simulated data offering controlled topology while real-data topology is difficult to determine in advance.

5. Methodology

The methodology combines controlled topological data generation, extensive neural-network training, and layerwise homology computation. Persistent homology selects analysis parameters, while intrinsic distances support comparisons across changing layer geometries.

The study generates three controlled point-cloud data sets with known, topologically complicated structures.
D-I, D-II, and D-III vary in dimension and topology, including nine holes, interlocked tori, spheres, and balls.
Experiments vary activation, depth from four to ten layers, and width from six to fifty neurons, using at least 30 well-trained networks per configuration.
Networks are trained for zero training error and approximately 0.01% generalization error using cross-entropy loss, Adam, and long training runs.
Topology is tracked at every layer by computing homology on point clouds, while persistent homology is used to select k and ε.
The graph-geodesic metric δk normalizes distances across layers, preserves nearest-neighbor connectivity, and enables comparisons between different layer dimensions.
Table 1 records each data set, activation, layer widths, and number of successfully trained networks.

6. Results and discussions

Across simulated experiments, well-trained networks simplify topology through the layers, with ReLU changing Betti numbers faster than smooth activations. Depth spreads simplification across layers, whereas reduced depth concentrates it near the output.

Activation effects: Individual training runs show consistent β0 decay, although hyperbolic tangent has greater variance than leaky ReLU and ReLU.
Activation effects: Across experiments, nonhomeomorphic ReLU produces the most rapid reductions in all Betti numbers, while hyperbolic tangent can reduce them slowly or occasionally increase them.
Depth effects: Initial layers primarily induce geometric changes, while deeper layers produce more of the observed topological changes.
Topological features: The interlocking tori in D-II are the hardest features to simplify, with β1 loops persisting across many layers, especially under hyperbolic tangent.
Width effects: A bottleneck layer forces large topological changes, while narrow networks simplify topology faster than wider networks on D-I.
Width effects: For the same activation, width differences are negligible on D-II and D-III, but widths below fifteen neurons make high-accuracy training more difficult.
Depth effects: Reducing depth concentrates topological simplification in final layers rather than distributing it evenly, making those layers work harder.

7. Consistency with real-world data

Experiments on real-world data sets show that neural networks simplify data topology across layers, with ReLU generally simplifying it faster than tanh. The real-data experiments use task-specific preprocessing and fixed network dimensions in response to computational constraints.

Experimental scope: Real-data experiments covered MNIST, HTRU2, banknotes, and sensorless-drive data sets, tracking topology through well-trained neural networks.The study selected real-valued data sets that could be trained to high accuracy.
Experimental constraints: Real-data experiments fixed network width at 10 neurons and depth at 10 layers because full persistent-homology calculations made broader architectural exploration prohibitively expensive.For real data, persistent homology had to be computed at every layer because a single observation scale could not be used across layers.
MNIST: MNIST images were projected from R784 onto their leading 50 principal components before persistent-homology analysis.The dimension reduction was required because persistent homology in the original 784-dimensional space exceeded available computational resources.
MNIST: For MNIST digit classification, topology decreased across layers, with ReLU reducing complexity fastest; tanh did not reduce Ma to a topological disk within ten layers.The binary task classified one chosen digit against all other digits.
HTRU2: HTRU2 used 3,278 balanced samples, with 80% for training and 20% for testing, in ten-layer networks activated by ReLU or tanh.The persistent barcodes compare the first, middle, and last layers, k = 1, 5, 10.
HTRU2: Across HTRU2, topology simplified through the layers, and ReLU simplified topological complexity more effectively than tanh.Figure 17 combines persistence barcodes with principal-component projections of pulsar and non-pulsar data.
Banknotes: For banknotes, β1 and β2 reached zero under both activations, while β0 reached one with ReLU but remained two with tanh.Betti-number reduction also occurred more rapidly with ReLU activation.

8. Concluding discussions

The concluding discussion interprets deep networks as progressively transforming complicated data topology toward linear separability. It connects this view to ReLU folding, excess width, and depth as distinct mechanisms for changing data representations.

Conclusions: The authors conclude that deep neural networks gradually simplify complicated data shapes until the classes become linearly separable in the output space.They present this as an empirically supported view of how deep networks operate.
Conclusions: The discussion identifies deep layers and rectified activations as mechanisms that aid topological changes.This conclusion is framed as an interpretation of the empirical study.
Mechanisms: Many-to-one maps can disentangle concentric circles in one step, while higher-dimensional embeddings can enable disentangling operations in an expanded space.Figure 21 illustrates both routes to transforming nonlinearly separable sets.
Mechanisms: The authors speculate that ReLU folds space, excess width supplies higher-dimensional room for transformation, and depth supplies additional time for transformation.These roles are presented as a proposed interpretation rather than as a proved mechanism.
Mechanisms: Affine maps provide translation, rotation or reflection, and stretching or shrinking, while ReLU adds folding to these transformation capabilities.The discussion uses torus-to-sphere transformations to motivate folding as an important added operation.