Source-linked AI summary
Approximation Capabilities of Neural ODEs and Invertible Residual Networks
Han Zhang, Xi Gao, Jacob Unterman, Tom Arodz
TL;DR
Whether Neural ODEs and i-ResNets can model arbitrary continuous invertible functions remained unresolved, despite the importance of constructing invertible models. The paper analyzes their limits and proves that extra dimensions yield universal approximation of homeomorphisms, while a linear layer yields universal approximation of continuous functions. These results are established under the stated model assumptions, including a universal-approximator assumption for the ODE vector field.
Problem
Whether Neural ODEs and i-ResNets can model arbitrary invertible mappings remained unresolved, despite their use as invertible network architectures.
Source: §1 · p. 2, §1.1 · p. 2Method
The paper analyzes approximation limits and proves universality results for higher-dimensional Neural ODEs and i-ResNets and for models capped by a linear layer.
Source: §1.2 · p. 3Results
Any homeomorphism on X ⊂ Rp can be modeled by a Neural ODE or i-ResNet in 2p dimensions, and q = p + 1 suffices for a Neural ODE followed by a linear layer to approximate Rp → R functions.
Source: §1.2 · p. 3Takeaways & Limitations
Within the stated scope, adding dimensions overcomes the identified limits for modeling homeomorphisms, while a linear output layer extends the models to non-invertible continuous functions.
Source: §1.2 · p. 3The ODE approximation analysis assumes its vector field network can be made large enough to approximate any desired function.
Source: §3 · p. 5
Abstract
from arXiv · showhide
Neural ODEs and i-ResNet are recently proposed methods for enforcing invertibility of residual neural models. Having a generic technique for constructing invertible models can open new avenues for advances in learning systems, but so far the question of whether Neural ODEs and i-ResNets can model any continuous invertible function remained unresolved. Here, we show that both of these models are limited in their approximation capabilities. We then prove that any homeomorphism on a $p$-dimensional Euclidean space can be approximated by a Neural ODE operating on a $2p$-dimensional Euclidean space, and a similar result for i-ResNets. We conclude by showing that capping a Neural ODE or an i-ResNet with a single linear layer is sufficient to turn the model into a universal approximator for non-invertible continuous functions.
1 Introduction
Residual networks can represent broad mappings, but invertibility constraints on Neural ODEs and i-ResNets leave their ability to represent arbitrary invertible maps unresolved. The paper establishes dimensionality limits and shows how extra dimensions or a linear output layer expand their approximation capabilities.
1 Introduction
- A residual block adds a differentiable nonlinear transformation to its input, and sequences of such blocks can represent mappings from inputs to outputs.A linear layer after a deep residual sequence is a universal approximator for Lebesgue integrable functions Rp → R [LJ18].Source: §1 · pp. 1–2
- Invertible networks are useful for normalizing flows, but whether their architectures approximate arbitrary invertible mappings remained an open question.Normalizing flows transform samples from a simple distribution to sample from a complicated, multi-modal distribution [RM15].Source: §1 · p. 2
1.1 Invertible Models
- i-ResNets constrain residual functions to have Lipschitz constant below 1, while Neural ODEs generalize residual sequences as continuous-time systems.The i-ResNet constraint guarantees that each residual layer is one-to-one; Neural ODE transformations are specified through solutions to an initial value problem.Source: §1.1 · pp. 2–3
- Neural ODEs and i-ResNets have limited approximation capabilities in their original dimensionality, leaving their universality for invertible continuous mappings unresolved.For Neural ODEs, the mapping x → −x on R cannot be realized by an ODE defined on R because continuous paths cannot intersect.Source: §1.1 · pp. 2–3
1.2 Our Contribution
- The paper proves that extra dimensions let Neural ODEs and i-ResNets model any homeomorphism, while a linear layer makes either model universal for continuous functions Rp → R.The homeomorphism result uses a 2p-dimensional Euclidean space embedding X as a linear subspace; the linear-layer result uses q = p + 1.Source: §1.2 · p. 3
2 Background on ODEs, Flows, and Embeddings
The background defines flows and explains why continuous-time evolution constrains which homeomorphisms a Neural ODE can realize. It then frames the flow-embedding question and notes that unrestricted solutions use non-Euclidean manifolds, complicating their application to gradient-trained ODE-Nets.
2.1 Flows
- A homeomorphism is a one-to-one mapping onto the same space whose inverse is continuous; a p-homeomorphism acts on X ⊂ Rp.Source: §2.1 · p. 4
- A flow is a time-indexed family of homeomorphisms with the composition rule φs+t(x) = φt(φs(x)); continuous flows have nonintersecting trajectories.A discrete flow uses integer time and can represent any homeomorphism, whereas continuous flows may not contain a given homeomorphism as φT.Source: §2.1 · p. 4
2.2 Continuous Flows and ODEs
- A continuous flow corresponds to an ODE whose vector field is the time derivative of the flow at time zero; under mild conditions its time-T map is a homeomorphism.An ODE with a Lipschitz-continuous right-hand side has a well-defined, unique local flow by the Picard-Lindelöf theorem.Source: §2.2 · pp. 4–5
- Adding time as an extra state coordinate converts a time-dependent ODE into an autonomous ODE.The added coordinate satisfies dx[τ]/dt = 1 and starts at zero, replacing explicit use of time in the vector field.Source: §2.2 · p. 5
2.3 Flow Embedding Problem for Homeomorphisms
- The flow-embedding problem asks whether a homeomorphism can occur as a time-T map, and restricted embeddings do not always exist even on simple domains.Unrestricted embeddings always exist, but use a smooth non-Euclidean manifold; this makes them difficult to apply to gradient-trained ODE-Nets.Source: §2.3 · p. 5
3 Approximation of Homeomorphisms by Neural ODEs
Assuming the ODE’s underlying network can approximate any desired function, this section identifies limits of p-dimensional Neural ODEs and gives a construction using extra dimensions. Any homeomorphism on X ⊂ R^p can be represented by a 2p-dimensional ODE-Net on inputs augmented with p zeros.
3.1 Restricting the Dimensionality Limits Capabilities of Neural ODEs
- No p-ODE-Net can model a mapping that fixes a separating set Z but sends a point across Z into another connected component.The result generalizes the x → −x example; the proof uses the need for a continuous flow from the identity to h.Source: §3.1 · p. 6
3 Approximation of Homeomorphisms by Neural ODEs
- At q = p, Neural ODEs are restricted in how they can rearrange distinct input regions, beyond the requirements of invertibility and continuity.The analysis assumes the underlying network is a universal approximator and considers q = p and q > p.Source: §3 · pp. 5–6, §3.1 · p. 6
3.2 Neural ODEs with Extra Dimensions are Universal Approximators for Homeomorphisms
- A 2p-ODE-Net can map [x, 0] to [h(x), 0] for any homeomorphism h on X ⊂ R^p.The construction represents h within the higher-dimensional space while returning the output to the embedded p-dimensional subspace.Source: §3.2 · pp. 6–8
- The construction routes each input through R^2p from x to h(x), with f and g chosen to meet endpoint and derivative conditions.The resulting paths do not intersect at the same position and time, and the induced vector field extends to R^2p.Source: §3.2 · pp. 7–8
- Training uses inputs augmented by p zeros and penalizes deviations from h(x) in the first p outputs and zero in the remaining outputs.After training, integrating with −fΘ instead of fΘ gives the continuous inverse h−1.Source: §3.2 · pp. 6, 8
4 Approximation of Homeomorphisms by i-ResNets
The section shows that p-dimensional i-ResNets cannot model certain homeomorphisms, including negation, but that adding dimensions enables approximation of homeomorphisms under a finite, known Lipschitz bound.
4.1 Restricting the Dimensionality Limits Capabilities of i-ResNets
- No p-i-ResNet with residual mappings Lip(f_i) < 1 can model x → −x, demonstrating a limitation even for a simple homeomorphism.The proof uses the preservation of order in one dimension and the nonintersection of extended paths.Source: §4.1 · pp. 8–9
- In p-i-ResNets, extended paths from distinct inputs do not intersect, and this constraint underlies the architectures’ limited ability to represent invertible mappings.The proof connects nonintersection and continuity to the impossibility of modeling the specified mapping.Source: §4.1 · p. 9
4.2 i-ResNets with Extra Dimensions are Universal Approximators for Homeomorphisms
- A 2p-i-ResNet can represent any homeomorphism h with Lip(h) ≤ k using ⌊k + 4⌋ residual layers.The construction applies to inputs [x, 0(p)] and produces [h(x), 0(p)].Source: §4.2 · pp. 9–10
- The construction stores the displacement δ(x) in the extra coordinates, advances toward h(x) in fixed steps, then restores the extra coordinates to zero.The first and last layers depend on h; the middle layers are fixed linear layers.Source: §4.2 · p. 10
5 Invertible Networks capped by a Linear Layer are Universal Approximators
A single linear layer after an invertible network extends its approximation capability to non-invertible functions. The paper gives a construction using p + r dimensions for p inputs and r outputs.
5 Invertible Networks capped by a Linear Layer are Universal Approximators
- A Neural ODE or i-ResNet followed by one linear layer can approximate non-invertible functions as well as traditional feedforward networks.The construction embeds the input and output in a joint space and extracts the output with a linear layer.Source: §5 · pp. 10–11
- For a compact input set, a linear layer-capped q-ODE-Net can perform any network F approximating a Lebesgue-integrable function; if f is Lipschitz, a q-i-ResNet can also perform F.Here q = p + r for p-dimensional inputs and r output dimensions.Source: §5 · p. 11
- The construction keeps the first p coordinates unchanged while the ODE accumulates F(x) in the last r coordinates, which the linear layer returns.For i-ResNets, the paper describes residual layers that accumulate F(x)/k in the output coordinates.Source: §5 · p. 11
6 Experimental Results
Experiments support the theoretical dimensionality results: an extra dimension enables an i-ResNet to learn negation, while added null channels affect Neural ODE classification loss differently depending on network capacity.
6.1 i-ResNets
- The i-ResNet test trained five residual blocks on 10,000 uniformly sampled training values for 100 epochs and evaluated on 2,000 test values.The network used linear residual mappings and spectral normalization to constrain the Lipschitz constant.Source: §6.1 · p. 11
- With one extra dimension, the i-ResNet learned [x, 0] → [−x, 0] with test MSE below 10−10 on each output; in one dimension, its MSE was 33.39.The one-dimensional model instead learned x → cx for a small positive c.Source: §6.1 · p. 11
6.2 Neural ODEs
- The Neural ODE experiment compared CIFAR10 test cross-entropy across added null channels and filter counts, using 50,000 training and 10,000 test images.The tested channel additions ranged from d = 0 to 7.Source: §6.2 · p. 12
- For ODE-Nets with at least 64 filters, adding more than three null channels slowed the decrease in test loss; the change in trend was significant (Mann-Whitney U test, p=.002).Three added channels correspond to q = 2p for the RGB inputs; the figure plots test cross-entropy against the number of added channels.Source: §6.2 · p. 12, Figure 2 · p. 12