PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Said Neural Networks Aren't Linear?

Nimrod Berman, Assaf Hallak, Assaf Shocher

arXiv:2510.08570v2cs.LG

TL;DR

Neural networks are nonlinear in standard coordinates, limiting access to linear-algebra methods for tasks such as projection and iterative simplification. The paper defines Linearizers by placing a linear operator between invertible neural networks, making the map linear in induced vector spaces. This framework collapses hundreds of diffusion sampling steps into one and supports architectural idempotency.

  • Problem

    Neural networks are nonlinear in standard coordinates, making tasks such as projection, idempotency, and collapsing iterative procedures difficult.

    Source: §1 · p. 1

  • Method

    A Linearizer sandwiches a linear operator between invertible neural networks, whose induced vector-space operations make the overall mapping linear.

    Source: §2.1 · p. 2, §2.2 · p. 2

  • Results

    The framework collapses hundreds of diffusion sampling steps into one, with one-step and multi-step generation MSE of 3.0 × 10−4 on MNIST and CelebA.

    Source: §1 · p. 1, §4.1 · p. 5

  • Takeaways & Limitations

    Architectural idempotency makes the Linearizer a global projector rather than a model trained to approximate projection.

    Source: §4.3 · p. 6

    Invertible networks are harder to train, the demonstrated applications are not scaled to state-of-the-art benchmarks, and the framework's precise expressivity remains open.

    Source: §5 · p. 7

Abstract

from arXiv · show

Neural networks are famously nonlinear. However, linearity is defined relative to a pair of vector spaces, $f:X \to Y$. Leveraging the algebraic concept of transport of structure, we propose a method to explicitly identify non-standard vector spaces where a neural network acts as a linear operator. When sandwiching a linear operator $A$ between two invertible neural networks, $f(x)=g_y^{-1}(A g_x(x))$, the corresponding vector spaces $X$ and $Y$ are induced by newly defined addition and scaling actions derived from $g_x$ and $g_y$. We term this kind of architecture a Linearizer. This framework makes the entire arsenal of linear algebra, including SVD, pseudo-inverse, orthogonal projection and more, applicable to nonlinear mappings. Furthermore, we show that the composition of two Linearizers that share a neural network is also a Linearizer. We leverage this property and demonstrate that training diffusion models using our architecture makes the hundreds of sampling steps collapse into a single step. We further utilize our framework to enforce idempotency (i.e. $f(f(x))=f(x)$) on networks leading to a globally projective generative model and to demonstrate modular style transfer.

1. Introduction

Neural networks are nonlinear in standard coordinates, putting classical linear-algebra tools out of reach for tasks such as projection and iterative simplification. The paper introduces Linearizers, which make a neural mapping linear in induced vector spaces while retaining nonlinear expressiveness.

1. Introduction

  • Linear operators support eigenvalue and singular-value analysis, inversion, and compositional manipulation; repeated application simplifies through operator powers.Continuous evolution can be represented by an operator exponential, and composition preserves linearity.Source: §1 · p. 1
  • Nonlinear mappings make projection, idempotency, and iterative procedure collapse difficult, motivating a latent space where the mapping is linear.The paper identifies these tasks as requiring engineered loss functions and optimization schemes in the nonlinear regime.Source: §1 · p. 1
  • Linearizers embed a linear operator between invertible neural networks, inducing vector spaces in which the overall mapping is linear.The framework is presented as a bridge between nonlinear expressiveness and linear-algebra tractability.Source: §1 · p. 1

2. Linearizer Framework

The framework uses transport of structure to define vector spaces in which a neural mapping is linear: invertible neural networks set the coordinates around a linear operator. Its induced-space linearity supports linear-algebra properties and operations, including composition, inner products, transposes, and SVD.

2.1. Definitions

  • Invertible coordinate transformations induce new vector-space operations, while the vectors and scalar field remain unchanged.A Linearizer wraps a linear operator between invertible networks; its input and output spaces use operations induced by the respective networks.Source: §2 · p. 2, §2.1 · p. 2

2.2. Linearity

  • The induced operations make the input and output spaces vector spaces, and the Linearizer is a linear map between them.The input-space operations come from gx and the output-space operations from gy.Source: §2.2 · p. 2

2.3. Intuition

  • In the coordinates defined by gx and gy, the function acts as a matrix multiplication, analogous to expressing a matrix in its eigenbasis.For a shared basis, repeated application corresponds to taking a power of the core matrix.Source: §2.3 · p. 2

2.4. Properties

  • The paper derives basic linear-transform properties from the vector-space isomorphism established by transport of structure.The properties are reviewed explicitly even though they follow from that isomorphism.Source: §2.4 · p. 2

2.4.1. COMPOSITION

  • Composing two Linearizers with compatible spaces yields another Linearizer.Source: §2.4.1 · p. 2

2.4.2. INNER PRODUCT AND HILBERT SPACE

  • The induced inner product makes the induced vector spaces Hilbert spaces.Figure 1 depicts the linear operation between invertible functions and the induced addition and scalar multiplication.Source: §2.4.2 · p. 3, Figure 1 · p. 3

2.4.3. TRANSPOSE

  • The transpose is defined with respect to the induced inner products.Source: §2.4.3 · p. 3

2.4.4. SINGULAR VALUE DECOMPOSITION

  • The paper states that spectral properties transfer under isomorphic changes of coordinates.The SVD is given by the same singular values, with input singular vectors transformed through the coordinate map.Source: §2.4.4 · p. 3

3. Theoretical Analysis

The analysis characterizes the topological constraint imposed by the linear bottleneck and examines when the core matrix A remains functionally important. Despite that constraint, Linearizers can fit any finite dataset to arbitrary precision.

3.1. Expressiveness

  • The kernel of a Linearizer is determined by A, ruling out some mappings, such as sending exactly three points to zero and a fourth to a nonzero value.This restriction follows from the subspace properties of the kernel in latent space.Source: §3.1 · p. 3
  • Linearizers can fit any finite dataset to arbitrary precision using suitable invertible networks and a linear map.The paper attributes this capacity to degrees of freedom outside the data support, including room for the required infinite null space.Source: §3.1 · p. 3
  • With distinct gx and gy, absorbing SVD factors and singular-value scaling into the invertible networks reduces A to zeros and ones, making its role the function’s rank.Source: §3.2 · p. 4
  • When gx=gy, diagonalizable A can be rewritten with eigenvalues in the core, but shared-basis families need non-diagonal A to retain functional diversity.In flow matching, the networks are shared while A varies by timestep.Source: §3.2 · p. 4

4. Applications

The applications use Linearizers for one-step flow matching, modular style transfer, and architecturally idempotent projection. The experiments are proof-of-concept, with scaling and engineering refinements left for future work.

4.1. One-Step Flow Matching

  • FID from 100 full iterative steps closely matches the one-step formulation; simulating 1000→1 rather than 100→1 improves FID by ∼8 points.The authors also report a 3.0 × 10−4 mean squared error between one-step and multi-step generations and note that absolute FID is not yet competitive with state-of-the-art systems.Source: §4.1 · p. 5, Figure 4 · p. 6
  • The Linearizer’s Moore–Penrose pseudoinverse enables encoding real data and latent interpolation between two real images, with high-quality information preservation reported using LPIPS and PSNR.Figure 3 shows interpolation between real, non-generated images; Figure 4(b) presents inversion-reconstruction consistency.Source: §4.1 · p. 5, Figure 3 · p. 5, Figure 4 · p. 6

4.2. Modular Style Transfer

  • Style-specific matrices generated from a style index separate content from style, and interpolating those operators yields transfers across the evaluated style pairs.The style operators are distilled from a pretrained Johnson-style network; reported examples include mosaic/candy, rain-princess/udnie, and candy/rain-princess.Source: §4.2 · p. 5, PSNR LPIPS · p. 6

4.3. Linear Idempotent Generative Network

  • Idempotency of f is equivalent to idempotency of A, allowing a binary projection matrix to enforce the property architecturally.The method assumes a diagonalizable projection matrix and uses thresholded binary values in the forward pass with continuous values for backpropagation.Source: §4.3 · p. 6
  • Unlike IGNs that are idempotent only near trained source or target distributions, this Linearizer is architecturally idempotent across the ambient input space without injected latent noise.The authors describe this construction as a step toward the global projector envisioned by Shocher et al. (2024).Source: §4.3 · p. 6

4.4. Additional Experiments

  • Distilled Linearizers closely match VAE, FactorVAE, and β-VAE encoders on FactorVAE, SAP, and DCI metrics, often slightly improving FactorVAE and SAP.The evaluation uses dSprites, which provides ground-truth factors of variation.Source: §4.4 · p. 7
  • On weather prediction from 96 past steps, the method surpasses Autoformer and is competitive with PatchTST, with ∼0.008–0.034 MSE gaps by horizon.The authors describe this as a small experiment in a new domain.Source: §4.4 · p. 7

5. Limitations

The Linearizer has practical training, evaluation, and theoretical boundaries: invertible networks are harder to train, applications are not yet benchmark-scaled, and expressivity remains unresolved.

  • Invertible networks are inherently harder to train than standard architectures and often require careful design.Source: §5 · p. 7
  • Applications demonstrate feasibility but have not yet been scaled to state-of-the-art benchmarks.Source: §5 · p. 7
  • The precise expressivity of the Linearizer remains an open theoretical question.Source: §5 · p. 7

6. Related Work

The related work distinguishes Linearizers from approaches that linearize dynamics, symmetries, manifolds, or parameter-space training: they target arbitrary learned mappings and exact finite-dimensional linearity in induced spaces.

  • Unlike Koopman methods that approximate dynamics on a chosen observable dictionary, Linearizers target arbitrary learned maps with distinct input and output coordinates.Their matrix A may be rectangular when input and output dimensions differ, and linear-algebraic structure in A transfers to f, including SVD and analytic inversion with A+.Source: §6 · pp. 7–8
  • Unlike the NTK's linear evolution in parameter space, Linearizers achieve input-output linearity in a learned basis at any network width, independent of training dynamics.Source: §6 · p. 8
  • Unlike Koopman Distillation Model (KDM), Linearizers train from scratch, enforce exact finite-dimensional linearity, and target uses beyond diffusion.The cited KDM distills a pretrained diffusion teacher into a one-step encoder-linear-decoder generator.Source: §6 · p. 8
  • Unlike equivariant representations that linearize symmetry actions, Linearizers make an arbitrary learned map exactly linear between induced spaces.Source: §6 · p. 8
  • Linearizers flatten mapping functions rather than data manifolds or deformation fields, and unlike deep linear networks, retain expressivity through nonlinear coordinate maps.Normalizing flows use invertible transformations to transport probability measures; Linearizers instead transport algebraic structure.Source: §6 · p. 8

7. Conclusion

The paper concludes that Linearizers make neural networks exact linear operators in induced spaces and demonstrate applications in generation, style transfer, and idempotency. It identifies scaling, dynamics modeling, and theoretical expressivity as future directions.

  • Linearizers learn invertible coordinate transformations that make neural networks exact linear operators in induced spaces.Source: §7 · p. 8
  • The demonstrated applications include one-step flow matching, modular style transfer, and architectural enforcement of idempotency.Source: §7 · p. 8
  • Scaling one-step flow matching and IGN to larger datasets and higher resolutions is proposed as a route toward competitive, efficient generative models.Source: §7 · p. 8
  • Matrix exponentials are proposed for continuous-time motion dynamics, while expressivity limits and integration with operator-learning frameworks remain open directions.Source: §7 · p. 8

Impact Statement

The impact statement notes potential inference-cost and energy benefits alongside generative-AI risks. The accompanying proof verifies transported vector-space axioms using the induced operations.

Impact Statement

  • One-step diffusion sampling can reduce inference cost and energy use, while retaining generative AI's risk of misleading synthetic content.The authors state their method introduces no additional ethical concerns beyond those already established for generative modeling.Source: Impact Statement · p. 9

A. Proof of Proposition 2.3 (Valid Vector Spaces)

  • The proof checks vector-space axioms by transporting them through g and g−1, using addition u ⊕g v := g−1(g(u) + g(v)) and scalar multiplication a ⊙g u := g−1(a g(u)).Source: A. Proof of Proposition 2.3 (Valid… · p. 11

B. Proof of Proposition 2.9 (SVD of a Linearizer)

The appendix transports Euclidean Hilbert-space tools through invertible coordinates, deriving an SVD and pseudoinverse for Linearizers, and adds style-transfer and inversion-interpolation examples.

B. Proof of Proposition 2.9 (SVD of a Linearizer)

  • The Euclidean SVD of A induces an SVD of the Linearizer, with orthonormal singular vectors and singular values defined in the induced inner-product spaces.The induced inner products are pulled back through gx and gy, and the Linearizer’s singular triplets satisfy the corresponding singular-vector relations.Source: B. Proof of Proposition 2.9 (SVD of a… · p. 12

C. Proof of Proposition 2.7 (Hilbert Space)

  • The inner product pulled back through a bijection g makes V a Hilbert space because g is an isometry onto Euclidean space, which is complete.The proof establishes completeness by mapping Cauchy sequences through g and using its surjectivity.Source: C. Proof of Proposition 2.7 (Hilbert… · p. 12

D. Proof of Lemma 4.1 (Pseudo-Inverse of a Linearizer)

  • The Linearizer pseudoinverse is obtained by wrapping the Euclidean Moore–Penrose pseudoinverse of A between the inverse coordinate maps.The proof verifies the four Penrose conditions in the induced inner-product spaces.Source: D. Proof of Lemma 4.1 (Pseudo-Inverse… · p. 13

E.1. Style Transfer

  • The appendix extends the main-paper style-transfer results with additional transformations following the same setup.Figure 7 presents examples using the left-side and right-side style images.Source: §E.1 · pp. 13–14, Figure 7 · p. 14

E.2. One-step Inverse Interpolation

  • Additional inversion-based interpolation examples are provided for CelebA and MNIST.The appendix identifies them as extensions of the interpolation examples in the main text.Source: §E.2 · p. 14

E.3. Additional Applications – Full Results

The additional applications report CelebA classification accuracy, compare distilled encoders with their originals, describe weather-forecasting evaluation, and provide further inversion interpolations.

CelebFaces Attributes (CelebA)

  • 87.6% test accuracy for the Linearizer versus 86.1% for ResNet-50 on CelebA’s 40-attribute classification task.The dataset includes large pose variation, background clutter, and diverse identities.Source: §E.3 · p. 14

Distillation of nonlinear disentanglement models

  • The distilled Linearizer closely matches the original VAE, FactorVAE, and β-VAE encoders on FactorVAE, SAP, and DCI metrics.Table 1 reports disentanglement metrics on DSPRITES, with higher values better.Source: §E.3 · p. 14, Table 1 · p. 17

Weather Prediction

  • Weather forecasting evaluates predictions from 96 past steps across future horizons using MSE, where lower scores are better.The comparison includes Autoformer and PatchTST; the supplied passages do not state the metric values.Source: §E.3 · p. 14, Table 2 · p. 17

Additional interpolations

  • Additional inversion-based interpolation examples are shown for CelebA and MNIST.The supplied figure captions identify the CelebA and MNIST interpolation examples.Source: Figure 8 · p. 15, Figure 9 · p. 16

F. Derivation for Runge–Kutta Collapse to One Step

In induced coordinates, the Runge–Kutta updates act linearly, so their composition can be collected into a single operator; the appendix also records implementation choices for the models.

F. Derivation for Runge–Kutta Collapse to One Step

G.1. Backbones and Block Structure

  • Both model architectures use six repeated invertible blocks and differ in their internal block types.The listed components include ActNorm, affine coupling, and invertible 1×1 convolutions for diffusion/style, and spatial splitting and pixel rearrangement for IGN.Source: §G.1 · p. 18

G.2. Time Path (Diffusion)

  • Diffusion and flow-matching models feed time embeddings to the coupling conditioners and the time-dependent core A_t; style and IGN models omit time.The diffusion core uses a low-rank parameterization, while style uses a hypernetwork-generated convolution kernel and IGN uses diagonal A.Source: §G.2 · p. 18, §G.3 · p. 18

G.4. Conditioners

  • Diffusion/style affine couplings use tiny U-Net conditioners, whereas IGN additive couplings use CNNs with exact inverses.The diffusion/style conditioners predict shift and log-scale; IGN’s forward coupling is additive.Source: §G.4 · p. 18

G.5. Losses (weights fixed) IGN.

  • IGN training combines reconstruction MSE, diagonal-A sparsity/rank, and a light isometry term on g.The diffusion/flow-matching setup adds perceptual reconstruction and alignment terms for stability and model quality.Source: §G.5 · p. 19

G.6. One-line Hyperparameter Summary

  • Table 6 is identified as a one-line summary of the scaffolding needed for reproduction.The supplied passage gives the table’s scope but not its contents.Source: Table 6 · p. 19

H. Linearizer Universal Final Fit Theorem

The theorem shows that Linearizers can approximate any target mapping on a finite dataset arbitrarily well; a flow-matching illustration depicts the induced-coordinate path as straightened for one-step generation.

H. Linearizer Universal Final Fit Theorem

  • The proof constructs an ideal exact fit by mapping distinct data points to distinct latent points and choosing a linear map between their latent representations.The stated diffeomorphism-extension fact supplies global coordinate maps for the finite point sets.Source: H. Linearizer Universal Final Fit… · pp. 19–20
  • For finite datasets in the theorem’s stated setting, invertible neural networks and a linear map yield a Linearizer with arbitrarily low dataset error.The result applies to finite inputs in R^N with N ≥ 2 and targets in R^M.Source: H. Linearizer Universal Final Fit… · p. 20

I. Additional Illustrations

  • In the Flow Matching illustration, the induced g-space represents the noise-to-data transport path as a straight line rather than a nonlinear curve.The caption says this rectification enables exact one-step generation.Source: Figure 10 · p. 21

Related papers

Loading 2510.08570v2…