PaperlayerView exact revision on arXiv

Source-linked AI summary

Partially Linear Autoencoders for Manifold Learning and Dimensionality Reduction

Louen Pottier, Louis Lesueur, Anders Thorin

arXiv:2608.29867v1cs.LG

TL;DR

The paper asks whether autoencoders can retain reconstruction quality with a linear encoder and nonlinear decoder. It compares four architectures across synthetic, mechanics, and image data, finding that Lenc-AE generally matches AE and outperforms architectures with linear decoders. The geometric explanation is that success depends on whether the manifold admits an injective linear projection into latent space.

  • Problem

    The paper investigates whether the encoder can be constrained to be linear without substantially reducing autoencoder reconstruction quality, and which component is more critical for nonlinear manifold representation.

  • Method

    The study compares AE, Lenc-AE, Ldec-AE, and LAE across synthetic manifolds, computational mechanics data, and image data, interpreting results through injective linear projections.

  • Results

    Across test cases, AE and Lenc-AE perform comparably, while Ldec-AE and LAE consistently lag in reconstruction accuracy and manifold unfolding quality.

  • Takeaways & Limitations

    A nonlinear decoder is the critical component, while a linear encoder can provide interpretable and computationally efficient latent representations when injective projection conditions hold or approximately hold.

  • Takeaways & Limitations

    Lenc-AE can fail when a linear projection is non-injective, and reconstruction MSE alone does not guarantee a self-intersection-free embedding.

Abstract

from arXiv · show

Autoencoders are widely used for nonlinear dimensionality reduction and manifold learning. While most common implementations rely on both nonlinear encoders and decoders, we investigate the specific role of the encoder and the extent to which it can be constrained to be linear without reducing accuracy. We conduct a comparative study on four autoencoder architectures: standard fully nonlinear autoencoders (AE), linear-encoder autoencoders (Lenc-AE), linear-decoder autoencoders (Ldec-AE), and fully linear autoencoders (LAE), evaluated on synthetic manifolds, computational mechanics data sets, and real-world image data sets including MNIST. We demonstrate that imposing a linear encoder preserves most of the representational capacity of the autoencoder, provided the decoder remains nonlinear. In particular, Lenc-AE consistently outperforms both Ldec-AE and LAE, and achieves reconstruction quality comparable to fully nonlinear AE, while offering advantages in terms of parsimony and interpretability of the latent representation. These results suggest that the nonlinear decoder is the critical component for manifold learning, rather than the encoder. A geometric interpretation of this finding is developed, which identifies the precise conditions under which a linear encoder is sufficient, and the specific manifold configurations that expose its limitations.

1 Introduction

Dimensionality reduction extracts patterns from high-dimensional data, but linear subspace assumptions can fail for complex structures that nonlinear manifold methods capture.

  • Dimensionality reduction enables meaningful pattern extraction from high-dimensional data such as images.
  • Traditional methods such as PCA assume that data lies in a low-dimensional linear subspace.
  • Complex variations in natural images, spectra, biomedical signals, and genomic data can exceed what a single linear subspace captures.
  • Manifold learning methods capture nonlinear structures such as clusters, trends, and multi-scale organization that PCA may miss.Classical methods generally provide implicit embeddings, making out-of-sample extension and model-reduction use challenging.

1.1 Autoencoders

Autoencoders learn explicit latent representations and reconstructions through encoder-decoder mappings, while their latent dimension and geometry determine whether manifold structure is preserved.

  • An autoencoder uses an encoder from R^D to R^d and a decoder from R^d to R^D to reconstruct data through a low-dimensional latent representation.Training minimizes a reconstruction loss.
  • Nonlinear mappings allow autoencoders to capture complex data structures beyond linear dimensionality-reduction methods.
  • The latent dimension must balance compression and underfitting, with Whitney’s theorem giving 2p as an embedding bound for p-dimensional manifolds.If the manifold is homeomorphic to R^d, d = p can suffice, although p is often unknown.
  • Without explicit constraints, latent representations may lack exploitable geometry needed for model coupling or differential equations in reduced coordinates.Distance-preserving regularization can encourage approximate local isometry.
  • With well-sampled manifold data, suitable architecture, and a theorem-consistent latent dimension, the latent space can faithfully preserve manifold structure.
  • Figure 1 depicts encoding samples from a p-manifold into a lower-dimensional latent space and decoding latent coordinates back into the reconstructed manifold.

1.2 Asymmetric Autoencoder Architectures and the Role of Encoder vs. Decoder Complexity

Encoder and decoder perform asymmetric manifold-learning tasks, so their computational complexity need not be matched and should depend on the application.

  • Symmetric autoencoder architectures are common for convenience, not because encoding and decoding fundamentally require equal complexity.
  • The encoder must produce a compact, injective manifold projection, whereas the decoder must reconstruct the manifold’s full geometric complexity.These asymmetric roles motivate task-dependent allocation of model capacity.

1.3 Motivation and Advantages of Partial Linearity

Partial linearity is motivated as an intermediate design space between PCA-like simplicity and fully nonlinear expressivity, with architecture choice shaped by geometry, efficiency, and interpretability.

  • The study independently varies encoder and decoder linearity across four autoencoder architectures.
  • Fully linear autoencoders are mathematically equivalent to PCA, while Lenc-AE, Ldec-AE, and AE provide distinct nonlinear-reduction approaches.
  • Partially linear architectures trade some representational capacity for interpretability, computational efficiency, and compatibility with physical constraints.
  • Related partially linear designs appear in signal recovery, edge computing, sparse recovery, computational mechanics, and Koopman-inspired dynamical analysis.
  • Linear decoders can encode known physical mixing constraints, such as endmember combinations in hyperspectral unmixing, and improve transparency in medical imaging.
  • Linear encoders expose input-feature contributions and enable lightweight matrix-vector execution, while linear decoders make output basis contributions transparent.
  • Linear encoders preserve input-space distances and affine structure, whereas linear decoders force learned coordinates adapted to data geometry.
  • The architectures are compared to determine whether a linear encoder can match nonlinear AE reconstruction and when encoder linearity becomes limiting.

2 Methodology

The study compares four autoencoder architectures across synthetic manifolds, computational mechanics examples, and image data using specified latent dimensions and consistent training conditions.

  • Test cases: Eight test cases span synthetic manifolds, computational mechanics displacement fields, and computer-vision data sets including MNIST and Office.The synthetic cases are helix, knot, Swiss roll, and S-curve; mechanics cases are cantilever beam and paperclip.
  • Experimental design: Paperclip and MNIST have unknown intrinsic dimensions, estimated through the minimum latent dimension needed for correct reconstruction.Prior MNIST estimates typically range from 10 to 15 depending on the methodology.
  • Architecture and implementation: The nonlinear components use three-layer fully connected networks for synthetic and mechanics cases, while image models use convolutional, fully connected, and transposed-convolutional layers.ELU activations are used for synthetic and mechanics networks, and ReLU activations for image models.
  • Training and comparison: No hyperparameter optimization is performed, because the goal is to compare architectures under consistent conditions rather than optimize each test case.
  • Experimental design: The experiments use latent dimensions specified in Table 2, with visualized latent spaces when d ≤3.For Beam, the intrinsic dimension is p = 2; for Office, the rotating-camera images theoretically form a closed curve with p = 1.
  • Training and comparison: All four architectures are trained by minimizing mean squared error reconstruction loss, with an additional UMAP-loss result for Office.The comparison includes LAE, Lenc-AE, Ldec-AE, and AE.

3.1 Synthetic Data Sets

Synthetic manifold experiments show that nonlinear decoders are essential for reconstructing curved geometries, while a linear encoder can suffice when its projection is injective.

  • Helix: The helix has intrinsic dimension 1, and its central-axis projection provides an injective linear encoding that a nonlinear decoder can invert.Thus, Lenc-AE with d = 1 can theoretically learn the helix perfectly.
  • Helix: Ldec-AE and LAE cannot accurately reconstruct the helix because their outputs are restricted to a line or plane.This limitation applies in both latent dimensions 1 and 2.
  • Helix: Lenc-AE reconstructs the helix perfectly for d = 1 and outperforms the standard AE in that setting.The result does not require the adapted training method suggested for nonlinear AE.
  • Knot: Minimizing reconstruction MSE alone does not guarantee a self-intersection-free knot embedding because it imposes no injectivity constraint.Distinct knot regions can collapse to identical latent coordinates, motivating topology-aware training strategies.
  • Knot: For the knot, linear decoders fail because lines or planes cannot represent its intricate topology, while AE and Lenc-AE recover only partial global structure.The AE is evaluated at d = 1 and d = 2; Lenc-AE behaves similarly at d = 2 but cannot meaningfully approximate the knot at d = 1.
  • S-curve and Swiss roll: The S-curve and Swiss roll are two-dimensional manifolds whose curved or rolled geometries force linear projections to overlap regions.Consequently, Lenc-AE is less accurate than AE, although both outperform architectures with linear decoders.
  • S-curve and Swiss roll: For the Swiss roll, AE reconstructs accurately but learns an arbitrary latent mapping, whereas Lenc-AE produces a more organized latent space reflecting global geometry.The Lenc-AE trade-off is reduced reconstruction accuracy, with discontinuities and distortions where the projection is non-injective.

3.2 Computational Mechanics Data Sets

The computational mechanics experiments show that Lenc-AE preserves the key reconstruction and latent-topology benefits of nonlinear autoencoders, while linear decoders substantially restrict performance. Across paperclip and beam data, AE and Lenc-AE outperform Ldec-AE and LAE, especially at small latent dimensions.

  • Beam: The beam data set contains 20-degree-of-freedom displacement fields forming a two-dimensional manifold, for which d = 4 should suffice theoretically for perfect nonlinear-AE reconstruction.The fields arise from varying a two-component tip load and are generated with a 10-node finite-element discretization.
  • Beam: At d = 2, AE and Lenc-AE produce visually accurate beam reconstructions and separate upward from downward bending in latent space.Ldec-AE and LAE mix these configurations instead of capturing the same topology.
  • Paperclip: The paperclip benchmark contains 1,000 sequences of 300 finite-element deformation steps generated under history-dependent elastoplastic deformation.The data represent quasi-static balanced wrench forces applied by two hands.
  • Visualization: The computational-mechanics reconstructions and latent spaces are visualized for d ∈{2, 3}, with each latent point color matched to its corresponding deformed beam.Reconstruction metrics are reported for d ∈{1, 2, 4, 8, 16, 32, 64}.
  • Paperclip: Across all tested d values, reconstruction losses rank AE best, Lenc-AE second, and Ldec-AE and LAE substantially worse and nearly identical.From d = 8 onward, reconstructions from all four architectures are almost indistinguishable from the targets despite the continuing decline in error.

3.3 Image Data Sets

The image experiments evaluate autoencoders on MNIST and a closed office-image curve, finding that Lenc-AE remains competitive while nonlinear decoders are important for high-quality reconstruction. An added UMAP-inspired loss reduces latent self-intersections and still permits a linear encoder to learn a smooth latent manifold.

  • MNIST: MNIST is a 28 × 28 grayscale-image benchmark whose intrinsic dimension has been estimated between 7 and 13, with one study estimating p = 9.These estimates motivate experiments with latent dimensions below 16.
  • MNIST: On MNIST, reconstruction losses consistently rank AE first, Lenc-AE second, and Ldec-AE and LAE significantly higher and nearly identical.The linear decoder acts as a hard bottleneck that limits reconstruction quality regardless of latent dimension.
  • Office Video: The office images form a theoretically closed one-dimensional curve, so a nonlinear autoencoder can in principle learn them with latent dimension d = 2.The data contain 477 images captured while rotating a horizontal camera through more than 360 degrees.
  • Office Video: Projecting the office images into two latent dimensions initially creates multiple self-intersections, whereas t-SNE successfully visualizes their circular topology.This motivates adding a UMAP-inspired loss that pulls neighboring latent points together and pushes randomly sampled negatives apart.
  • Office Video: With d = 3, all architectures consistently reconstruct the office manifold, but linear decoders generally fail to produce high-quality images.The figure compares reconstructed images and latent visualizations for the four architectures.
  • Office Video: Including the UMAP loss makes AE slightly better than Lenc-AE, while a linear encoder still retains enough degrees of freedom to learn a smooth latent manifold.Without the loss, Lenc-AE slightly outperforms AE in reconstruction error.

4 Synthesis and Discussion

Across all test cases, AE and Lenc-AE perform comparably and outperform Ldec-AE and LAE, indicating that nonlinear decoders contribute more to representational capacity than nonlinear encoders. The geometric explanation is injectivity: a linear encoder can suffice when its projection is injective on the manifold, although higher latent dimensions can resolve some failures.

  • Cross-test results: Across all test cases, AE and Lenc-AE perform comparably, while Ldec-AE and LAE consistently lag in reconstruction accuracy and manifold unfolding quality.Lenc-AE occasionally matches or surpasses AE, so linearity in the encoder incurs only a marginal cost.
  • Geometric interpretation: A linear encoder can match a fully nonlinear AE when its projection onto R^d is injective on the data manifold and the decoder is sufficiently expressive.Injectivity requires distinct manifold points to map to distinct latent coordinates.
  • Geometric interpretation: The injectivity condition holds for the Helix at d = 1, the Beam at d = 2, and the Office manifold at d = 2, but fails for rolled or folded manifolds such as the Swiss roll and S-curve.For MNIST and Paperclip, competitive Lenc-AE performance suggests approximate injectivity may be achievable in practice.
  • Architectural asymmetry: Linear decoders require the data manifold itself to be a linear subspace, whereas linear encoders require only an injective linear projection compatible with curved or deformed structures.This asymmetry explains why nonlinear encoders cannot compensate for a decoder that is too rigid.
  • Implications: Lenc-AE offers an interpretable, analytically tractable latent representation while retaining nonlinear decoding capacity, making it a possible efficient alternative when injectivity is met or approximately met.The conclusion is conditioned on the geometric suitability of the input manifold.
  • Limitations and scope: Satisfactory reconstruction may still occur when injectivity fails at d = p if an overcomplete latent space d > p gives the decoder additional degrees of freedom.This behavior is reported for Paperclip and for MNIST at large d.

5 Conclusion and Further Work

The comparison identifies Lenc-AE as a structurally principled alternative to fully nonlinear autoencoders: a linear encoder can retain representational capacity when paired with a nonlinear decoder. The conclusion also outlines structural-mechanics applications and calls for sharper tests of when the required injectivity condition holds.

  • Conclusion: A linear encoder with a nonlinear decoder preserves most of the representational capacity of fully nonlinear AE while improving interpretability, computational efficiency, and latent-space regularity.The result depends on an injective linear projection of the data manifold onto the latent space, exactly or approximately.
  • Conclusion: Linearizing the decoder is substantially more restrictive and explains the consistently poor performance of Ldec-AE and LAE across the tested cases.The restriction prevents these architectures from capturing the relevant nonlinear data geometry.
  • Further Work: Lenc-AE extends linear substructuring models by retaining interface compatibility while using a nonlinear decoder to represent complex nonlinear deformation modes.This makes the architecture relevant to model-order reduction in structural mechanics, including regimes with large displacements or material nonlinearities.
  • Further Work: A Craig–Bampton-style Lenc-AE reduction method with interface guarantees and nonlinear structural benchmarks is proposed as a concrete direction for future research.The conclusion also calls for characterizing the injectivity condition more precisely and developing practical tests for its approximate validity.
Loading 2608.29867v1…