PaperlayerView exact revision on arXiv

Source-linked AI summary

An Input Skip Can Reparameterize a VAE Without Improving the Generator

Kevin Lee

arXiv:2610.37456v1cs.LGphysics.ao-phstat.ML

TL;DR

An input skip can reduce reconstruction error without establishing whether the model’s distribution of new samples or generative scores improve. The paper derives a Gaussian rescaling that preserves the generator, then compares separately trained skip and no-skip VAEs and tests reconstruction-based variance estimation. In the Gaussian experiment, MSE fell from 0.0616 to 0.00368 while median energy score rose from 5.723 to 7.123; DART energy-score differences varied across records, and PPCA had lower mean scores than every neural fit.

  • Problem

    An input skip can reduce reconstruction error without establishing whether the model’s distribution of new samples or generative scores improve.

    Source: §1 · p. 1

  • Method

    The paper derives a Gaussian rescaling that preserves the generator, then compares separately trained skip and no-skip VAEs and tests reconstruction-based variance estimation.

    Source: §1 · p. 1, §4.3 · p. 4

  • Results

    In the Gaussian experiment, MSE fell from 0.0616 to 0.00368 while median energy score rose from 5.723 to 7.123; DART energy-score differences varied across records, and PPCA had lower mean scores than every neural fit.

    Source: §4.1 · p. 3, §4.2 · p. 4

  • Takeaways & Limitations

    Assessing generation requires considering observation variance and generative scores alongside reconstruction error.

    Source: §5 · p. 4

    Score differences depend on fitting and, for log estimates, approximate inference.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

An input skip can reduce reconstruction error while leaving a model's distribution of new samples unchanged. For a Gaussian variational autoencoder (VAE) with constant input skip $0\leq\alpha<1$, a model without a skip can be written in skip form by multiplying its decoder output by $b=1-\alpha$ and its noise variance by $b^2$. The distribution of observations and latent variables and its exact generative scores stay the same, while reconstruction mean squared error (MSE) is multiplied by $b^2$. Separate training can produce different generators because optimization depends on the parameter coordinates. In a Gaussian experiment with known density, changing skip and observation variance together reduces MSE but worsens energy score and the estimate of negative log density from importance sampling. On seven annual Deep-ocean Assessment and Reporting of Tsunamis (DART) records, energy score differences at the same observation variance have both signs on four original records and favor the skip on three records from 2018. Probabilistic principal component analysis (PPCA) has lower mean energy scores than every neural DART fit. With the encoder and decoder kept the same, estimating observation variance from training reconstruction error can also worsen energy score. Assessing generation requires more than reconstruction error.

1 Introduction

An input skip lets the observation contribute directly to its reconstruction, potentially lowering reconstruction error without improving generation. The paper separates an exact Gaussian reparameterization from the different outcomes that can arise when models are trained separately.

1 Introduction

  • An input skip adds part of the observation directly to the decoder reconstruction, and increasing its weight can reduce reconstruction error while generative performance stays the same or worsens.The DART application uses a constant skip to isolate this effect; earlier work used an adaptive skip and iterative reconstruction at station 23461 in 2022 [7].Source: §1 · p. 1
  • For a constant skip with Gaussian noise, rescaling the decoder and noise preserves the generator while multiplying reconstruction MSE by (1 − α)^2.Separate training can nevertheless change the generator because Adam steps and gradient clipping depend on parameter scaling [2] [3].Source: §1 · p. 1
  • The DART records capture both oscillations and changes in mean level, while varying observation variance changes the relative weight of reconstruction and KL divergence during training.The HEIGHT variable expresses bottom pressure as equivalent water column height; comparisons use windows of 192 consecutive observations.Source: §1 · pp. 1–2

2 Input skip and generative evaluation

The paper defines an implicit Gaussian VAE with an input skip and shows how to express a no-skip model in skip form. This exact transformation preserves the generator and generative scores while scaling reconstruction error.

2 Input skip and generative evaluation

  • The implicit model writes X = αX + fθ(Z) + ϵ; solving for X gives a conditional density and a rule for generating samples.Its conditional observation mean is gθ(z) = fθ(z)/b, and its variance is σ²_r/b².Source: §2 · p. 2
  • Setting the residual decoder to f = b gθ and its variance to b²σ²₀ preserves the joint distribution of observations and latent variables, the marginal, and the exact posterior.Here b = 1 − α, with 0 ≤ α < 1.Source: §2 · p. 2
  • With the same encoder, the transformation preserves the ELBO and importance weights while multiplying reconstruction MSE by b².The proof uses the change of variables x to bx and its Jacobian factor; separate fits can follow different training paths under the same Adam settings and gradient norm limit.Source: §2 · p. 2

3 Experiments

The experiments combine a known-density Gaussian simulation with annual DART records, comparing skip and variance conditions, neural fits, and PPCA. Evaluation uses energy scores and importance-weighted density estimates, and also tests variance estimation with network weights fixed.

3.1 Data

  • DART records are analyzed in windows of 192 observations, with four original records and three 2018 records using distinct window-weighting and partition schemes.The Gaussian experiment uses eight independent normal latent factors, orthogonal sinusoidal loadings, and isotropic noise with standard deviation 0.25.Source: §3.1 · pp. 2–3

3.2 Models and comparisons

  • Each record uses four conditions and three training seeds; neural encoders and PPCA use at most 16 latent coordinates or loading directions.PPCA determines base variance, while 2018 PPCA variance is selected on a validation grid before neural fitting.Source: §3.2 · p. 3

3.3 Evaluation

  • Energy scores compare sample-to-observation distances against half the pairwise sample distance, with lower scores indicating better agreement; negative importance-weighted log density estimates use K = 256.DART scores cover full windows, mean-subtracted windows, and time differences; low effective sample size leaves density-estimate accuracy uncertain.Source: §3.3 · p. 3

3.4 Observation variance estimation

  • Observation variance is estimated from encoder samples by minimizing the expected Gaussian reconstruction term, including its log normalizer, while encoder and decoder weights remain unchanged.The estimated variance is used for evaluation and sample generation.Source: §3.4 · p. 3

4 Results

Across the experiments, lower reconstruction error does not ensure better generative scores: the Gaussian fit comparison worsens both reported generative scores, DART results vary by records and PPCA, and reconstruction-based variance estimation can worsen test energy scores.

4.1 Synthetic Gaussian experiment

  • MSE fell from 0.0616 to 0.00368 while median energy score rose from 5.723 to 7.123 when skip and observation variance both changed.Median estimated negative IW log density also rose from 0.133 to 1.201 nats per coordinate; these were separate fits, whereas exact rescaling of one fit preserves its generator and exact proper scores.Source: §4.1 · p. 3
  • The high observation variance, 25σ²₀ ≈ 1.53929, exceeds data variance in 185 of 192 covariance directions, leaving unavoidable excess variance under independent observation noise.The passage states this restriction applies to nonlinear decoders as well as linear ones.Source: §4.1 · pp. 3–4

4.2 DART case studies

  • At base variance, DART energy-score differences have both signs across four original records, while all three 2018 cases favor the skip; PPCA has lower mean scores than every neural fit.For 46407/2011, subtracting each window’s mean reverses the median difference’s sign; within two 2018 records, full-window differences also take both signs across intervals and seeds.Source: Figure 1 · p. 4, §4.2 · p. 4

4.3 Estimating observation variance

  • Estimating variance from training reconstruction error improved all three mean energy scores in 42 fits trained at σ²₀ but worsened all three in 42 fits trained at 25σ²₀.Thus, minimizing Gaussian reconstruction loss on training windows can worsen energy score on test windows.Source: §4.3 · p. 4

5 Conclusion

Exact rescaling can lower reconstruction error without changing a Gaussian VAE’s generator. Separate DART fits can differ in generative scores, and PPCA scores lower than every neural fit.

5 Conclusion

  • Exact rescaling can make reconstruction MSE arbitrarily small while preserving the generator.The conclusion cautions that reconstruction error must be interpreted with observation variance and generative scores.Source: §5 · p. 4
  • Separate DART fits show energy score differences of both signs.Source: §5 · p. 4
  • PPCA has lower mean energy scores than every neural DART fit.Source: §5 · p. 4

A Density, MSE, and energy score

This section distinguishes exact reparameterizations of one model from separately trained fits: the former preserve the generator while scaling reconstruction error, but the latter require generative evaluation. It also discusses density and energy-score estimates and their scope.

A Density, MSE, and energy score

  • The objective sums squared residuals before division by dimension, and its KL coefficient depends on observation variance and the chosen residual scaling.At α = 0.8, the reconstruction term is multiplied by 0.04 and the relative KL coefficient becomes 25βα.Source: A Density, MSE, and energy score · pp. 7–8
  • Exact rescaling preserves the observation and latent-variable distribution, exact posterior, ELBO, and importance weights while multiplying squared reconstruction error by b^2.The mapping uses fθ = bgθ and rescales the Gaussian variance by b^2.Source: A Density, MSE, and energy score · p. 7
  • For a nonlinear decoder, mean-latent MSE can lie above or below encoder-averaged MSE; both scale by b^2, but separately trained models need samples or densities for generation assessment.Positive scaling preserves error rankings and thresholds for the two representations of one model.Source: A Density, MSE, and energy score · p. 7
  • The study treats constant scalar α < 1; input-dependent skips, matrix coefficients, and the singular α = 1 case require separate analysis.For the constant skip, a change of variables gives the generative density through its Jacobian.Source: A Density, MSE, and energy score · p. 7
  • Raising α from zero to 0.8 lowers expected reconstruction MSE from one to 0.04 while raising expected negative log likelihood from approximately 1.419 to 2.548 nats.The likelihood calculation includes the density normalizer and both quantities are population expectations.Source: A Density, MSE, and energy score · pp. 7–8
  • Importance-sampling estimates depend on encoder sampling and marginal density; poor posterior approximation can concentrate weights, and a single estimate can exceed log pα(x).The expectation is at most log pα(x); finite-K comparisons also change both the density and the latent sampling distribution.Source: A Density, MSE, and energy score · p. 8
  • A density in meter coordinates includes the scale correction −d log s, while a complete time-series density also requires dependence modeling between windows.The reported log-density comparisons use standardized DART coordinates and original Gaussian coordinates.Source: A Density, MSE, and energy score · p. 8
  • Energy score is strictly proper under finite first moments, but finite ensembles add Monte Carlo variability; the Gaussian covariance restriction rules out recovering the full covariance under the stated noise floor.The covariance calculation applies to nonlinear as well as linear decoders, while its contribution to score differences remains uncertain.Source: A Density, MSE, and energy score · p. 8

B Adam under decoder rescaling

Decoder rescaling changes the parameter coordinates used by Adam, so matching updates requires scaling its learning rate and stabilizer. A numerical check verifies equality under a common parameterization, while the cause of separately trained DART score differences remains open.

B Adam under decoder rescaling

  • Matching an Adam step after decoder rescaling requires ηr = bηw and ϵr = ϵw/b; clipping can also change updates elsewhere in the network.With negligible epsilon, scaling the learning rate alone approximates the matching step; at b = 0.2, the corresponding rate in w is five times larger.Source: B Adam under decoder rescaling · pp. 8–9
  • A common-parameterization check produced identical parameter and optimizer tensors after every step, but it does not explain score differences in separately trained DART fits.The check covered 48 tensors across four Gaussian runs of 10,000 Adam steps.Source: B Adam under decoder rescaling · p. 9
  • Under linear Gaussian assumptions with exact inference, moving away from zero loading reduces population NLL; whether the nonlinear fits reached a local optimum remains unresolved.The result shows that a small encoder KL can occur away from a likelihood optimum.Source: B Adam under decoder rescaling · p. 9

C Windows, splits, and training

The section defines the original-record splits, windows, and training setup, then extends the DART analysis to three additional 2018 records. The skip lowers mean energy scores on all three additional records in two evaluation checks, though differences from original records cannot be attributed to deployment history without accounting for other changes.

C Windows, splits, and training

  • The Gaussian simulator uses eight latent factors with orthogonal sinusoidal loadings in 192 dimensions, while the study fits 60 original cases and 36 additional 2018 cases.The 60 original fits cover the Gaussian family and four DART records; the additional fits cover three DART records from 2018.Source: C Windows, splits, and training · p. 9
  • Original DART partitions exclude two days around split dates, and windows advance by 16 observations for training and 192 for validation and test.Training uses a ReLU encoder and decoder, Adam with a 10^-3 learning rate, and a global gradient norm cap of 10; validation checks every 250 steps determine retained weights.Source: C Windows, splits, and training · pp. 9–11

C.1 Additional DART records

  • Additional records required at least 240 days of coverage, valid routine observations in 80% of UTC slots, and minimum non-overlapping window counts in each partition.The model fits and evaluations use larger overlapping window sets than the required subsets.Source: §C.1 · p. 11
  • The 2018 records use six UTC start hours with equal total weight, while normalization gives equal weight to unique training observations in eligible windows.This observation set and weighting differ from those used for the original records.Source: §C.1 · p. 11
  • The two added stations were selected in identifier order before neural results were available, but hardware continuity within catalogue deployment periods remains uncertain.The 44402/2018 record was chosen after examining that station’s 2016 record.Source: §C.1 · p. 11
  • The skip lowers mean energy scores on all three additional records both at validation-selected weights and when every window receives equal weight.Original and additional records differ in calendar splits, normalization data, evaluation stride, training duration, and weight selection, so deployment history alone cannot explain score differences.Source: §C.1 · p. 12

D ELBO terms and Gaussian baselines

The analyses separate reconstruction error, density estimates, and energy scores when comparing skip conditions and observation variances. Results show that lower reconstruction error need not accompany better generative scores, while finite-sample importance estimates and distinct shape metrics require cautious interpretation.

D ELBO terms and Gaussian baselines

  • In the Gaussian comparison from no skip at base variance to skip at high variance, mean conditional NLL rises from 0.04692 to 1.17735 nats per coordinate, while analytic KL falls from 0.09338 to 0.02497.The KL decrease is smaller than the NLL increase; the skip fit’s encoder KL at the same variance is 4.81 nats per window, versus about 0.0003 for one no-skip fit.Source: D ELBO terms and Gaussian baselines · p. 12
  • For 46407 in 2011, mean NLL falls from 27.0860 to −0.6335 while mean MSE rises from 0.006154 to 0.007217 when moving from no skip at base variance to skip at high variance.Both averages use the encoder distribution; NLL is in nats per coordinate and MSE is in squared dimensionless coordinates.Source: D ELBO terms and Gaussian baselines · p. 12
  • The exact reparameterization identity holds mathematically, but the single-precision check found a maximum conditional log-density difference of 1 nat and does not establish marginal IW estimate accuracy.Marginal densities are exact for Gaussian baselines but unknown for neural fits.Source: D ELBO terms and Gaussian baselines · pp. 12, 14

D.1 Importance sampling estimates of log density

  • At 16,384 samples, median DART ESS per fit ranged from 1.0001 to 23.4, and marginal log-density estimate accuracy remained unknown.At base variance the skip estimate was higher for three original records at every tested sample count and encoder distribution; at high variance, differences took both signs for three records.Source: §D.1 · p. 15

E Window means, shape, and energy scores

  • Within-window variance accounts for 99.14–99.68% of original-record test variance, and full-window energy scores assess mean levels as well as shape and their dependence.Subtracting window means changes the Euclidean distance and isolates a shape-distribution assessment.Source: E Window means, shape, and energy scores · p. 16
  • Mean-subtracted windows and time increments assess shape distributions using different distances; the increment representation determines shape up to an additive constant.Both shape scores are proper under finite first moments, while full-window scores also assess mean levels and dependence on shape.Source: E Window means, shape, and energy scores · p. 16

E.1 Baseline distributions

At the same observation variance, PPCA scores better than every neural DART fit, while changing observation variance with fixed network weights can alter energy scores. The leading 16 covariance directions account for nearly all empirical covariance trace, but this does not establish Gaussianity or neural convergence.

E.1 Baseline distributions

  • PPCA has lower mean energy scores than all 84 neural fits on all three scores, including after window-mean removal and for time increments.The comparison changes both model class and fitting procedure.Source: §E.1 · p. 16
  • The leading 16 covariance directions account for more than 99.97% of empirical covariance trace on every record, including after mean subtraction and for time increments.This does not determine Gaussianity, dependence beyond covariance, or whether neural training converged.Source: §E.1 · p. 17

E.2 Observation variance estimation

  • With encoder and decoder weights fixed, training-window reconstruction error selects the variance minimizing the expected Gaussian reconstruction term, but energy score evaluates prior-generated samples against test windows.Variance replacement also changes the posterior; a lower negative ELBO can coexist with increased negative marginal log density.Source: §E.2 · p. 19
  • Changing observation variance with the encoder and decoder fixed can change energy scores, while its effects on exact marginal likelihood and predictive calibration remain unresolved.The variance is used both to generate noise and evaluate density; the comparisons keep encoder and decoder weights and outputs unchanged.Source: §E.2 · p. 19

E.3 Network weights, window weights, and variation through time

The time-varying DART comparisons evaluate skip and no-skip fits at the same original observation variance, across alternative weight and window-weight choices and time intervals. The 2018 full-test partitions and all reported 28-day intervals favor the skip, although some 14-day intervals do not.

E.3 Network weights, window weights, and variation through time

  • Alternative 2018 evaluations compare validation-selected weights with equal UTC start-hour weights against final weights with equal window weights, at the same original observation variance.The evaluations were specified before inspection of the new test results, and the two weight choices use different samples.Source: §E.3 · p. 22, Table 11 · p. 22
  • Each test window is assigned to an interval by its start time, and interval score contributions sum to the overall mean; population confidence intervals would need to account for serial dependence and sparse coverage.Adjacent 2018 windows start four hours apart and share 176 of 192 observations.Source: §E.3 · p. 22
  • For full windows in 2018, every 28-day interval containing test windows favors the skip for all three seeds, while each of records 21415 and 21418 has one 14-day interval with a positive difference.The positive interval carries 0.44% of test weight at 21415 and 5.26% at 21418.Source: §E.3 · p. 22

F Feature ECDFs and spectra

The feature ECDFs compare distributions of window-level means, within-window standard deviations, and increment norms, while the spectra describe variation within windows after subtracting their means. On 2018 records, neural prior samples show less within-window variation than observations, whereas PPCA reproduces more of it, with model-class and fitting differences unresolved.

F Feature ECDFs and spectra

  • The three ECDF features are window mean relative to the training mean, within-window standard deviation, and the Euclidean norm of time increments, each scaled in meters.Matching these feature distributions does not establish matching distributions of complete windows or their dependence between mean and shape.Source: F Feature ECDFs and spectra · p. 26
  • The spectra show variation after subtracting each window mean; tidal oscillations remain, but nearby constituents are unresolved at 0.5-cycle-per-day bin spacing.The plotted median is computed separately at each frequency, so its summed spectrum can differ from the median variance across fits.Source: F Feature ECDFs and spectra · pp. 26, 28
  • On 2018 records, neural prior samples have smaller within-window standard deviations and increment norms than observations, while PPCA reproduces more variation but also differs from the data.PPCA is broader for window standard deviations at 44402/2018; model class, inference method, and fitting procedure differ, so their separate contributions remain unresolved.Source: F Feature ECDFs and spectra · p. 29

G Records, tides, and HEIGHT

The analysis uses selected annual DART station records and routine observation windows, so its comparisons describe those records and procedures. Deployment dates do not establish the causes of height changes, and instrument response and sampling constrain interpretation of spectral power.

G Records, tides, and HEIGHT

  • The seven records cover six sites, with 95 original-record test windows and 773 overlapping test windows from additional records, and the comparisons apply to those selected records and procedures.Serial dependence persists within the observations.Source: G Records, tides, and HEIGHT · p. 29
  • All three listed deployment dates precede their training cutoffs, but deployment dates alone do not establish the causes of height changes; drift, environmental variability, data availability, and reporting types can also affect distributions.The four original records were chosen and inspected before the evaluation.Source: G Records, tides, and HEIGHT · p. 29
  • The routine windows use observations 900 seconds apart, and nearby diurnal or semidiurnal constituents remain unresolved in 192-observation DFT bins spaced 0.5 cycles per day.Interpreting high-frequency power also requires accounting for instrument depth and finite sampling; warning or noise-removal evaluations need independent event or measurement-error evidence.Source: G Records, tides, and HEIGHT · p. 29

Related papers

Loading 2610.37456v1…