PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring the Intrinsic Dimension of Objective Landscapes

Chunyuan Li, Heerad Farkhoor, Rosanne Liu, Jason Yosinski

arXiv:1804.08838v1cs.LGcs.NEstat.ML

TL;DR

The paper asks how many parameters are really needed to solve neural-network problems and whether parameter count measures problem difficulty. It trains models in randomly oriented subspaces, increasing subspace dimension until solutions appear, and uses the resulting intrinsic dimension to compare tasks, compress networks, and approximate solution description length. Across problems, intrinsic dimension can be far below direct parameter count, while the method’s threshold and solution-set interpretations impose stated limitations.

  • Problem

    The paper asks how accurately parameter count reflects problem difficulty and how many parameters are really needed for good performance.

  • Method

    The method trains networks in randomly oriented subspaces of the full parameter space, increasing subspace dimension until solutions appear and using that dimension as measured intrinsic dimension.

  • Results

    Intrinsic dimension is often much lower than direct parameter dimension and can compare difficulty across supervised and reinforcement-learning problems.

  • Takeaways & Limitations

    The method provides an upper bound on solution minimum description length and supports network compression, while intrinsic dimension can indicate whether models are well suited to a problem.

  • Takeaways & Limitations

    The authors caution that intrinsic dimension does not establish solution-set volume coverage, and d_int100 can vary widely because of baseline matching and regularization effects.

Abstract

from arXiv · show

Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty of a problem. But how accurate are such notions? How many parameters are really needed? In this paper we attempt to answer this question by training networks not in their native parameter space, but instead in a smaller, randomly oriented subspace. We slowly increase the dimension of this subspace, note at which dimension solutions first appear, and define this to be the intrinsic dimension of the objective landscape. The approach is simple to implement, computationally tractable, and produces several suggestive conclusions. Many problems have smaller intrinsic dimensions than one might suspect, and the intrinsic dimension for a given dataset varies little across a family of models with vastly different sizes. This latter result has the profound implication that once a parameter space is large enough to solve a problem, extra parameters serve directly to increase the dimensionality of the solution manifold. Intrinsic dimension allows some quantitative comparison of problem difficulty across supervised, reinforcement, and other types of learning where we conclude, for example, that solving the inverted pendulum problem is 100 times easier than classifying digits from MNIST, and playing Atari Pong from pixels is about as hard as classifying CIFAR-10. In addition to providing new cartography of the objective landscapes wandered by parameterized models, the method is a simple technique for constructively obtaining an upper bound on the minimum description length of a solution. A byproduct of this construction is a simple approach for compressing networks, in some cases by more than 100 times.

1 INTRODUCTION

The paper frames neural-network training as movement across high-dimensional objective landscapes whose structure remains difficult to interpret. It proposes studying these landscapes through randomly oriented lower-dimensional slices and measuring the dimension at which solutions appear.

  • Neural-network training traverses an objective landscape determined completely once the dataset and architecture are specified.
  • High-dimensional landscapes can defy intuitions formed from familiar two-dimensional hills, valleys, and local optima.
  • Random subspace training restricts optimization to d-dimensional slices of the full D-dimensional parameter space.
  • The measured intrinsic dimension is the smallest subspace dimension at which solutions first appear during an increasing-dimension sweep.
  • The paper measures intrinsic dimensions across neural-network types and datasets, including MNIST, CIFAR-10, ImageNet, and reinforcement-learning tasks.

2 DEFINING AND ESTIMATING INTRINSIC DIMENSION

Intrinsic dimension is defined by the codimension of a solution set and estimated by repeatedly optimizing in randomly oriented subspaces of increasing dimension. The method uses performance thresholds to identify approximate solutions in practical neural-network problems.

  • 2 DEFINING AND ESTIMATING INTRINSIC DIMENSION: A toy problem with D = 1000 has a 990-dimensional solution manifold, leaving intrinsic dimension d_int = 10.The ten intrinsic dimensions correspond intuitively to the ten sum constraints imposed on the parameter vector.
  • 2 DEFINING AND ESTIMATING INTRINSIC DIMENSION: Random subspace optimization estimates intrinsic dimension for complicated, data-dependent objectives where direct algebraic calculation is unavailable.
  • 2.1 MEASURING INTRINSIC DIMENSION VIA RANDOM SUBSPACE TRAINING: The parameterization uses a frozen random D × d projection matrix and a trainable d-dimensional vector, leaving only d degrees of freedom.The projected initialization preserves the region selected by standard neural-network initialization schemes.
  • 2.1 MEASURING INTRINSIC DIMENSION VIA RANDOM SUBSPACE TRAINING: When d = D, the construction recovers direct optimization or its rotated equivalent; when d < D, solutions are sought by repeatedly increasing d and rerunning optimization.
  • 2.1 MEASURING INTRINSIC DIMENSION VIA RANDOM SUBSPACE TRAINING: In the toy example, both 90% and 100% solvability first occur at random subspace dimension 10.
  • 2.2 DETAILS AND CONVENTIONS: The practical measure d_int90 defines solutions as achieving at least 90% of baseline performance, while d_int100 can vary widely near a well-tuned baseline.The 90% threshold is chosen as a trade-off between solution quality and robustness to training noise.
  • 2.2 DETAILS AND CONVENTIONS: Changing the performance threshold somewhat is expected to leave most conclusions qualitatively unchanged.

3 RESULTS AND DISCUSSION

Experiments show that intrinsic dimension can be far below native parameter count, remains relatively stable across model sizes, and supports comparisons of task difficulty and network compressibility.

  • MNIST: 750 subspace dimensions achieve 90% of the directly trained MNIST FC network’s validation accuracy.The network has D = 199,210 parameters.
  • Compression: 750 degrees of freedom, or 0.4% of the full parameter count, suffice to obtain 90% of the direct baseline’s performance.The resulting representation compresses the network from 793kB to 3.2kB, a 260× reduction.
  • Robustness: 24.1× variation in native parameters across 20 FC MNIST models corresponds to only 1.33× variation in measured intrinsic dimension.The authors report that this stability holds across networks varying in width and depth, though some variation may reflect noise.
  • Parameter efficiency: Random subspace training uses fewer trainable parameters than directly trained FC networks at comparable performance levels.For convolutional networks, the gap between direct and subspace training is closer at fixed parameter budgets, while direct-training efficiency varies by architecture.
  • MNIST: 290 intrinsic dimensions suffice for LeNet on MNIST, versus 750 for the FC network, yielding about 150× compression for LeNet.The comparison supports lower intrinsic dimension as an indicator of greater parameter efficiency for this task and model pair.
  • Cross-task comparisons: 4, 700, and 6k intrinsic dimensions are measured for InvertedPendulum, Humanoid, and Atari Pong, respectively.Humanoid is similar to MNIST FC classification, while Pong is on the same order as CIFAR-10; shuffled-label MNIST requires dint90 = 190,000.

4 CONCLUSIONS AND FUTURE DIRECTIONS

The paper defines intrinsic dimension through random subspace training and uses it to compare objective-landscape difficulty and enable network compression. It also identifies better subspace constructions and broader applications as future directions.

  • Random subspace training approximates intrinsic dimension for neural-network modeling problems and supports comparisons across learning domains.
  • In some problems, intrinsic dimension is much lower than direct parameter dimension, enabling network compression.
  • Future work could replace random linear subspaces with carefully constructed linear or nonlinear subspaces more likely to contain solutions.
  • Intrinsic-dimension methods may help assess components of larger, heterogeneous networks trained with multiple modules and losses.

S5 ADDITIONAL MNIST RESULTS AND INSIGHTS

Additional MNIST experiments examine variance, baselines, shuffled labels, optimizer effects, and optimization stability. They show that intrinsic dimension can remain relatively stable across model sizes while increasing with memorization demands.

  • Sweep and variance: 20 FC networks spanning 1–5 hidden layers and widths of 50–400 were evaluated using three runs per subspace dimension and bootstrap variance estimates.
  • Shuffled labels: On 50k shuffled-label MNIST, the L=5, W=400 network has dint90 = 190k, compared with dint90 = 750 on standard MNIST.
  • Shuffled labels: As shuffled-label dataset size increases, intrinsic dimension increases sublinearly, suggesting shared memorization machinery across training examples.
  • Optimization stability: Direct training fails at depths above five, whereas subspace training maintains stable performance across the tested depths.
  • Optimizers: Adam generally produces higher intrinsic dimensions than SGD because it achieves higher baselines, especially with individual baselines.

S6 ADDITIONAL REINFORCEMENT LEARNING RESULTS AND DETAILS

The reinforcement-learning experiments measure intrinsic dimension across CartPole, evolutionary-strategy tasks, and Atari Pong. Results vary with task and algorithm, with noise affecting reward curves and evaluation requiring repeated runs in some settings.

  • DQN on CartPole: CartPole DQN uses an FC network with L=2 and W=400, averages at least five runs, and sets the baseline reward to 195.07.
  • DQN on CartPole: CartPole and Pole-and-Cart intrinsic dimensions are reported as dint90 = 25 and 23, respectively.
  • Evaluation: RL reward curves are not necessarily monotonic because the games are sensitive to noise, while the figure reports rewards averaged over the last 100 episodes.
  • Evolution Strategies: InvertedPendulum trained with ES has dint90 = 4, smaller but of the same order of magnitude as CartPole, despite richer dynamics.
  • Evolution Strategies: Humanoid walking has dint90 = 700, similar to MNIST FC at dint90 = 650 and below CIFAR-10 convnet at dint90 = 2,500.
  • Atari Pong: Atari Pong from pixels with a roughly 1M-parameter convnet has dint90 = 6,000.

S7 THREE METHODS OF RANDOM PROJECTION

The paper compares dense, sparse, and Fastfood random projections for mapping low-dimensional parameters into large network spaces. These methods trade computational and memory costs while enabling larger intrinsic-dimension measurements.

  • Projection setup: Subspace training maps v ∈ R^d into R^D using an orthonormal random matrix and an offset vector, rather than optimizing all D parameters directly.
  • Dense projections: Dense random projections require O(Dd) time and storage, creating scaling limits for large models or subspaces.
  • Sparse projections: Very sparse projections reduce storage and multiplication costs and enabled measurement at d=2,500 for CIFAR-10.
  • Sparse projections: The sparse implementation achieved closer to a constant 10× speed improvement than the theoretical D-factor improvement and could not scale to million-parameter models with large intrinsic dimensions.
  • Fastfood: Fastfood implicitly generates projections using O(D) space and fast transforms, avoiding explicit storage of the transformation matrix.
  • Compute cost: Subspace training is more computationally expensive per update because it propagates through both the network and the projection module.

S8 ADDITIONAL CIFAR-10 RESULTS

CIFAR-10 experiments show that intrinsic dimension varies less than native network size across FC models, while architecture and regularization affect the dimensions needed for successful training.

  • FC networks: A 12.16× change in native dimension D across FC networks corresponds to a 5.0× change in dint90.Much of the change is attributed to differences in baseline performance.
  • Problem difficulty: CIFAR-10 is harder than MNIST under the paper’s baselines of 58% and 99% accuracy, respectively.
  • ResNet vs. LeNet: ResNet reaches the LeNet baseline at dint90 = 1000 ∼2000, but requires dint90 = 20,000 ∼50,000 to reach its higher baseline.The tested ResNet has 280k parameters.
  • The role of regularizers: Subspace training acts as a regularizer by restricting the solution set, with especially strong effects when d is small.It is compared with ℓ2 weight penalties and Dropout.
  • The role of regularizers: Subspace training tends to overfit less than direct training under testing NLL comparisons with Dropout.
  • The role of regularizers: When ℓ2 penalty < 5×10^-4 or dropout rate < 0.1, properly chosen subspace training outperforms direct training on testing NLL.With large d, subspace training also overfits the training dataset.

S9 IMAGENET

The ImageNet experiment used SqueezeNet to probe a larger problem, but limited training time prevented a reliable estimate of intrinsic dimension.

  • ImageNet: ImageNet training produced no reliable dint90 estimate, except that it is over 500k.Each intrinsic-dimension setting took about 6 to 7 days across 4 GPUs.

S10 INVESTIGATION OF CONVOLUTIONAL NETWORKS

Control experiments on LeNet variants indicate that both local connectivity and weight tying contribute to convolutional networks’ parameter efficiency.

  • Regularization comparisons: Validation-accuracy comparisons also examine ℓ2 penalty, Dropout, and subspace training as alternative regularization effects.
  • ImageNet context: At d = 500k, SqueezeNet reaches 34.34% ImageNet accuracy, below the threshold required to estimate dint90.
  • Architectural variants: The LeNet control study varies local connectivity and weight sharing using Standard, Untied, FCTied, and FC variants.
  • Results: On MNIST, the four LeNet variants require dint90 values of 290, 600, 425, and 2000, respectively.
  • Results: On CIFAR-10, the corresponding dint90 values are 1000, 2750, 2500, and 35000.
  • Results: Experiments show that both tied weights and local connections are important to the model.

S11 SUMMARIZATION OF dint90

The paper summarizes dint90 across datasets and architectures as an estimate of the minimum trainable dimension needed to solve each objective landscape.

  • Definition and scope: dint90 indicates the minimum number of trainable-parameter dimensions required to properly solve a problem and reflects its difficulty level.
  • Summary: Table S5 and Fig. S20 summarize intrinsic dimensions across the different datasets and neural-network architectures tested.
  • Notation: The summary uses SP for shuffled pixel, SL for shuffled label, and FC-5 for a 5-layer fully connected network.
  • Interpretation: The reported intrinsic dimensions provide a cross-problem comparison of objective-landscape difficulty.
Loading 1804.08838v1…