PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring the Intrinsic Dimension of Objective Landscapes

Chunyuan Li, Heerad Farkhoor, Rosanne Liu, Jason Yosinski

arXiv:1804.08838v1cs.LGcs.NEstat.ML

TL;DR

The paper asks how many parameters are really needed to solve neural-network problems and whether parameter count measures problem difficulty. It trains models in randomly oriented subspaces, increasing subspace dimension until solutions appear, and uses the resulting intrinsic dimension to compare tasks, compress networks, and approximate solution description length. Across problems, intrinsic dimension can be far below direct parameter count, while the method’s threshold and solution-set interpretations impose stated limitations.

  • Problem

    The paper asks how accurately parameter count reflects problem difficulty and how many parameters are really needed for good performance.

    Source: §1 · p. 2

  • Method

    The method trains networks in randomly oriented subspaces of the full parameter space, increasing subspace dimension until solutions appear and using that dimension as measured intrinsic dimension.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Intrinsic dimension is often much lower than direct parameter dimension and can compare difficulty across supervised and reinforcement-learning problems.

    Source: §4 · p. 9, S6.2 EVOLUTIONARY STRATEGIES (ES)… · pp. 16–17

  • Takeaways & Limitations

    The method provides an upper bound on solution minimum description length and supports network compression, while intrinsic dimension can indicate whether models are well suited to a problem.

    Source: §3.1 · p. 7, §4 · p. 9

    The authors caution that intrinsic dimension does not establish solution-set volume coverage, and d_int100 can vary widely because of baseline matching and regularization effects.

    Source: §2.2 · p. 4, §3.1 · p. 6

Abstract

from arXiv · show

Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty of a problem. But how accurate are such notions? How many parameters are really needed? In this paper we attempt to answer this question by training networks not in their native parameter space, but instead in a smaller, randomly oriented subspace. We slowly increase the dimension of this subspace, note at which dimension solutions first appear, and define this to be the intrinsic dimension of the objective landscape. The approach is simple to implement, computationally tractable, and produces several suggestive conclusions. Many problems have smaller intrinsic dimensions than one might suspect, and the intrinsic dimension for a given dataset varies little across a family of models with vastly different sizes. This latter result has the profound implication that once a parameter space is large enough to solve a problem, extra parameters serve directly to increase the dimensionality of the solution manifold. Intrinsic dimension allows some quantitative comparison of problem difficulty across supervised, reinforcement, and other types of learning where we conclude, for example, that solving the inverted pendulum problem is 100 times easier than classifying digits from MNIST, and playing Atari Pong from pixels is about as hard as classifying CIFAR-10. In addition to providing new cartography of the objective landscapes wandered by parameterized models, the method is a simple technique for constructively obtaining an upper bound on the minimum description length of a solution. A byproduct of this construction is a simple approach for compressing networks, in some cases by more than 100 times.

1 INTRODUCTION

The paper frames neural-network training as movement across high-dimensional objective landscapes whose structure remains difficult to interpret. It proposes studying these landscapes through randomly oriented lower-dimensional slices and measuring the dimension at which solutions appear.

  • Neural-network training traverses an objective landscape determined completely once the dataset and architecture are specified.Source: §1 · p. 1
  • High-dimensional landscapes can defy intuitions formed from familiar two-dimensional hills, valleys, and local optima.Source: §1 · pp. 1–2
  • Random subspace training restricts optimization to d-dimensional slices of the full D-dimensional parameter space.Source: §1 · p. 2
  • The measured intrinsic dimension is the smallest subspace dimension at which solutions first appear during an increasing-dimension sweep.Source: §1 · p. 2
  • The paper measures intrinsic dimensions across neural-network types and datasets, including MNIST, CIFAR-10, ImageNet, and reinforcement-learning tasks.Source: §1 · p. 2

2 DEFINING AND ESTIMATING INTRINSIC DIMENSION

Intrinsic dimension is defined by the codimension of a solution set and estimated by repeatedly optimizing in randomly oriented subspaces of increasing dimension. The method uses performance thresholds to identify approximate solutions in practical neural-network problems.

2 DEFINING AND ESTIMATING INTRINSIC DIMENSION

  • A toy problem with D = 1000 has a 990-dimensional solution manifold, leaving intrinsic dimension d_int = 10.The ten intrinsic dimensions correspond intuitively to the ten sum constraints imposed on the parameter vector.Source: §2 · p. 2
  • Random subspace optimization estimates intrinsic dimension for complicated, data-dependent objectives where direct algebraic calculation is unavailable.Source: §2.1 · p. 2

2.1 MEASURING INTRINSIC DIMENSION VIA RANDOM SUBSPACE TRAINING

  • The parameterization uses a frozen random D × d projection matrix and a trainable d-dimensional vector, leaving only d degrees of freedom.The projected initialization preserves the region selected by standard neural-network initialization schemes.Source: §2.1 · p. 3
  • When d = D, the construction recovers direct optimization or its rotated equivalent; when d < D, solutions are sought by repeatedly increasing d and rerunning optimization.Source: §2.1 · p. 3
  • In the toy example, both 90% and 100% solvability first occur at random subspace dimension 10.Source: Figure 1 · p. 3, §2.1 · p. 4

2.2 DETAILS AND CONVENTIONS

  • The practical measure d_int90 defines solutions as achieving at least 90% of baseline performance, while d_int100 can vary widely near a well-tuned baseline.The 90% threshold is chosen as a trade-off between solution quality and robustness to training noise.Source: §2.2 · p. 4
  • Changing the performance threshold somewhat is expected to leave most conclusions qualitatively unchanged.Source: §2.2 · p. 4

3 RESULTS AND DISCUSSION

Experiments show that intrinsic dimension can be far below native parameter count, remains relatively stable across model sizes, and supports comparisons of task difficulty and network compressibility.

MNIST

  • 750 subspace dimensions achieve 90% of the directly trained MNIST FC network’s validation accuracy.The network has D = 199,210 parameters.Source: §3.1 · p. 4

Compression

  • 750 degrees of freedom, or 0.4% of the full parameter count, suffice to obtain 90% of the direct baseline’s performance.The resulting representation compresses the network from 793kB to 3.2kB, a 260× reduction.Source: §3.1 · pp. 4–5

Robustness

  • 24.1× variation in native parameters across 20 FC MNIST models corresponds to only 1.33× variation in measured intrinsic dimension.The authors report that this stability holds across networks varying in width and depth, though some variation may reflect noise.Source: §3.1 · pp. 5–6

Parameter efficiency

  • Random subspace training uses fewer trainable parameters than directly trained FC networks at comparable performance levels.For convolutional networks, the gap between direct and subspace training is closer at fixed parameter budgets, while direct-training efficiency varies by architecture.Source: §3.1 · p. 7

MNIST

  • 290 intrinsic dimensions suffice for LeNet on MNIST, versus 750 for the FC network, yielding about 150× compression for LeNet.The comparison supports lower intrinsic dimension as an indicator of greater parameter efficiency for this task and model pair.Source: §3.1 · p. 7

Cross-task comparisons

  • 4, 700, and 6k intrinsic dimensions are measured for InvertedPendulum, Humanoid, and Atari Pong, respectively.Humanoid is similar to MNIST FC classification, while Pong is on the same order as CIFAR-10; shuffled-label MNIST requires dint90 = 190,000.Source: §3.1 · p. 8, §3.3 · p. 9

4 CONCLUSIONS AND FUTURE DIRECTIONS

The paper defines intrinsic dimension through random subspace training and uses it to compare objective-landscape difficulty and enable network compression. It also identifies better subspace constructions and broader applications as future directions.

  • Random subspace training approximates intrinsic dimension for neural-network modeling problems and supports comparisons across learning domains.Source: §4 · p. 9
  • In some problems, intrinsic dimension is much lower than direct parameter dimension, enabling network compression.Source: §4 · p. 9
  • Future work could replace random linear subspaces with carefully constructed linear or nonlinear subspaces more likely to contain solutions.Source: §4 · p. 9
  • Intrinsic-dimension methods may help assess components of larger, heterogeneous networks trained with multiple modules and losses.Source: §4 · p. 9

S5 ADDITIONAL MNIST RESULTS AND INSIGHTS

Additional MNIST experiments examine variance, baselines, shuffled labels, optimizer effects, and optimization stability. They show that intrinsic dimension can remain relatively stable across model sizes while increasing with memorization demands.

Sweep and variance

Shuffled labels

Optimization stability

Optimizers

S6 ADDITIONAL REINFORCEMENT LEARNING RESULTS AND DETAILS

The reinforcement-learning experiments measure intrinsic dimension across CartPole, evolutionary-strategy tasks, and Atari Pong. Results vary with task and algorithm, with noise affecting reward curves and evaluation requiring repeated runs in some settings.

DQN on CartPole

  • CartPole DQN uses an FC network with L=2 and W=400, averages at least five runs, and sets the baseline reward to 195.07.Source: S6.1 DQN EXPERIMENTS · p. 15
  • CartPole and Pole-and-Cart intrinsic dimensions are reported as dint90 = 25 and 23, respectively.Source: S6.1 DQN EXPERIMENTS · p. 15

Evaluation

Evolution Strategies

Atari Pong

S7 THREE METHODS OF RANDOM PROJECTION

The paper compares dense, sparse, and Fastfood random projections for mapping low-dimensional parameters into large network spaces. These methods trade computational and memory costs while enabling larger intrinsic-dimension measurements.

Projection setup

Dense projections

Sparse projections

Fastfood

Compute cost

S8 ADDITIONAL CIFAR-10 RESULTS

CIFAR-10 experiments show that intrinsic dimension varies less than native network size across FC models, while architecture and regularization affect the dimensions needed for successful training.

FC networks

  • A 12.16× change in native dimension D across FC networks corresponds to a 5.0× change in dint90.Much of the change is attributed to differences in baseline performance.Source: S8 ADDITIONAL CIFAR-10 RESULTS · p. 18

Problem difficulty

ResNet vs. LeNet

  • ResNet reaches the LeNet baseline at dint90 = 1000 ∼2000, but requires dint90 = 20,000 ∼50,000 to reach its higher baseline.The tested ResNet has 280k parameters.Source: S8 ADDITIONAL CIFAR-10 RESULTS · p. 19

The role of regularizers

  • Subspace training acts as a regularizer by restricting the solution set, with especially strong effects when d is small.It is compared with ℓ2 weight penalties and Dropout.Source: S8 ADDITIONAL CIFAR-10 RESULTS · p. 19
  • Subspace training tends to overfit less than direct training under testing NLL comparisons with Dropout.Source: S8 ADDITIONAL CIFAR-10 RESULTS · p. 19
  • When ℓ2 penalty < 5×10^-4 or dropout rate < 0.1, properly chosen subspace training outperforms direct training on testing NLL.With large d, subspace training also overfits the training dataset.Source: S8 ADDITIONAL CIFAR-10 RESULTS · p. 19

S9 IMAGENET

The ImageNet experiment used SqueezeNet to probe a larger problem, but limited training time prevented a reliable estimate of intrinsic dimension.

ImageNet

  • ImageNet training produced no reliable dint90 estimate, except that it is over 500k.Each intrinsic-dimension setting took about 6 to 7 days across 4 GPUs.Source: S9 IMAGENET · p. 20

S10 INVESTIGATION OF CONVOLUTIONAL NETWORKS

Control experiments on LeNet variants indicate that both local connectivity and weight tying contribute to convolutional networks’ parameter efficiency.

Regularization comparisons

ImageNet context

  • At d = 500k, SqueezeNet reaches 34.34% ImageNet accuracy, below the threshold required to estimate dint90.Source: Figure S18 · p. 22

Architectural variants

Results

S11 SUMMARIZATION OF dint90

The paper summarizes dint90 across datasets and architectures as an estimate of the minimum trainable dimension needed to solve each objective landscape.

Definition and scope

  • dint90 indicates the minimum number of trainable-parameter dimensions required to properly solve a problem and reflects its difficulty level.Source: S11 SUMMARIZATION OF dint90 · p. 22

Summary

Notation

Interpretation

Related papers

Loading 1804.08838v1…