PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, Jeff Clune

arXiv:1912.07768v1cs.LGstat.ML

TL;DR

Training neural networks and evaluating many NAS candidates can require substantial data and computation, motivating methods that generate more efficient teaching data. The paper introduces GTNs, which meta-learn generators that train learners through synthetic-data inner loops and target-task outer-loop evaluation. GTNs accelerate learning and enable faster NAS evaluation, while the experiments and discussion remain bounded by supervised settings and unresolved questions about synthetic data that differs from real data.

  • Problem

    Determining whether learning algorithms can generate synthetic training data that helps learners rapidly acquire target-task performance matters because evaluating many architectures on full datasets is expensive.

  • Method

    GTNs meta-learn a generator that produces synthetic learner data, differentiating through nested learner-training and outer-loop target-task evaluation.

  • Results

    GTNs enable much faster learning, including a 9-fold training speed-up, and GTN-NAS finds higher-performing architectures than comparable methods when controlling for the search algorithm.

  • Takeaways & Limitations

    GTNs provide a route to rapidly training independent architectures and may support on-demand networks with specified performance, efficiency, or skill trade-offs.

  • Takeaways & Limitations

    The experiments focus on supervised learning, while the broader applicability of GTNs and the significance of their dissimilarity from real data remain open questions.

Abstract

from arXiv · show

This paper investigates the intriguing question of whether we can create learning algorithms that automatically generate training data, learning environments, and curricula in order to help AI agents rapidly learn. We show that such algorithms are possible via Generative Teaching Networks (GTNs), a general approach that is, in theory, applicable to supervised, unsupervised, and reinforcement learning, although our experiments only focus on the supervised case. GTNs are deep neural networks that generate data and/or training environments that a learner (e.g. a freshly initialized neural network) trains on for a few SGD steps before being tested on a target task. We then differentiate through the entire learning process via meta-gradients to update the GTN parameters to improve performance on the target task. GTNs have the beneficial property that they can theoretically generate any type of data or training environment, making their potential impact large. This paper introduces GTNs, discusses their potential, and showcases that they can substantially accelerate learning. We also demonstrate a practical and exciting application of GTNs: accelerating the evaluation of candidate architectures for neural architecture search (NAS), which is rate-limited by such evaluations, enabling massive speed-ups in NAS. GTN-NAS improves the NAS state of the art, finding higher performing architectures when controlling for the search proposal mechanism. GTN-NAS also is competitive with the overall state of the art approaches, which achieve top performance while using orders of magnitude less computation than typical NAS methods. Speculating forward, GTNs may represent a first step toward the ambitious goal of algorithms that generate their own training data and, in doing so, open a variety of interesting new research questions and directions.

1 INTRODUCTION AND RELATED WORK

The paper asks whether synthetic data can teach learners more efficiently than real data and proposes GTNs as a scalable approach, motivated especially by the cost of evaluating many NAS candidates.

  • Motivation: Training neural networks need not use all available data, because intelligently sampled surrogate datasets can achieve competitive test performance with less training effort.This motivates replacing full datasets with more efficient training inputs.
  • Motivation: NAS evaluation is expensive because thousands or millions of candidate architectures may require training on full datasets.Rapid architecture evaluation is therefore a central target for training-efficiency methods.
  • Core idea: GTNs learn to generate synthetic data that teaches a target task efficiently, without requiring the generated data to resemble the original data distribution.The paper contrasts this teaching objective with methods that select or generate data to represent the original distribution.
  • Core idea: GTN training uses nested inner and outer loops to optimize a generator that produces learner training data.The learner trains on synthetic data while the generator is updated through the learner’s target-task performance.
  • NAS relevance: GTNs are agnostic to learner initialization and architecture, enabling their synthetic data to accelerate evaluation across candidate architectures in NAS.This distinguishes GTNs from approaches whose usefulness depends on similar learner parameterizations.
  • Future directions: GTNs may support on-demand creation of networks meeting particular performance, speed, energy, or skill requirements.These applications are presented as future possibilities enabled by rapidly training new architectures.

2 METHODS

GTNs train a generator and learner through nested meta-learning loops, using synthetic data for inner-loop learning and real task data for outer-loop evaluation. The methods section also addresses learner generality, meta-gradient instability, curriculum control, and NAS evaluation speed.

  • GTN method: GTNs train a data-generating network so that a learner trained on its outputs achieves high target-task accuracy quickly.The generator and learner cooperate because both are optimized toward learner performance on the target task.
  • Inner loop: In the inner loop, G(z, y) maps Gaussian noise and labels to synthetic data, which trains the learner for a fixed number of optimizer steps.The paper uses SGD with momentum, while allowing optimizers such as SGD or Adam in the general formulation.
  • Outer loop: The outer loop evaluates the learner trained on synthetic data against real training data and uses the resulting meta-loss to update the generator.This differentiates through the learner-training process to optimize the data generator.
  • Learner agnosticism: Each new outer-loop training selects and randomly initializes a learner architecture from a predefined set, encouraging synthetic data to generalize across architectures and initializations.This procedure operationalizes the goal of learner-agnostic GTN data.
  • Optimization: Meta-gradient training is unstable and hyperparameter-sensitive, with optimization diverging when the number of inner-loop steps increases.The authors introduce weight normalization to mitigate this instability and simplify the optimization procedure.
  • Optimization: Weight normalization parameterizes weights as W = g · V/∥V∥ and removes the need to calculate and combine Evolution Strategies gradients with backpropagation gradients.The paper hypothesizes that this stabilization may apply beyond GTNs, but leaves that broader evaluation to future work.
  • Curriculum learning: GTNs encode curricula through ordered generator inputs, allowing curriculum optimization without the nondifferentiable sample indexes used in conventional dataset curricula.The experiments compare variants with increasing control over the ordering of generator inputs.
  • NAS acceleration: GTN evaluation replaces real search-evaluation data with synthetic data to reduce the time required to train each NAS candidate.The CIFAR10 setup uses a smaller architecture and shorter search evaluation before final evaluation.

3 RESULTS

GTNs improve few-step learning by jointly learning synthetic data and its presentation curriculum, then use this accelerated training as a proxy for neural architecture search. Across MNIST and CIFAR10, the approach outperforms relevant controls and provides a substantially cheaper architecture-evaluation signal.

  • 3.3 GTNS FOR SUPERVISED LEARNING: GTNs enable more rapid learning than real data across MNIST, CIFAR10, and cart-pole, while MNIST GTNs significantly outperform dataset distillation and real-data controls after 32 SGD steps.For MNIST, the gain is attributed to synthetic data compressing information from the real training set into examples that support faster learning.
  • 3.1 IMPROVING STABILITY WITH WEIGHT NORMALIZATION: Weight normalization substantially improves GTN meta-learning robustness and final learner performance across hyperparameter-tuning runs.The comparison used 20 MNIST runs with and without weight normalization.
  • 3.2 IMPROVING GTNS WITH A CURRICULUM: Learning both which synthetic samples to present and their order makes the full-curriculum GTN outperform the other curriculum variants.The full curriculum is used for the remaining GTN experiments.
  • 3.4 ARCHITECTURE SEARCH WITH GTNS: GTN evaluation correlates 0.5582 with 100-epoch real-data evaluation among the top 50% of architectures, slightly exceeding 3-epoch real-data correlation of 0.5235.This corresponds to an approximately 9× cost reduction per trained model.
  • 3.4 ARCHITECTURE SEARCH WITH GTNS: Using GTN-generated data to evaluate 800 randomly generated CIFAR10 architectures achieves state-of-the-art results when controlling for the search algorithm.GTN-NAS remains competitive with methods using more advanced architecture-proposal techniques, though that comparison is described as apples-to-oranges.

4 DISCUSSION, FUTURE WORK, AND CONCLUSION

GTNs support rapid training across architectures and suggest applications in NAS, on-demand model creation, reinforcement learning, and AI systems that generate training environments. The paper concludes that GTNs improve few-step learning and architecture evaluation while opening broader research directions.

  • Future applications: GTNs can rapidly train independent models, making them useful when many models must be trained, as in neural architecture search.The paper also suggests using GTNs to create models with specified trade-offs among accuracy, inference time, and memory.
  • Open questions: GTN synthetic data can differ substantially from real MNIST and CIFAR10 images, raising questions about how neural networks learn from unnatural inputs.The paper presents possible explanations as speculative hypotheses rather than established conclusions.
  • Open questions: Weight normalization improved GTN training stability and motivates testing whether it similarly stabilizes other meta-gradient methods.The authors identify this as a hypothesis for future investigation.
  • Future work: Future extensions include combining GTNs with weight sharing or HyperNetworks, learning unsupervised embeddings, stabilizing GANs, and using closed-loop curricula.A recurrent GTN could condition samples on prior learner outputs to adjust difficulty during training.
  • Future work: GTNs can generate training environments for reinforcement-learning agents, demonstrated on the simple CartPole task.The paper also considers combining this capability with open-ended systems that generate new learning tasks.
  • Conclusion: GTN-generated data can produce better few-step accuracy than equivalent real data while generalizing across architectures and random initializations.The authors use this efficient training data to build a fast NAS method that finds state-of-the-art architectures when controlling for the search algorithm.
  • Conclusion: GTNs address the computational bottleneck of evaluating candidate architectures and may enable algorithms that generate their own training data.The authors present this as an intriguing research direction rather than a completed capability.

APPENDIX A ADDITIONAL EXPERIMENTAL DETAILS

The appendix describes GTN optimization as nested learner and generator training, with domain-specific outer losses and synthetic-data inner losses. Noise and labels drive the generator, while learned noise can encode a curriculum.

  • Losses: The outer-loop loss is domain specific, while supervised MNIST and CIFAR experiments use cross-entropy on real data.The inner-loop loss matches the outer-loop loss but uses synthetic data instead of real data.
  • Optimization: The learner’s inner-loop update uses stochastic gradient descent with momentum and parameters θt, α, and β.The cited equation defines the update, while α and β are the learning-rate and momentum hyperparameters.
  • Generator inputs: The generator receives Gaussian noise and uniformly sampled class labels to produce synthetic training inputs.The noise can instead be optimized directly, with labels fixed, to learn a curriculum.

A.1 MNIST EXPERIMENTS:

The MNIST and CIFAR10 experiments use sampled convolutional learners and fully connected generators, with dataset-specific architectural details and normalization or augmentation choices documented in hyperparameter tables.

  • MNIST: MNIST learners use two convolutional layers with uniformly sampled filter counts, max pooling, and a subsequent fully connected layer.The first and second convolutional layers sample filters from U([32, 128]) and U([64, 256]), respectively.
  • MNIST: The MNIST generator uses two fully connected layers followed by two convolutional layers, ending with a one-filter Tanh output.The first generator convolution has 64 filters, and the fully connected widths depend on the synthetic image size H.
  • MNIST: The MNIST experiment hyperparameters are listed in Table 2.
  • CIFAR10: CIFAR10 learners use five convolutional layers, global average pooling, and a fully connected layer, with downsampling in the second and fourth convolutions.The first convolution samples its filter count from U([32, 128]); later convolutions use U([64, 256]).
  • CIFAR10: CIFAR10 training augments real data with random crops and horizontal flips, and its generator outputs three channels rather than one.Weight normalization is used during GTN-based architecture evaluation but not added to the final architectures found by search.
  • CIFAR10: The CIFAR10 experiment hyperparameters are listed in Table 3.

APPENDIX B REASONS GTNS ARE NOT EXPECTED TO PRODUCE SOTA ACCURACY VS. ASYMPTOTIC PERFORMANCE WHEN TRAINING ON REAL DATA

GTN learners are not expected to match state-of-the-art accuracy because they train for far fewer steps and use compressed data optimized for rapid evaluation across architectures. Meta-gradient memory costs also grow with learner size and inner-loop length, though checkpointing extends feasibility.

  • Accuracy expectations: GTN learners use only 32 or 128 SGD steps, compared with tens of thousands for state-of-the-art training.
  • Accuracy expectations: GTN data is compressed and optimized to generalize across diverse architectures for rapid evaluation, rather than to maximize accuracy for one manually designed architecture.
  • Accuracy expectations: State-of-the-art methods may use data beyond the benchmark and complex augmentation schemes, unlike the constrained GTN comparison.
  • Computational constraints: Meta-gradient memory requirements grow with network size and the number of inner-loop steps because activations and weights must support second-order gradients.Gradient checkpointing reduces storage by retaining learner weights and recomputing activations, enabling larger networks and longer inner loops.

APPENDIX E EXTENDED NAS RESULTS

GTN-based architecture search is competitive with leading methods under limited computation and uses GTN-generated data to evaluate many architectures efficiently.

  • 2.92% mean error places GTN third among GHN, ENAS, GTN, and NAONet in the less-than-1-day computation regime.The corresponding errors are 2.84%, 2.89%, 2.92%, and 2.93%, respectively.
  • GTN is the only one of the four limited-computation methods relying on Random Search for architecture proposal.
  • 16h total search used 8h to train the GTN and 8h to evaluate 800 architectures with GTN-produced synthetic data.Each architecture was evaluated five times with different initializations, reporting mean ± SD.
  • GTN found better architectures than Random-WS and GHN-Top and was competitive with NAONet and ENAS.NAONet and ENAS use non-random architecture proposals, whereas GTN relies on random search in the comparison.

APPENDIX F CONDITIONED GENERATOR VS. XY-GENERATOR

The appendix compares GTN generators conditioned on one-hot labels with generators that jointly produce images and soft labels. The soft-label variant under-performed, despite its expected informational advantage and sensitivity to tuning.

  • The dark-knowledge generator was expected to outperform the conditional generator because soft labels encode more information than one-hot labels.
  • Jointly generating images and soft labels under-performs generating only images.The authors note that different hyperparameters or improved training stability could change this result.
  • Initialization and training of the dark-knowledge variant were more sensitive than those of the conditional generator.The authors therefore suggest that more rigorous tuning could produce a different comparison.

APPENDIX G GTN GENERATES (SEEMINGLY) ENDLESS DATA

GTNs can generate additional synthetic data beyond the batch size and optimization steps used during meta-training. More generated data improves learner performance, although real data becomes more effective after enough inner-loop steps.

  • Larger evaluation batch sizes improve GTN learner performance beyond the meta-trained batch size of 128, with diminishing returns.This indicates that GTNs can sample larger training sets without being limited by the training batch size.
  • Increasing inner-loop optimization steps improves performance on GTN-generated data, showing that the data is not overfit to the meta-training step count.
  • GTN-generated data is significantly better than real data in the very low-data regime, whereas real data becomes significantly more effective after more inner-loop steps.Both comparisons are reported with p < 0.05.
  • GTN samples without curriculum are shown as a separate evaluation condition in Figure 6.
  • The learner-distribution experiment tests whether different initializations trained from one GTN collapse to functionally similar solutions.The motivation is to assess whether ensembling diverse GTN learners remains possible.

APPENDIX H GTN FOR RL

A CartPole experiment extends GTNs to reinforcement learning, where GTN-generated synthetic data can train a learner to maximum performance in one SGD step after meta-training.

  • The CartPole inner loop uses supervised regression, while the outer loop is reinforced because the simulator is non-differentiable.
  • GTNs could generate an entire synthetic MDP by producing initial states, subsequent states, and optionally rewards for an agent's actions.The paper presents synthetic MDPs as a future research direction.
  • GTN and A2C effectively solve CartPole, while GTN training uses the same number of simulator steps as training one policy-gradient learner.
  • One SGD step on GTN-generated synthetic data trains a learner to maximum CartPole performance.The authors caution that this is unlikely to hold for harder target reinforcement-learning tasks.
  • The CartPole learner uses a single-layer network with 64 hidden units, tanh activation, batch size 512, and one inner-loop training iteration.

APPENDIX I SOLVING MODE COLLAPSE IN GANS WITH GTNS

GTNs offer a different route to addressing GAN mode collapse by optimizing synthetic data for learner performance rather than realism. The paper presents GTN–GAN combination as an interesting option, while explicitly leaving its comparative advantage unresolved.

  • GANs tended to generate images from only one class, exemplifying mode collapse caused by training instabilities.
  • The proposed GTN–GAN approach is presented as an interesting option rather than as better than existing mode-collapse techniques.
  • GTNs are motivated as a possible step toward algorithms that generate their own training environments and curricula.
  • Combining GTNs with GANs produced more realistic images than GTNs alone and stabilized GAN training, preventing mode collapse.

APPENDIX K ON THE REALISM OF IMAGES

GTN-generated images can be largely unrecognizable while still supporting strong MNIST performance, with recognizable images appearing sharply near the curriculum’s end. The paper tests several explanations but does not yet explain this realism pattern.

  • Most curriculum-generated images were alien or unrecognizable, followed by a qualitative increase in recognizability during the final four or five rows.
  • Learners already exceeded 98% MNIST accuracy after about 23 inner-loop iterations, before seeing the recognizable final batches.
  • The paper considers three explanations: realism may improve performance but be hard to optimize, unrealism may compress information, or realism may not affect performance.
  • Class-wise pixel averages of GTN-generated images were recognizable, consistent with class features being distributed across multiple synthetic images.
  • Label-flip analysis found recognizable images averaged 2.0 label flips, but many unrecognizable images also flipped infrequently.
  • The paper reports no satisfactory explanation for why realism rises at the curriculum’s end and notes that the effect differs between MNIST and CIFAR.
Loading 1912.07768v1…