PaperlayerView exact revision on arXiv

Source-linked AI summary

Bootstrap your own latent: A new approach to self-supervised Learning

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, Michal Valko

arXiv:2006.07733v3cs.LGcs.CVstat.ML

TL;DR

Self-supervised image representation methods often depend on carefully managed negative pairs. BYOL instead bootstraps representations between online and target networks, achieving state-of-the-art results without negative pairs, including 79.6% top-1 accuracy on ImageNet with a larger ResNet.

  • Problem

    State-of-the-art contrastive methods require careful treatment of negative pairs to learn useful image representations for downstream tasks.

  • Method

    BYOL trains an online network to predict representations from an exponentially averaged target network across differently augmented image views.

  • Results

    BYOL performs on par or better than the state of the art across transfer and semi-supervised benchmarks, reaching 79.6% top-1 accuracy versus 76.8% previously on ImageNet.

  • Takeaways & Limitations

    BYOL demonstrates that self-supervised image representations can achieve state-of-the-art benchmark performance without negative pairs.

  • Takeaways & Limitations

    BYOL depends on vision-specific augmentations, so extending it to other modalities requires suitable augmentations that may demand substantial expertise.

Abstract

from arXiv · show

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.

1 Introduction

BYOL is introduced as a self-supervised image-representation algorithm that directly bootstraps network outputs without negative pairs. It achieves strong ImageNet linear-evaluation results and performs on par with or better than the state of the art in semi-supervised and transfer settings.

  • Contribution: BYOL introduces a self-supervised algorithm for learning image representations.It is presented as a new approach to self-supervised learning.
  • Method: BYOL directly bootstraps network outputs as targets for an enhanced representation without using negative pairs.The method iteratively uses one network’s outputs as targets.
  • Method: BYOL is more robust to image-augmentation choices than contrastive methods.The paper suspects that avoiding negative pairs contributes to this improved robustness.
  • Results: 74.3% top-1 accuracy is achieved with ResNet-50 under ImageNet linear evaluation.The evaluation trains a linear classifier on top of the frozen representation.
  • Results: 79.6% top-1 accuracy is achieved with a larger ResNet under ImageNet linear evaluation.The result is reported alongside the ResNet-50 result in Figure 1.
  • Results: BYOL obtains results on par with or superior to the current state of the art in ImageNet semi-supervised and transfer settings.The representation is evaluated on ImageNet and other vision benchmarks using ResNet architectures.

2 Related work

Prior self-supervised representation-learning methods are mainly generative or discriminative, with contrastive approaches achieving state-of-the-art performance but relying on positive and negative pairs. BYOL relates to earlier bootstrapping and moving-average target networks while using these mechanisms to stabilize prediction targets without negative pairs.

  • Generative and discriminative methods: Unsupervised representation-learning methods are categorized as generative or discriminative.Generative methods model data and latent embeddings, often through auto-encoding or adversarial learning.
  • Generative and discriminative methods: Contrastive methods achieve state-of-the-art self-supervised performance by bringing views of the same image together and separating views from different images.They use positive pairs for same-image views and negative pairs for different-image views.
  • Bootstrapping and target networks: DeepCluster avoids negative pairs by bootstrapping targets from prior representations, but requires costly clustering and precautions against trivial collapse.It assigns cluster indices from a prior representation as classification targets for a new representation.
  • Alternative self-supervised methods: Handcrafted auxiliary prediction tasks, including colorization, inpainting, jigsaw puzzles, and geometric transformations, are outperformed by contrastive methods.These tasks have nonetheless been shown to produce useful representations, even with suitable architectures.
  • Bootstrapping and target networks: BYOL uses a moving-average network to produce prediction targets, unlike MoCo’s momentum encoder, which maintains consistent negative-pair representations in a memory bank.The related moving-average target idea was inspired by deep reinforcement learning, where target networks stabilize bootstrapping updates.

3 Method

BYOL learns image representations by training asymmetric online and target networks on cross-view prediction, updating only the online network while tracking it with an exponential moving average. Its dynamics and predictor architecture are intended to avoid collapse without explicit negative examples, while retaining only the encoder for downstream use.

  • Architecture: BYOL uses online and target networks with identical encoder-projector-predictor architectures, but applies the predictor only to the online branch.The online weights are θ and target weights are ξ; the target network supplies prediction targets.
  • Training objective: BYOL predicts one augmented view’s target projection from another view, symmetrizes the loss, and optimizes θ while stopping gradients through ξ.The loss combines both view orderings, and the target parameters are not updated by gradient descent.
  • Parameter updates: ξ is updated as an exponential moving average of θ, and BYOL retains only encoder fθ as the final image representation.Inference-time comparisons count only the weights in the final representation encoder.
  • Collapse and dynamics: BYOL does not use an explicit anti-collapse term, and its target dynamics are not generally gradient descent on the joint loss over θ and ξ.The authors hypothesize that no jointly minimized loss describes BYOL’s dynamics, analogous to GAN training.
  • Collapse and dynamics: Under an optimal predictor, the authors hypothesize that undesirable equilibria are unstable, and report that experiments did not converge to such equilibria.They specifically argue that BYOL avoids constant features in zθ.

4 Experimental evaluation

The evaluation tests BYOL representations on ImageNet linear and semi-supervised classification, transfer classification, and broader vision tasks. BYOL achieves 74.3% ImageNet top-1 accuracy with ResNet-50 and improves over supervised and SimCLR baselines on segmentation and detection.

  • Evaluation scope: BYOL is evaluated after ImageNet pretraining on linear and semi-supervised ImageNet classification, transfer classification, segmentation, detection, and depth estimation.The evaluation also assesses generality by pretraining on Places365-Standard and reproducing the protocol.
  • Linear evaluation on ImageNet: 74.3% top-1 accuracy and 91.6% top-5 accuracy are obtained by BYOL with a standard ResNet-50 (×1) under linear ImageNet evaluation.These results improve over the previous self-supervised state of the art by 1.3% top-1 and 0.5% top-5 accuracy, respectively.
  • Semi-supervised training on ImageNet: BYOL is evaluated with fine-tuning on fixed 1% and 10% labeled subsets of ImageNet under a semi-supervised protocol.Both top-1 and top-5 test accuracies are reported.
  • Transfer to other classification tasks: BYOL representations are evaluated for generic transfer across other classification datasets using both linear evaluation and fine-tuning.The protocol examines whether ImageNet-learned features transfer across image domains rather than being ImageNet-specific.
  • Transfer to other vision tasks: +1.9 mIoU over Supervised-IN and +1.1 mIoU over SimCLR are achieved by BYOL on VOC2012 semantic segmentation.The task classifies each image pixel.
  • Transfer to other vision tasks: +3.1 AP50 over Supervised-IN and +2.3 AP50 over SimCLR are achieved by BYOL on object detection.Evaluation uses Faster R-CNN, fine-tuning on trainval2007, and testing on test2007.

5 Building intuitions with ablations

BYOL’s ablations show robustness to batch size and reduced augmentations, while identifying the target network and predictor as central to avoiding collapse. The experiments also clarify BYOL’s advantage over SimCLR and the importance of regularization.

  • Experimental setup: Ablations average each configuration over three seeds and use 300-epoch training because improvements at 100 epochs may not persist longer.The authors report half the best–worst run difference when it exceeds 0.25.
  • Batch size: BYOL remains stable from batch sizes 256 to 4096, whereas SimCLR rapidly deteriorates; BYOL drops only at smaller sizes because of encoder batch normalization.The training procedure averages gradients and delays target updates when reducing batch size, preserving other hyperparameters.
  • Image augmentations: With random crops only, BYOL reaches 59.4% (−13.1 points from 72.5%), while SimCLR reaches 40.3% (−27.6 points from 67.9%).Removing color distortions reduces BYOL by −9.1 accuracy points and SimCLR by −22.2 accuracy points.
  • Bootstrapping: The moving-average target network performs best at decay 0.99 with 72.5%, compared with 69.8% at 0.999, 68.4% at 0.9, and 18.8±0.7 for a constant random network.A decay of 1 leaves the target fixed at initialization, while decay 0 updates it directly from the online network.
  • Ablation to contrastive methods: Without negative examples, only BYOL with both a bootstrap target network and predictor performs well; adding negatives without retuning temperature hurts performance.Adding a target network to SimCLR alone improves performance by +1.6 points, indicating a stabilization benefit beyond supplying negatives.
  • Predictor and regularization: Removing the predictor causes collapse in an unsupervised Mean Teacher variant, while removing the target network can avoid collapse with a near-optimal predictor reaching 52.5% top-1 accuracy.The ablations also show that removing weight decay causes divergence in BYOL or SimCLR, emphasizing weight regularization.

6 Conclusion

BYOL is a self-supervised image-representation algorithm that predicts previous versions of its outputs without negative pairs, achieving state-of-the-art benchmark results. Its broader use remains limited by the need for modality-specific augmentations, motivating automated augmentation search.

  • 6 Conclusion: BYOL learns image representations by predicting previous versions of its outputs without using negative pairs.The authors introduce BYOL as a new algorithm for self-supervised learning.
  • 6 Conclusion: BYOL achieves state-of-the-art results on various benchmarks, including a new state of the art under ImageNet linear evaluation with ResNet-50 (1×).The method bridges most of the remaining gap between self-supervised methods and supervised learning baselines.
  • 6 Conclusion: BYOL remains dependent on augmentation sets specific to vision applications.Generalizing it to audio, video, text, and other modalities requires similarly suitable augmentations for each modality.
  • 6 Conclusion: Automating augmentation search is an important next step for generalizing BYOL to other modalities.Designing suitable augmentations may require significant effort and expertise.

Broader impact … C Evaluation on ImageNet training

BYOL is presented as unsupervised-learning research with broad vision applicability but potential dual-use and dataset-bias risks. The supplied appendices specify its online/target-network algorithm and image augmentation and evaluation procedures.

  • Broader impact: BYOL may inspire new unsupervised-learning algorithms and investigations across different vision applications.The passage identifies both positive and negative potential impacts from particular uses.
  • Broader impact: Potential risks include dual use and replication of biases present in vision datasets.The learned representation may inherit biases from the datasets used to train it.
  • A Algorithm: The algorithm initializes online and target parameters, encoders, projectors, and predictor components, then optimizes online parameters over K steps and batch size N.A target-network update schedule and learning-rate schedule are also specified.
  • A Algorithm: The online network computes projections from two transformed views of each input image.The algorithm applies encoder and projector functions to transformations t and t′.
  • A Algorithm: Target projections are computed by applying the target encoder and projector to the opposite transformed views.This pairs each target projection with the corresponding online-view transformation.
  • B Image augmentations: Training augmentations compose random cropping, optional flips, color changes, blurring, and solarization with predetermined probabilities.The augmentation parameters are listed in Table 6.
  • B Image augmentations: Random crops sample 8%–100% of the original image area, then resize patches to 224 × 224 using bicubic interpolation.The crop aspect ratio is logarithmically sampled between 3/4 and 4/3.
  • C Evaluation on ImageNet training: Evaluation resizes images to 256 pixels on the shorter side before applying a 224 × 224 center crop and ImageNet-based channel normalization.The same channel normalization is used in training after augmentations.

C.1 Self-supervised learning evaluation on ImageNet · C.2 Linear evaluation on larger architectures and supervised baselines · D Transfer to other datasets

BYOL is evaluated with frozen-representation linear protocols and fine-tuning on ImageNet, including modified evaluation procedures and semi-supervised settings. Larger encoders bring BYOL within 0.4 accuracy points of supervised performance.

  • C.1 Self-supervised learning evaluation on ImageNet: The standard ImageNet linear evaluation trains a classifier on frozen representations without updating network parameters or batch statistics.Training uses random crops resized to 224 × 224 pixels and random flips; testing resizes the shorter side to 256 pixels before center cropping.
  • C.1 Self-supervised learning evaluation on ImageNet: A modified linear protocol replaces spatial augmentations with pre-train augmentations and regularizes the classifier logits.The classifier uses a hyperbolic tangent clipping function and a logit-regularization penalty, with α = 20 and β = 1e−2.
  • C.1 Self-supervised learning evaluation on ImageNet: 74.8% top-1 accuracy is achieved by BYOL with a ResNet-50 (1×) under the modified protocol, versus 74.3% under the original protocol.The reported increase results from modifying the evaluation protocol.
  • C.1 Self-supervised learning evaluation on ImageNet: Fine-tuning BYOL on balanced ImageNet label subsets from 1% through 100% always outperforms a supervised baseline trained from scratch.The comparison uses a ResNet-50 (1×) and also includes a fine-tuned SimCLR representation.
  • C.1 Self-supervised learning evaluation on ImageNet: 77.7 top-1 is obtained after fine-tuning BYOL on full ImageNet, compared with 76.5 top-1 from random initialization.The BYOL initialization also exceeds the vanilla supervised baseline, whereas the reproduced SimCLR checkpoint does not improve over random initialization.
  • C.2 Linear evaluation on larger architectures and supervised baselines: BYOL’s larger ResNet-50 (2×) and ResNet-50 (4×) encoders come within 0.4 accuracy points of the best supervised baselines.The comparison covers deeper and wider ResNet architectures and uses supervised baselines when available.
  • C.2 Linear evaluation on larger architectures and supervised baselines: The larger-encoder study reports ImageNet top-1 and top-5 accuracies for linear evaluation of BYOL.Results are presented for larger ResNet encoders alongside fully supervised baselines.
  • C.2 Linear evaluation on larger architectures and supervised baselines: Supervised baselines for the larger-architecture comparison train for up to 200 epochs with Nesterov momentum, cosine-annealed learning rates, weight decay, and label smoothing.The learning rate includes a five-epoch linear warmup, and the baselines use RandAugment augmentations.

D.1 Datasets … D.7 Further comparisons on PASCAL and NYU v2 Depth

The appendix specifies transfer datasets and evaluation protocols across classification, fine-tuning, segmentation, detection, and depth estimation. Additional comparisons report that BYOL generally outperforms published baselines, with RMS error on NYU Depth as a notable exception.

  • D.1 Datasets: BYOL transfer uses the same dataset suite as [8], spanning Food-101, CIFAR-10/100, Birdsnap, SUN397, Stanford Cars, FGVC Aircraft, PASCAL VOC 2007, DTD, Oxford-IIIT Pets, Caltech-101, and Oxford 102 Flowers.Validation sets specified by dataset creators are used for hyperparameter selection.
  • D.2 Transfer via linear classification: Linear evaluation trains a regularized multinomial logistic regression classifier on frozen pretrained representations without recomputing batch-normalization statistics.Images receive no augmentation and are resized to 224 pixels on the shorter side before ImageNet-statistics normalization.
  • D.3 Transfer via fine-tuning: Fine-tuning initializes the network with pretrained representation parameters, using random resized crops and flips during training and a 224 × 224 center crop at test time.Test images are resized to 256 pixels along the shorter side before center cropping.
  • D.4 Implementation details for semantic segmentation: Semantic segmentation uses an FCN-16s architecture with a ResNet-50 convolutional backbone, dilated conv5 blocks, extra dilated convolutions, and a per-pixel classification layer.Training uses per-pixel softmax cross-entropy with random scaling, cropping, and horizontal flipping.
  • D.5 Implementation details for object detection: Object detection uses Faster R-CNN with an R50-C4 backbone and frozen representations, applying multi-scale image resizing without other augmentation.The R50-C4 backbone ends at ResNet-50’s conv4 stage, while the prediction head uses conv5 with global pooling.
  • D.6 Implementation details for depth estimation: Depth estimation uses a ResNet-50 backbone, four fast up-projection blocks, and a reverse Huber training loss.NYU Depth v2 frames are down-sampled by 0.5, center-cropped to [304, 228], randomly flipped, and color-transformed.
  • D.7 Further comparisons on PASCAL and NYU v2 Depth: BYOL outperforms published baselines in almost all comparable semantic-segmentation, object-detection, and NYU Depth settings, including comparisons with more data or deeper models.Tables 11 and 12 extend prior comparisons with additional published baselines using comparable networks.
  • D.7 Further comparisons on PASCAL and NYU v2 Depth: RMS error for NYU Depth prediction is a notable exception, and its sensitivity to outliers may relate to higher-variance predictions.The passage states that the reason for this exception is unclear.

E Pretraining on Places 365 … F.3 Image augmentations

BYOL transfers effectively beyond ImageNet, outperforming SimCLR on Places365 while retaining strong performance across ablations. Additional experiments show that BYOL benefits from default two-layer projector and predictor designs, remains robust to batch-size and augmentation changes, and uses a specified multi-seed evaluation setup.

  • E Pretraining on Places 365: BYOL was pretrained for 1000 epochs on Places365-Standard using the same parameters as Section 4, alongside a SimCLR reproduction, before linear evaluation.Places365-Standard contains 1.80 million training images and 36,500 labeled validation images, making it roughly similar to ImageNet in scale.
  • E Pretraining on Places 365: BYOL-PL consistently outperforms SimCLR on Places365, but BYOL-IN performs better on every evaluated task except Places365 and SUN397.All three unsupervised methods achieve relatively high performance on the Places365 task.
  • F Additional ablation results: Additional ablations use 300 epochs, three seeds, initial learning rate 0.3, batch size 4096, weight decay 10^-6, and base target decay rate τ_base 0.99 unless specified otherwise.Reported confidence intervals are half-differences between the maximum and minimum seed scores, omitted below 0.25 accuracy points.
  • F.1 Architecture settings: The default depth-2 projector and predictor yield the best performance among the tested architectural settings.Depth denotes the number of linear layers, and depth 1 corresponds to a linear layer because the final layer has neither ReLU nor batch normalization.
  • F.1 Architecture settings: BYOL’s performance is quite robust across a range of learning rates and weight-decay values, although the optimal learning rate depends on training duration.Setting weight decay to zero may produce unstable results, as observed for SimCLR.
  • F.2 Batch size: BYOL’s performance deteriorates more slowly than SimCLR’s as batch size decreases from 4096 to 64 under gradient accumulation and less frequent target updates.The batch-size sweep compares BYOL with the authors’ SimCLR reproduction using the same adjustment procedure for reduced batch sizes.
  • F.3 Image augmentations: BYOL is more resilient than SimCLR to changes in individual image augmentations across the tested transformations.Using symmetric parameters across both views, with Gaussian blurring probability 0.5 and solarization probability 0.2, recovers results very similar to the baseline.

F.4 Details on the relation to contrastive methods · F.5 SimCLR baseline of Section 5 · F.6 Ablation on the normalization in the loss function

The appendix relates BYOL’s loss to InfoNCE and SimCLR, examines how predictors, target networks, and loss parameters affect performance, and reports implementation and normalization ablations. BYOL remains strongest in the tested variants, while modified SimCLR and alternative normalizations provide concrete baselines and comparisons.

  • F.4 Details on the relation to contrastive methods: BYOL’s loss can be derived from the factored InfoNCE loss used by contrastive methods such as SimCLR.The derivation connects Eq. 2 to InfoNCE through algebraic transformations and parameter choices.
  • F.4 Details on the relation to contrastive methods: SimCLR adds negative examples, whereas BYOL corresponds to β = 0 in the predictor-and-target-network formulation.Setting β = 1 recovers the usual InfoNCE loss after the stated rescaling and transformations.
  • F.4 Details on the relation to contrastive methods: No tested α, β combination significantly outperforms BYOL, and adding negative examples with β = 1 hurts performance in the BYOL setting.The best SimCLR temperature is α = 0.1, while the predictor-and-target-network setting performs best at α higher than 0.3.
  • F.4 Details on the relation to contrastive methods: A target network both stops gradients through prediction targets and stabilizes them through averaging, changing the objective while making targets stable and stale.The ablation separates the effects of using an online prediction target from using a target network.
  • F.5 SimCLR baseline of Section 5: 69.2% top-1 accuracy at 300 epochs was achieved by the reproduced SimCLR baseline after rescaling the loss and enlarging the projector, versus 67.9% without those changes.The projector hidden and output sizes were increased to 4096 and 256, respectively.
  • F.5 SimCLR baseline of Section 5: Loss rescaling matters because LARS updates for biases and batch-normalization parameters remain sensitive to multiplicative scaling.Updates on preconditioned weights are invariant to loss scaling, unlike these parameter updates.
  • F.6 Ablation on the normalization in the loss function: 67.4% top-1 accuracy was obtained without normalization despite projection norms increasing rapidly and stabilizing around 3 · 10^6 during the first 100 epochs.BYOL uses squared error between ℓ2-normalized predictions and targets; the ℓ2 normalization performs best among the tested choices.

G Training with smaller batch sizes … J.3 Loss function

The appendix reports that BYOL remains effective with smaller batches, clarifies its gradient update and predictor-dependent stability, and provides implementation details for its JAX loss and networks.

  • G Training with smaller batch sizes: 73.7% top-1 accuracy is achieved after 1000 epochs with batch size 512 across 64 TPU cores, taking approximately 4 days.The setup uses the same linear evaluation procedure as Section 3.
  • H Details on Equation 5 in Section 3.2: BYOL’s online update follows Eq. 5 while back-propagating gradients only through the predictor’s input, matching the direction derived from predictor optimality.The optimal predictor is written as q⋆(θ, ξ, zθ), explicitly depending on online and target parameters.
  • I Importance of a near-optimal predictor: The target network’s exponential moving average can be replaced by directly copying online weights without collapse, provided the predictor remains sufficiently good.This experiment builds on the importance of keeping the predictor near-optimal.
  • I.1 Predictor learning rate: 0.01% top-1 accuracy occurs with predictor learning-rate multiplier λ = 0 when the target network is directly copied from the online network.Performance is reasonably good for sufficiently large λ values and sharply decreases as λ approaches zero.
  • I.1 Predictor learning rate: The representation typically collapses or performs poorly when the predictor learning rate is lower than or equal to the projector learning rate.The experiment applies separate multipliers λ and µ to predictor and projector learning rates.
  • I.2 Optimal linear predictor in closed form: A closed-form optimal linear predictor also permits removing the slowly moving target network by hard-copying online weights into the target.The predictor has shape (F, F), while online and target projections have shape (B, F).
  • J.1 Hyper-parameters: The hyperparameters specify 1000 epochs with batch size 4096, a 4096-unit hidden MLP, 256-dimensional projections, and target EMA coefficient 4e-3.Optimization uses LARS with cosine annealing after linear warmup.
  • J.2 Network definition / J.3 Loss function: The JAX implementation builds a ResNetV1_50x1 encoder followed by projector and predictor MLPs, and computes cosine regression between online predictions and stopped-gradient target projections.The loss evaluates two image transformations and does not back-propagate through the target network.

J.4 Training loop · J.5 Update function

The training loop initializes BYOL state, repeatedly creates two augmented image views, and updates the model for each global step. The update function differentiates the loss only through the online network, applies optimizer updates, and updates the target network before returning state.

  • J.4 Training loop: The loop seeds JAX randomness, batches the dataset, obtains a dummy input, and initializes BYOL state before training.Initialization uses the configured seed and the first dataset batch to construct the initial state.
  • J.4 Training loop: For each global step, the loop splits randomness and generates two independently augmented views of the input image.The two views are produced by simclr_augmentations with separate random keys and image numbers.
  • J.4 Training loop: The loop passes both augmented views and the current state into update_fn, then returns the updated online parameters.The returned value is byol_state['online_params'].
  • J.5 Update function: The update function receives online and target parameters, optimizer state, the current training step, and two transformed images.The loss is differentiated with respect to the online parameters only.
  • J.5 Update function: The online update computes loss gradients, evaluates the learning-rate schedule, applies optimizer updates, and writes the resulting online parameters.Gradients are obtained with jax.grad(loss_fn, argnums=0), and updates are applied through optix.
  • J.5 Update function: The target update uses target_ema to compute tau and moves target parameters toward online parameters with an exponential-moving-average expression.The update uses base_ema from the configured hyperparameters.
  • J.5 Update function: Initialization creates online and target parameters with net_init and initializes optimizer state from the online parameters using a dummy input.The dummy image is used to compute intermediate output shapes.
Loading 2006.07733v3…