Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions

1 INTRODUCTION

• Few-shot image density estimation asks a model to learn a probability distribution from only a few examples, a task that had received comparatively little attention.The introduction contrasts it with the more-studied few-shot setting for discriminative models.Source: §1 · p. 1

WHY AUTOREGRESSIVE MODELS?

• Autoregressive models offer fast, stable training and tractable likelihoods, making objective comparisons across model variants practical.Their structure also allows complexity to be added in directions orthogonal to the generative model itself.Source: WHY AUTOREGRESSIVE MODELS? · p. 1

WHY LEARN TO LEARN DISTRIBUTIONS?

• With tiny target datasets, training or fine-tuning from scratch risks memorization, motivating learning an algorithm from experience when target distributions share structure.The paper frames this as learning to learn distributions rather than designing a rapid-learning algorithm by hand.Source: WHY LEARN TO LEARN DISTRIBUTIONS? · p. 2

FEW-SHOT LEARNING AS INFERENCE OR AS A WEIGHT UPDATE?

• Sample-conditional density models treat meta-learning as inference: support samples vary while the model parameters remain fixed.A contrasting Meta PixelCNN approach adapts an unconditional model's weights using a gradient step on the few-shot samples.Source: FEW-SHOT LEARNING AS INFERENCE OR AS A… · p. 2
• The paper compares attentive and gradient-based PixelCNN variants on image mirroring, Omniglot, and Stanford Online Products, reporting state-of-the-art likelihoods on Omniglot.The study also describes attention as a way to transmit texture information from support images onto the target canvas.Source: FEW-SHOT LEARNING AS INFERENCE OR AS A… · p. 2

Meta-learning

• Other work learned optimizers for black-box optimization and used attentive recurrent comparison for one-shot Omniglot classification.These examples situate the paper among broader meta-learning and attention-based approaches.Source: §2 · pp. 2–3
• Earlier meta-learning work learned optimizers or initial parameters for rapid task adaptation, including few-shot learning and imitation learning.The cited approaches include learned optimizers and fixed one-step gradient descent from learned base parameters.Source: §2 · p. 3

Few-shot density estimation

• Few-shot density estimation had been explored with matching networks, variational autoencoders, and sequential generative models before this work focused on extending PixelCNN.The cited examples include memory addressing with variational inference and a sequential model extending DRAW.Source: §2 · p. 3

Attentive autoregressive models

• Attention had been used in recurrent language, image-captioning, and motor-control models, while convolutional translation models offered batched attention over input sentences.The paper describes its attentive PixelCNN as similar in high-level design to convolutional machine translation, but applied to pixels and image generation.Source: §2 · p. 3

3.1 FEW-SHOT LEARNING WITH ATTENTION PIXELCNN

• Attention PixelCNN queries support-image memory at every generated pixel, using global features and patch textures to condition image generation.The support images are encoded into spatially indexed key and value vectors; attention context is combined with global context by channel-wise concatenation.Source: §3.1 · pp. 3–4
• A context-sensitive attention mechanism replaces the shared encoder because relevant support-image information changes with the generated pixel and a single vector must carry information across salient regions.The attention weights are shared across generation steps.Source: §3.1 · pp. 3–4
• Support images are encoded as spatially indexed key and value vectors, while PixelCNN features represent the support set and pixels generated so far for querying that memory.The support-image encoder is shallow, typically two convolutional layers, with hidden units covering small image patches.Source: §3.1 · p. 4
• Relative-position channels and a support-image label channel augment support images to encode patch locations and identify the image from which each patch came.The label channel may help when assembling patches from multiple views of an object.Source: §3.1 · pp. 4–5

3.2 FEW-SHOT LEARNING WITH META PIXELCNN

• Meta PixelCNN implicitly conditions on support images by adapting parameters with gradient descent, while its learned inner objective can differ from the outer likelihood objective.The learned inner function need not compute likelihood or respect the pixels’ causal ordering, yet can learn gradients that increase the target log probability.Source: §3.2 · p. 5

4 EXPERIMENTS

• The experiments use a shared support-set encoder that processes support images in parallel, concatenates their encodings, and maps them to an embedding.The encoder uses a 5 × 5 convolution, further convolutions and max-pooling, then two fully connected layers.Source: §4 · p. 5

4.1 IMAGENET FLIPPING

• On ImageNet, Attention PixelCNN learns to consistently mirror images, whereas the baseline without attention fails to flip them; attention still makes slight upper- and left-border mistakes, where context is least.Validation results are 0.90 nats/dim for attention versus 2.65 for the baseline. During sampling, attention proceeds right-to-left over the input while output is written left-to-right.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 OMNIGLOT

• Attention PixelCNN significantly outperforms other methods, including PixelCNN without attention, across 1-, 2-, and 4-shot Omniglot; all PixelCNN variants achieve state-of-the-art likelihood results.Meta PixelCNN also reaches state-of-the-art likelihoods, surpassed only by Attention PixelCNN.Source: §4.2 · pp. 6–7, Table 1 · p. 7
• Omniglot attention follows corresponding support-image regions while generating in raster order, and the reported likelihood difference correlates with improved sample quality.The experiments evaluate held-out examples of a character from the same alphabet as the support images.Source: §4.2 · pp. 6–8, Figure 4 · p. 7

4.3 STANFORD ONLINE PRODUCTS

• On Stanford Online Products, Attention PixelCNN captures objects and sometimes copies support-set textures, while the baseline can miss their style and texture.The task conditions on three images of one object, despite arbitrary camera viewpoints and potentially changing backgrounds.Source: §4.3 · p. 8, Figure 5 · p. 8
• Unlike other datasets, the authors observe no quantitative test-likelihood benefit from attention; baseline and attention models achieve 2.15 and 2.14 nats/dim, respectively, on the validation set.The authors suggest that criteria besides likelihood may be needed to evaluate few-shot visual concept learning.Source: §4.3 · p. 9

• Attention PixelCNN achieves state-of-the-art few-shot density estimation on Omniglot and promising results on natural images.Source: §5 · p. 9
• Attention weights reveal sensible generation algorithms for image mirroring and handwritten character drawing.Source: §5 · p. 9
• Meta PixelCNN applies gradient-based meta-learning to few-shot density estimation and achieves state-of-the-art likelihood results on Omniglot.Source: §5 · p. 9
• The paper adapts PixelCNN to the task of few-shot density estimation.Source: §5 · p. 9

6.1 ADDITIONAL SAMPLES

• Figure 6 compares global-conditional, attention-conditional, and gradient-conditional PixelCNN on 24×24 image flipping.Source: Figure 6 · p. 11

6.2 QUALITATIVE COMPARISON TO CONVDRAW

• Likelihood and sample quality give conflicting rankings: prior methods sometimes produce clearly better samples despite PixelCNN variants outperforming prior state of the art in likelihood.Attention PixelCNN has 286K parameters, compared with ConvDRAW's 53M.Source: §6.2 · p. 11
• The conditional ConvDRAW model encodes support images with four convolution layers without attention, then concatenates them to the ConvL-STM state at each of 12 Draw steps.Source: §6.2 · p. 11
• Figure 7 presents a 4-shot qualitative comparison with ConvDRAW.Source: Figure 7 · p. 11