PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiple Object Recognition with Visual Attention

Jimmy Ba, Volodymyr Mnih, Koray Kavukcuoglu

arXiv:1412.7755v2cs.LGcs.CVcs.NE

TL;DR

Recognizing multiple objects in large, uncropped images is difficult because ConvNets scale poorly and prior systems often required separately trained localization and recognition components. This paper uses recurrent visual attention to process selected image regions sequentially and learn localization and recognition from label sequences. On multi-digit house-number recognition, it reports better accuracy than state-of-the-art ConvNets with fewer parameters and less computation.

  • Problem

    ConvNets scale poorly with image size, and recognizing objects in uncropped images has often required separately trained detection and recognition components.

    Source: §1 · p. 1

  • Method

    A deep recurrent attention network processes multi-resolution glimpses sequentially, predicting locations and object labels and learning localization and recognition from label sequences.

    Source: §1 · p. 1, §3 · p. 2, §3.2 · p. 5

  • Results

    The model outperformed state-of-the-art ConvNets on multi-digit house-number recognition while using fewer parameters and less computation.

    Source: §6 · p. 8

  • Takeaways & Limitations

    DRAM can handle variable-length label sequences, and a model trained on fixed-length sequences can be transferred and fine-tuned for similar tasks with longer targets.

    Source: §5 · p. 8

Abstract

from arXiv · show

We present an attention-based model for recognizing multiple objects in images. The proposed model is a deep recurrent neural network trained with reinforcement learning to attend to the most relevant regions of the input image. We show that the model learns to both localize and recognize multiple objects despite being given only class labels during training. We evaluate the model on the challenging task of transcribing house number sequences from Google Street View images and show that it is both more accurate than the state-of-the-art convolutional networks and uses fewer parameters and less computation.

1 INTRODUCTION

ConvNets perform well on recognition tasks but scale poorly with image size, while multi-object recognition from uncropped images has often required separately trained detection and recognition components. The paper introduces a recurrent attention model trained end-to-end to localize and recognize objects from label sequences, and reports improved accuracy with fewer parameters and less computation on house-number transcription.

1 INTRODUCTION

  • ConvNets scale poorly as input images grow, motivating costly implementations across multiple GPUs or machines.For uncropped multi-object images, prior approaches combined a recognizer with a separately trained detector or proposal generator.Source: §1 · p. 1
  • A deep recurrent network processes multi-resolution image crops called glimpses, updates its internal representation, and predicts the next location and possibly the next object.It can be trained end-to-end to localize and recognize multiple objects using only label sequences.Source: §1 · p. 1
  • The model outperforms state-of-the-art ConvNets on both tightly cropped and larger, less tightly cropped house-number inputs, with a larger margin in the latter setting.The evaluation uses multi-digit house numbers from publicly available Google Street View imagery.Source: §1 · p. 1

2 RELATED WORK

Prior sequence-recognition approaches combined detectors and classifiers, while ConvNets achieved strong results on cropped text. Earlier end-to-end and attention-based systems provided precedents, but left open the challenge of scaling recurrent attention to real-world images and multiple objects.

2 RELATED WORK

  • Sliding-window character detectors and classifiers were typically trained separately, while LeCun et al. (1998) trained a digit-sequence system end-to-end but retained ad-hoc candidate-location extraction.The separate components used different loss functions.Source: §2 · p. 2
  • ConvNets on cropped character sequences achieved state-of-the-art house-number and natural-scene text recognition, including systems using shared character-position weights or synthetic text training.The house-number approach is Goodfellow et al. (2013); the synthetic-text approach is Jaderberg et al. (2014a).Source: §2 · p. 2
  • This work extends the recurrent attention model of Mnih et al. (2014), which had not been shown to scale to real-world images or multiple objects.The authors relate their variational-lower-bound learning approach to RAM's reinforcement-learning procedure.Source: §2 · p. 2

3 DEEP RECURRENT VISUAL ATTENTION MODEL

The model processes images through recurrent attention: each glimpse updates an internal state that guides the next location, while a coarse whole-image view initializes attention. It learns class predictions and glimpse locations jointly, then extends the same recurrent process to ordered sequences of objects.

3 DEEP RECURRENT VISUAL ATTENTION MODEL

  • Each glimpse updates a recurrent state that guides the next location, with per-glimpse processing cost independent of original image size.A coarse, down-sampled image initializes the top recurrent layer and guides the first glimpse.Source: §3 · pp. 2–3
  • The glimpse network combines image features and location features element-wise, and recurrent layers aggregate these features while preserving spatial information.The context network and classifier connect to separate recurrent layers, so context affects glimpse locations but cannot directly inform classification.Source: §3 · p. 3

3.1 LEARNING WHERE AND WHAT

  • Training marginalizes over glimpse locations using a variational free-energy lower bound, approximated with Monte Carlo samples of location sequences.The classifier uses image class labels; a 0/1 reward and state-based baseline reduce gradient-estimator variance, and the resulting rule is equivalent to REIN-FORCE.Source: §3.1 · p. 4
  • At inference, the model can use deterministic location predictions or average class predictions over sampled glimpse sequences.Averaging log probabilities performed best in practice; glimpse locations use image-centered Cartesian coordinates, with a location-scale setting around 15% of image width working well.Source: §3.1 · p. 5

3.2 MULTI-OBJECT/SEQUENTIAL CLASSIFICATION AS A VISUAL ATTENTION TASK

  • For multiple objects, the model predicts an ordered label sequence one object at a time, using fixed glimpses per target and an end-of-sequence label for variable-length outputs.With S targets and N glimpses per target, training uses N × (S + 1) glimpses.Source: §3.2 · p. 5
  • The multi-object objective accumulates gradients across targets, while the recurrent model remains compact and handles images with variable numbers of objects.The sequential objective combines each target’s location and class-label probabilities.Source: §3.2 · p. 5
  • For multi-object learning, reward is proportional to correctly predicted targets, and gradients ignore targets after the first labeling error.The paper describes this first-error restriction as crucial for high-performing sequential prediction.Source: §3.2 · p. 5

4 EXPERIMENTS

Experiments progress from MNIST multi-object tasks to multi-digit SVHN comparisons with ConvNets. The results include distinct learned glimpse policies and state-of-the-art performance from forward-backward reading on house numbers.

4.1 LEARNING TO FIND DIGITS

  • With four glimpses on noisy 100×100 MNIST pairs, the context-network DRAM significantly outperforms the other models on 55 combination classes.The table reports error rates for the pairs-classification task.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 LEARNING TO DO ADDITION

  • On two-digit addition with weaker label supervision, DRAM achieves 2.5% error while ConvNets learn longer and perform worse; its policy toggles between digits.The addition task has 19 targets, and different digit pairs can map to the same sum.Source: §4.2 · p. 6, Figure 2 · p. 6

4.3 LEARNING TO READ HOUSE NUMBERS

  • A single left-to-right DRAM remains behind the best ConvNet and often overpredicts digits instead of the terminal class.It predicts sequentially with three glimpses per digit and runs until it predicts a terminal label or reaches the dataset's maximum digit length.Source: §4.3 · p. 7
  • The forward-backward recurrent attention model reaches state-of-the-art performance on SVHN by combining forward and backward sequence predictions.Table 3 reports whole-sequence recognition error rates on multi-digit SVHN.Source: §4.3 · p. 7, Table 3 · p. 7
  • On enlarged 110×110 SVHN, focusing a 54×54-trained attention model significantly outperforms a deep ConvNet after only a few hours of fine-tuning.The attention model is applied to enlarged images without modification, then uses a glimpse-centered crop for focusing.Source: §4.3 · pp. 7–8

5 DISCUSSION

DRAM outperforms state-of-the-art ConvNets on standard SVHN sequence recognition, with a larger advantage as image area increases or signal-to-noise ratio falls. It also uses selected input scales, supports variable-length sequences, and is reported as less prone to over-fitting.

5 DISCUSSION

  • DRAM outperforms state-of-the-art deep ConvNets on standard SVHN sequence recognition, with a larger advantage as image area increases or signal-to-noise ratio falls.The advantage becomes more significant as the image area around house numbers increases or signal-to-noise ratio decreases.Source: §5 · p. 8
  • Selected input scales let recurrent attention models scale better than ConvNets processing entire images, with computational cost independent of input dimensionality.The reported DRAM estimate uses the dataset's maximum sequence length; expected cost is lower in practice because most house numbers are 2–3 digits long.Source: §5 · p. 8
  • DRAM is reported as less prone to over-fitting than ConvNets, likely because its glimpse policy is stochastic during training.Dropout gives the attention model a marginal 0.1% boost on multi-digit SVHN; a 10-layer ConvNet achieves 5.5% error when dropout is applied only to its last two fully connected hidden layers.Source: §5 · p. 8
  • DRAM handles variable-length label sequences and can be transferred and fine-tuned from fixed-length training data to longer target sequences.The authors note this is especially useful when data for longer sequences is scarce.Source: §5 · p. 8

6 CONCLUSION

The paper presents an end-to-end attention model that sequentially classifies multiple objects in images. On multi-digit house number recognition, it outperforms state-of-the-art ConvNets with fewer parameters and less computation.

6 CONCLUSION

  • The attention model decides where to focus computation and is trained end-to-end to classify multiple objects sequentially.Source: §6 · p. 8
  • On multi-digit house number recognition, the model outperforms state-of-the-art ConvNets while using fewer parameters and less computation.Source: §6 · p. 8
  • The authors conclude that attention can improve both accuracy and efficiency on this real-world task.Source: §6 · p. 8

8 APPENDIX

The appendix gives training settings and model configurations for learning to find digits and read house numbers.

8.1 GENERAL TRAINING DETAILS

  • Training uses ReLU hidden units, SGD with Nesterov momentum, mini-batches of 128, momentum 0.9, and a learning rate that starts at 0.01 and decays by 0.97 each epoch.The authors report that ReLU units significantly speed up training.Source: §8.1 · p. 10

8.2 DETAILS OF LEARNING TO FIND DIGITS

  • For learning to find digits, the model uses 512 LSTM units, 256-unit fully connected layers, and a single 256-unit hidden layer in the glimpse network.The Cartesian coordinate unit width is 20 and glimpse-location sampling standard deviation is 0.03.Source: §8.2 · p. 10

8.3 DETAILS OF LEARNING TO READ HOUSE NUMBERS

  • For house-number reading, the model uses a three-layer convolutional glimpse network with 64, 64, and 128 filters, plus 512-unit LSTM layers and 1024-unit fully connected hidden layers.The Cartesian coordinate unit width is 12 pixels and glimpse-location sampling variance is 0.03.Source: §8.3 · p. 10

Related papers

Loading 1412.7755v2…