AutoFocus: Efficient Multi-Scale Inference

1. Introduction

• Human vision uses high-resolution foveal vision alongside lower-resolution peripheral vision, with eye movements guiding where to look next.The paper uses this as motivation for coarse-to-fine detection; it describes saccadic masking as ignoring regions between fixation points.Source: §1 · pp. 1–2
• Modern object detectors process every pixel of a multi-scale image pyramid, spending equal energy at each scale despite the differing object sizes.The paper notes that batch-mode computational savings can provide substantial monetary benefits for applications such as large-scale image and video indexing.Source: §1 · p. 1
• 40% of COCO object instances are small, but they occupy only 0.3% of image area; adding a scale of 3 entails 9 times more computation to detect them.Even with padding for spatial context, the upsampled regions would remain small relative to the original image.Source: §1 · pp. 1–2
• AutoFocus predicts category-agnostic FocusPixels at coarse scales, generates chips around them, and processes only those regions at finer scales.On COCO, it processes 20% of the image at the largest scale without performance loss, or as little as 5% with a 1% performance drop.Source: §1 · p. 2

2. Related Work

• Recent accelerated multi-scale methods had been evaluated only in constrained settings such as pedestrian or face detection, or object detection in videos.AutoFocus is presented as a framework for generic object detection evaluated on benchmark datasets.Source: §2 · p. 2
• Image pyramids help recognize objects of different sizes because convolutional neural networks are not scale invariant, but inference on high-resolution pyramids remains a bottleneck.Single-low-resolution detectors such as YOLO [56], SSD [43], and RetinaNet [37] are commonly deployed because pyramid inference is prohibitively expensive.Source: §2 · p. 2
• Earlier acceleration approaches included rejecting easy regions with filters in Viola-Jones, reducing scales with feature approximation, cascades, and feature pyramids.Deep-learning detectors also use feature pyramids to improve accuracy without compromising speed [45, 43, 70, 9, 36, 44, 52, 24, 37, 41].Source: §2 · p. 2
• AutoFocus predicts small-object presence at coarser scales to avoid computation in large regions at finer scales, unlike object proposals that require tight overlap and instance-level reasoning.Its targets need not be salient, distinguishing it from saliency- and reinforcement-learning-based guided search methods.Source: §2 · p. 2

3. Background

• SNIP trains on objects within scale-appropriate ranges and ignores large detections in high-resolution images when merging detections across scales.Because it is not trained on large objects at high resolution, it is unlikely to detect them there during inference.Source: §3 · p. 2
• Inference chips around small objects could avoid processing full high-resolution images, but cropping an object into multiple parts can increase the error rate.Cropping during training did not hurt performance and can act as data augmentation, unlike the stated inference concern.Source: §3 · p. 2
• Figure 2 shows FocusPixel labels changing with scale: two small elephants generate FocusPixels at scale 1, while only the smallest does at scale 2.The largest elephant at scale 1 is background, and a borderline case is ignored during training.Source: Figure 2 · p. 3
• Merging detections from chips at multiple scales is needed in addition to predicting where to look at the next scale.The concern follows from the possibility that inference chips crop objects into multiple parts.Source: §3 · pp. 2–3

4. The AutoFocus Framework

• AutoFocus has three components: FocusPixel prediction, FocusChip generation for efficient inference, and focus stacking to merge detections across scales.Source: §4 · p. 3

4.1. FocusPixels

• Feature-map cells overlapping objects sized 5×5–64×64 pixels in resized chips are positive FocusPixels; cells for objects below 5×5 or between 64×64 and 90×90 are invalid.The object sizes are measured in the resized chip input to the network.Source: §4.1 · p. 3
• A binary softmax classifier predicts FocusPixels from the conv5 feature map using two added convolutional layers.Source: §4.1 · p. 4

4.2. FocusChip Generation

• At inference, threshold t selects foreground pixels; components are dilated, merged when connected, enclosed in chips, and constrained to a minimum height and width k.The threshold controls the speed–accuracy trade-off; overlapping chips are also merged.Source: §4.2 · p. 4

5 Merge chips C if they overlap

• For each component in S, generate an enclosing chip whose size exceeds k.Source: §3 · p. 4

5 Merge chips C if they overlap

• Overlapping chips are merged; valid detections from each scale are projected to image coordinates and aggregated with Non-Maximum Suppression.Source: §5 · p. 4, §4.3 · p. 5

4.3. Focus Stacking for Object Detection

• Dilating predicted maps helps keep objects inside chip boundaries; detections observed at a zoomed-in chip boundary are discarded to avoid false positives from cropped objects.Source: §4.3 · p. 4
• Figure 4 shows the sequence from original image, to FocusPixels and FocusChip, to network detections, and finally pruned chip detections.Source: Figure 4 · p. 5

5. Datasets and Experiments

• COCO experiments add fully convolutional FocusPixel-prediction layers to SNIPER without changing its architecture or training schedule, using three inference scales.Chips of similar size and aspect ratio are grouped for batch throughput; padding can slightly change pixels processed per image, but its overhead is negligible on large datasets.Source: §5 · p. 5

5.1. Stats for FocusPixels and FocusChips

• At scale 3, FocusPixels occupy about 4% of image area, rising to 7% after dilation; reducing minimum chip size from 512 to 64 gives a theoretical ~10× speedup but risks detection context.The speedup bound assumes error-free FocusPixels based on ground truth.Source: §5.1 · pp. 5–6

5.2. Quality of FocusPixel prediction

• At scale 2, FocusPixels cover all confident detections using 5% of image area, versus 25% at scale 1; recall improves at higher resolutions.Confident detections are those overlapping ground-truth boxes with IoU > 0.5 and score > 0.5.Source: §5.2 · p. 6

5.3. Quality of FocusChips

• To enclose all confident detections, FocusChips use 5% more area than FocusPixels at scale 2 and 10% more at scale 1, because objects are often nonrectangular.Source: §5.3 · p. 6

5.4. Speed Accuracy Trade-off

• AutoFocus processes 2.8× fewer pixels without performance loss, or 5× fewer for about 1% mAP loss; processing one-fifth of scale 3 retains 47.5% mAP.The latter is 1.5% better than using only scales 1 and 2.Source: §5.4 · pp. 6–7, Figure 7 · p. 7
• On COCO test-dev, AutoFocus matches SNIPER at 47.9% mAP (68.3% at 0.5 IoU) while running 6.4 versus 2.5 images per second.On PASCAL VOC, AutoFocus achieves 4.6% better AP at 0.7 IoU than DeformableV2 while processing the same area.Source: Table 1 · p. 7, §5.4 · p. 7, Table 2 · p. 7

• The reported results use only a Faster R-CNN detector; proposed extensions include YOLOv2, RetinaNet, pose estimation, and instance segmentation.Source: §6 · p. 9
• The authors identify efficient multi-scale algorithms for stuff segmentation as a future direction.Source: §6 · p. 9
• Combining tracking with multi-scale inference could further accelerate video processing.Source: §6 · p. 9