PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaMixer: A Fast-Converging Query-Based Object Detector

Ziteng Gao, Limin Wang, Bing Han, Sheng Guo

arXiv:2203.16507v2cs.CV

TL;DR

Query-based object detectors use learnable queries, but their decoders have limited adaptability and often rely on extra networks. AdaMixer combines adaptive feature sampling with query-guided channel and spatial mixing. It reports strong detection performance, including 45.0 AP for ResNet-50 on MS COCO validation after 12 epochs.

  • Problem

    Query-based detectors have limited decoder adaptability in sampling and processing features, and commonly add encoders or pyramid networks that increase pipeline complexity.

    Source: §1 · p. 1

  • Method

    AdaMixer combines adaptive 3D feature sampling over space and scales with query-guided adaptive channel and spatial mixing.

    Source: §1 · p. 2, §3.3 · p. 4, §3.4 · p. 4

  • Results

    AdaMixer achieves superior performance, especially on small-object detection, with less computational cost; with ResNet-50, it reaches 45.0 AP on MS COCO validation after 12 epochs.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors propose AdaMixer as a strong baseline for future query-based object-detection research.

    Source: §5 · p. 8

    The total parameter number remains a little bit large, mainly because linear layers produce dynamic mixing weights.

    Source: §5 · p. 8

Abstract

from arXiv · show

Traditional object detectors employ the dense paradigm of scanning over locations and scales in an image. The recent query-based object detectors break this convention by decoding image features with a set of learnable queries. However, this paradigm still suffers from slow convergence, limited performance, and design complexity of extra networks between backbone and decoder. In this paper, we find that the key to these issues is the adaptability of decoders for casting queries to varying objects. Accordingly, we propose a fast-converging query-based detector, named AdaMixer, by improving the adaptability of query-based decoding processes in two aspects. First, each query adaptively samples features over space and scales based on estimated offsets, which allows AdaMixer to efficiently attend to the coherent regions of objects. Then, we dynamically decode these sampled features with an adaptive MLP-Mixer under the guidance of each query. Thanks to these two critical designs, AdaMixer enjoys architectural simplicity without requiring dense attentional encoders or explicit pyramid networks. On the challenging MS COCO benchmark, AdaMixer with ResNet-50 as the backbone, with 12 training epochs, reaches up to 45.0 AP on the validation set along with 27.9 APs in detecting small objects. With the longer training scheme, AdaMixer with ResNeXt-101-DCN and Swin-S reaches 49.5 and 51.3 AP. Our work sheds light on a simple, accurate, and fast converging architecture for query-based object detectors. The code is made available at https://github.com/MCG-NJU/AdaMixer

1. Introduction

AdaMixer targets limits in query-based detection by making query decoding adapt to object location, scale, and content. It reports strong MS COCO results with both a short training schedule and longer training.

1. Introduction

  • Dense detectors rely on grid-based priors, which bring challenges including anchor design, training-sample selection, and post-processing of redundant detections.Source: §1 · p. 1
  • Query-based decoders have limited adaptability in both spatial feature sampling and processing, while extra encoders or pyramid networks add pipeline complexity and training demands.Source: §1 · pp. 1–2
  • AdaMixer adapts feature sampling over space and scales and uses query-guided dynamic channel and spatial mixing to decode sampled features.Source: §1 · p. 2
  • AdaMixer uses a backbone and its proposed decoder without extra attentional encoders or explicit pyramid networks.Source: §1 · p. 2
  • With ResNeXt-101-DCN and Swin-S, AdaMixer reaches 49.5 and 51.3 AP; with ResNet-50 at 12 epochs, it reaches 45.0 AP and 27.9 APs.The 12-epoch setting uses random flipping as the only augmentation.Source: §1 · p. 2

2. Related Work

Related work contrasts dense detection with query-based methods and locates AdaMixer within the latter. AdaMixer focuses on query-decoder adaptability across images, while Table 1 compares decoder adaptability across methods.

2. Related Work

  • Dense detectors use anchors or anchor points to search feature maps, while query-based detectors predict object sets by attending queries to image features.Source: §2 · p. 2
  • Prior query-based approaches address issues such as limited spatial resolution, small-object performance, or slow convergence through changes to attention, positional modeling, or query initialization.Source: §2 · p. 2
  • AdaMixer centers its approach on query-decoder adaptability across images through adaptive 3D feature sampling and adaptive content decoding, without extra encoders or explicit pyramid networks.Source: §2 · p. 2
  • Table 1 compares decoder adaptability across query-based object detectors; its scope includes trainable networks inserted between pretrained backbones and query decoders.Source: Table 1 · p. 3

3. Approach

AdaMixer improves query decoding by adapting both where queries sample features and how they decode sampled content. Its decoder treats semantic content and position separately while adapting both to each image.

3. Approach

  • AdaMixer focuses on decoder adaptability, using semantic and spatial adaptation to relate queries to each image.The approach introduces adaptive 3D feature sampling and adaptive content decoding.Source: §3 · p. 3

3.1. Object Query Decoder Revisited

  • DETR-style decoders adapt sampling locations but decode sampled content statically, while Sparse R-CNN adapts content processing but limits spatial structure and sampling locations.The review motivates a decoder that adapts both sampling locations and sampled content to image content.Source: §3.1 · p. 3

3.2. Our Object Query Definition

  • Each object query pairs a content vector q with a positional vector (x, y, z, r), from which its bounding box is decoded.The positional vector encodes box center coordinates and logarithmic scale and aspect ratio; the base stride is set to 4 in the experiments.Source: §3.2 · p. 3

3.3. Adaptive Location Sampling

  • Queries predict 3D offsets that select locations across aligned feature-map scales, with interpolation combining bilinear spatial sampling and Gaussian weighting across scales.Offsets are not restricted to the query box; group sampling splits channels to generate more diverse sampling points.Source: §3.3 · p. 4, Figure 2 · p. 4

3.4. Adaptive Content Decoding

  • Adaptive mixing uses query-conditioned channel mixing followed by spatial mixing to decode sampled features’ semantics and structures.The dynamic channel weights are shared across sampling points, while the spatial operation mixes across sampled positions.Source: §3.4 · pp. 4–5, Figure 3 · p. 4

3.5. Overall AdaMixer Detector

  • The decoder updates query content and position through separate streams, with position-aware self-attention incorporating positional embeddings and box-overlap information.The detector consists of a backbone and AdaMixer decoder, without explicit feature pyramid networks or attentional encoders between them.Source: §3.5 · p. 5, Figure 4 · p. 5

4. Experiments

Experiments evaluate AdaMixer on MS COCO under short- and longer-training recipes, then test its decoder choices through ablations. The results report strong performance with limited training and comparisons against other query-based detectors.

4.1. Implementation Details

  • Experiments use COCO trainval35k/minival and 1× (12 epochs, flipping only) or 3× (36 epochs, crop and multi-scale augmentation) recipes; ResNet-50 runs take about 9 and 29 hours.The default configuration uses six decoder stages and four mixer groups, with 32 sampling points and 128 mixing output patterns per group.Source: §4.1 · p. 6

4.2. Fast Convergence with Limited Budgets

  • 42.7 AP with 100 queries; 44.1 and 45.0 AP with 300 and 500 queries, alongside 27.0 and 27.9 APs for small objects, after 12 epochs with random flipping only.The 100-query result outperforms state-of-the-art traditional and query-based detectors under the limited-budget comparison.Source: Table 3 · p. 6, §4.2 · pp. 6–7

4.3. Ablation Studies

  • Enabling both adaptive sampling locations and adaptive content mixing beats non-adaptive decoding by 7.0 AP, while channel-then-spatial mixing performs best among the tested mixing designs.The ablations associate the two adaptability components with query-dependent offsets and mixing weights; the non-adaptive variants hold those weights constant across queries and images.Source: Table 4 · p. 7, §4.3 · p. 7
  • Combining sinusoidal position embeddings with box IoF in query self-attention notably improves results, while four mixer groups yield the fewest FLOPs and parameters with promising performance.The authors argue that IoF encodes containment relations between query boxes, which is important for self-attention to imitate NMS.Source: Table 4 · p. 7, §4.3 · pp. 7–8, Table 6 · p. 8
  • AdaMixer avoids extra pyramid networks by modeling semantics and scales in the decoder; performance generally tracks sampling and mixing pattern counts, with 32 points and 128 patterns identified as a good balance.Performance saturates beyond 32 sampling points and decreases when mixing output patterns exceed 128; models with extra pyramid networks might need more training time and samples.Source: §4.3 · p. 7

4.4. Comparison with Other Query-based Detectors

  • With longer 3× training, AdaMixer significantly outperforms competitive query-based detectors at lower computational cost; Swin-S reaches 51.3 AP and 34.2 APs with single-scale testing.Among the compared query-based detectors, AdaMixer is the only one that does not require extra attentional encoders or explicit pyramid networks.Source: Table 5 · p. 8, §4.4 · p. 8

5. Conclusion and Limitation

AdaMixer improves query-based object decoding through adaptive sampling and channel/spatial mixing, while avoiding extra modeling networks between backbone and decoder. The authors report strong performance, especially for small objects, fast convergence, and a remaining parameter-count limitation.

Conclusion

  • AdaMixer achieves superior performance, especially on small-object detection, with less computational cost than other query-based detectors and fast convergence under limited training budgets.The authors identify AdaMixer as a potential strong baseline for future research.Source: §5 · p. 8
  • Adaptive 3D sampling and adaptive channel and spatial mixing improve query-based decoding without extra network modeling between the backbone and decoder.Source: §5 · p. 8

Limitation

  • Despite grouping, AdaMixer retains a somewhat large parameter count, mainly because the linear layer producing dynamic mixing weights has many parameters.Further parameter reduction is left for future work.Source: §5 · p. 8

A. 1. Detection Performance on COCO test set

Table 7 reports AdaMixer performance on COCO test-dev under a longer training scheme and single-model, single-scale testing. The reported models are the same as those in Table 5, whose performance is reported on COCO minival.

A. 1. Detection Performance on COCO test set

A. 2. More Studies on Adaptive 3D Sampling

The sampling study compares adaptive 3D sampling with alternatives on COCO minival under 1× training without FPNs. The authors report that single-feature sampling and RoIAlign perform worse, while adaptive 3D sampling also extends spatial offset sampling to scale and uses sparse queries.

A. 2. More Studies on Adaptive 3D Sampling

  • Single-feature sampling and RoIAlign produce inferior results to AdaMixer's adaptive 3D sampling in the reported study.The comparison uses COCO minival, 1× training, and no FPNs.Source: A. 2. More Studies on Adaptive 3D… · p. 9, Table 8 · p. 9
  • Adaptive 3D sampling lets queries adapt sampling locations and feature levels, whereas RoIAlign restricts locations to the bounding box and discretizes the sampling level.Source: A. 2. More Studies on Adaptive 3D… · p. 9
  • RoIAlign lacks multi-level feature modeling and therefore necessitates multi-scale feature interaction necks, according to the authors.Source: A. 2. More Studies on Adaptive 3D… · p. 9
  • Adaptive 3D sampling uses offsets for spatial sampling like deformable convolutions, extends adaptation to scale, and operates sparsely through queries rather than densely.The authors also describe it as a continuous, interpolable generalization of scale-equalizing operators on discretized feature maps.Source: A. 2. More Studies on Adaptive 3D… · p. 9

A. 3. Model Details

The implementation uses PyTorch primitives, and the reported speed comparison covers training time, inference FPS, and theoretical FLOPs. The authors note that the current adaptive sampling implementation could be optimized.

A. 3. Model Details

  • The model is implemented in the mmdetection framework using PyTorch primitives, without customized CUDA code.Source: A. 3. Model Details · p. 9
  • The speed table reports FPS measured on one Nvidia V100; training-hour estimates for other detectors use eight V100 cards and a few training iterations.Source: Table 9 · p. 9
  • Backbones, including Swin-S, are initialized from ImageNet-1K classification pre-trained models.Source: A. 3. Model Details · p. 9
  • AdaMixer shows advantages in theoretical FLOPs, actual training time, and inference FPS compared with other detectors.The cited passage gives no specific speed values.Source: A. 3. Model Details · p. 9
  • The current adaptive 3D sampling procedure uses PyTorch's grid_sample primitive, which the authors say could be optimized.Source: A. 3. Model Details · p. 9

A. 4. Visualizations

The visualizations show how AdaMixer’s sampling varies across groups, stages, and queries, with the authors suggesting that its changing focus may aid bounding-box estimation.

A. 4. Visualizations

  • Different queries show different sampling-point patterns, which the authors interpret as evidence of AdaMixer’s enhanced sampling adaptability and flexibility.The visualizations use COCO minival samples.Source: A. 4. Visualizations · p. 10
  • Sampling points can widen after focusing on small regions, then focus again across stages 2→3→4→5→6.The authors suspect this non-monotonic pattern helps estimate bounding boxes more accurately.Source: A. 4. Visualizations · p. 10
  • Point size encodes the sampling z-axis coordinate, with larger points indicating larger values; colors distinguish groups.Source: A. 4. Visualizations · p. 10
  • The first row shows the input image and detections, while subsequent rows show sampling points for each stage.Source: A. 4. Visualizations · p. 10
  • The visualizations show sampling points extending outside objects and different groups preferring different semantics.Source: A. 4. Visualizations · p. 10

Related papers

Loading 2203.16507v2…