RelationNet++: Bridging Visual Representations for Object Detection via Transformer Decoder

1 Introduction

• BVR uses an attention-based decoder to let auxiliary visual representations enhance the main detector representation through appearance and geometry interactions.The main representation supplies query features, while other representations act as auxiliary keys.Source: §1 · pp. 1–2
• Key sampling and shared location embeddings reduce the computation and memory costs of attention over numerous key instances and geometric relations.These techniques support efficient computation of the proposed bridging module.Source: §1 · p. 2
• BVR bridges center and corner representations into anchor-based detection, replacing classification and regression inputs with enhanced features while working in-place.The detector retains its overall inference process and convenience of use.Source: §1 · p. 2
• 1.5∼3.0 AP improvements are achieved across RetinaNet, Faster R-CNN, FCOS and ATSS, with RelationNet++ reaching 52.7 AP on COCO test-dev.The result comes from improving a strong ATSS detector by about 2.0 AP.Source: §1 · p. 2
• The work contributes a general in-place BVR module, efficient bridging techniques, and demonstrations across four prevalent object detectors.These contributions summarize the proposed module, its efficiency designs, and its broad applications.Source: §1 · p. 2

2.1 Object / Part Representations

• Bounding boxes use four coordinates and commonly extract features through pooling, while center points reduce the hypothesis space from O(H^2 × W^2) to O(H × W).Center representations typically use the image feature at the center point.Source: §2.1 · p. 3
• Corner representations detect points such as top-left and bottom-right corners, then use their locations as part features to determine a bounding box.This representation treats object parts separately before composing the final object.Source: §2.1 · p. 3
• Bounding-box and center representations generally favor category classification, whereas corner representations favor finer object localization.Object-based representations are also more compatible with end-to-end learning because corner-based methods require composing objects from parts.Source: §2.1 · p. 3

2.2 Object Detection Frameworks in a Representation View

• Detection frameworks evolve intermediate representations toward final bounding boxes, motivating categories such as anchor-based versus anchor-free and object-based versus part-based flows.Figure 2 summarizes representation flows for several typical frameworks.Source: §2.2 · p. 3, Figure 2 · p. 3
• Faster R-CNN and RetinaNet use bounding boxes as intermediate representations, but Faster R-CNN is multi-stage whereas RetinaNet is one-stage with denser anchor hypotheses.Faster R-CNN refines anchors into proposals, while RetinaNet directly applies a localization-refinement head.Source: §2.2 · p. 3
• FCOS uses center points and directly regresses four sides, while CornerNet groups predicted corners to compose final bounding boxes.Corner-based detectors generally reveal better localization than object-level representations in the cited comparison.Source: §2.2 · p. 4

3.1 A General Attention Module to Bridge Visual Representations

• BVR uses the master representation as queries and auxiliary representations as keys, combining appearance and geometric relationships to strengthen query features.The module is inspired by Transformer decoders and generates strengthened features for the master representation.Source: §3.1 · p. 4
• The attention module handles heterogeneous geometry by mapping query-key relative locations through cosine/sine embeddings and a two-layer MLP with eight attention heads.For bounding boxes and points, a center or corner is extracted so their geometric descriptions can be compared.Source: §3.1 · p. 4

3.2 BVR for RetinaNet

• RetinaNet uses center and corner predictions as auxiliary keys to enhance classification and regression features, respectively, with top-k key selection shared across pyramid levels.A default top-k value of 50 limits computation and suppresses low-quality background candidates.Source: §3.2 · pp. 5–6
• Shared relative location embedding reduces BVR geometry-term costs to O(time) = (d0+d0d1+d1G)·400^2+GKHW and O(memory) = (2+d0+d1+G)·400^2+GKHW.The approach replaces direct pairwise geometry computation with a fixed relative-location map and bilinear sampling.Source: §3.2 · pp. 5–6, Table 3 · p. 7

3.3 BVR for Other Frameworks

• BVR extends to ATSS, FCOS, and Faster R-CNN by adapting the master and auxiliary representations while retaining the same bridging principle.For Faster R-CNN, proposal RoIAlign features predict center and corner representations, while direct geometry computation is used because the query count is smaller.Source: §3.3 · p. 6

3.4 Relation to Other Attention Mechanisms in Object Detection

• Unlike non-local networks and RelationNet, which model dependencies within pixels or bounding boxes, BVR connects different representation forms and is complementary to those methods.Its novelty includes heterogeneous similarity computation, key selection, and shared relative location embedding.Source: §3.4 · p. 6

4.1 Method Analysis using RetinaNet

• Sharing keys yields +1.6 and +1.5 AP over independent keys for 20 and 50 keys, respectively, with 50 keys achieving the best accuracy.Source: §4.1 · p. 7
• The BVR-enhanced RetinaNet outperforms the original RetinaNet by 2.9 AP, while an auxiliary point head alone improves the baseline by 0.8 AP.Source: §4.1 · pp. 7–8
• Shared location embeddings save 42× memory and 102× FLOPs versus direct position computation while slightly improving performance from 38.3 to 38.5 AP.Source: §4.1 · p. 7
• Center representations are more suitable auxiliaries for classification, whereas corner representations are more suitable auxiliaries for regression.Source: §4.1 · p. 8
• Combining appearance and geometry terms improves AP by 1.1 over appearance alone and 0.9 over geometry alone, with geometry helping more at larger IoU criteria.Source: §4.1 · p. 8

4.2 BVR is Complementary to Other Attention Mechanisms

• BVR is mostly complementary to non-local and object-relation modules: on Faster R-CNN, BVR adds 1.9 AP alone and 2.0 AP on top of ORM.Source: §4.2 · p. 9

4.3 Generally Applicable to Representative Detectors

• BVR improves RetinaNet, Faster R-CNN, FCOS, and ATSS by 1.5–2.0 AP using a strong backbone, multiscale training, and longer training.Source: §4.3 · p. 9

4.4 Comparison with State-of-the-Arts

• 52.7 AP is reached on COCO test-dev by improving a strong ATSS detector by 2.0 AP.Source: §4.4 · p. 9

• BVR bridges heterogeneous visual representations through attention, enhancing main detector representations with other representations.Source: §5 · p. 9
• BVR achieves about 1.5–3.0 AP improvements across RetinaNet, Faster R-CNN, FCOS, and ATSS, reaching 52.7 AP on COCO test-dev.Source: §5 · p. 9

• The method is intended for object-oriented visual applications, where object detection is usually an indispensable component.Source: Broader Impact · p. 10
• Unpredictable failures and data biases constrain use, especially where failures could lead to serious consequences.Source: Broader Impact · p. 10