REACT++: Efficient Cross-Attention for Real-Time Scene Graph Generation

1 Introduction

• REACT++ addresses the need to balance object detection accuracy, relation prediction accuracy, and latency for real-time SGG.The paper frames SGG as combining object detection and relation prediction while requiring low-cost inference for downstream applications.Source: §1 · p. 2
• REACT previously improved latency and object detection but retained relation-head bottlenecks in feature extraction, global scene modeling, and directional prototype interactions.The prior REACT model used YOLO, prototype learning, and Dynamic Candidate Selection, but its relation prediction component remained limited by two-stage biases.Source: §1 · p. 2
• DAMP replaces slow ROI Align with direct multi-scale pooling based on one-stage detector features for more efficient feature extraction.The contribution summary states that DAMP improves both latency and accuracy in the SGG task relative to traditional ROI Align.Source: §1 · pp. 2–3
• AIFI provides low-cost global scene context, complementing subject and object representations for relation prediction.The module is inspired by RT-DETR and is intended to extract global information about the scene.Source: §1 · p. 3
• CARPE uses subject-to-object cross-attention and Rotary Position Embedding to encode asymmetric relations and spatial information in prototype space.The design replaces symmetric prototype interactions with dedicated subject and object cross-attention layers and incorporates spatial biases through RoPE.Source: §1 · p. 3
• The paper evaluates REACT++ experimentally on three datasets and separately studies DAMP, AIFI, and Dynamic Candidate Selection through ablations.The paper organization identifies experimentation across three datasets and ablation analyses for the principal architectural components.Source: §1 · p. 3

2.1 Scene Graph Generation

• Scene graph research has expanded from language-prior and context-based relation modeling toward de-biasing, long-tail learning, and Transformer-based efficiency.The related work discusses Visual Genome, language priors, contextual reasoning, dataset biases, alternative datasets, and direct triplet modeling.Source: §2.1 · pp. 3–4

2 Related Work

• SGG research spans two-stage pipelines that prioritize detection performance and one-stage pipelines that often prioritize efficiency.Two-stage methods sequentially use an object detector and relation predictor, whereas one-stage methods infer objects and relations directly from image features.Source: §2.1 · p. 4

2.2 Real-Time SGG

• Two-stage systems commonly rely on large Faster-RCNN backbones, while REACT++ decouples detection from relationship prediction and preserves the detector’s original class probabilities.The design removes a dedicated classifier and uses YOLO-derived subject and object features to improve first-stage efficiency without compromising the second stage.Source: §2.2 · pp. 4–5
• Prior REACT improved the first stage but did not address efficiency in its relation-modeling stage.This limitation motivates extending the architecture with a more efficient relation head.Source: §2.2 · p. 5
• One-stage models can reduce latency but may underperform in object detection because they learn object and relation representations jointly.The paper argues that replacing Faster-RCNN with YOLO in a two-stage pipeline can combine lower latency with stronger object detection than one-stage approaches.Source: §2.2 · p. 5

Object Detection Accuracy

• Two-stage SGG redundantly decodes object labels twice, and the second prediction depends on each model’s context-learning strategy.This design may create substantial differences from the original detector’s object-detection accuracy.Source: §3 · pp. 5–6
• Object detection can drop by almost 13 mAP@50 points after relation prediction with GPS-NET, showing harmful dependency between the two stages.The authors motivate replacing Faster-RCNN with YOLO for both detection and box-specific feature extraction.Source: §3 · p. 6

Global vs Local Context

• ROI-based local pooling and union features provide limited global scene context, motivating global-to-local fusion for context-dependent predicates.Global information can help distinguish predicates associated with scenes such as kitchens, mountains, and beaches.Source: §3 · pp. 6–7

Asymmetry of Relations

• Prior methods use the same object representation across relation roles, although relations such as <person, eating, pizza> and <pizza, eating, person> are asymmetric.The authors motivate dedicated cross-attention to encode subject–object asymmetry in the shared embedding space.Source: §3 · p. 7

Proposal Candidate Selection

• Fixed top-proposal selection increases graph-learning cost, while ranking scores weight subject and object confidence more heavily than predicate confidence.Filtering low-confidence proposals before relation prediction can reduce computation with minimal overall accuracy impact.Source: §3 · pp. 7–8

4 Toward Real-Time SGG

• REACT++ replaces Faster-RCNN with a YOLO-based decoupled two-stage design and redesigns the relation head to remove multi-scale alignment and RoIAlign pooling.The changes target a major computational bottleneck while retaining two-stage modeling and adding global context and dedicated cross-attention.Source: §4 · pp. 8–9

4.1 Decoupled Two-Stage SGG

• The DTS design freezes detector regression and classification heads, performs NMS before relation prediction, and uses detector classes for final object predictions.The relation stage therefore predicts object pairs and predicates rather than re-predicting object class labels.Source: §4.1 · p. 9

4.2 DAMP: Detection-Anchored Multi-scale Pooling

• RoI Align contributes up to 40% of relation-head latency, motivating DAMP’s direct retrieval and pooling of YOLO feature vectors using retained detection indices.DAMP leverages YOLO’s anchor-free grid representation to eliminate explicit RoI Align operations.Source: §4.2 · p. 10

(1) Multi-scale gather with Gaussian neighbourhood.

• DAMP gathers a Gaussian-weighted 3×3 neighborhood around each detection anchor across FPN levels P3, P4, and P5.The anchor is snapped to each pyramid level before neighborhood aggregation.Source: (1) Multi-scale gather with Gaussian… · p. 10

(2) Projection and fusion.

• Each gathered vector is projected to dimension D before the three feature levels are fused, reducing gathering from N × 7 × 7 × 3 to N × 9 × 3 operations.This is a 5.4× reduction with no additional parameters in the gather stage, although the fixed Gaussian neighborhood may introduce noise.Source: (2) Projection and fusion · p. 10

4.3 CARPE: Cross-Attention Rotary Prototype Embedding

• CARPE models relation triplets in a shared prototype space and explicitly distinguishes subject and object representations through separate cross-attention roles.This redesign addresses the asymmetry between subject and object representations in relation modeling.Source: §4.3 · p. 10

(1) Semantic lifting.

• REACT++ replaces three independent ReLU MLPs with SwiGLU lifting blocks and ties subject/object weights to reduce parameters while gating activations multiplicatively.The multiplicative gate is intended to eliminate ReLU dead neurons, and tied subject/object weights halve the lifting parameter count.Source: (1) Semantic lifting · p. 11

(2) Visual–semantic fusion.

• Cross-attention replaces fixed visual–semantic gates, letting visual features select and blend relevant object prototypes with geometry-aware logits.Queries are visual features, while keys and values come from the full object prototype bank; this is described as more expressive at equal or lower parameter cost.Source: (2) Visual–semantic fusion · p. 11

(3) Geometry encoding.

• Geometry RoPE encodes normalized box widths, heights, centers, and area, adding a head-specific bias to attention logits before softmax.The resulting attention pattern can vary smoothly with box position.Source: (3) Geometry encoding · p. 11

(4) Prototype bank.

• An EMA shadow buffer supplies stabilized prototypes for initialized classes, while learnable weights remain a fallback for uninitialized classes.The method uses current-batch mean normalized relation representations to update the EMA prototypes, helping rare-class prototypes remain stable across batches.Source: (4) Prototype bank · p. 12

4.4 Dynamic Candidate Selection

• DCS computes an inference threshold xopt from the first proposal count where the generated metric slope falls below ϵ.Training uses a maximum of θ = 100 proposals, while evaluation samples candidate counts starting from 0 before selecting the threshold.Source: §4.4 · p. 12

5 Experiments

• Experiments use PSG, IndoorVG, and VG150, with R@K, mR@K, F1@K, and mAP@50 measuring relation and object-detection performance.The study also evaluates DTS, REACT++ components, and DCS using YOLOV8m and retrained comparison models.Source: §5 · p. 12, §5.1 · p. 12, §5.2 · p. 13

5.4 Results

• On PSG, REACT++ improves meanRecall@K by 5 points or 20% and Recall@K by 6 points or 17.9% over REACT, while using 20% less latency and 17% fewer parameters.The comparison is reported alongside PSG results in Table 2.Source: Table 2 · p. 13, §5.4 · p. 14
• DTS-YOLOV8 improves PSG mAP by 54.37% over Faster-RCNN two-stage approaches and by 120% on average over one-stage approaches, while improving F1@K by 58.47% over Faster-RCNN.These results compare object detection and relation prediction performance on PSG.Source: Table 2 · p. 13, §5.4 · p. 14
• On IndoorVG, REACT++ improves meanRecall@K to 20.7 versus REACT’s 18.0, although REACT remains higher in Recall@K at 30.9 versus 28.1.Table 3 covers IndoorVG and VG150 comparisons.Source: §5.4 · p. 14, Table 3 · p. 15
• VG150 results are constrained by low-quality bounding-box annotations and ambiguous classes; REACT++ surpasses REACT and some two-stage methods but trails PE-NET and SQUAT.The authors attribute the sensitivity to noisy data partly to REACT++ having fewer parameters.Source: §5.4 · pp. 14–15, Table 3 · p. 15
• 20% lower latency than REACT, from 32.5 to 25.9, while DCS adds 66.5% average latency reduction with a relative 1% F1@K loss.REACT++ with DCS is reported below 20ms and has 35.8M parameters.Source: §5.4 · p. 15

6 Ablation Studies

• The ablation program covers DAMP, AIFI, DCS, and experiments with alternative YOLO variants to assess REACT++ design choices.The experiments use IndoorVG for the DAMP and AIFI studies and vary proposal counts and detector backbones in later studies.Source: §6 · p. 15, §6.1 · p. 15, §6.3 · p. 17

6.1 DAMP vs. RoI Align for Object Feature Extraction

• DAMP improves the accuracy–latency balance over RoI Align, delivering a 0.9-point F1@K gain with relatively low latency.Direct DA extraction is 9.3ms, or 32%, faster than RoI Align but loses 1.9 F1@K points; DAMP remains stable across training epochs.Source: §6.1 · p. 16, Figure 3 · p. 16

6.2 Impact of the AIFI block for global context

• AIFI increases F1@K by 0.42 points, or 1.8%, while removing it improves overall parameter count and latency.The ablation uses DAMP box features and the CARPE predictor with geometryRoPE bias.Source: §6.2 · pp. 16–17, Table 4 · p. 17

6.3 Latency and Number of Proposals: the DCS case

• DCS achieves an optimal average F1@K of 23.89 at k = 47, just before performance starts to saturate.With 130 proposals, latency is similar to original REACT with 100 proposals (32.5ms).Source: §6.3 · pp. 17–18

6.4 Scaling REACT++

• YOLO12-medium achieves 30 in F1@K on PSG, while its overall performance surpasses YOLOV8-medium; latency consistently increases across YOLO12 model sizes.YOLO12-large is slightly worse in F1@K than YOLO12-medium, and YOLOV8-medium is faster than YOLO12-nano and YOLO12-small.Source: §6.4 · p. 18

7 Conclusion

• REACT++ attains competitive SGG performance at 25.9ms latency by combining DTS, DAMP, AIFI, CARPE, and DCS.The architecture replaces Faster-RCNN with YOLO and uses efficient prototype learning for relation prediction.Source: §7 · p. 19
• REACT++ identifies proposal quality and quantity as strongly correlated with SGG performance and latency, with a usable trade-off available through its design.The conclusion also identifies RoI Align as a bottleneck replaceable by direct index-based pooling on one-stage detectors.Source: §7 · p. 19

Appendix A Hyperparameters

• Table A1 lists the main hyperparameter differences between REACT and REACT++ on PSG.Source: Table A1 · p. 20

Appendix B Ablation: DAMP

• Table B2 compares DA, DAM, and DAMP box feature extractors on IndoorVG, with DAMP adding Gaussian-weighted neighbourhood gathering across scales.DA means Detection-Anchored; DAM adds multi-scale features without gathering; DAMP adds 3×3 Gaussian-weighted neighbourhood gathering.Source: Table B2 · p. 20

• REACT++ is evaluated with different YOLO backbone sizes on the PSG dataset.Source: Table C3 · p. 21