RTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generation

1 Introduction

• The baseline retains an outdated predicate when a persistent person–object pair changes relation, illustrating relational semantic inertia.On relation-transition frames, its SGDET R@10 is 34.3, versus 45.9 on non-transition frames under the with constraint setting.Source: §1 · p. 2
• RTAformer separates entity and relation temporal refinement, using calibrated relation-query differences as explicit cues for predicate updates.Cross-frame instance alignment encourages consistent query grounding.Source: §1 · p. 2
• Experiments on Action Genome show improved scene graph generation over state-of-the-art methods, with larger gains on relation-transition than non-transition frames in a backbone-controlled comparison against OED.Source: §1 · p. 3

2 Related Work

• VidSGG methods shifted from multi-stage pipelines to one-stage frameworks that integrate instance and relation prediction, including query-based OED and ARN and slot-attention UNO.Source: §2 · p. 3
• Temporal methods use cross-frame context through temporal transformers, anticipatory objectives, tracklets, query cross-attention, or entity-predicate reasoning.Other reliability approaches include memory-guided debiasing in TEMPURA and selective spatio-temporal denoising in TD2-Net.Source: §2 · p. 3
• TR2 addresses time-varying relations with text-guided supervision of inter-frame relation-feature differences and a short-term message token.Source: §2 · p. 3
• Unlike TR2, RTAformer injects adjacent-frame relation-query differences before temporal aggregation, while separate streams model entity persistence and relation changes.Training-time instance alignment encourages consistent cross-frame query grounding to reduce representational drift in transition cues.Source: §2 · pp. 3–4, Figure 2 · p. 4

3.1 Problem Formulation

• VidSGG maps a video sequence to dynamic frame-wise graphs whose nodes are detected objects and whose relations are multi-label subject–predicate–object triplets.The objective is to learn a mapping from video to graphs that maximizes P(G | V).Source: §3.1 · p. 4

3.2 Framework Overview

• RTAformer uses a one-stage set-prediction framework with frame-wise parsing, disentangled temporal aggregation, and training-time instance alignment.A vision backbone and Transformer encoder feed cascaded instance-pair and relation-context decoders; DTA and SSIA handle temporal refinement and alignment.Source: §3.2 · p. 5

3.3 Frame-Wise Spatial Triplet Parsing

• Learnable triplet queries represent candidate subject–object interactions; the instance-pair decoder extracts spatial context before the relation decoder produces relation-aware representations.Decoder layers use self-attention among queries and cross-attention to retrieve frame features.Source: §3.3 · p. 5

3.4 Disentangled Temporal Aggregation

• DTA separates instance and relation streams: instance queries aggregate persistent entity evidence, while TADI injects calibrated adjacent-frame relation-query changes before temporal aggregation.TADI decomposes displacement into RMS magnitude and normalized direction, with a magnitude-dependent scale gate and foreground confidence modulating the update.Source: §3.4 · pp. 6–7

3.5 Spatio-Semantic Instance Alignment

• SSIA applies contrastive regularization to adjacent-frame instance-pair queries, using matched triplets and ground-truth object persistence to support same-index alignment and reduce representational drift.The alignment is applied during spatial training and adds no inference overhead.Source: §3.5 · p. 8

3.6 Optimization and Inference

• Training uses two phases: spatial representations are learned first, then the frozen backbone and spatial modules leave DTA and temporal heads to optimize sequence-level clips.At inference, the model performs spatial triplet parsing, DTA, and final prediction without external trackers or temporal post-processing.Source: §3.6 · p. 9

4.1 Comparison with State-of-the-Art Methods

• With ViT-L/14, RTAformer improves R@10 over UNO by 5.3 and 6.4 percentage points under with-constraint and no-constraint settings, respectively.It also achieves the highest R@K among methods reported in Table 1.Source: §4.1 · pp. 10–11

4.2 Ablation Studies

• Combining SSIA and DTA achieves the highest recall across evaluated cutoffs and settings, while SSIA raises R@10 from 41.2 to 42.2 with constraint when DTA is enabled.The module ablation uses ViT-B/14 and compares against a frame-wise spatial baseline.Source: Table 2 · p. 11, §4.2 · p. 11

4.1 Comparison with State-of-the-Art Methods

• With ViT-L/14, RTAformer reaches mR@K of 30.5/38.3/44.2 with constraint and 37.2/53.5/64.4 without constraint, the highest reported across evaluated cutoffs and settings.The comparison includes methods reporting mR@K, and the gains complement overall R@K improvements.Source: §4.1 · p. 11

4.3 Analysis of Relation Transitions

• Transition frames are more challenging for matched-backbone OED, and RTAformer’s R@10 gains are larger on transition than non-transition frames: 4.8 versus 1.7 points with constraint and 4.1 versus 1.9 without.The comparison uses frozen DINOv2 ViT-B/14; Table 4 reports recall entries for R@10/R@20.Source: Table 4 · p. 12, §4.3 · p. 12
• RTAformer exceeds OED’s transition-frame R@10 by 8.0 and 8.6 percentage points with and without constraint, respectively, in comparisons using released SGDET configurations.It has higher recall than OED and TEMPURA at every evaluated cutoff under both constraint settings.Source: Figure 4 · p. 12, §4.3 · p. 13
• At transition boundaries, RTAformer updates predicates for persistent entities, whereas OED preserves previous-frame relations in the illustrated person–broom and person–door cases.The examples show changes to holding and not contacting predicates.Source: Figure 5 · p. 13, §4.3 · p. 14
• Transition prevalence rises from 20.9% to 55.4% across increasing relation-query displacement deciles, versus an overall 33.7% rate; displacement yields AUROC 0.621 as a transition score.The passage describes this cue as informative but imperfect.Source: Figure 4 · p. 12, §4.3 · p. 13

5 Limitations

• Evaluation covers only Action Genome under SGDET, and DETR-style decoding does not guarantee exact cross-frame query correspondence.Broader evaluation across VidSGG datasets and protocols would strengthen evidence for generalization.Source: §5 · p. 14

• Relational semantic inertia describes outdated predicate predictions persisting after relationships between visible entities change.Source: §6 · p. 14
• RTAformer uses calibrated relation-query changes to guide predicate updates and improves SGDET performance on Action Genome.Source: §6 · p. 14
• A backbone-controlled comparison shows larger gains on relation-transition frames than on non-transition frames.Source: §6 · p. 14

S1.1 Architecture Details

• RTAformer uses frozen DINOv2 backbones and DETR-style Transformer modules with 256-dimensional model width, 2,048-dimensional feed-forward layers, 8 attention heads, and 100 triplet queries.Each Temporal Aggregation Decoder block contains self-attention, temporal cross-attention, and a feed-forward network.Source: S1.1 Architecture Details · p. 18

S1.2 Training and Evaluation Details

• Training first optimizes the frame-wise spatial parser with frame-level supervision and SSIA, then freezes the backbone and parser while training temporal modules on sequence clips.The SSIA loss is used only during spatial-module training.Source: S1.2 Training and Evaluation Details · pp. 18–19
• The SSIA validity threshold is IoU 0.3, combined with category consistency to retain cross-frame anchors despite displacement between sparsely sampled annotated frames.This threshold applies to SSIA positive-anchor mining, not SGDET evaluation.Source: S1.2 Training and Evaluation Details · p. 19

S1.3 Relation-Transition Subset Construction

• The evaluation-only transition subset includes persistent subject–object pairs whose predicate sets differ without overlap in at least one relation family.Partially overlapping predicate sets are excluded; the 47,357 eligible target frames partition into 25,883 transition frames and 21,474 non-transition frames, where non-transition means no transition under this criterion.Source: S1.3 Relation-Transition Subset… · pp. 19–20
• Transition-subset comparisons are reported only for TEMPURA and OED because compatible checkpoints, hyperparameters, and evaluation resources were available for controlled re-evaluation.Previously reported full-test-set results are not directly comparable to this newly constructed subset evaluation.Source: S1.3 Relation-Transition Subset… · p. 21

S2 Additional Analysis of Spatio-Semantic Instance Alignment

• Each query index links instance-pair and relation predictions into one triplet, and SSIA stabilizes that instance-pair grounding so relation-query displacement is less likely to reflect representational drift.Hungarian matching supervises subject, object, and predicate predictions jointly at the assigned query index.Source: S2 Additional Analysis of… · p. 21

S2.1 Illustration of SSIA Validity Masking

• SSIA uses a same-index pair as a positive anchor only when the corresponding object has a same-category match with sufficient spatial overlap in the previous frame.Validity masking avoids imposing alignment when persistence is unverified.Source: S2.1 Illustration of SSIA Validity… · pp. 21–22, Figure S1 · p. 22

S2.2 Representation-Space Visualization of Same-Index Query Consistency

• In t-SNE visualizations, SSIA-trained same-slot embeddings form more compact clusters with shorter cross-frame links than embeddings trained without SSIA.The comparison uses 900 sampled temporal instance pairs per model.Source: S2.2 Representation-Space… · p. 22, Figure S2 · p. 23

S2.3 Temporal Grounding Consistency of Same-Index Queries

• Under independent frame-wise inference without temporal aggregation or cross-frame association, SSIA-trained same-index queries maintain object-side grounding across annotated frames.The examples show consistent localization of objects such as a sandwich and dish despite hand motion and object displacement.Source: S2.3 Temporal Grounding Consistency of… · p. 23, Figure S3 · p. 24