PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videos

Ziqi Gao, Jieyu Zhang, Wisdom Oluchi Ikezogwo, Jae Sung Park, Tario G. You, Daniel Ogbu, Chenhao Zheng, Weikai Huang, Yinuo Yang, Winson Han, Quan Kong, Rajat Saini, Ranjay Krishna

arXiv:2602.23543v3cs.CV

TL;DR

Existing spatio-temporal scene graph datasets are limited in scale and annotation completeness, motivating a resource that can support broader video reasoning. The paper introduces SVG2 through an automated synthesis pipeline and trains TraSeR with trajectory-aligned resamplers. TraSeR improves scene graph prediction across tasks and yields additional video question answering accuracy when its graphs are provided to a VLM.

  • Problem

    Existing spatio-temporal scene graph datasets are limited in scale, difficult to expand, and often incomplete for objects and temporal relationships.

  • Method

    The paper builds SVG2 with automated segmentation, tracking, semantic parsing, and GPT-5 relation inference, then trains TraSeR with temporal-window and object-trajectory resamplers.

  • Results

    +15∼20% relation prediction, +30% to +40% object prediction over open-source baselines, +13% object prediction over GPT-5, and +15% attribute prediction are reported for TraSeR.

  • Takeaways & Limitations

    +1.5%∼4.6% absolute video question answering accuracy gains are obtained when TraSeR’s generated scene graphs are supplied to GPT-4.1.

  • Takeaways & Limitations

    SVG2 inherits the current boundaries of the automated models used to synthesize it, while TraSeR remains trajectory-grounded rather than fully end-to-end.

Abstract

from arXiv · show

We introduce Synthetic Visual Genome 2 (SVG2), a large-scale panoptic video scene graph dataset. SVG2 contains over 636K videos with 6.6M objects, 52.0M attributes, and 6.7M relations, providing an order-of-magnitude increase in scale and diversity over prior spatio-temporal scene graph datasets. To create SVG2, we design a fully automated pipeline that combines multi-scale panoptic segmentation, online-offline trajectory tracking with automatic new-object discovery, per-trajectory semantic parsing, and GPT-5-based spatio-temporal relation inference. Human verification of SVG2 annotation accuracy confirms its reliability (objects: 93.8%, attributes: 88.3%, relations: 85.4%). Building on this resource, we train TRaSER, a video scene graph generation model. TRaSER augments VLMs with a trajectory-aligned token arrangement mechanism and new modules: an object-trajectory resampler and a temporal-window resampler to convert raw videos and panoptic trajectories into compact spatio-temporal scene graphs in a single forward pass. The temporal-window resampler binds visual tokens to short trajectory segments to preserve local motion and temporal semantics, while the object-trajectory resampler aggregates entire trajectories to maintain global context for objects. On the PVSG, VIPSeg, VidOR and SVG2 test datasets, TRaSER improves relation detection by +15 to 20%, object prediction by +30 to 40% over the strongest open-source baselines and by +13% over GPT-5, and attribute prediction by +15%. When TRaSER's generated scene graphs are sent to a VLM for video question answering, it delivers a +1.5 to 4.6% absolute accuracy gain over using video only or video augmented with Qwen2.5-VL's generated scene graphs, demonstrating the utility of explicit spatio-temporal scene graphs as an intermediate representation.

1 Introduction

Existing video scene graph datasets are small, sparse, and difficult to generalize beyond fixed distributions. SVG2 addresses this gap with a large automated dataset and TraSeR, which generates temporally structured graphs and improves downstream recognition and question answering.

  • Research gap: Limited high-quality spatio-temporal scene graph data constrain scale, completeness, and expansion because dense frame-level annotation is costly.Sparse frame sampling can miss objects that emerge or disappear, while long-tail bias and inconsistent temporal labels hinder generalization.
  • SVG2: 636K videos, 6.6M objects, 52M attributes, and 6.7M relations make SVG2 an order-of-magnitude expansion in scale and diversity over prior datasets.Human verification reports 93.8% object, 88.3% attribute, and 85.4% relation annotation accuracy.
  • SVG2: SVG2 combines panoptic segmentation, online–offline tracking with new-object discovery, per-trajectory semantic parsing, and GPT-5-based spatio-temporal relation inference.The tracking mechanism is designed to detect newly emerging instances and preserve identity consistency across videos.
  • TraSeR: TraSeR uses temporal-window and object-trajectory resamplers to preserve local temporal variation and global object context in a single forward pass.The model addresses rapidly changing attributes and relations alongside long token sequences and many relation candidates.
  • Results: +15∼20% relation prediction, +30% to +40% object prediction over open-source baselines, +13% object prediction over GPT-5, and +15% attribute prediction demonstrate TraSeR’s benchmark gains.The reported improvements span relation, object, and attribute prediction tasks.
  • Results: +1.5%∼4.6% absolute video question answering accuracy gains occur when TraSeR scene graphs are supplied to GPT-4.1.The comparison is against video-only baselines or video augmented with Qwen2.5-VL scene graphs.

2 Related work

Scene graph research progressed from large-scale image annotations toward cleaner, mask-grounded, and panoptic representations, while video methods still face limitations in fine-grained temporal grounding.

  • Image scene graphs: Visual Genome enabled large-scale object, attribute, and relational image annotations, followed by datasets targeting cleaner labels, broader coverage, or stronger grounding.VRD improved predicate labels, Open Images V6 expanded mask-grounded relations, and PSG introduced panoptic segmentation for graph nodes.
  • Video scene graphs: Video scene graph methods seek temporally grounded object–relation graphs, but bounding-box approaches struggle with fine-grained spatial details for non-rigid objects and amorphous regions.These limitations can produce incomplete or imprecise relational reasoning.

3 Synthetic Visual Genome 2

SVG2 is built through an automated pipeline for dense, temporally grounded video scene graphs, combining trajectory generation, semantic parsing, verification, and relation inference at large scale.

  • Design challenges: Video scene graph construction requires consistent tracking through object emergence and disappearance and precise temporal localization of spatial and temporal interactions.These requirements distinguish video scene graphs from static image scene graphs.
  • Automatic pipeline: The synthesis pipeline integrates SAM2, DAM, GPT-5, and a two-stage online–offline tracker to produce dense, temporally grounded, semantically rich graphs.SAM2 supplies multi-scale panoptic masks, while tracking addresses new-object discovery and temporal consistency.
  • Automatic pipeline: Phase 2 generates trajectory descriptions, extracts object names and attributes, and uses SAM3 trajectory matching with spatiotemporal IoU to verify labels.Original masks are retained, while objects unsupported by matched SAM3 trajectories are discarded.
  • Dataset construction: 636K automatically annotated videos form SVG2, supplemented by SVG2test, a 100-video expert-annotated diagnostic benchmark.The dataset combines 43K videos from SA-V with 593K videos from PVD.
  • Dataset comparison: Table 1 compares related benchmarks by annotation density, category counts, and object, attribute, and relation statistics, distinguishing sparse from dense annotations.Its caption notes that AG reports frame-level relation instances, whereas other datasets report unique trajectory-level instances.
  • Dataset statistics: 5.89M object instances, 46.95M attributes, and 5.99M spatiotemporal relations come from the fully automatic PVD subset.The SA-V subset contributes 0.66M objects, 5.01M attributes, and 0.72M relations after hybrid tracking refinement.

4 TraSeR

TraSeR converts raw videos and panoptic trajectories into structured video scene graphs by aligning visual tokens with object trajectories and compressing them through complementary global and temporal resamplers.

  • Trajectory-Aligned Token Arrangement: TraSeR grounds ViT tokens to instance trajectories, preserving object identities before decoding a structured video scene graph.Trajectory-aligned token streams are separated by object and ordered temporally.
  • Trajectory-Aligned Token Arrangement: Coverage scores assign visual tokens to objects when segmentation overlap exceeds the effective threshold τ_eff.Coverage averages mask values over each token’s spatial footprint and applies a temporal maximum.
  • Dual Resampler Module: The object-trajectory resampler aggregates each object’s full trajectory into compact global semantic representations.This reduces variable-length token sequences before language-model decoding.
  • Dual Resampler Module: The temporal-window resampler independently summarizes object tokens within time windows, retaining local motion and temporal cues.Absent objects produce no window summary, while timestamp embeddings expose temporal positions.
  • Evaluation: Table 2 evaluates four VidSGG benchmarks using triplet recall, relation recall, object accuracy, and attribute recall under a lenient semantic criterion.Relations and triplets use a temporal IoU threshold of 0.5.
  • Dual Resampler Module: The two resamplers provide complementary object-centric and time-aware representations suited to structured language-model decoding.Training combines SVG2 supervision with video segmentation and relation datasets unified through task-specific serialization templates.

5 Experiment

The experiments evaluate open-vocabulary video scene graph generation with semantic and strict metrics across multiple benchmarks, then test how training data, architecture, and scene graphs affect performance. TraSeR achieves substantial gains in scene graph prediction and improves downstream video question answering.

  • Evaluation Setup: The evaluation covers object, attribute, relation, and triplet prediction across PVSG, VidOR, VIPSeg, and SVG2test.SVG2test uniquely provides complete object–attribute–relation annotations, while other benchmarks cover narrower task combinations.
  • Evaluation Setup: The two-tiered metric combines exact string matching with LLM-assisted semantic alignment for open-vocabulary predictions.Relations additionally require temporal IoU above a threshold, and triplets must satisfy all subject, object, relation, and temporal criteria.
  • Scene Graph Generation Results: Approximately +15% triplet detection, +15% relation detection, +35% object prediction, and +15% attribute recognition improvements are reported over open-source baselines.These results use the lenient semantic criterion with temporal IoU threshold 0.5; TraSeR also surpasses GPT-5 on object and attribute prediction.
  • Video Question Answering: +0.4% on AGQA 2.0 and +4.6% on Perception Test are obtained when GPT-4.1 receives TraSeR’s scene graphs with video.In blind-video AGQA evaluation, TraSeR’s text-only graphs reach 13.22% accuracy, 6.5 percentage points above Qwen2.5-VL’s graphs.
  • Ablation Study: The full training configuration performs best, while academic-only training overfits and generalizes poorly to PVSG and SVG2test.Adding SVG2 data supplies dense, temporally aligned annotations, diverse categories, and visual variability.
  • Ablation Study: Combining both resamplers yields the best balance of object-level and temporal alignment, whereas bounding-box supervision degrades performance on dynamic PVSG videos.The temporal-window resampler is essential for dynamic relation localization, and segmentation trajectories provide more stable grounding than coarse boxes.

6 Limitations and Discussion

SVG2 inherits limitations from the automated models used to synthesize it, while TraSeR currently assumes trajectory-grounded inputs rather than proposing and tracking objects end to end.

  • Dataset and Model Boundaries: SVG2 inherits the current boundaries of the automated models used to synthesize its annotations.The authors contrast this constraint with the scale, density, and consistency that manual annotation cannot practically provide.
  • Dataset and Model Boundaries: TraSeR is currently trajectory-grounded and does not yet provide a unified end-to-end framework that natively proposes and tracks objects.Extending the approach in this direction is identified as future work.

7 Conclusion

The paper presents SVG2 as a large-scale synthetic video scene graph resource and TraSeR as a VLM-based model that generates complete graphs in one forward pass.

  • Contributions: SVG2 contains over 636K videos with dense object, attribute, and relation annotations, representing an order-of-magnitude expansion over prior resources.The dataset is built entirely through an automated pipeline.
  • Contributions: TraSeR augments a VLM with object-trajectory and temporal-window resamplers to produce complete video scene graphs in a single forward pass.The model operates on raw videos and panoptic trajectories.

A.1 Panoptic Trajectory Generation

SVG2 generates dense, temporally consistent panoptic trajectories through multi-scale mask proposals, coverage filtering, and online–offline tracking that discovers new objects while preserving identities.

  • Proposal generation: Multi-scale SAM2 prompts produce dense per-frame panoptic segmentation proposals for broad spatial coverage.The generator uses 32×32, 16×16, and 4×4 point grids to capture large regions and fine details.
  • Proposal generation: Coverage-optimized filtering removes redundant masks while preserving full proposal coverage, reducing memory use and improving downstream tracking stability.Masks are sorted by area and retained when overlap with selected masks remains below 90%.
  • Online–offline tracking: Online–offline tracking detects newly appearing objects, records their entry frames and masks, and propagates identities across the full sequence.Online tracking monitors breakpoints, while offline propagation restarts objects at their true entry frames and continues through all frames.
  • Online–offline tracking: The pipeline uses overlap-based identity matching to continue existing trajectories or assign new IDs when overlap is insufficient.The asymmetric overlap measure is designed to handle partial occlusion more robustly than IoU.
  • Tracking evaluation: Online+offline propagation raises mask coverage from 0.435 to 0.486 on 100 sampled SVG2 videos.Offline propagation improves continuity after online tracking breakpoints.
  • Trajectory semantics: DAM generates localized trajectory descriptions that support fine-grained semantic parsing of small, partially occluded, or complex objects.DAM combines region-grounded descriptions with global scene context.

B.1 Data Statistics

SVG2 combines hundreds of thousands of videos with millions of object, attribute, and relation annotations, spanning diverse categories and temporally grounded scene structures.

  • Dataset scale: SVG2 contains approximately 6.6M instance labels and trajectories, 52M attributes, and 6.7M relationships across 636K videos.The videos are sourced from 43K SA-V samples and 593K PVD samples.
  • Objects and attributes: Object categories span persons, vehicles, animals, furniture, and everyday objects.Figure 4 presents the distribution across these semantic classes.
  • Annotation structure: Visualization examples combine panoptic trajectories, DAM-derived object semantics, and GPT-5-derived spatial and non-spatial relations.Figures 8–10 illustrate the resulting structured annotations.
  • Objects and attributes: Attributes capture fine-grained visual properties including color, material, and state.Figure 5 summarizes the attribute distribution.
  • SVG2test: SVG2test uses expert annotators and model-in-the-loop review to produce dense segmentation masks and scene graph annotations.GPT-5 proposes attributes and relationships, which expert crowd workers review and edit.
  • Relations: Relations cover spatial, motion, functional, stateful, social, attentional, and event-level categories.Figures 6 and 7 show category-level and predicate-level distributions.

D Training Details

Training uses trajectory-aligned token organization and dual resampling, with standardized video inputs, evaluation metadata, and structured output formatting.

  • Optimization setup: The resamplers use three layers and 32 learnable queries, with τeff set to 0.5.RMSNorm is applied before and after cross-attention blocks.
  • Optimization setup: Training uses 1-fps videos, 4-second temporal windows, frozen ViT features, and jointly trained projectors, language models, and resamplers.The object-trajectory resampler processes each object trajectory in one pass.
  • Token arrangement: Trajectory tokens are delimited by <obj_traj_start> and <obj_traj_end>, include explicit object identities, and retain Qwen2.5-VL separator tokens.This arrangement makes individual object trajectories explicit in the input sequence.
  • Evaluation protocol: Evaluation gives every model identical object bounding-box trajectories, coordinates, timestamps, and structured prompting conditions.Outputs are constrained by a predefined JSON schema.

E.2 Comparison under different evaluation criteria

TraSeR remains strong under relaxed temporal grounding and exact-match object evaluation, while larger training sets consistently improve performance across benchmarks.

  • Baseline comparison: Proprietary models outperform open-weight VLMs particularly on triplet and relation prediction, highlighting the difficulty of temporally grounded extraction.GPT-5 is the strongest API-access model in the reported comparison.
  • Evaluation criteria: Under relaxed temporal IoU 0.1, TraSeR surpasses all open-weight baselines by large margins on triplet and relation performance.The comparison uses relaxed temporal grounding for relations and triplets.
  • Evaluation criteria: TraSeR achieves the highest object-level performance across datasets under the strict exact-string-match score, often exceeding proprietary API models.This strict score requires exact matches between predicted and ground-truth object labels.
  • Architecture ablation: Three resampler layers with 32 temporal-window queries provide slightly better overall results and inference stability than tested alternatives.Architectural differences across variants are relatively small.
  • Data scale ablation: Increasing training data consistently improves average performance across most metrics and benchmarks.Triplet and relation gains are clear on VidOR and SVG2test as data scale increases.
  • Data scale ablation: SVG2test attribute performance rises from 22.3 to 27.1 when training uses the full dataset.The ablation identifies the full 593K-video PVD subset as providing diverse object-level annotations.

E.4 Trajectory Robustness and Temporal Scalability

TraSeR is evaluated under realistic trajectory noise and on videos far longer than its training clips. It remains robust to predicted segmentation, while longer videos mainly expose weaknesses in fine-grained temporal grounding.

  • Trajectory Robustness: IoU ≥0.3 predicted masks are used to test TraSeR’s sensitivity to upstream trajectory quality.The predicted masks come from the automated Phase-1 tracking pipeline and replace ground-truth inputs.
  • Trajectory Robustness: Even with noisy predicted masks, TraSeR outperforms GPT-4.1, Gemini-2.5 Pro, and open-source models across triplet, relation, object, and attribute metrics.The comparison uses models with perfect ground-truth inputs for GPT-4.1 and Gemini-2.5 Pro.
  • Temporal Scalability: TraSeR processes videos up to 166 seconds directly without truncation, despite training videos averaging approximately 15 seconds.The temporal generalization evaluation covers VidOR at 48 seconds, Epic-Kitchens at 120 seconds, and Ego4D at 166 seconds.
  • Temporal Scalability: Object prediction remains highly robust across video lengths, whereas relation and triplet performance declines on longer videos.The decline reflects the difficulty of extrapolating fine-grained temporal grounding far beyond the training distribution.
Loading 2602.23543v3…