Click2Graph: Interactive Panoptic Video Scene Graphs from a Single Click

1. Introduction

• Existing video SGG and PVSG pipelines do not let users intervene, while promptable segmentation locates objects but does not identify them or infer their interactions.This limits user control when models overlook objects, misclassify interactions, or drift during tracking.Source: §1 · p. 1
• A single visual cue prompts subject tracking across time, discovery and segmentation of interacting objects, and prediction of ⟨subject, object, predicate⟩ relationships.The resulting scene graph is temporally consistent.Source: §1 · p. 1
• Prompting different entities can yield distinct scene graphs, making the system controllable through the user's choice of subject.The examples show a dog prompt associated with carpet and sitting, and a child prompt associated with dog and playing.Source: §1 · p. 2, Figure 1 · p. 2
• The Dynamic Interaction Discovery Module generates subject-conditioned object prompts, while the Semantic Classification Head jointly infers subject, object, and predicate labels.Together, these modules add semantic and relational reasoning to promptable segmentation.Source: §1 · p. 2
• OpenPVSG experiments show strong spatial grounding and object discovery, while fine-grained semantic classification remains the primary bottleneck.Most errors concern visually similar object categories or predicates.Source: §1 · pp. 1–2

2.1. Advances in Video Scene Graph Generation

• Video SGG methods advance temporal reasoning and predicate modeling, but remain closed systems in which users cannot intervene after detection or classification errors.Prior work includes transformer-based temporal approaches and methods addressing long-tail predicates.Source: §2.1 · p. 2

2.2. Panoptic-Level Scene Understanding

• PVSG extends panoptic scene graphs to temporally consistent video annotations, while Click2Graph guides interaction-centric graph construction from a user-specified subject.The paper positions its framework as unifying temporal reasoning, panoptic precision, and visual prompting.Source: §2.2 · pp. 2–3, Table 1 · p. 3

2.3. Promptable and Interactive Scene Analysis

• Interactive scene analysis includes text-prompted and visually guided approaches, but visually guided interaction for scene graph generation remains underexplored.Existing interactive video object segmentation tracks a prompted object without discovering interactions or reasoning about semantic relationships.Source: §2.3 · p. 3, §2.3.2 · p. 3

2.3.1. Text-Prompted Generation

• Text-prompted SGG methods use language guidance for predicate detection or semantic priors, but text does not uniquely ground pixel-level subjects.These systems also depend on language availability and may not generalize across settings.Source: §2.3.1 · p. 3

2.3.2. Visually-Guided Interaction

• Click2Graph uses direct visual prompts, which the paper describes as universal, unambiguous, and spatially precise.It is presented as the first framework to use direct visual prompts for end-to-end PVSG, including object discovery, segmentation, and predicate prediction.Source: §2.3.1 · p. 3, §2.3.2 · p. 3

2.4. Foundation Models for Segmentation

• SAM2 provides promptable video segmentation and mask propagation, but lacks category, interaction, and interacting-entity discovery; Click2Graph adds modules for scene graph outputs.The Dynamic Interaction Discovery Module and Semantic Classification Head transform geometric outputs into pixel-accurate, temporally consistent scene graphs.Source: §2.4 · p. 3

3.1. Problem Formulation

• Each interaction tracklet links a subject, an interacting object, their relationship, and panoptic masks over the active temporal window.The formulation supports multiple subjects and allows users to introduce new prompts at any time.Source: §3.1 · p. 3

3.2. Network Architecture

• SAM2 provides prompted, temporally consistent masks, while DIDM discovers objects and SCH predicts subject, object, and predicate labels.The architecture uses the two modules to extend a class-agnostic segmentation backbone into a scene graph generator.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3. Dynamic Interaction Discovery Module

• DIDM combines a subject-mask feature with learnable object queries, then uses Transformer cross-attention to predict object prompt locations.The module uses Nq = 3 queries, whose predicted points prompt SAM2 to segment candidate interacting objects.Source: §3.3 · p. 4

3.4. Semantic Classification Head

• SCH aggregates SAM2 vision features over predicted masks to classify entities, then predicts predicates from joint subject–object features.Separate MLPs predict subject and object classes and the relationship predicate.Source: §3.4 · p. 4

3.5. Training Objective

• The total objective combines mask, point-localization, subject, object, and relation losses, with set-based Hungarian matching aligning predictions to ground-truth interactions using DETR [2].The stated total loss is Lmask + LL2 + Lsub + Lobj + Lrel.Source: §3.5 · pp. 4–5
• Mask supervision combines binary cross-entropy, IoU, and Dice losses; DIDM points are supervised against ground-truth object interior points, and semantic predictions use cross-entropy.Source: §3.5 · p. 5

3.6. Training and Inference Details

• Training freezes the 224M-parameter SAM2.1-Large backbone and adds about 5M trainable DIDM and SCH parameters; inference runs at about 10 FPS on an A100.Training uses 400 epochs with 8-frame clips; the reported inference memory footprint is about 7GB at 1024x1024 resolution.Source: §3.6 · p. 5

3.7. Ground-Truth Point Sampling Strategy

• Ground-truth object points are sampled with probabilities proportional to distance from mask boundaries, favoring stable interior targets for DIDM supervision.The passage states that this distance-weighted strategy improves object discovery accuracy.Source: §3.7 · p. 5

Scale and Composition

• OpenPVSG contains 400 videos totaling approximately 150k frames at 5 FPS, drawn from VidOR, EPIC-Kitchens, and Ego4D.The dataset includes third-person and egocentric perspectives, and subjects typically interact with no more than two objects per frame.Source: §4 · p. 6

Annotations

• Its annotations cover 126 object categories, 57 relationship predicates, pixel-accurate panoptic masks, and temporally consistent relation trajectories.The masks ground both thing and stuff classes.Source: §4 · p. 6

Relevance to Click2Graph

• The benchmark spans camera motion, occlusion, varied environments, and fine-grained semantic classes, testing prompting, segmentation, and relation prediction together.The paper motivates OpenPVSG as a testbed for evaluating those capabilities in a unified framework.Source: §4 · p. 6

• Standard SGG metrics such as PREDCLS, SGCLS, and SGDET are described as inappropriate for evaluating Click2Graph’s integrated pipeline.Source: §5 · p. 6
• The evaluation uses three complementary recall-based metrics to characterize Click2Graph’s performance.Source: §5 · p. 6
• Together, the metrics address spatial precision, prompt generation reliability, and overall scene graph accuracy.Source: §5 · p. 6

• Recall@K counts a prediction as correct when subject, object, and predicate labels match and both masks reach IoU ≥ 0.5 with ground truth.Predictions are ranked by confidence, and only the top K are counted; the metric evaluates the full pipeline.Source: §1 · p. 6
• SpIR evaluates spatial grounding regardless of predicted class or predicate labels, measuring DIDM object prompts together with SAM2 mask propagation.Source: §1 · p. 6
• PLR measures whether DIDM’s predicted object prompt points fall inside the ground-truth object mask, independently of later segmentation or semantic prediction.Source: §1 · p. 6
• Each experiment is repeated 25 times with a unique initial point sampled from the subject’s ground-truth mask, and metrics are reported as mean ± standard deviation.Source: §1 · p. 6

End-to-End Performance

• Nq=3 versus approximately 100 proposals per frame, Click2Graph achieves competitive R@K while predicting only interactions associated with the prompted subject.The comparison is with automated PVSG approaches that generate full-frame proposals and detect subjects.Source: Table 2 · p. 7, §6 · p. 7

Robustness to Prompt Type

• Point, box, and mask prompts all yield stable results with low run-to-run variance, although masks score slightly higher.Point and box prompts are sampled with probability 0.49 each during training, while mask prompts are sampled with probability 0.02.Source: §6 · p. 7, Table 3 · p. 7

Contribution of System Components

• High PLR and strong SpIR indicate reliable object-prompt generation and panoptic masks, while R@K remains challenging because of fine-grained semantic confusions.Reported confusions include child versus baby, box versus bag, and floor versus ground.Source: §6 · p. 7
• Replacing subject-conditioned DIDM prompts with dataset-level object-prior sampling severely degrades PLR, SpIR, and R@K across datasets.The heuristic is not conditioned on the prompted subject.Source: Table 4 · p. 7, §6 · p. 7

Qualitative Analysis

• Qualitative examples show coherent interacting-object triplets and continued predictions through partial occlusion or momentary subject disappearance, with errors often involving predicate granularity or visually similar categories.Examples include confusion between “on” and “sitting,” and between “gift” and “box.”Source: §6 · p. 7, Figure 4 · p. 8
• Qualitative examples corroborate the reported pattern: segmentation and interaction discovery are reliable, while semantic classification is the primary bottleneck.Source: §6 · pp. 7–8

7. Conclusions and Future Work

• Click2Graph combines a user-specified subject with conditioned object-prompt generation and semantic reasoning to produce structured, interaction-centric scene graphs.The system transforms SAM2 into a PVSG pipeline capable of triplet prediction.Source: §7 · p. 8
• OpenPVSG experiments show strong spatial grounding and reliable object discovery, but fine-grained semantic classification remains challenging in a large, diverse label space.Most errors involve visually similar object categories or predicates, making semantic reasoning the primary bottleneck rather than segmentation or interaction discovery.Source: §7 · p. 8
• Real-time user intervention is limited to segmentation correction; users cannot modify predicted labels during inference, and corrections do not yet update the model.The proposed feedback mechanism would let label corrections update learnable class embeddings, adapting semantic predictions over time and maintaining consistency across future frames.Source: §7 · pp. 8–9
• Future directions include language models for predicate reasoning, multisubject prompting for complex interactions, and interactive supervision for long-tail predicate learning.The language-model direction is intended to reduce fine-grained semantic confusion.Source: §7 · p. 9