PaperlayerView exact revision on arXiv

Source-linked AI summary

TempoGround: State-Aware Streaming Visual Grounding with Vision-Language Models

Leqian Ding, Junning Qiu, Manwen Yang, Yu Guo, Fei Wang

arXiv:2609.02359v1cs.CV

TL;DR

Streaming visual grounding must maintain object identity and localization as images arrive causally, despite cross-frame inconsistency and partial occlusion. TempoGround addresses this gap with state-aware correspondence, curriculum prediction, and Streaming Grounding Reinforcement, improving 2D and 3D grounding across challenging benchmarks.

  • Problem

    Streaming visual grounding must continuously track language-matched objects and produce 2D and camera-frame 3D boxes despite identity drift, cross-frame inconsistency, and partial-occlusion failures.

  • Method

    TempoGround uses state-aware cross-frame correspondence and curriculum prediction from 2D association and presence estimation to camera-frame 3D grounding, optimized with Grounding, Identity, and Consistency rewards.

  • Results

    TempoGround improves F1_2D@0.5 and F1_2D@0.95 by 4.4 and 0.5 on average, and F1_3D@0.25 and AP_3D by 6.2 and 7.5 across challenging causal-streaming benchmarks.

  • Takeaways & Limitations

    TempoGround provides a practical foundation for accurate and consistent visual grounding under causally arriving image streams.

  • Takeaways & Limitations

    At timestep t, predictions may depend only on the query, observations I1:t, and previous predictions Y1:t−1.

Abstract

from arXiv · show

Visual grounding maps language referents to spatial targets and is central to open-vocabulary perception with vision-language models. Existing methods have made substantial progress on single-frame and video-based visual grounding, yet under streaming inputs they still suffer from identity drift, cross-frame inconsistency, and fragile localization under partial occlusion. To address these issues, we present TempoGround, a VLM-native framework that detects cross-frame object correspondence and explicitly models object presence states, thereby enabling accurate and consistent visual grounding under streaming inputs. The key is a curriculum prediction mechanism guided by state-aware cross-frame correspondence: TempoGround resolves 2D instance association, predicts whether each object newly enters, continues in, or leaves the view, decodes the 2D box, and then lifts it to a camera-frame 3D box. As token-level supervision alone cannot capture the geometric objectives of streaming grounding, we further introduce Streaming Grounding Reinforcement (SGR), which optimizes TempoGround with verifiable Grounding, Identity, and Consistency rewards, jointly reinforcing persistent localization and temporally consistent predictions. We carefully design a three-stage training strategy and train TempoGround on large-scale data. We evaluate visual grounding under causally streaming inputs on multiple challenging benchmarks: TempoGround improves F1_2D@0.5 and F1_2D@0.95 by 4.4 and 0.5 on average, and F1_3D@0.25 and AP_3D by 6.2 and 7.5, respectively. These results demonstrate that TempoGround provides a practical foundation for visual grounding under streaming inputs.

1 Introduction

Streaming visual grounding requires causal, temporally consistent localization of language-referred objects in 2D and camera-frame 3D. TempoGround addresses identity drift, unstable boxes, and occlusion-related failures with state-aware correspondence, curriculum prediction, reinforcement, and staged training.

  • Motivation: Streaming visual grounding continuously tracks language-matched objects from incrementally arriving images and outputs 2D and camera-frame 3D boxes.The setting is motivated by embodied perception and downstream control.
  • Motivation: Identity drift, cross-frame inconsistency, and fragile localization under partial occlusion are characteristic streaming failure modes.Intermittent visibility can cause missed or degraded detections that causal cross-frame reasoning may correct.
  • Approach: TempoGround uses state-aware cross-frame correspondence to associate instances, estimate entry, persistence, or exit states, and progress from 2D association to camera-frame 3D grounding.The method matches a lightweight cue from the previous prediction against corresponding regions in the current image before estimating presence and localization.
  • Training: Streaming Grounding Reinforcement optimizes geometric streaming grounding with verifiable Grounding, Identity, and Consistency rewards beyond token-level imitation.The framework is built on a standard VLM decoder and autoregressive outputs, but supervised fine-tuning alone does not directly provide box-level streaming supervision.
  • Training: A three-stage recipe progresses from single-frame geometric competence to streaming supervision and sequence-level reinforcement using 26M and 5.1M supervised samples in its first two stages.Stage 1 strengthens 2D detection and 2D-to-3D lifting; Stage 2 establishes curriculum prediction.
  • Results: F1_2D@0.5 and F1_2D@0.95 improve by 4.4 and 0.5 on average, while F1_3D@0.25 and AP_3D improve by 6.2 and 7.5 under causal streaming inputs.The experiments span multiple challenging benchmarks.

2 Related Works

Related work advances spatial understanding with VLMs, reconstructed geometry, multi-view video, and generative grounding. TempoGround instead focuses on object-level localization in the current camera frame from causally arriving image streams.

  • 3D scene understanding: Prior 3D scene-understanding methods encode reconstructed geometry or multi-view video for relational, layout-level, embodied, and spatial reasoning.Their supervision and inference remain largely scan- or clip-centric.
  • 3D scene understanding: Many prior object predictions use a world or scene coordinate system, whereas TempoGround targets localization in the current camera frame under causal streams.World-frame predictions can require an additional alignment step for ego-centric viewpoints.
  • Visual grounding: Visual grounding research spans open-vocabulary detectors, autoregressive multimodal LLM boxes, pixel-aligned grounding, generative detection, and staged camera-frame 2D-to-3D prediction.These approaches strengthen the generative formulation of visual object grounding.

3 Method

TempoGround performs causal streaming visual grounding by combining state-aware cross-frame correspondence with a curriculum from 2D localization to camera-frame 3D lifting. Streaming Grounding Reinforcement and progressive training align the model with geometric, identity, and consistency objectives.

  • Problem formulation: At each timestep, TempoGround causally predicts a possibly empty set of query-matching instances with 2D and camera-frame 3D boxes.Predictions depend only on the query, observations through the current timestep, and previous predictions.
  • Curriculum prediction: The decoder follows the curriculum 2D correspondence → presence estimation → 2D box decoding → camera-frame 3D lifting.This structured autoregressive order is used during inference and training.
  • Cross-frame correspondence: Correspondence uses labels and previous 2D boxes from adjacent frames while discarding previous 3D boxes to avoid stale viewpoint-dependent geometry.The cue is matched against corresponding regions in the current image, after which 3D is re-estimated in the current camera frame.
  • Presence state estimation: Presence states classify associated objects as new, track, or lost, indicating entry, persistence, or exit from the field of view.Objects that re-enter after being lost are treated as new under contiguous visibility.
  • Streaming Grounding Reinforcement: SGR optimizes streamed rollouts with a deterministic verifier and rewards for grounding, identity, and consistency.The rewards encourage persistent boxes, correct enter-and-exit behavior, and resistance to failure streaks propagated through self-generated cues.
  • Progressive training: Training progresses from about 26M single-frame samples to about 5.1M streaming samples and finally SGR refinement on streamed trajectories.The stages establish geometric competence, state-aware correspondence, and sequence-level objectives respectively.

4 Experiments

TempoGround is evaluated under causal streaming protocols for 2D and 3D grounding, with ablations examining curriculum components, SGR rewards, and robustness to corrupted correspondence cues.

  • Evaluation setup: All methods are evaluated under the same causal streaming protocol, using F1_2D@0.5/@0.95 and F1_3D@0.25/AP_3D.Single-frame methods process frames independently, while video methods use the clip observed so far and report the current-frame prediction.
  • Implementation details: Stage 1 uses 15.6M 2D and 10.5M camera-frame 2D+3D samples, while Stage 2 uses 5.1M samples and Stage 3 draws a subset.Stage 3 training uses GRPO on approximately 4.1K streaming trajectories.
  • Main results: TempoGround consistently surpasses general-purpose VLMs and specialist grounders on both 3D and 2D causal-streaming results.The 2D gains indicate benefits beyond camera-frame 3D lifting.
  • Ablation studies: Both cross-frame correspondence and presence supervision improve AP3D in curriculum-prediction ablations.Using camera-frame 3D boxes as the prior underperforms the 2D-only cue because previous-view geometry can bias current-frame localization.
  • Ablation studies: Removing Grounding, Identity, or Consistency rewards lowers accuracy, indicating that all three SGR rewards contribute positively.The ablations use the same backbone, data budget, and causal-streaming evaluation protocol.
  • Robustness analysis: Mixed rollout and synthetic cue perturbations achieve the best AP3D accuracy across the full corruption-intensity range.Rollouts model realistic closed-loop residuals, while synthetic perturbations cover rarer failure modes.

5 Conclusion

TempoGround combines state-aware cross-frame correspondence with curriculum prediction and SGR for visual grounding under causally arriving image streams. Experiments support its use as a practical foundation for streaming visual grounding.

  • Conclusion: TempoGround grounds objects in causal image streams by modeling cross-frame correspondence and object presence states.Its curriculum proceeds from 2D association and presence estimation to camera-frame 3D grounding.
  • Conclusion: Streaming Grounding Reinforcement optimizes predictions with verifiable Grounding, Identity, and Consistency rewards beyond token-level imitation.These rewards target streaming grounding behavior directly.
  • Conclusion: TempoGround provides a practical foundation for streaming visual grounding.The conclusion presents this as the outcome of extensive experiments.

A Implementation Details

TempoGround serializes each visible instance with a reference, presence state, 2D box, and camera-frame 3D box, while exited or absent objects receive special outputs.

  • Output schema: TempoGround uses special tokens to delimit reference, state, 2D bounding-box, and 3D bounding-box fields.The inventory includes <ref>, <state>, <2d_bbox>, and <3d_bbox>.
  • Visible instances: Visible instances with state new or track are serialized with their label, state, 2D coordinates, and camera-frame 3D box.2D coordinates are integers in [0, 1000].
  • Exited instances: Exited instances are serialized as a reference and lost state without boxes.A previously exited instance therefore has no serialized spatial boxes.
  • Special cases: When no query-matching object is visible, the model emits <null>; multiple instances use a canonical concatenation order.Inference disables thinking-mode generation and serves the model with vLLM.

A.2 Evaluation Protocol

Evaluation scores visible instances under causal streaming using 2D-IoU matching shared by 2D and 3D F1, while AP3D pools scored-frame predictions at multiple 3D-IoU thresholds.

  • Metrics: The protocol reports F1_2D@0.5, F1_2D@0.95, F1_3D@0.25, and AP3D, averaging reported numbers equally across datasets.The previous-frame correspondence cue comes from the model’s own preceding prediction.
  • Scored instances: Instances marked lost are removed from ground truth and predictions before scoring, and frames without visible ground-truth boxes are excluded.This defines the scored set for F1 and AP3D.
  • Matching: Predictions and ground-truth boxes are greedily matched by descending axis-aligned 2D IoU using one-to-one assignments.The same assignments support both 2D and 3D F1, after which oriented 3D IoU is evaluated.
  • F1 score: Per-frame F1 uses matched pairs above the relevant overlap threshold, with zero assigned when the precision-recall denominator is zero.Dataset scores average F1 over scored frames before the unweighted cross-dataset mean.
  • Average precision: AP3D pools predictions and ground-truth boxes across scored frames at oriented 3D-IoU thresholds 0.15, 0.25, and 0.50.AP uses Pascal VOC 11-point interpolation; detection averages over categories, while referring uses one target.

A.3 Runtime

TempoGround performs continuous object grounding with sub-second average single-image inference on one NVIDIA H200 GPU.

  • 0.65 s and 0.45 s are the average single-image inference latencies for the 9B and 4B models, respectively.Each value is averaged over 10 runs and covers vision encoding, prefilling, and autoregressive grounding-output generation.
  • The reported latency covers the full forward path for one observation.
  • Table A.1 reports average inference latency on one NVIDIA H200 GPU over 10 runs.

B Streaming Grounding Reinforcement

Streaming Grounding Reinforcement defines verifiable trajectory-level training rewards for geometric grounding quality and identity-aware behavior.

  • SGR computes reward terms from closed-loop trajectories rather than token-level outputs alone.The appendix describes the computation of the Stage 3 reward terms introduced in the main paper.
  • Shared geometric quality is computed after pairing a visible ground-truth instance with a prediction.
  • Unmatched instances receive q=0, and this scalar is reused by Rg and Rid.

B.1 Grounding Reward Rg

The reward design separates trajectory localization credit, event-level identity correctness, and consistency penalties for persistent streaming failures.

  • Grounding Reward Rg: Rg separates localization credit from false-positive cost instead of averaging all per-frame quality values globally.This prevents long tracks from dominating short ones and penalizes unmatched predictions.
  • Grounding Reward Rg: Each ground-truth track receives equal weight through the mean quality over frames where its presence is new or track.Frames labeled lost are excluded because no box is required there.
  • Identity Reward Rid: Rid scores per-instance presence events, rewarding matched new states, accepting matched track states, and penalizing misses or incorrect states.Lost-state events are handled separately from visible-state events.
  • Consistency Reward Rcon: Rcon penalizes long contiguous chains of frame failures and phantom-after-exit predictions under self-conditioned cue feedback.It avoids treating ordinary box changes under ego-motion as inconsistency.
  • Trajectory Objective: Stage 3 mixes Rg, Rid, and Rcon with weights 0.70, 0.15, and 0.15, then subtracts 0.02 ffmt.The format term penalizes empty outputs when the ground truth is nonempty.

C Data Curation Details

The data-curation pipeline constructs streaming examples using geometric visibility gates, explicit state transitions, query uniqueness checks, and perturbed correspondence cues.

  • C Data Curation Details: Streaming data construction uses geometric gates, transition rules, and sequence-acceptance criteria.
  • Visibility and Presence: An instance is in view when its camera-frame depth is positive, its clipped area is at least 64 pixels, and clipped completeness is at least 0.05.
  • Visibility and Presence: Positive localization supervision applies stricter present criteria, including clipped completeness of at least 0.30 and clipped image area of at least 2 × 10^-4.
  • State Transitions: State labels assign new at first supervision or re-entry, track during continued visibility, and lost on the one-frame transition out of view.
  • Sequence Acceptance: Accepted windows must contain an enter event and either an exit event or a leading empty prefix, with training lengths of 24–64 and test lengths of 32–64.
  • Annotation Sources: Referring supervision uses object indices from public corpora while pixels and camera-frame boxes come from underlying RGB-D scans.
  • Query Curation: Candidate expressions undergo candidate-index and contrastive uniqueness checks when same-category instances are co-visible.
  • Referring Tracks: Referring tracks require linguistic dependencies to be co-visible at entry and re-entry, while geometrically visible but undecidable frames remain empty.

D Additional Qualitative Results

Additional qualitative visualizations illustrate TempoGround’s streaming detection and referring behavior, including tracking across frames and distinguishing queried object instances.

  • Streaming Detection: Figures D.1 and D.2 provide qualitative visualizations of streaming detection.The examples cover persistent tracking and detection of multiple instances.
  • Referring: Figures D.3 and D.4 provide qualitative visualizations of referring.They show tracking of an entering object and interpretation of a queried object category.
  • Streaming Detection: TempoGround maintains correct 2D and 3D tracking while a bed remains in view.
  • Streaming Detection: TempoGround accurately detects both large and small trash-can instances in the scene.
  • Referring: When the frame above the sofa enters view, TempoGround tracks it accurately in 2D and 3D.
  • Referring: For the query “loveseat,” TempoGround identifies the two-seat couch while avoiding single-seat and three-seat detections.
Loading 2609.02359v1…