Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation

1 INTRODUCTION

• The video reference gap arises when language leaves unclear which observations support a target choice and when the queried event holds, as in confusing the hand-clasp target coach with contextual or later-event athletes.The example distinguishes the dyed-hair athlete, who provides identifying context, from the coach who is the hand-clasp target.Source: §1 · p. 1
• TCoVP interleaves chronological, timestamped entity primitives with language, using entity references, temporal spans, bounding boxes, and semantic descriptions to ground reasoning.Selected target primitives provide timestamped box prompts for mask propagation, with temporal support restricting output when the query specifies an action interval.Source: §1 · pp. 1–2
• With Qwen2.5-VL-3B and data volume fixed, TCoVP raises five-benchmark macro-average J &F from 57.8 to 62.7 over text-only CoT with the same answer-primitive format.The introduction reports evaluation across five VRS benchmarks and multiple MLLM backbones, with consistent improvements especially on queries requiring explicit temporal event localization.Source: §1 · p. 2
• The authors construct TCoVP-SFT-15k and TCoVP-RL-13k and train with chain-aligned rewards for structural validity, temporal alignment, spatiotemporal grounding, and target primitive accuracy.Evaluation spans five VRS benchmarks and Qwen2.5-VL, Qwen3-VL, and Molmo2 backbones.Source: §1 · p. 2

2 RELATED WORK

• TCoVP addresses a complementary interface problem by making relevant entities and events explicit in a single-turn, temporally ordered primitive chain before dense mask propagation.Earlier VRS approaches include segmentation tokens or mask decoders, while RL-based methods optimize intermediate reasoning, keyframe selection, or spatial grounding.Source: §2 · p. 3
• Unlike video reasoning methods that generate timestamped evidence, bounding-box CoT, or selected events, TCoVP organizes identity, temporal order, and target primitives for implicit-query segmentation.The selected target primitives are exposed to a video mask propagator.Source: §2 · p. 3

3.1 TEMPORAL PRIMITIVE CHAIN REPRESENTATION

• Each primitive links an entity identity to semantic state, timestamp, bounding box, temporal support, and an optional action or relation, while chronological ordering records evolving evidence.Reusing entity identifiers across timestamps preserves entity continuity.Source: §3.1 · p. 3

3.2 DATA CONSTRUCTION

• TCoVP-SFT-15k contains complete serialized reasoning trajectories, while TCoVP-RL-13k retains examples with reward-compatible primitive annotations.Instances include target and contextual boxes, temporal supports, and a text-only counterpart for controlled comparison.Source: §3.2 · p. 4
• Chronological-chain annotations are generated from sampled videos, with entity identifiers reused across observations and primitives interleaved with language.Consistency checks validate identifiers, timestamp order, temporal support, and boxes; unresolved references or targets are filtered out.Source: §3.2 · p. 5

3.3.1 COLD START INITIALIZATION VIA SFT

• Supervised fine-tuning teaches canonical chain serialization, emitting primitives only for relevant observations, chronological ordering, and identifier reuse.The SFT stage also provides a text-only counterpart for the representation ablation.Source: §3.3.1 · pp. 5–6

3.3.2 CHAIN-ALIGNED REINFORCEMENT LEARNING

• Chain-aligned reinforcement learning rewards structural validity, temporal alignment, spatiotemporal grounding, and target-primitive accuracy, then compares candidate chains within each query group while regularizing against a reference policy.Temporal precision and recall penalize irrelevant and missing evidence; target matching penalizes hallucinated and missed targets.Source: §3.3.2 · pp. 6–7

3.4 PRIMITIVE-TO-MASK EXECUTION

• At inference, selected target primitives supply timestamped box prompts for SAM3 mask propagation, and multiple initialized masks are merged by pixel-wise union.GroundMoRe output masks are restricted to the predicted query-relevant action interval; general VRS masks span the full video.Source: §3.1 · p. 4, §3.4 · p. 7

4.1 IMPLEMENTATION DETAILS

• The three backbones use 16 uniformly sampled frames, one SFT epoch, two reinforcement-learning epochs, and eight sampled responses per query.Target primitives in the final answer provide timestamped bounding-box prompts to SAM3 for video-mask propagation.Source: §4.1 · p. 8

4.2 EVALUATION DATASETS AND METRICS

• Evaluation uses Long-RVOS, ReasonVOS, ReVOS, MeViS, and GroundMoRe, reporting region similarity J, contour accuracy F, and their average J &F.GroundMoRe additionally tests identifying the relevant action interval rather than only the longer interval when an entity is visible.Source: §4.2 · p. 8

4.3 MAIN RESULTS

• 55.37 J &F is TCoVP-M-4B’s highest overall score on GroundMoRe, versus 31.96 for the strongest listed prior VRS baseline.TCoVP-4B scores 55.21 and TCoVP-3B scores 53.67 on the benchmark.Source: §4.3 · p. 8
• Molmo2-4B leads Long-RVOS, ReVOS, and MeViS with 67.4, 68.3, and 65.2 J &F, while Qwen3-VL-4B leads ReasonVOS at 72.1.All three variants beat the strongest listed prior baseline by 7.7–10.0 points on Long-RVOS and 6.7–7.3 points on ReasonVOS.Source: §4.3 · p. 8

4.4 ABLATION STUDIES

• Ablations use Qwen2.5-VL-3B and examine the reasoning representation, reinforcement-learning stage, and temporal-alignment and spatiotemporal-grounding rewards.Source: §4.4 · p. 9
• 62.7 versus 57.8 is the five-benchmark macro average for TCoVP versus text-only CoT under the same data volume, training configuration, and backbone.TCoVP gains 7.4 points on GroundMoRe and 2.5–6.4 points on each other benchmark.Source: §4.4 · p. 9
• Adding chain-aligned reinforcement learning to SFT gains 8.8, 8.5, 5.4, and 9.9 points on Long-RVOS, ReasonVOS, ReVOS, and GroundMoRe, respectively.The largest improvement is on GroundMoRe.Source: §4.4 · p. 9
• Removing either temporal-alignment or spatiotemporal-grounding rewards lowers performance across all four benchmarks; using both yields the best performance.Source: §4.4 · p. 9

• Brief events may occur between the 16 uniformly sampled frames TCoVP uses and remain unobserved.Source: §5 · p. 9
• Cross-entity actions and relations are expressed in natural language rather than a dedicated structured representation.Source: §5 · p. 9
• Final masks inherit errors from SAM3 propagation; more complex scenarios may require adaptive sampling, denser spatial evidence, and structured interaction representations.Source: §5 · p. 9

6 CONCLUSION

• TCoVP links query-relevant observations chronologically with repeated entity identifiers, temporal support, and spatial grounding, then uses target primitives as timestamped prompts for SAM3 mask propagation.The authors report improvements across multiple backbones and benchmarks, particularly on motion-grounded queries.Source: §6 · p. 10
• Generative AI produced initial temporal-chain annotations and assisted with manuscript language editing; structural checks, visual verification, and author review followed.The authors state they take responsibility for the final content.Source: AI USE STATEMENT · p. 10
• Derived annotations may inherit errors and social biases from source datasets and annotation models, although the study introduces no new human-subject data collection.Any released derived data will retain source attribution and follow underlying dataset licenses and redistribution terms.Source: ETHICS STATEMENT · p. 10
• The paper directs readers to sections and appendices for training, evaluation, implementation, ablation, and oracle-analysis details.Appendix A.2 includes analyses of mask-propagation backends and intermediate spatial and temporal grounding.Source: REPRODUCIBILITY STATEMENT · p. 10

A APPENDIX

• The appendix adds implementation details on dataset construction, training, and inference.Source: A APPENDIX · p. 14
• Additional experiments examine mask-propagation backends, oracle analyses, intermediate spatial and temporal grounding, and conventional referring VOS performance.Source: A APPENDIX · p. 14
• The appendix also includes more visualizations and failure cases.Source: A APPENDIX · p. 14

A.1.1 DATA CONSTRUCTION

• The annotation prompt supplies sampled frames, video duration, question, target counts, annotated boxes, and an expected answer, then requests keyframes, entities, temporal intervals, and grounded and text-only reasoning traces.It requires structured JSON output with the completed answer, keyframes and boxes, temporal intervals, and both reasoning traces.Source: §A.1.1 · p. 14
• For zero-target samples, the prompt directs the model to state that no video object satisfies the query; for GroundMoRe, target intervals are restricted to the query-relevant action.Non-target intervals in GroundMoRe retain their definition as the periods when objects appear.Source: §A.1.1 · p. 14
• TCoVP-SFT-15k draws approximately 5.8k ReVOS, 4.8k MeViS, 2.6k Long-RVOS, and 1.8k GroundMoRe samples; TCoVP-RL-13k draws approximately 5.1k, 4.0k, 2.2k, and 1.5k, respectively.Source: §A.1.1 · p. 14, Figure 4 · p. 14
• Single-target instances comprise 82.8% of the data, compared with 14.7% multi-target and 2.5% zero-target instances.The stated composition retains multi-target and zero-target cases for relational reasoning and queries without a valid target.Source: §A.1.1 · p. 14

A.1.2 TRAINING SETTINGS

• Training first uses supervised fine-tuning to initialize canonical TCoVP serialization, then chain-aligned reinforcement learning to optimize structural validity, temporal alignment, grounding, and target accuracy.Source: §A.1.2 · p. 14
• The maximum image-pixel budgets are 128 × 28 × 28 for Qwen2.5-VL-3B and 128 × 32 × 32 for Qwen3-VL-4B and Molmo2-4B.The same budgets apply in both training stages.Source: §A.1.2 · p. 14
• Reinforcement learning samples eight completions per query, uses a temporal-alignment tolerance of σ = 1.0, and applies a temporal matching gate of τ = 1s.The KL coefficient is β = 0.04 for Qwen2.5-VL-3B and β = 0.1 for Qwen3-VL-4B and Molmo2-4B.Source: §A.1.2 · p. 15
• Target-accuracy rewards emphasize spatial matching outside GroundMoRe (wt = 1.0, ws = 2.0) and temporal matching for GroundMoRe (wt = 2.0, ws = 1.0).Source: §A.1.2 · p. 15
• For conventional VRS, prompts permit multiple target objects; GroundMoRe prompts require one target and limit its temporal span to the query-relevant action.GroundMoRe contextual objects retain full-video appearance spans, while conventional VRS prompts require consistent identities, appearance intervals, timestamps, and boxes.Source: Figure 6 · p. 15, §A.1.2 · p. 16

A.1.3 INFERENCE DETAILS

• Inference uniformly samples 16 frames and applies model-specific image-pixel constraints matching training.The constraints are 128 × 28 × 28 for Qwen2.5-VL-3B and 128 × 32 × 32 for Qwen3-VL-4B and Molmo2-4B.Source: §A.1.3 · p. 16
• The model generates a TCoVP reasoning trace and final answer, whose target primitives provide timestamps and bounding boxes for SAM3.Each timestamped box initializes a mask track on its corresponding video frame.Source: §A.1.3 · p. 16
• SAM3 propagates the initialized tracks through the video, and their masks are merged by pixel-wise union.Source: §A.1.3 · p. 16
• For GroundMoRe, propagation is restricted to the predicted query-relevant action interval; without a valid target primitive, the output is an empty mask trajectory.Source: §A.1.3 · p. 16

A.2.1 MASK PROPAGATION BACKENDS AND ORACLE ANALYSIS

• With SAM3, oracle boxes raise J &F to 76.4, 87.3, and 85.6 on Long-RVOS, ReasonVOS, and ReVOS; full-video oracle propagation performs below TCoVP-3B on GroundMoRe.Table 6 further shows that restricting propagation to the predicted timespan improves GroundMoRe J &F from 39.22 to 54.06 with predicted boxes and from 47.14 to 60.41 with oracle boxes under SAM2; oracle timespans yield larger gains than oracle boxes.Source: Table 5 · p. 16, §A.2.1 · pp. 16–17, Table 6 · p. 17

A.2.2 INTERMEDIATE SPATIAL GROUNDING QUALITY

• Chain-aligned reinforcement learning improves think- and answer-box IoU over SFT-only training across all four benchmarks, with GroundMoRe answer-box IoU rising from 52.6 to 59.4.Removing either temporal-alignment or spatiotemporal-grounding rewards degrades intermediate grounding; removing temporal alignment lowers GroundMoRe answer-box IoU to 55.4.Source: Table 7 · p. 17, §A.2.2 · p. 17

A.2.3 INTERMEDIATE TEMPORAL GROUNDING QUALITY ON GROUNDMORE

• On GroundMoRe, chain-aligned reinforcement learning raises overall tIoU from 44.29 to 50.20 and J &F from 43.76 to 53.67 over SFT-only training.Removing only the temporal-alignment reward causes a larger overall tIoU decline than removing only the spatiotemporal-grounding reward.Source: Table 8 · p. 18, §A.2.3 · p. 18

A.2.4 PERFORMANCE ON CONVENTIONAL REFERRING VOS BENCHMARKS

• In zero-shot transfer, TCoVP-4B achieves 81.7 J &F on Ref-DAVIS17 and 73.6 on Ref-YouTube-VOS, leading reported Ref-DAVIS17 methods and remaining competitive with VideoSEG-O3 on Ref-YouTube-VOS.Neither evaluation benchmark's training split was used to train TCoVP.Source: Table 9 · p. 18, §A.2.4 · p. 18

A.3.1 COMPARISON WITH VIDEOSEG-O3-4B

• For the shot-attempt query, TCoVP-4B distinguishes the basketball from contextual participants and grounds it over 9.3–11.7 seconds, unlike VideoSEG-O3-4B's player-following mask.Timestamped basketball boxes provide prompts for mask propagation.Source: Figure 7 · p. 19, §A.3.1 · p. 19

A.3.2 COMPARISON WITH VEASON-R1-3B

• In two examples, TCoVP-3B rejects an absent turtle and grounds the basketball rather than the acting person, while Veason-R1-3B hallucinates the turtle and selects the person.For the basketball query, TCoVP tracks the ball from player to basket over 7.7–10.3 seconds.Source: §A.3.2 · pp. 19–20, Figure 8 · p. 20, Figure 9 · p. 20

A.3.3 FAILURE CASE ANALYSIS

• TCoVP-3B tracks both players but reverses their defender and offensive roles, selecting the man in white instead of the ground-truth target, the man in red.The role inversion leads the mask to follow the wrong player despite accurate identity tracking and localization.Source: §A.3.3 · pp. 20–21, Figure 10 · p. 21