PaperlayerView exact revision on arXiv

Source-linked AI summary

Ask About the Right Moment: Training-Free Object Identity Reasoning for Video-Language Models

Di Yang, Xiaochang Li, Muyang Liu, Yufei Wang, Yanhai Xiong

arXiv:2610.31471v1cs.CV

TL;DR

Video-LMs struggle to preserve the identity of an object when it is hidden or resembles other objects. Track, Then Bind uses a static question to bind identity when evidence is visible, then tracks that identity and reads answers from tracked state. Without training, it solves both VET-Bench tasks and improves over direct answers on real cup games and TOC-Bench.

  • Problem

    Video-LMs recognize objects in individual frames but struggle to keep their identities distinct over time, especially when identifying evidence disappears or objects look alike.

    Source: §1 · p. 2

  • Method

    Track, Then Bind asks the VLM about identity at an identifying moment, carries the binding with object tracks, and answers from tracked state or a track-derived timeline.

    Source: §1 · p. 2, §3 · p. 4, §4 · p. 5

  • Results

    Without training, it solves both VET-Bench tasks and adds 17–18 points over direct answers on real cup games and 12–18 points on TOC-Bench.

    Source: §7 · p. 8

  • Takeaways & Limitations

    A single configuration applies across benchmarks, separating identity binding from temporal identity maintenance in the reported evaluations.

    Source: §1 · p. 2, §7 · p. 8

    The agent inherits tracker errors: identical cups can swap tracks when lifted and crossed by hand, and very small hidden objects are rarely tracked.

    Source: §6 · p. 8

Abstract

from arXiv · show

Many questions about a video concern one physical object. Where does the ball end up after the cups are shuffled? Which card is the queen? How often was the mug covered? Video-language models (Video-LMs) recognise such objects in single frames but do not keep them apart over time. On real shell-game videos, open models from 8B to 235B parameters, their thinking variants, a reinforcement-learned tool agent and three training-free tool agents all stay within ten points of chance. We separate semantic identity binding from temporal identity maintenance. An object's identity should be fixed at a moment when the evidence for it is visible, and carried afterwards by a tracker. \emph{Track, Then Bind} implements this principle without training. A promptable video tracker follows every candidate object and draws it with a track number. A frozen vision-language model answers a static question about the identifying evidence, which binds the target to a track. The answer is then derived from the tracked state in text, without asking the model to reinterpret the original video during read-out. A single configuration covers all benchmarks. With Qwen3-VL-32B it solves both VET-Bench tasks, where a fine-tuned trajectory model reaches 92\% and 90\% and the same VLM voting over 16 samples stays at 30--32\%. It answers 55.4\% of the hardest Perception Test cup games, where the best published model answers 41.5\%, and, using track-derived timelines, 48.6\% of TOC-Bench questions, against 37.1\% for the same model answering directly. Drawing the same tracks on the video and asking the question directly falls 12--19 points short of the agent on real cup games.

1. Introduction

Video-LMs often lose track of a physical object when identifying evidence disappears or look-alike objects move. Track, Then Bind fixes identity when it is visible, then uses tracking to maintain it and answer from tracked state.

1. Introduction

  • Video-LMs, reasoning variants, and training-free tool agents remain near chance on shuffled real cup games, while answering object-identity questions requires following one object through occlusion, containment, and look-alike copies.The paper connects this challenge to assistants remembering item locations, robots whose arms hide objects, and laboratory rows of identical tubes.Source: §1 · p. 2
  • Identity can be fixed only when identifying evidence is visible; a tracker can then carry it, while the answer can be read from tracked state rather than reinterpreting the video.Track, Then Bind uses one configuration across benchmarks without training.Source: §1 · p. 2
  • Table 1 reports accuracy across VET-Bench, Perception Test cup-game sets, and TOC-Bench, with the best result per column bolded.The caption defines PT-65 as 65 videos with three cups and at least one swap.Source: Table 1 · p. 3
  • On real cup games, the agent improves direct answers by 16.8 points at 32B and by 11.5 points on TOC-Bench.It binds identity with a static question about the identifying moment and reads the answer from tracked state; a direct question over the same track overlays trails the agent on real cup games.Source: §1 · p. 3

2. Related Work

Prior work studies temporal reasoning, object permanence, tracking, visual prompts, and tool-using agents, while object identity across time remains a distinct challenge. Track, Then Bind differs by fixing identity when evidence is visible and deriving later answers from tracked state.

2. Related Work

  • VET-Bench [1] and TOC-Bench [3] directly target object identity, amid broader evidence of temporal, event-binding, and spatial limitations in Video-LMs.TempCompass finds many models barely exceed chance without single-frame and language shortcuts, while TOMATO [29] reports a 57-point human gap on frame-order questions.Source: §2 · p. 3
  • Chain-of-thought [32], self-consistency [34], reinforcement-learned models [35] [5], and trajectory or grounding approaches [1] [6] add reasoning traces, but learned representations or generated tokens remain insufficient for look-alike and hidden objects.Self-consistency is also used as a call-matched baseline (Table 3).Source: §2 · pp. 3–4
  • Object-permanence research spans infant expectations, synthetic benchmarks [38] [39], and models for containment, occlusion, intuitive physics, and segmentation [40] [41] [16].Xie and Alanwar [68] find that video world models retain visible objects but drop hidden ones.Source: §2 · p. 4
  • Video trackers such as SAM2 [4] and SAM3 [44] maintain object identities, but do not determine which object a question refers to; this agent gets boxes from the VLM and identities from SAM2.1.Related methods include memory-based segmentation [45] [46], point tracking [47], tracking-by-detection [48], and open-vocabulary detectors [49].Source: §2 · p. 4
  • Visual prompts mark regions or tracks [10] [11], while trained systems track, refer to, or segment objects [54]; here the model identifies the target from one moment and answers from tracks.The same track overlay alone recovers little of the gain at 8B and part at 32B (Section 5.3).Source: §2 · p. 4
  • Prior video agents retrieve frames, summarize captions, plan over symbolic memory, or query tracking states [8] [7], but leave final identity judgment to the model.STRAND [9] links object states by attribute and position similarity, making identical objects difficult to separate; this design differs in when identity is queried and what determines the answer.Source: §2 · p. 4

3. Problem Formulation

The formulation separates identifying a target from maintaining its identity over time. A direct Video-LM answer must bridge sparse observations after identifying evidence may disappear, whereas the proposed factorization binds once and reads from tracked state.

3. Problem Formulation

  • A video contains physical objects, and a question may ask about a target's final location, identity, or visibility; objects that hide, cover, or resemble it are its carriers.A Video-LM answers directly from a sparse frame sample.Source: §3 · p. 4
  • An evidence moment is a time whose local observation window makes the target or its containing carrier identifiable among the carriers.For a visible target, the identifying evidence may be its appearance; for a hidden target, it may be the moment it is covered.Source: §3 · p. 4
  • When evidence disappears before the final frame, a direct answer must carry identity through sparse frames in which identical carriers may cross; visibility events may also fall between sampled frames.The target can be identified while a card is face up or as a ball is covered.Source: §3 · p. 4
  • The answer factors into binding a target to a carrier track at an evidence moment and reading the answer from tracked state in text, without showing the video again.Questions without a target to bind instead use a timeline of per-object visibility segments and events computed from mask areas.Source: §3 · pp. 4–5

4. Method

Track, Then Bind plans the needed operations, tracks candidate objects, identifies an evidence moment, binds the target to a numbered track, and answers from tracked facts or a timeline. The same thresholds apply across benchmarks.

4. Method

  • A plan selects which operations run, while one set of thresholds serves every benchmark.The framework is organized into five steps.Source: §4 · p. 5

Step 1: Plan

  • The VLM plans which entities and roles matter, how the target is recognised, whose left and right apply, and whether the question concerns final location.The plan uses a fixed schema and is based on video viewed at 2 fps.Source: §4 · p. 5

Step 2: Perceive

  • The VLM boxes entities on sampled frames, then SAM2.1 [4] propagates candidate tracks forward and backward, with a native-rate second pass when carriers overlap, move sharply, or end early.Joint tracking helps separate identical objects by trajectory; all questions about one video share its boxes and tracks.Source: §4 · p. 5

Step 3: Find evidence moments

  • The system locates evidence moments from detections or track disappearance; if the tracker cannot locate one, the bind clip spans the whole video.Small hidden objects are often lost or do not vanish cleanly, and a mask track alone cannot distinguish a face-up card from its back.Source: §4 · p. 5

Step 4: Bind

  • The VLM answers one static question about a 4 fps clip with numbered carrier tracks, identifying the target or the carrier it is under at the specified moment.The clip spans three seconds around a located moment or the whole video; the question does not ask the VLM to follow the target through the shuffle.Source: §4 · p. 5

Step 5: Read out

  • For a bound target, the VLM answers from plain-text tracked facts without the video; for other questions, it answers using a timeline of visibility segments, events, and coarse motion.A person-relative question adds a VLM call to establish whether the person is opposite the camera; an unparseable answer falls back to the direct answer.Source: §4 · p. 5

5. Experiments

The experiments test Track, Then Bind against direct-answer and tool-agent baselines across synthetic and real object-identity benchmarks. The results favor binding identity at visible evidence and reading answers from tracked states, while revealing tracking and viewpoint errors as remaining failure sources.

5.1. Experimental Setup

  • The evaluation covers 50 synthetic shell-game videos, 50 card-monte videos, 107 real Perception Test cup videos, and 2,323 TOC-Bench questions split by video.Design choices use development data, and the frozen configuration is run once on test sets.Source: §5.1 · pp. 5–6

5.2. Main Results

  • 55.4% on PT-65 and 48.6% on TOC-Bench are reported for the 32B agent, which also solves both VET-Bench tasks without training.On the 107 real cup games, it reaches 51.4% at 8B and 52.3% at 32B; every baseline stays within ten points of chance on PT-65.Source: §5.2 · p. 6
  • 21–34-point gains occur in event existence, reappearance versus disappearance, and relative spatial change; conditional-state accuracy falls from 49.8% to 44.7% at 32B.The agent improves on the direct answer in nine of ten TOC-Bench dimensions at 8B and eight at 32B.Source: §5.2 · p. 6, Table 2 · p. 6
  • 46.7% for the 8B vote over 13 samples compares with 33.6% for one answer, while the 32B agent is 15.0 points above the vote.The 8B agent reads 25k tokens against 157k for the vote; on 1,800 Video-MME questions, it scores 70.9% versus 71.9% for direct answering.Source: §5.2 · pp. 6–7, Table 3 · p. 6

5.3. Ablation Study

  • At 32B, asking about the whole numbered clip yields 72.0% on the cup game but 54.2% on real cup games; a located moment shortens the cup-game bind clip from about 47 to 12 frames.On real cup games, the 8B agent finds a hidden-object moment in only 21% of videos and the 32B agent in 11%; other videos use the whole numbered clip.Source: §5.3 · p. 7, Table 5 · p. 7, Table 6 · p. 7
  • Reintroducing the shuffled video for read-out lowers accuracy even when the tracked facts are unchanged.With facts appended, the 8B cup-game result drops from 100% to 38.0%, and card monte drops to 72.0%.Source: §5.3 · p. 7, Table 5 · p. 7
  • One configuration stays within the confidence intervals of benchmark-tuned variants on all four test sets, using 13.4 rather than 20 calls per cup game.On TOC-Bench at 8B, it uses 10.8 rather than 20.2 calls per question; TOC-Bench gains come from timeline read-out.Source: §5.3 · p. 8, Table 7 · p. 8

5.4. Qualitative Analysis

  • The card-monte example binds the queen while face up and tracks it to the right, whereas real-cup errors include lost tracks, viewpoint mistakes, and unbound questions.At 32B, 14 of 51 errors come from binding or tracking, 20 from viewpoint, and 12 from questions that are never bound.Source: §5.4 · p. 8, Figure 3 · p. 8

6. Limitations

The agent inherits tracker errors: identical cups can swap tracks when lifted and crossed, and very small hidden objects are rarely tracked. Tracking every question also costs about 11 VLM calls, including questions that need no tracking.

6. Limitations

  • Identical cups lifted and crossed by hand can still swap tracks.The agent inherits these tracker errors.Source: §6 · p. 8
  • Very small hidden objects are rarely tracked, limiting the agent’s object coverage.Most real cup games are therefore bound on the whole clip.Source: §6 · p. 8
  • The agent makes about 11 VLM calls per question, including general questions that need no tracking.Skipping tracking for those questions is left to future work.Source: §6 · p. 8

7. Conclusion

Track, Then Bind separates recognizing an object from maintaining its identity over time: the VLM identifies it when evidence is visible, then tracks carry that identity. Without training, it solves both VET-Bench tasks and improves on direct answers for real cup games and TOC-Bench.

7. Conclusion

  • Without training, Track, Then Bind solves both VET-Bench tasks and improves over direct answers on real cup games and TOC-Bench.The conclusion reports gains of 17–18 points on real cup games and 12–18 points on TOC-Bench.Source: §7 · p. 8
  • The method asks the VLM about identity when identifying evidence is visible, then carries identity with tracks.The final answer comes from the tracked state in text.Source: §7 · p. 8
  • The conclusion frames the central problem as Video-LMs losing object identity when evidence disappears or objects look alike.Source: §7 · p. 8

A. Algorithm and Implementation Details

Track, Then Bind uses one program and configuration across benchmarks, with the plan selecting the question-specific branch. The appendix also reports implementation details and per-question computation costs.

A. Algorithm and Implementation Details

B. Benchmarks, Splits and Evaluation

The evaluation describes its development and test data, split rules, and scoring procedures, then details three tool-agent baselines. SpatialClaw propagates masks on many real cup-game questions but remains below chance.

B. Benchmarks, Splits and Evaluation

  • Development data include Perception Test validation cup videos and generated VET card-monte videos; test data are run once with the frozen configuration.The evaluation-data table defines duration as mean video length and development data as the basis for design and threshold selection.Source: B. Benchmarks, Splits and Evaluation · p. 12, Table 10 · p. 12
  • TOC-Bench uses 2,323 questions over 1,951 videos, split by video into development and test; each format has its own parser.The test set contains four-way multiple-choice, two-statement, numerical, and ordering questions; unparsable answers count as wrong.Source: B. Benchmarks, Splits and Evaluation · p. 12
  • The evaluation sets are disjoint across cup-game development and test, general Perception Test questions, and TOC-Bench splits.Rendered card videos also share no files with VET-Bench test videos.Source: B. Benchmarks, Splits and Evaluation · p. 12

C. Baselines

  • SpatialClaw is run with its official code, prompts, tools, and model configuration; unavailable tool-server sessions are rerun once.Its baseline uses SAM3 and Depth Anything 3 at the pinned commits, and benchmark parsers score returned answers.Source: C. Baselines · pp. 12–13
  • 77% of real cup-game questions use SpatialClaw mask propagation, yet its accuracy remains below chance.Table 12 reports test-set results and the share of questions with successful SAM3 mask propagation.Source: Table 12 · p. 13, C. Baselines · p. 13
  • In the shown SpatialClaw card-monte run, the queen’s mask disappears after the card turns over and the model answers from the last frame.Track, Then Bind answers the same video correctly.Source: Figure 4 · p. 13
  • VideoAgent combines segment captions, video and track features, and re-identification features; Qwen3-VL-32B-Instruct replaces GPT-4 as its agent LLM.The implementation also replaces unavailable APIs and averages 9.8 LLM calls per question.Source: C. Baselines · pp. 13–14
  • VideoINSTA samples one frame per second, clusters clips, and combines clip summaries and ratings into an answer.Its implementation uses Qwen3-VL-32B-Instruct instead of GPT-3.5 and averages 14.8 LLM calls per question.Source: C. Baselines · p. 14

D. Additional Results

Paired same-VLM test comparisons favor the agent in every row, although two real-cup-game intervals include zero; further analyses show dimension- and subset-specific variation.

D. Additional Results

  • The agent leads every same-VLM baseline comparison, but two real-cup-game intervals include zero.The two intervals are for the 8B self-consistency vote and VideoINSTA.Source: D. Additional Results · p. 14
  • On TOC-Bench event counts, VideoINSTA and VideoAgent score 39.4% and 35.3%, respectively, versus 32.0% for ours.The thinking variant's overall accuracy is within 0.2 points of the direct answer; its largest dimension shift is 10.1 points.Source: D. Additional Results · p. 14
  • On Perception Test cup games with four or more swaps, the 8B and 32B agents gain 2.8 and 11.4 points over the direct answer, respectively.The four four-cup games are answered once at 8B and never at 32B.Source: D. Additional Results · p. 14

E. Design Choices on Development Data

Development analyses examine how question timing, tracking and grounding settings, and read-out choices relate to accuracy.

E. Design Choices on Development Data

  • 48.7% vs 40.0% development cup-game accuracy at 8B (+8.7 points), and 52.3% vs 44.1% at 32B (+8.2), favor asking which cup held the target just after hiding.The intervals are [0.5, 16.9] at 8B and [1.5, 14.9] at 32B; the moment-of-question comparisons have p=0.046 and p=0.026.Source: E. Design Choices on Development Data · p. 14
  • The overlay adds 6.7 and 4.1 points over direct answers, while the target's own track hands off identity in 56 of 195 videos and is right in 48% of them.With tracks and overlays fixed, the question moment adds the remaining gain.Source: E. Design Choices on Development Data · p. 14
  • Card-monte accuracy falls from 56 to 21 of 56 videos at 10 fps versus 30 fps; the dense pass reproduces the 30 fps result without a fixed rate.Identity breaks where identical cards cross quickly.Source: E. Design Choices on Development Data · p. 14
  • Grounding on 16 rather than 8 frames lowers TOC-Bench accuracy by 2.5 points and card-monte accuracy by 3 videos, while nearly doubling VLM calls.Source: E. Design Choices on Development Data · p. 14
  • 57.9% versus 48.2% development cup-game accuracy favors asking whether the person is opposite the camera on four frames rather than faces the camera on the last frame.Among six wordings tested on the same videos, the single test configuration is 6.5 points above tuned variants at 8B and 1.9 points above at 32B.Source: E. Design Choices on Development Data · p. 14
  • 42.1% of TOC-Bench development questions are answered with visibility segments, event list, and motion direction, versus 37.4% with visibility segments alone (+4.6, CI [1.5, 7.7]).Source: E. Design Choices on Development Data · p. 14

F. Additional Qualitative Examples

The examples illustrate located identity moments, tracking-based timelines, and prompt configurations across the additional qualitative examples and worked cases.

F. Additional Qualitative Examples

G. Worked Examples

  • Each worked-example panel lists what the VLM sees, its prompt and reply, and the call count; each run reproduces the logged output of the single configuration.The examples appear alongside the same VLM's direct answers.Source: G. Worked Examples · p. 18
  • Figures 7–9 show hidden-target, face-up queen, and timeline examples; the timeline branch skips binding when no target needs to be distinguished.Source: Figure 7 · p. 18, Figure 8 · p. 19, Figure 9 · p. 20

H. Prompts

  • Table 17 gives the unified agent's verbatim prompts, while Table 18 lists verbatim prompts for the per-benchmark configurations used in the Table 7 comparison.Source: Table 17 · p. 21, Table 18 · p. 22

Related papers

Loading 2610.31471v1…