PaperlayerView exact revision on arXiv

Source-linked AI summary

VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings

Ruizhi Chen, Zining Fan, Yudi Guo, Jiang Liu, Zhiyu Lin, Zhongjiang He, Shouqian Shi, Hao Sun

arXiv:2610.22842v1cs.CV

TL;DR

Existing video embeddings often compress multiple objects and their dynamics into a global representation, limiting direct object-level text grounding. VTrajEmbed encodes object trajectories for text retrieval and trains with verified relation masks, while Text2Traj-Bench supplies human-verified relations. It improves trajectory retrieval over Qwen3-VL-Embedding-8B and also reports results on five global retrieval datasets.

  • Problem

    Video embedding models typically compress multiple objects and their dynamics into a global representation, whereas object-level grounding requires identifying which object matches a description.

    Source: §1 · p. 1

  • Method

    VTrajEmbed encodes object trajectories and text in a shared space, using Slow-Fast features and mask-guided contrastive learning over verified query-trajectory relations.

    Source: §1 · pp. 1–2

  • Results

    15.48 and 18.05 percentage-point R@1 gains over Qwen3E-8B for text-to-trajectory and trajectory-to-text, respectively; trajectory-derived representations average 49.48% R@1 across five global datasets versus 42.86%.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The reported evaluation spans object-level trajectory retrieval and trajectory-derived representations on five global retrieval datasets.

    Source: §1 · p. 2, §5 · p. 9

    Sparse-context preservation assumes a semantic-to-context condition and resampler stability; deleting similar frames is not automatically lossless.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Fine-grained alignment between text and video object trajectories is a core challenge in multimodal understanding. Existing embedding models excel at global retrieval, yet their single global representation entangles multiple objects and their spatiotemporal dynamics, limiting direct object-level grounding. To address this issue, we propose VTrajEmbed, an embedding model that decomposes a video into object trajectories and aligns each trajectory embedding with text for object-level grounding through retrieval. Our main contributions: (1)Slow-Fast Trajectory Encoding, a dual-branch architecture that establishes trajectory representations by densely sampling low-resolution frames for high-frequency motion cues and sparsely selecting high-resolution frames for scene context, aggregated by a shared LLM backbone; (2)Mask-Guided Contrastive Learning, a training objective built upon these representations that couples an intra-group loss and a batch-level bidirectional loss under a relation mask, excluding unverified relations and preventing confirmed matches from becoming false negatives; (3)Text2Traj-Bench, a trajectory-level text-video benchmark with model-assisted construction and human-verified query--trajectory relations. Experimental results show that VTrajEmbed-5B improves text-to-trajectory and trajectory-to-text R@1 over Qwen3-VL-Embedding-8B by 15.48 and 18.05 percentage points, respectively, and achieves 49.48% average R@1 across five global retrieval benchmarks, compared with 42.86% for Qwen3-VL-Embedding-8B.

1 Introduction

Video embedding models commonly compress a clip into one representation, while object-level grounding asks which object matches a description. VTrajEmbed aligns text with object trajectories and introduces an architecture, a masked training objective, and a human-verified benchmark.

1 Introduction

  • Global video representations entangle multiple objects and their dynamics, leaving object-level alignment less directly addressed than video-level retrieval.Prior video grounding systems output localized tubes, while a concurrent embedding approach still produces single-vector representations without object-level alignment.Source: §1 · p. 1
  • VTrajEmbed uses object trajectories as alignment units, encoding them and text in a shared semantic space for retrieval.A query parser prompts segmentation and tracking; the resulting trajectories are encoded with dense low-resolution observations and sparse high-resolution scene context.Source: §1 · p. 1
  • Its contributions include Slow-Fast encoding, mask-guided contrastive learning, and Text2Traj-Bench with model-assisted construction and human-verified relations.The objective couples local discrimination with bidirectional retrieval, while the benchmark includes within-video and cross-video relation annotations.Source: §1 · p. 2
  • 69.67% text-to-trajectory and 66.34% trajectory-to-text R@1; VTrajEmbed-5B exceeds Qwen3-VL-Embedding-8B at 54.19% and 48.29%, respectively.Its trajectory-derived representations average 49.48% R@1 across five global retrieval datasets, versus 42.86% for the same baseline.Source: §1 · p. 2

2 Related Work

Prior work develops shared multimodal embeddings, fine-grained image alignment, and open-vocabulary video perception. VTrajEmbed differs by embedding video object trajectories for text-aligned retrieval.

Multimodal Embedding Models

  • Multimodal embedding systems support shared-space retrieval, but video methods primarily target clip-level rather than individual-trajectory alignment.The cited work includes CLIP and SigLIP for image-text pretraining, alongside video-text methods such as LanguageBind, InternVideo2, and Qwen3-VL-Embedding.Source: §2 · p. 2

Object-Level Multimodal Alignment

  • Image alignment methods progress from region-language pretraining to object representations, while VTrajEmbed targets video trajectories with appearance, motion, and temporal context.FG-CLIP and ObjEmbed are identified as especially relevant static-image approaches.Source: §2 · p. 2

Open-Vocabulary Object Tracking and Segmentation

  • Open-vocabulary tracking and spatiotemporal grounding systems output localized tracks, masks, or tubes; VTrajEmbed instead provides reusable text-aligned trajectory embeddings for bidirectional retrieval.This separates candidate generation from semantic matching.Source: §2 · p. 2

2. Verified Description ��

Figure 1 summarizes VTrajEmbed’s architecture, training objective, and benchmark construction. Its panels connect trajectory-text retrieval to verified relation supervision and evaluation.

Model architecture

  • Panel (a) combines dense low-resolution object observations with sparse high-resolution scene context, mapping trajectories and text into a shared space for similarity retrieval.Source: Figure 1 · p. 3

Training objective

  • Panel (b) shows mask-guided contrastive learning for within-video discrimination and bidirectional retrieval, using verified relations while excluding uncertain pairs.Source: Figure 1 · p. 3

Data

  • Panel (c) depicts Text2Traj-Bench construction through trajectory curation, verified descriptions, and relation construction, producing fixed candidate pools and human-verified annotations.The annotations include within-video and cross-video query-trajectory relations.Source: Figure 1 · p. 3

3 Method

VTrajEmbed ranks object trajectories against text in a shared embedding space, with candidate generation frozen for learning. Its method combines Slow-Fast encoding, masked contrastive learning, and a human-verified benchmark.

3 Method

  • The method uses a query parser and frozen segmentation-and-tracking module to propose trajectories, then ranks them against the original query by shared-space similarity.For controlled trajectory-level evaluation, Text2Traj-Bench instead uses query-independent candidate pools fixed before query generation.Source: §3 · p. 3, §3.1 · p. 3

3.1 Slow-Fast Trajectory Encoding

  • Dense low-resolution Fast observations encode appearance and geometry in ordered blocks; sparse high-resolution context is semantically pruned and fused into those blocks.The shared language backbone aggregates trajectory features and separately embeds the original query; timestamps preserve irregular observation gaps.Source: §3.1 · pp. 3–4
  • Sparse-context preservation depends on semantic-to-context and resampler-stability assumptions; deleting similar frames is not automatically lossless.The stated top-1 preservation condition requires a correct reference result with cosine margin γ > 2 maxr δr.Source: §3.1 · p. 4

3.2 Mask-Guided Contrastive Learning

  • A relation mask includes confirmed matches and mismatches but excludes unverified pairs, supporting within-video discrimination and bidirectional batch retrieval over multiple positives.The joint objective weights the local group and bidirectional losses; confirmed positives enter both numerator and denominator.Source: §3.2 · pp. 4–5

3.3 Text2Traj-Bench: Human-Verified Trajectory Relations

  • Text2Traj-Bench supports training and three held-out settings: within-video text-to-trajectory, trajectory-to-text, and object-centric video retrieval.Source recordings and derived records are assigned to splits before benchmark construction, and models propose annotations that humans review.Source: §3.3 · p. 5

Trajectory curation.

  • The benchmark freezes query-independent candidate pools before query generation and admits within-video queries only after human review confirms their target relation.Cross-video development and test relations required for retrieval galleries are individually verified as matches or mismatches; unverified pairs remain unresolved in training.Source: Trajectory curation · p. 6

4 Evaluation

VTrajEmbed improves text-to-trajectory and trajectory-to-text retrieval over Qwen3E-8B and retains strong global retrieval after adaptation. Ablations show contributions from both Slow-Fast branches and the joint training objectives.

4.2 Overall Performance

  • 69.67% T2Traj R@1 vs 54.19% for Qwen3E-8B, and 66.34% Traj2T R@1 vs 48.29%; global average R@1 is 49.48% vs 42.86%.The global average is after benchmark-specific alignment.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 7

4.1 Experimental Setup

  • Across T2Traj, Traj2T, and OCVR, evaluation uses human-verified relevance; T2Traj and Traj2T share query-independent trajectories, and all settings report R@1, R@3, and mAP.T2Traj ranks trajectories within a video's fixed pool; Traj2T ranks test queries for a trajectory; OCVR ranks videos for an object-centric query.Source: §4.1 · pp. 6–7

4.3 Ablation Study

  • Removing the Fast branch lowers T2Traj R@1 to 41.08%, versus 69.67% for the full model; removing Slow lowers it to 64.58%.The Fast-branch removal also lowers R@3 to 87.01% and mAP to 64.94%; Slow-branch removal lowers mAP to 79.86%.Source: §4.3 · p. 8, Table 3 · p. 9
  • Removing Lbi lowers T2Traj R@1 from 69.67% to 49.29% and mAP from 83.72% to 71.10%; removing Lgroup lowers them to 48.06% and 66.27%.The group-only variant has higher R@3 than the bidirectional-only variant, 93.55% versus 81.45%.Source: §4.3 · p. 9, Table 3 · p. 9
  • Reducing Fast observations from 32 to 16 lowers T2Traj R@1 to 64.31% and mAP to 79.81%.The passage describes this as a smaller degradation than removing the Fast branch entirely.Source: §4.3 · p. 9, Table 3 · p. 9

5 Conclusion and Future Work

The conclusion reports trajectory-level and global retrieval results, then outlines evaluation and benchmark extensions. The notation appendix defines symbols for the encoder, assumptions, losses, and retrieval metrics.

5 Conclusion and Future Work

  • 15.48 and 18.05 percentage-point gains in T2Traj and Traj2T R@1, respectively, accompany 49.48% average R@1 across five global datasets versus 42.86% for Qwen3E-8B.The authors interpret these results as finer object discrimination while preserving broader video-text retrieval transferability; future work targets query-conditioned evaluation, learned similarity aggregation, and broader benchmark coverage.Source: §5 · p. 9

A Notation

  • Encoder notation distinguishes appearance and geometry-enhanced features, Fast-block and Slow-frame counts, and pooled Slow context, projections, gates, and visual/text embeddings.Table 4 groups related symbols by role and states context-dependent meanings explicitly.Source: A Notation · pp. 12–13

Slow-context preservation (Lemma 1)

  • The Slow-context lemma bounds trajectory-embedding distortion from preselection, frame pruning, and pooling multiplicities, preserving the dense-reference decision when the correct-target margin exceeds twice the maximum distortion.The bound is δ_r = K_c(ε_pre + K_Rρ_τ + ε_mult).Source: Slow-context preservation (Lemma 1) · p. 13

Training groups, relations, and losses (Section 3.2)

  • The training objective combines within-query-group discrimination with bidirectional batch retrieval, using a relation mask to include confirmed matches and exclude unverified pairs.The group loss is zero when no batch group has confirmed negatives; the joint objective weights group and bidirectional losses separately.Source: Training groups, relations, and losses… · p. 14

Multi-positive ranking control (Lemma 2)

  • The multi-positive ranking lemma relates group, text-to-trajectory, and trajectory-to-text losses to conservative top-1 error across those comparison families.Ties count as errors, while empty comparison families contribute zero to the joint bound.Source: Multi-positive ranking control (Lemma 2) · p. 14

Experimental diagnostics and retrieval metrics

  • R@1 and R@3 measure the fraction of single-target queries retrieving the target within the corresponding rank; mAP summarizes average precision over queries.The appendix also distinguishes measured embedding distortion from its theoretical bound and specifies development-set pruning thresholds.Source: Experimental diagnostics and retrieval… · p. 14

Operators and notation conventions

  • The notation conventions define list and set handling, including that indexed lists retain repeated-record multiplicities, and specify common operators such as normalization, norm, and arg max.Source: Operators and notation conventions · p. 15

B Experimental and Reproducibility Details

The appendix records the model, optimization, runtime, and expression-group settings used for the reported experiments. It details frozen pretrained components, ROI feature processing, training configuration, and noisy-ground-truth augmentation.

B.1 Model and Encoder Configuration

  • The frozen visual encoder uses intermediate Transformer features from layers 12, 18, and 24, while the ROI projector and language backbone are initialized from ObjEmbed-4B and Qwen3-VL-4B-Instruct.The pretrained visual encoder, ROI projector, and base language-model parameters remain frozen during training.Source: §B.1 · p. 15
  • Fast-path ROI features from three scales are merged into a 7 × 7 × 2560 region representation, flattened, and passed through an MLP to produce the object appearance vector.The three projected feature maps contribute 640, 1280, and 2560 channels before their concatenation and projection.Source: §B.1 · p. 15

B.2 Optimization and Runtime Configuration

  • Training uses fused AdamW with a 5 × 10−5 initial learning rate, linear decay without warm-up, eight A800 GPUs, and a three-epoch budget.Architectural ablations keep the full model's data split, optimizer, schedule, batch configuration, duration, and seed; objective ablations change only the loss terms.Source: §B.2 · pp. 15–16

B.3 Expression-Group Construction and Noisy-GT Augmentation

  • Each expression-centered group has four trajectory records, retaining a clean ground truth and confirmed negative alongside matched positives or a noisy-GT variant.Noisy-GT augmentation removes a contiguous 10%–20% segment, independently drops remaining observations with probability 0.1, and perturbs boxes within specified spatial bounds.Source: §B.3 · p. 16

C.1 Candidate generation and task scope

VTrajEmbed separates trajectory proposal from embedding-based ranking, then encodes each trajectory with ordered object observations and sparse scene context. Separate trajectory and text passes produce representations for similarity ranking.

C.1 Candidate generation and task scope

  • Query-conditioned proposals use appearance prompts and frozen tracking, while Text2Traj-Bench freezes query-independent candidate pools; complete tracks retain available identity observations without implying perfect tracking.The encoder neither predicts boxes nor regresses event boundaries.Source: §C.1 · p. 16

C.2 Sampling and temporal blocks

  • Fast observations are uniformly sampled from each trajectory, while the Slow branch preselects one to four frames and semantically prunes redundant context without changing Fast observations.The retained Slow frames are not guaranteed to be uniformly spaced in physical time; Fast blocks contain at most four observations and are not extended by copying frames.Source: §C.2 · p. 17

C.3 Fast branch: ROI features and geometry

  • The Fast branch extracts multiscale ROI appearance features, adds an MLP encoding of box geometry, and uses a shared attention resampler to aggregate chronological observation blocks.The geometry input includes normalized box center, width, height, area, displacement, and log-area change; displacement is not time-normalized velocity.Source: §C.3 · pp. 17–18

C.4 Slow branch and fusion

  • The Slow branch pools retained high-resolution scene features using Fast appearance as the cross-attention query, providing aggregate context rather than aligned event order.Fusion adds a gated Slow context residual to Fast block features; the gate is initialized at zero, making the initial representation Fast-only.Source: §C.4 · p. 18

C.5 Language-model inputs and adaptation

  • Separate prompts and masked passes encode trajectory steps for the visual readout and the query alone for the text readout; the language backbone is frozen with LoRA adapters in its last eight layers.The text representation depends only on the query, while the trajectory pass reads the valid fused trajectory steps.Source: §C.5 · pp. 18–19

D Training Data and Relation Annotations

Training groups combine query-relative positive and confirmed-negative trajectories, with overlap screening and identity checks used to select or verify records.

D.1 Annotated positives and overlap-based screening

  • Each query-centered training group targets four records, retaining one clean ground-truth trajectory and one confirmed negative, with remaining slots filled by noisy ground truth or matched raw positives.Annotations may cover only the query event, while matched tracker trajectories retain their full available extent.Source: §D.1 · p. 19
  • Low overlap proposes a negative but does not certify mismatch; same-identity fragments and unresolved cases are excluded, and only confirmed nonmatches enter the negative set.The low-overlap screening threshold is maxIoU < 0.2; intermediate-overlap or uncertain-identity tracks are ignored.Source: §D.1 · p. 19
  • Noisy-ground-truth augmentation removes a contiguous segment, drops observations, and perturbs boxes, while preserving the clean ground-truth trajectory as an independent positive record.The removed segment is approximately 10%-20% of observations, and each remaining observation is independently dropped with probability 0.1.Source: §D.1 · p. 19

D.2 Model-assisted, human-verified training data

Model proposals are converted into training and benchmark relations through human verification, while unverified pairs remain distinct from confirmed matches and mismatches. Text2Traj-Bench preserves candidate pools through curation, description, and query review.

D.2 Model-assisted, human-verified training data

  • LLMs verbalize human-approved trajectory facts into queries, but human verification determines accepted queries and core relations; unverified or unresolved pairs remain unlabeled.Video captions or action-class labels alone do not provide tube supervision, and source-specific supervision or added annotation must be documented.Source: §D.2 · pp. 19–20

D.3 Batch mask, duplicate records, and empty comparisons

  • The batch mask propagates positive labels only when a trajectory supports the query, marks only established mismatches as negative, and leaves other relations unverified.Cross-video comparisons enter the batch objective only where verified relations exist; multi-positive training rewards total positive probability rather than requiring uniform probability across positives.Source: §D.3 · p. 20

E Text2Traj-Bench Construction Details

  • Text2Traj-Bench uses trajectory curation, verified description, and relation construction, with models proposing candidates and human review determining core annotations.Source assignments are fixed before construction, and cross-video training relations do not connect to held-out sources.Source: E Text2Traj-Bench Construction Details · p. 20

E.1 Trajectory Curation

  • Annotators review proposal usability and identity redundancy before the candidate pool is frozen; later description or query rejection does not remove trajectories.Unresolved identity pairs remain separate candidates, and queries matching multiple fragments are rejected.Source: §E.1 · pp. 20–21

E.2 Verified Description

  • Descriptions and structured facts are proposed from trajectory observations and verified by annotators; queries must derive from approved facts and preserve temporal and relational qualifiers.Queries are model-generated rather than independently collected user utterances, so fluency alone does not establish coverage of natural user queries.Source: §E.2 · p. 21

E.3.1 Within-video Verification

  • Within-video verification checks every query against the complete candidate pool, labeling confirmed matches, confirmed mismatches, and uncertain relations; admitted queries require one target and confirmed mismatches for every alternative.This establishes an annotation-based decision within the candidate pool, not infallible uniqueness in the physical scene.Source: §E.3.1 · pp. 21–22

E.3.2 Cross-video Expansion

Cross-video expansion mines candidate comparisons, but human verification—not mining strata or similarity—determines core relation labels. The resulting batch objective and evaluation remain bounded by available relations and the specified retrieval protocol.

E.3.2 Cross-video Expansion

  • Cross-video mining stratifies candidate pairs as easy negatives, hard negatives, or potential matches, but these strata guide sampling rather than determine relation labels.Human review confirms core relations; unverified pairs remain unlabeled, and optional automatic easy-negative pseudo-labels are kept separate from benchmark ground truth.Source: §E.3.2 · p. 22
  • Only verified relations available in a batch contribute cross-video comparisons; missing labels remain unverified, and positive-only rows or columns contribute zero loss.Reporting should include realized cross-video positive and negative counts and zero-negative row and column rates.Source: §E.3.2 · p. 23
  • Independent audits record annotation reliability and provenance, while majority approval remains an annotation rule rather than a guarantee of correctness.Audits include residual errors, pre-adjudication agreement, corrections, rejection reasons, and reviewer time.Source: §E.3.2 · p. 23
  • Within-video R@1 measures ranking within a fixed candidate pool, not proposal recall or end-to-end success; reverse, cross-video, and global retrieval require separate protocols.Estimating proposal recall, post-curation retention, and end-to-end success requires an independently sampled source subset.Source: §E.3.2 · p. 23

F Lemma 1: Task-Relevant Preservation under Slow-Fast Sampling

The lemma compares a pruned Slow context with dense-context references while holding the Fast observations fixed. Its guarantee depends on explicit semantic-similarity, resampler-stability, and preselection-approximation conditions.

F.1 Reference representations and semantic coverage

  • The analysis fixes a trajectory and trained parameters, then compares contexts derived from its Fast observation set F.S0 is uniformly preselected from F and then greedily pruned to S; excluded-frame features are reference quantities, not deployed computation.Source: §F.1 · p. 23
  • The dense-context reference encodes high-resolution context at every Fast observation, while the temporal resampler summarizes global context rather than ordered scene interactions.The Fast temporal sequence remains fixed throughout this comparison.Source: §F.1 · p. 24
  • The guarantee assumes bounded context differences for assigned pairs, stable resampling at the preselected input length, and preselection error at most εpre.The semantic-similarity condition allows residual η; the passage cautions that these conditions do not establish sufficient context features or automatically accurate sparse pooling.Source: §F.1 · p. 24

F.2 Proof of semantic-pruning and representation bounds

The appendix bounds representation changes from Slow-context pruning and derives a condition for preserving a reference top-1 ranking. It also accounts for attention-mass changes, preselection and encoding costs, and the multi-positive retrieval objective.

F.2 Proof of semantic-pruning and representation bounds

  • The pruned context differs from the preselected context by at most KRρτ + εmult, and from dense context by at most εpre + KRρτ + εmult.A reference top-1 decision is preserved when its cosine margin exceeds twice the maximum candidate representation error.Source: §F.2 · p. 24

F.3 Attention mass after deleting repeated representatives

  • Pruning repeated representatives changes attention mass as well as feature values, so a bound based only on semantic feature distortion can miss representation error.With equal logits and values consisting of three zeros and one one, pruning duplicates changes the output from 0.25 to 0.5 despite zero within-cluster feature distortion.Source: §F.3 · p. 25

F.4 Preselection coverage, stability, and Fast features

  • Preselection error depends on temporal coverage and changing attention multiplicities; finite feature-stability bounds require bounded projections and a stable feature domain.Uniform selection in the valid-observation list does not guarantee a fixed physical-time radius, particularly across occlusions.Source: §F.4 · p. 25
  • The preservation argument keeps the low-resolution Fast sequence unchanged, and tiny objects or fast, detailed interactions may violate its small-error regime.It isolates Slow-context compression rather than assuming away low Fast resolution.Source: §F.4 · p. 26

F.5 Encoding cost and scope of the design argument

  • Removing Slow passes can save (ns −ms)Cs before pruning overhead, but positive end-to-end savings are not guaranteed.Measured cost must include auxiliary modules, resampling, ROI reuse, and caching; the stated frame budgets are implementation budgets, not information-saturation points.Source: §F.5 · p. 26

G.1 A common retrieval objective

  • The retrieval objective aggregates softmax probability over valid answers, targeting a highest-ranked valid answer without requiring uniform probabilities across positives.Its semantic interpretation depends on included labels correctly identifying matches and non-matches; excluded, unconfirmed pairs are not covered.Source: §G.1 · p. 26

G.2 Proof of the per-comparison bound

The proof bounds top-1 errors for comparisons with labeled positive and negative sets, then extends the control to local and bidirectional retrieval families. The interpretation depends on accurate relation labels and does not cover unconfirmed or unseen candidates.

G.2 Proof of the per-comparison bound

  • For a comparison with mA positives, log(1 + 1/mA) times its top-1 error is bounded by the loss; a loss below that threshold guarantees the best positive outranks every included negative.The threshold is sharp, and duplicate records count toward mA.Source: §G.2 · p. 27

G.3 Joint control of the three retrieval families

  • The joint objective bounds empirical errors across local selection, text-to-trajectory batch rows, and trajectory-to-text batch columns, including empty comparison families.The bounds apply to included comparisons and do not require independent comparisons or disjoint score pairs.Source: §G.3 · p. 27

G.4 Relation masking and direct supervision

  • Local and batch terms supply conditional sources of negatives, while unknown relations are excluded and inaccurate labels undermine the semantic interpretation of the bound.The result does not bound errors on unconfirmed or unseen candidates, tracker misses, annotation mistakes, or train-test shifts; the adopted loss weighting requires empirical validation.Source: §G.3 · p. 27, §G.4 · p. 28
Loading 2610.22842v1…