Source-linked AI summary
Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking
Han Wang, Yuxuan Liu, Yuhan Sun, Jian Yang, Xiaotong Xu, Yixuan Lv, Zhuang Zhou, Shengyang Li
TL;DR
Referring single-object tracking must use language strongly for grounding without allowing it to cause semantic drift during later tracking, while existing approaches can require costly vision-language alignment training. LVTrack uses a mode-conditioned Gated Feature Injector with a frozen vision-language backbone, supplemented by spatially and temporally aware localization components. The paper reports strong benchmark performance with substantially reduced training cost, while noting fixed-resolution inputs and limited exploration of dynamically updated language guidance.
Problem
Existing tracking methods face semantic drift when linguistic guidance is overemphasized, while VLP features lack explicit instance-level binding and spatial selection and alignment training is costly.
Method
LVTrack freezes a vision-language pretrained backbone and regulates textual fusion with a mode-conditioned Gated Feature Injector, GS-KL supervision, hybrid positional encoding, and appearance memory.
Results
LVTrack achieves strong benchmark performance across visual grounding and language-initialized tracking settings with substantially reduced training cost.
Takeaways & Limitations
The framework directly leverages pretrained cross-modal representations for unified grounding and tracking while mitigating language-dominated drift.
Takeaways & Limitations
LVTrack currently relies on fixed input resolutions and mainly studies language guidance supplied at initialization rather than dynamically updated instructions during tracking.
Abstract
from arXiv · showhide
Referring single-object tracking enables language-grounded target initialization and subsequent tracking by jointly leveraging semantic cues and visual templates. The core difficulty is to use language differently across stages: it is indispensable for grounding but can induce semantic drift during tracking when overemphasized. Meanwhile, current methods often require costly vision-language alignment training. We present LVTrack, a pure transformer framework that introduces a mode-conditioned Gated Feature Injector to adaptively regulate textual guidance and alleviate semantic drift. Together with targeted adaptations, it directly harnesses a frozen vision-language pretrained model, greatly reducing training cost and preserving strong language understanding. To further improve temporal localization, LVTrack integrates hybrid relative-absolute positional encodings with a lightweight memory mechanism and optimizes autoregressive box prediction using a Gaussian-smoothed KL loss. Extensive experiments on standard benchmarks demonstrate that LVTrack achieves strong performance.
1 Introduction
LVTrack addresses the training-cost and semantic-drift challenges of referring single-object tracking with a frozen vision-language backbone and mode-adaptive textual injection. It combines this design with spatially aware autoregressive localization and temporal appearance modeling.
- Referring SOT extends conventional tracking by using natural-language expressions to specify targets and improve robustness through semantic cues.
- Existing end-to-end approaches require alignment training, while VLP features lack explicit instance-level binding and spatial selection for object localization.
- LVTrack freezes the VLP backbone and uses a lightweight feature injector for conditional cross-attention fusion, reducing training cost.
- A dual-mode, amplitude-calibrated Gated Feature Injector allows stronger language-vision coupling for grounding and conservative textual injection during tracking to suppress semantic drift.
- GS-KL supplies distance-aware soft targets for coordinate tokens, while hybrid positional encoding and appearance memory improve geometric and temporal localization.
- LVTrack is reported to deliver strong performance across visual grounding, NL-initialized SOT, and NL+BBOX-initialized SOT, with substantially lower training cost.
2 Related Work
Related work develops multimodal tracking through language-visual fusion, adaptive switching, and unified formulations, but often underuses pretrained vision-language models because effective alignment training is costly.
- Language-assisted SOT treats language as an auxiliary cue alongside visual templates to improve semantic awareness and object discriminability.
- Prior methods include Siamese language guidance, adaptive vision-language representations, unified multimodal encoders, and support for additional modalities.
- Language-initialized SOT evolved from two-stage grounding-then-tracking pipelines toward adaptive switching and joint frameworks.
- Existing approaches often underexploit VLP generalization because establishing effective vision-language alignment requires costly training.
3 Method
LVTrack combines frozen vision-language features with mode-conditioned textual injection, temporal memory, hybrid positional modeling, and distance-aware autoregressive localization. Its design regulates language during tracking while supporting grounding and fine-grained coordinate prediction.
- Overall Framework: LVTrack extracts language, template, and search features with a frozen Perception Encoder backbone, then predicts normalized bounding boxes through transformer processing.The framework uses an 8-layer encoder and 4-layer autoregressive decoder after feature adaptation.
- Overall Framework: LVTrack supports template-only tracking, language-based grounding, and joint language-template tracking through three operational modes.Learnable mode prompts and dynamic input masking support unified training across tracking and grounding data.
- Mode-Conditioned Gated Feature Injector: MGFI injects language into visual features through cross-attention and channel-wise gating conditioned on task mode, while freezing the pretrained backbone.The gate controls injection amplitude, and its norms constrain language-induced perturbations that can cause semantic drift.
- Spatiotemporal Tracking Modeling: LVTrack fuses search features with a gated appearance memory built from historical target regions, updating memory only when prediction confidence exceeds a threshold.This confidence-controlled refresh is intended to avoid noisy accumulation during inference.
- Spatiotemporal Tracking Modeling: Hybrid positional modeling combines shared learnable 2D absolute embeddings with 2D Axial RoPE to align spatial features across template and search grids.The autoregressive decoder also uses RoPE when predicting discretized coordinate tokens.
- Training Objectives: GS-KL replaces one-hot coordinate targets with Gaussian-smoothed distributions, assigning higher weights to nearby bins and adding distance sensitivity to token classification.The loss addresses the equal treatment of near and distant coordinate errors under standard cross-entropy.
4 Experiments
LVTrack is evaluated across tracking, grounding, efficiency, and ablation settings using frozen-backbone configurations and multiple initialization protocols. Results show strong cross-dataset performance, effective grounding, lower training cost, and benefits from GS-KL, gated injection, memory, and 2D RoPE.
- Implementation Details: LVTrack uses a frozen PE-Base backbone, lightweight injectors, 64 memory tokens, and 4000 coordinate bins, with 256 × 256 or 384 × 384 search inputs.The tracking encoder and decoder use 8 and 4 layers, respectively.
- Efficiency Analysis: The frozen VLP design optimizes only 100M parameters and reaches best performance within 80 epochs, reducing optimization steps and compute relative to longer-trained baselines.The reported setup uses 30K samples per epoch and avoids an additional vision-language alignment fine-tuning stage.
- Comparison with State-of-the-art: LVTrack-384 achieves the best TNL2K AUC and Prec (58.7/59.9), the best OTB99 results across all metrics (63.6/83.0/76.3), and second-best LaSOT results (61.3/64.5/71.2) under NL initialization.LVTrack-256 ranks second on OTB99 and reaches 57.6 AUC on TNL2K.
- Comparison with State-of-the-art: Under NL+BBOX initialization, LVTrack-384 obtains the best TNL2K AUC (66.5), the best OTB99 AUC and NPrec (72.4/88.9), and second-best OTB99 Prec (95.3).LVTrack retains NL-only tracking ability while remaining competitive when visual templates are available.
- Comparison with State-of-the-art: LVTrack-256 achieves 74.88% RefCOCOg validation accuracy using the same unified model deployed for tracking, without retraining a dedicated grounding model.The results indicate that MGFI constrains language reliance during tracking without degrading grounding capability.
5 Conclusion
LVTrack unifies grounding and tracking in an autoregressive framework that adaptively regulates language and combines several temporal-localization designs. These choices enable frozen vision-language backbones, reduced training cost, and strong benchmark performance.
- LVTrack unifies grounding and tracking in an autoregressive formulation.
- Its mode-conditioned gated feature injector calibrates textual guidance for language-sensitive grounding while mitigating language-dominated tracking drift.
- GS-KL supervision, hybrid positional encoding, and appearance memory improve localization accuracy and temporal robustness.
- LVTrack directly leverages a frozen vision-language pretrained backbone with substantially reduced training cost and strong benchmark performance.
Limitations
The implementation remains bounded by fixed input resolutions and initialization-time language guidance. Its inference procedure also uses confidence-controlled memory updates and directive stripping to stabilize tracking.
- LVTrack relies on fixed input resolutions and does not fully explore arbitrary-resolution inputs.
- Language guidance is mainly provided at initialization, leaving dynamically updated instructions and additional tracking constraints for future work.
- Under language initialization, grounding creates the first-frame box and template, while later frames use tracking-mode autoregressive decoding in a local search region.
- Appearance memory updates occur only when aggregated coordinate-head confidence is sufficiently high, reducing unreliable template refreshes under occlusion or distraction.
- Directive words are retained for first-frame grounding but stripped and re-encoded for subsequent tracking frames.
B Hyperparameter Analysis
The hyperparameter analysis identifies balanced settings for feature selection, language gating, localization smoothing, and data sampling. It also shows that backbone choice alone does not explain LVTrack’s advantage over prior fusion pipelines.
- Backbone and feature selection: The frozen VLP ViT uses an intermediate layer because later layers increasingly compress patch information into global representations.
- Amplitude ratio: ρ = 0.6 achieves the best balance across tracking and grounding tasks in the amplitude ratio loss.
- Amplitude ratio: A looser ρ = 0.7 constraint increases language influence during tracking and lowers OTB99 from 63.3 to 61.1.
- GS-KL bandwidth: σ = 4 provides the best trade-off between locality and sharpness in GS-KL supervision and is adopted by default.
- Unified sampling ratios: The sampling ratio [0.45, 0.40, 0.15] gives the strongest tracking performance while maintaining high grounding accuracy.
- VLP backbone comparison: PE’s advantage is linked to its progressive-resolution training, 2D RoPE, and deeper text encoder, while replacing prior trackers’ backbones with PE still degraded tracking.
D MGFI Perturbation Analysis
The MGFI perturbation analysis bounds how language injection can alter tracking predictions. Under the stated local Lipschitz assumption, gate norms directly upper-bound language’s impact on tracking decisions.
- MGFI analysis models decoder-output changes as perturbations caused by language injection into visual features.
- Assuming the decoder is locally L_h-Lipschitz, joint-tracking deviation is bounded by the perturbation magnitude and residual terms.
- Gate norms directly upper-bound the maximum impact of language on tracking decisions.
E Positional Encoding and Position Modeling Details
LVTrack uses hybrid positional modeling to preserve both a shared global coordinate reference and geometric structure between template and search-region features. The scheme maintains a 2D grid and derives lower-resolution template positions from the search-region grid.
- Position Modeling: LVTrack combines shared 2D absolute positional encoding with 2D axial rotary positional embedding for visual feature interaction.The absolute component provides a unified coordinate reference, while the rotary component models relative geometry.
- Shared Coordinate System: The method preserves the 2D image-grid structure instead of flattening patches into a 1D sequence.Template and search features use a shared canonical token-grid coordinate system.
- Absolute Encoding: The learnable absolute positional grid corresponds to the search feature map, which contains 16 × 16 tokens in the model.Search-region positions are directly indexed from this grid.
- Template Encoding: Template positional codes are bilinearly interpolated from the search-region grid because the template uses a smaller 8 × 8 feature map.This treats template positions as a resampled subset of the search-region coordinate system.
E.2 2D Axial Rotary Positional Embedding
LVTrack applies axial 2D rotary positional embedding to attention queries and keys so relative positions are represented along both image axes. Its hybrid absolute-relative design preserves global coordinate consistency and local translation geometry.
- 2D Axial RoPE: 2D RoPE splits feature channels into height and width halves, encoding relative positions independently along the y- and x-axes.The transformed axial components are concatenated for each spatial token.
- Relative Geometry: The rotary transformation is applied independently to template and search tokens, preserving each region’s intra-frame local topology.This supports relative spatial reasoning within both visual regions.
- Ablation: 2D RoPE preserves neighborhood structure relevant to object translation better than 1D sequence encoding.In 1D, vertical neighbors can be distant in sequence order, whereas 2D RoPE represents offsets along both spatial axes.
- Design Rationale: The hybrid strategy uses Pabs for a globally unified coordinate reference and PRoPE for translation-invariant local geometry.Together, they facilitate feature interaction between the template and search region.
F BBOX-initialized Tracking Performance
LVTrack also supports conventional box-initialized tracking by bypassing language injection while retaining the core tracking pipeline. Under this protocol, it achieves competitive results across TNL2K, LaSOT, and OTB99.
- Protocol: In the BBOX-initialized setting, the first-frame ground-truth box initializes tracking and the language injector is bypassed.The core tracking pipeline remains unchanged without linguistic conditioning.
- Results: 62.9/80.4 gives LVTrack-384 the second-best AUC and NPrec on TNL2K.These values are reported as AUC/NPrec under BBOX initialization.
- Results: 78.9 is LVTrack-384’s second-best NPrec on LaSOT, while 71.0 is its second-best AUC on OTB99.The comparison covers representative box-initialized trackers and recent vision-language trackers.
- Results: 93.9/87.8 gives LVTrack-384 the best Prec and NPrec on OTB99.LVTrack-256 also ranks second on OTB99 Prec and NPrec at 93.8/87.6.
- Overall Comparison: LVTrack remains competitive with recent vision-language trackers and surpasses most earlier siamese and transformer baselines.SAVLT-B leads LVTrack on LaSOT under the pure BBOX protocol.
G More Visualizations
Additional visualizations illustrate LVTrack’s behavior under language-guided grounding, appearance variation, occlusion, and distractors. The examples show accurate localization, robustness to changing appearance, and successful re-localization after target loss.
- Challenging Cases: In a kangaroo example, LVTrack accurately localizes the target while competing trackers drift to nearby persons.The failure of competing trackers is attributed to incorrect language-guided grounding in the cited passage.
- Challenging Cases: LVTrack remains robust to a flying kite whose appearance and shape vary continuously over time.The visualization is presented as evidence of high-level semantic understanding and adaptation capability.
- Challenging Cases: When a squirrel reappears after severe distractors and occlusion, LVTrack successfully re-localizes and locks onto the target.The target is described as almost completely lost before reappearance.
- Figure Overview: Figure 7 provides additional tracking visualizations under NL initialization.The cited examples cover occlusion and appearance variations.