PaperlayerView exact revision on arXiv

Source-linked AI summary

Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance

Liting Lin, Heng Fan, Zhipeng Zhang, Yaowei Wang, Yong Xu, Haibin Ling

arXiv:2403.05231v2cs.CV

TL;DR

Visual tracking needs large computational resources, while direct LoRA transfer faces positional-encoding and convolutional-head challenges. LoRAT adapts a one-stream ViT tracker with shared spatial and token type embeddings plus an MLP-only head, achieving strong benchmark performance with reduced resource demands.

  • Problem

    Applying parameter-efficient fine-tuning to visual tracking is under-explored and complicated by separate template/search positional encodings and convolutional-head inductive biases.

  • Method

    LoRAT adapts LoRA to a one-stream tracker using shared positional embeddings, independent token type embeddings, and an MLP-only head.

  • Results

    LoRAT-g-378 achieves 0.762 SUC on LaSOT, while LoRAT-L-224 reaches 0.742 SUC and LoRAT-B-224 reaches 0.717 SUC.

  • Takeaways & Limitations

    LoRAT achieves state-of-the-art tracking performance while making large ViT trackers trainable with manageable computational resources.

Abstract

from arXiv · show

Motivated by the Parameter-Efficient Fine-Tuning (PEFT) in large language models, we propose LoRAT, a method that unveils the power of large ViT model for tracking within laboratory-level resources. The essence of our work lies in adapting LoRA, a technique that fine-tunes a small subset of model parameters without adding inference latency, to the domain of visual tracking. However, unique challenges and potential domain gaps make this transfer not as easy as the first intuition. Firstly, a transformer-based tracker constructs unshared position embedding for template and search image. This poses a challenge for the transfer of LoRA, usually requiring consistency in the design when applied to the pre-trained backbone, to downstream tasks. Secondly, the inductive bias inherent in convolutional heads diminishes the effectiveness of parameter-efficient fine-tuning in tracking models. To overcome these limitations, we first decouple the position embeddings in transformer-based trackers into shared spatial ones and independent type ones. The shared embeddings, which describe the absolute coordinates of multi-resolution images (namely, the template and search images), are inherited from the pre-trained backbones. In contrast, the independent embeddings indicate the sources of each token and are learned from scratch. Furthermore, we design an anchor-free head solely based on MLP to adapt PETR, enabling better performance with less computational overhead. With our design, 1) it becomes practical to train trackers with the ViT-g backbone on GPUs with only memory of 25.8GB (batch size of 16); 2) we reduce the training time of the L-224 variant from 35.0 to 10.8 GPU hours; 3) we improve the LaSOT SUC score from 0.703 to 0.742 with the L-224 variant; 4) we fast the inference speed of the L-224 variant from 52 to 119 FPS. Code and models are available at https://github.com/LitingLin/LoRAT.

1 Introduction

LoRAT brings parameter-efficient fine-tuning to visual tracking, addressing positional-encoding and convolutional-head issues that hinder direct LoRA transfer. It achieves strong benchmark performance while reducing training and resource requirements.

  • 1 Introduction: LoRAT reduces the cost of tracking large ViT models by fine-tuning a small parameter subset while leaving most pretrained components frozen.LoRA adds trainable low-rank modules without additional inference latency.
  • 1 Introduction: Direct LoRA transfer is hindered by separate template/search positional encodings and convolutional-head inductive biases.The positional design disrupts pretrained structure, while the convolutional head impedes convergence with LoRA.
  • 1 Introduction: The proposed solution uses a one-stream tracker with LoRA and two designs intended to better adapt parameter-efficient fine-tuning to tracking.The baseline minimizes changes to the pretrained ViT and limits additional trainable parameters.
  • 1 Introduction: LoRAT-B-224 can be trained on one RTX 4090 within 11 hours while reaching 605 inference FPS.Across variants, training requires 5.9 to 60 hours on eight Nvidia V100 GPUs.
  • 1 Introduction: LoRAT applies LoRA to one-stream tracking and achieves state-of-the-art performance across multiple benchmarks with manageable resource requirements.The method is evaluated with various pretrained ViT backbones on five large-scale tracking benchmarks.

2 Related Work

The paper situates LoRAT within Transformer-based one-stream tracking and parameter-efficient fine-tuning. Its architecture combines joint template-search processing with LoRA-style adaptation of pretrained models.

  • 2 Related Work: Transformer trackers improve visual tracking by modeling long-range dependencies and accommodating diverse pretrained models.The related-work discussion places one-stream tracking within broader Transformer-based tracking progress.
  • 2 Related Work: One-stream trackers jointly extract and fuse template and search features earlier through Transformer self-attention.MixFormer, SimTrack, and OSTrack are identified as representative one-stream trackers.
  • 2 Related Work: LoRAT freezes most pretrained ViT components while training LoRA modules, token type embeddings, and the head network.The architecture uses an MLP-only head for classification and anchor-free box regression.
  • 2 Related Work: PEFT reduces fine-tuning cost by updating a small parameter subset instead of the full model.Prompt-tuning and adapter-based approaches are the two principal PEFT categories described.
  • 2 Related Work: LoRA approximates weight updates with low-rank matrices that can be merged into pretrained weights without extra inference burden.This makes LoRA an adapter-based PEFT method suited to efficient deployment.

3 Adapting ViT for LoRA-friendly Tracker

The tracker adapts a one-stream ViT architecture for LoRA by preserving pretrained positional structure, adding token type embeddings, and replacing convolutional processing with an MLP-only head. LoRA represents task-specific weight updates through low-rank trainable factors.

  • One-Stream Tracker: The one-stream baseline projects template and search images into tokens, concatenates them, and processes them with a Transformer encoder for joint representation learning.The encoder output is de-concatenated before the search-region representation is sent to the head network.
  • Low-Rank Adaptation: LoRA approximates each weight update with BA, reducing trainable parameters from d × k to (d + k) × r when r is much smaller than the matrix dimensions.The low-rank updates can be added to original weights without increasing inference latency.
  • Decoupled Input Embedding: LoRAT shares spatial positional embeddings across template and search inputs while using token type embeddings to distinguish token sources.This decouples spatial position from token identity and preserves the pretrained ViT structure more closely.
  • Decoupled Input Embedding: The input embeddings combine patch embeddings with positional and token type embeddings before Transformer encoding.Token type embeddings also support foreground-object indication within template tokens.
  • Decoupled Input Embedding: Slicing-based positional adaptation outperforms interpolation-based adaptation in ablation experiments and is therefore used as the primary strategy.The slicing method selects a submatrix of the search-region positional embedding for template tokens.
  • MLP-only Head Network: An MLP-only head is used to mitigate convolutional inductive biases that may impede convergence during LoRA fine-tuning.The head performs target classification and anchor-free bounding-box regression.

4 Experiments

Experiments evaluate LoRAT across five tracking benchmarks, efficiency settings, and component ablations. Results show strong accuracy, reduced trainable parameters, and practical training and inference efficiency.

  • Efficiency comparison: 80 million trainable parameters for LoRAT-g-378 replace full fine-tuning of the 1.1-billion-parameter ViT-g model.The trainable components are LoRA, token type embeddings, and the head network.
  • Ablation experiments: LoRA improves tracker performance across all tested variants and is reported to mitigate catastrophic forgetting during downstream adaptation.The comparison uses full fine-tuning versus LoRA-based fine-tuning on three benchmarks.
  • Ablation experiments: The convolutional head performs competitively under full fine-tuning but fails to converge with LoRA, whereas the MLP-only head is computationally more efficient.The authors attribute this difference to convolutional inductive biases conflicting with low-rank adaptation’s intrinsic-dimension assumption.
  • Backbone pre-training: DINOv2 pre-training consistently outperforms MAE, CLIP, and EVA02 across the tested datasets.The comparison evaluates four ViT pre-training methods as tracker backbones.

5 Conclusion

LoRAT integrates LoRA into one-stream visual tracking to adapt large pre-trained models with reduced resource requirements. It achieves strong benchmark performance while making advanced tracking models more accessible to manageable training resources.

  • The approach is designed to reduce the resource requirements for adapting pre-trained models to tracking.
  • LoRAT achieves state-of-the-art performance on multiple visual-tracking benchmarks.

A More Implementation Details

The implementation crops template-search image pairs, uses IoU-aware classification and point-based box regression, and combines classification and regression losses equally.

  • Input Data: Template and search regions are cropped from input images, with background factors of 2 and 4 or 5 depending on the model variant.The search-region factor is 4 for -224 variants and 5 for -378 variants.
  • Classification Branch: The classification branch assigns foreground scores based on IoU between predicted and ground-truth boxes while targeting zero for background points.This IoU-aware classification score is called IACS.
  • Regression Branch: Each feature-map point predicts left, top, right, and bottom offsets, and the point with the highest classification score supplies the final box.
  • Loss Functions: The overall objective sums binary cross-entropy classification loss and GIoU box-regression loss with equal weights of 1.0.

B Results on Additional Benchmarks

The tracker is additionally evaluated on OTB100, NFS, and UAV123, with results reported in Table 11.

  • The tracker is evaluated on three additional benchmarks: OTB100, NFS, and UAV123.
  • The additional-benchmark results are presented in Table 11.
  • Table 11 reports comparative tracker performance across these three benchmarks.The supplied passage identifies the evaluation scope but does not provide individual scores.

C.1 Impact of Different Pre-training Methods

Pre-training strongly affects LoRA-based tracking, with larger and more strongly pre-trained backbones benefiting most. DINOv2 variants perform best overall, while EVA-02 remains competitive but is slower at inference.

  • Impact of Different Pre-training Methods: LoRA-based fine-tuning performs better on larger ViT-L backbones, according to the Table 12 ablation.
  • Impact of Different Pre-training Methods: DINOv2-pre-trained variants work better with LoRA-based fine-tuning across all datasets, consistent with reduced catastrophic forgetting.
  • Impact of Different Pre-training Methods: DINOv2 backbones consistently outperform other variants under both LoRA-based and full fine-tuning configurations.
  • Impact of Different Pre-training Methods: EVA-02 variants are competitive but have significantly lower inference FPS because of custom structures differing from standard ViT.

C.2 Detailed Comparison with OSTrack

The comparison shows that LoRAT trackers use less training time and GPU memory while achieving higher performance than OSTrack variants.

  • OSTrack requires significantly longer training times and higher GPU memory usage while achieving lower performance than the compared trackers.

C.3 Performance under Different Initialization Methods of LoRA Weights

The experiments compare three LoRA weight initialization strategies on LoRAT-L-224 and identify truncated normal initialization as the strongest overall choice.

  • Truncated normal initialization achieves the best overall performance among the tested LoRA initialization methods.The comparison includes kaiming uniform, gaussian normal, and truncated normal initialization.

C.4 Comparison with other PEFT methods

LoRA outperforms the compared PEFT methods in the tracking framework, with performance depending on where LoRA is inserted and which attention mechanism is used.

  • LoRAqv with r = 4 surpasses all compared PEFT methods using only 1.5% trainable parameters.
  • LoRAall with r = 16 matches full fine-tuning performance using 3.6% trainable parameters.
  • Applying LoRA to both MHSA and MLP produces the best performance among the evaluated LoRA configurations.The configurations include LoRA on attention projections, MLP layers, or every linear layer.
  • Full self-attention consistently outperforms S-MAM under both LoRA-based and full fine-tuning settings.S-MAM prunes template-to-search cross-attention, yet the tracker remains adaptable to it under LoRA fine-tuning.
Loading 2403.05231v2…