Source-linked AI summary
Referring Multi-Object Tracking in Moving-Camera Videos via Global Motion Compensation
Hsin-Chen Pai, Jyun-Kai Wang, Yi-Cheng Peng, Wei-Ta Chu
TL;DR
Moving-camera video can make object motion difficult to distinguish from camera motion, and RMOT has not fully integrated compensated motion cues with language. The paper adds residual-motion matching through late fusion as a plug-in; across three host methods it improves pooled HOTA in all settings, with a moving-class gain on iKUN.
Problem
Moving-camera video mixes camera and object motion, while integrating compensated motion cues with language for RMOT remains underexplored.
Source: §1 · p. 1, §2.2 · p. 1Method
The plug-in aligns residual motion with language and late-fuses a weighted motion-matching score with the host RMOT model's matching score.
Source: §1 · p. 1, §3.4 · p. 3Results
Across iKUN, FlexHook, and TransRMOT, the module improves pooled HOTA in all settings; on iKUN, moving-class HOTA rises from 27.697 to 37.139 ± 0.923.
Source: §1 · p. 1, §4.2 · p. 4Takeaways & Limitations
The module improves host RMOT methods without modifying or retraining them.
Source: §1 · p. 1The road-plane homography assumes flat ground, direction and turning expressions are routed to the other-class weight, and evaluation is limited to three frameworks on Refer-KITTI.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Referring multi-object tracking (RMOT) takes a video and a language expression as input and tracks all referred objects. Many tracking requirements involve how an object moves rather than how it appears. However, in a video captured by a moving camera, a parked vehicle may appear to move, while a moving vehicle may show little displacement. Existing RMOT methods relate motion with text but do not explicitly remove camera-induced motion. In this paper, we propose extracting residual motion across frames by estimating camera motion in driver-view videos and compare motion characteristics with the query expression. We consider the motion-matching extent and integrate it with the RMOT method's prediction result through late fusion. In the evaluation, we verify the performance gain of taking the motion compensation module as a plug-in across different RMOT hosts.
1. INTRODUCTION
RMOT motion cues can confuse camera-induced movement with object movement, while integrating motion compensation with language remains underexplored. The paper adds residual-motion matching as a plug-in and reports gains across three host methods.
1. INTRODUCTION
- In moving-camera video, observed motion mixes object and camera motion, so a static car can appear to move while a moving car appears nearly static.Source: §1 · p. 1
- The method estimates camera motion, computes residual velocity across frames, aligns motion features with language, then adds a host-weighted motion score to the host matching score.Source: §1 · p. 1
- The work connects motion compensation with motion-language modeling for RMOT.Source: §1 · p. 1
- The plug-in aligns an object's residual velocity with the referring expression and improves its host RMOT method without modification or retraining.Source: §1 · p. 1
- Across iKUN, FlexHook, and TransRMOT, the module improves pooled HOTA in all settings; on iKUN, moving-class HOTA rises from 27.697 to 37.139 ± 0.923.The reported iKUN increase is approximately +9.44 points.Source: §1 · p. 1
2. RELATED WORKS
Prior RMOT work matches language to tracked objects and some methods model motion from bounding-box movement. Conventional MOT compensates for camera motion, but integrating compensated cues with language for RMOT remains underexplored.
2.1. Referring Multi-Object Tracking
- Recent RMOT methods match object descriptions, with VMRMOT explicitly representing motion through position, direction, distance changes, and speed changes from bounding-box movement.Source: §2.1 · p. 1
2.2. Motion Compensation in Tracking
- MOT methods estimate or separate camera motion, but combining compensated motion cues with language for RMOT remains underexplored; the proposed framework adds a GMC matching score to adjust the host score.Source: §2.2 · p. 1, Figure 1 · p. 2
3. METHOD
The method attaches a global motion compensation module to a host RMOT method, estimates camera motion, extracts residual object motion, aligns it with language, and fuses its score with the host prediction. Its stages cover motion compensation, multi-scale residual velocity extraction, motion-language matching, and late fusion.
3.1. Motion Compensation
- The method estimates frame-to-frame homographies from tracked road-surface keypoints, accumulates them across frames, and applies the accumulated transformation once to object centroids to reduce numerical drift.Source: §3.1 · p. 2
3.2. Multi-Scale Residual Velocity
- Residual velocity subtracts camera-induced velocity from raw velocity; measurements at 2-, 5-, and 10-frame gaps are combined with box data into a 12-dimensional motion feature.Source: §3.2 · pp. 2–3
- A track contributes a motion feature only after 11 contiguous frames; for shorter histories, the module abstains and the host score passes through unchanged.Source: §3.2 · p. 3
3.3. Motion-Language Matching
- A motion encoder and fixed MiniLM text encoder project motion and language into a shared feature space, trained with InfoNCE; cosine similarity supplies the motion-matching score.Source: §3.3 · p. 3
3.4. Fusion
- The motion-matching score is weighted and added to the host prediction score; the weight is selected per host by LOSO, though FlexHook uses one weight across expression classes.Source: §3.4 · p. 3
- The GMC module can augment two-stage or end-to-end query-based RMOT frameworks without changing or retraining their detector, tracker, visual encoder, or language encoder.Source: §3.4 · p. 3
4. EXPERIMENTS
Across Refer-KITTI settings, the GMC module improves pooled and moving-class tracking scores across host models. Ablations show that ego-motion compensation and multi-scale residual-velocity gaps contribute to the moving-class gains.
4.1. Setup
- The evaluation uses HOTA on Refer-KITTI V1 and V2; fusion weights are selected by leave-one-sequence-out, and moving-class HOTA is primary because moving expressions are a minority.The V1 and V2 test splits contain three and four sequences, respectively.Source: §4.1 · p. 3
4.2. Main results
- GMC improves pooled HOTA across all host settings; fused iKUN and both FlexHook settings also exceed their published scores, while TransRMOT is compared with our reproduction.Table 2 reports pooled HOTA, DetA, and AssA; the TransRMOT comparison uses the host’s own frame convention.Source: Table 2 · p. 4, §4.2 · p. 4
- Moving-class HOTA improves in all four settings, with gains decreasing as baseline moving-class HOTA rises across three architectures.Table 3 reports moving-class HOTA, based on 21/150 expressions for iKUN, TransRMOT, and FlexHook V1, and 111/862 for FlexHook V2.Source: Table 3 · p. 4, §4.2 · p. 4
4.3. Ablation Studies
- On iKUN, removing ego-motion compensation lowers moving-class HOTA from 36.99 to 32.37, while removing multi-scale gaps lowers it to 35.14; both drops are significant (p<0.01).The five-seed ablation also reports pooled HOTA of 44.54 for the host and 45.28 for the full module.Source: §4.3 · p. 4, Table 4 · p. 4
- In the example, ego-compensated residuals distinguish the parked car’s camera-induced sweep from the moving car’s small image displacement.Without compensation, the two cases are confused; after compensation, the parked car has nearly zero residual velocity and the moving car nonzero residual velocity.Source: Figure 2 · p. 4, §4.3 · p. 4
5. CONCLUSION AND LIMITATIONS
The GMC plug-in separates object motion from camera motion and improves pooled HOTA across different host models on Refer-KITTI. Its scope is limited by the road-plane assumption, expression routing, and evaluation on Refer-KITTI alone.
5. CONCLUSION AND LIMITATIONS
- On moving-class expressions, iKUN shows the largest gain (+9.44 ± 0.92), while pooled HOTA improves for different host models on Refer-KITTI.The module is evaluated with different RMOT hosts.Source: §5 · p. 4
- The plug-in uses road-region correspondences to estimate camera motion, matches residual motion features with language, and late-fuses its score with the host prediction.It does not modify the detector, tracker, or appearance-language matching model.Source: §5 · p. 4
- The road-plane homography assumes flat ground, placing scenes with steep slopes or uneven terrain outside the model’s scope.Source: §5 · p. 5
- Direction and turning expressions (214 of 818 in Refer-KITTI V1) receive αoth rather than αmot, so the module does not help them.Source: §5 · p. 5
- Evaluation covers three RMOT frameworks but only Refer-KITTI benchmarks; the authors call for evaluation on other datasets to validate generality.Source: §5 · p. 5