PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous Ground-Truth Construction and a Recovery Policy for Air--Water Robotic Tracking

Jiangong Xiao, Zhe Sun, Kanzhong Yao, Yuanbo Bi, Haofei Zhao, Ruixuan Hu, Guan Huang, Xuelong Li

arXiv:2610.11096v1cs.CVcs.AI

TL;DR

Interface disturbances can temporarily hide or distort targets, while unreliable observations can corrupt online template updates and propagate tracking errors. The paper builds continuous physical-position labels from synchronized motion-capture poses and introduces CMRP, which selects fixed, pre-trigger, and trigger-frame references without retraining MixFormerV2’s visual backbone. 49.90 Macro Success AUC, 2.95 points above MixFormerV2 Official, while selected-interval coverage rises from 47.91% to 50.43% and mean recovery latency falls from 55.3 to 49.3 frames.

  • Problem

    Interface disturbances can temporarily hide or distort targets, while unreliable observations can corrupt online template updates and propagate tracking errors.

    Source: §I · p. 1

  • Method

    The paper builds continuous physical-position labels from synchronized motion-capture poses and introduces CMRP, which selects fixed, pre-trigger, and trigger-frame references without retraining MixFormerV2’s visual backbone.

    Source: §I · p. 2

  • Results

    49.90 Macro Success AUC, 2.95 points above MixFormerV2 Official, while selected-interval coverage rises from 47.91% to 50.43% and mean recovery latency falls from 55.3 to 49.3 frames.

    Source: E. Standard Cross-Medium Tracking… · p. 6, F. Tracking Coverage and Recovery · p. 6

  • Takeaways & Limitations

    The test set supports analysis of localization, drift, and stable recovery against continuous physical-position references on cross-medium sequences.

    Source: D. Manual Review and Evaluation Set · p. 4

    The policy decreases AUC by 0.35 and 0.42 points on LaSOT and UAV123, respectively.

    Source: G. Stability on Conventional Tracking… · p. 6

Abstract

from arXiv · show

Visual tracking across the air-water interface is challenged by splashes, bubbles, refraction, reflections, and abrupt appearance changes that can temporarily invalidate observations. This setting poses two coupled difficulties: first, for evaluation, image-only annotation cannot reliably describe the target's physical location during visual blindness; second, for online tracking, corrupted observations can contaminate motion estimates and appearance templates. We address the first difficulty with a construction pipeline that synchronizes camera frames with motion-capture poses, projects known target geometry, corrects underwater projection with a medium-gated residual, and subjects the annotations to manual review. This yields an evaluation-only cross-medium test set of 22,346 frames. We further introduce a Cross-Medium Recovery Policy (CMRP) centered on confidence-triggered template selection. It supplies MixFormerV2 with the fixed initial template, a window-best pre-trigger template, and a trigger-frame Kalman-guided image crop, together with their associated weights, without retraining the visual backbone. In the accuracy evaluation, CMRP achieves 49.90 Macro Success AUC, 2.95 points above MixFormerV2 Official. On selected cross-medium transition and occlusion-recovery intervals, CMRP increases MixFormerV2 tracking coverage from 47.91\% to 50.43\% relative to Official updating, while mean loss-to-recovery latency over successfully recovered videos decreases from 55.3 to 49.3 frames.

I. INTRODUCTION

Air–water transitions make image-based localization unreliable precisely when continuous target position is needed for evaluation. The paper constructs continuous physical-position labels and proposes a confidence-triggered recovery policy for online tracking.

I. INTRODUCTION

  • Splashes, bubbles, reflections, and refraction can distort or hide a target that remains physically present and moving.Image-only labels can be ambiguous during partial visibility and cannot reliably establish physical position when the target is fully obscured.Source: §I · p. 1
  • Motion-capture projection and manual review produce continuous physical-position boxes for an evaluation-only cross-medium set of 22,346 frames.The pipeline synchronizes camera frames and target poses, projects known geometry, and applies a medium-gated residual correction.Source: §I · pp. 1–2
  • CMRP combines the fixed initial template, a window-best pre-trigger template, and a trigger-frame Kalman-guided crop without retraining MixFormerV2’s visual backbone.The selected references are retained during sustained low confidence, with motion prediction and recovery search supporting position estimates.Source: §I · p. 2

A. Cross-Medium Robotic Tracking

  • Prior air–water robotics studies address mobility and physical transition constraints, while this work adds continuous target annotations and tracking evaluation through interface-induced degradation –, –.The cited examples include AquaMAV and Dipper,, flapping-wing robots,, and a morphing-propeller platform.Source: A. Cross-Medium Robotic Tracking · p. 2

B. Visual Object Tracking

The section situates CMRP among visual trackers and template-adaptation methods, then describes the cross-medium evaluation setting and its accuracy metric.

B. Visual Object Tracking

  • Transformer trackers model target-template and search-region relations, while template-adaptation methods select or combine appearance references over time.Examples include OSTrack, MixFormer, MixFormerV2, ODTrack, UpdateNet, and local trusted templates.Source: B. Visual Object Tracking · p. 2
  • CMRP focuses on selecting pre-trigger appearance and a motion-guided crop at low confidence, then retaining them during sustained low confidence.Earlier clear appearances may poorly represent changing appearance, while repeated updates during severe degradation risk contamination.Source: B. Visual Object Tracking · p. 2

C. Underwater Perception and Cross-Medium Tracking Data

  • Cross-medium sequences add splashes, reflections, bubbles, partial entry, and abrupt exposure changes; evaluation therefore needs continuous physical-position references when targets are obscured.The section distinguishes conventional benchmarks, and underwater-specific benchmarks, from this cross-medium need.Source: C. Underwater Perception and… · p. 2

III. PROBLEM FORMULATION AND EVALUATION

  • The online tracker predicts a box and model confidence from the initial box and image stream, without motion-capture poses, medium labels, or future frames.Macro Success AUC averages sequence-level success AUC values equally; success rate counts frames whose IoU reaches a threshold.Source: §III · p. 2, A. Standard Tracking Accuracy · p. 2

B. Tracking Coverage and Recovery

The section defines frame-level tracking coverage and video-level recovery latency for evaluating tracking through cross-medium transitions and occlusions.

B. Tracking Coverage and Recovery

  • Coverage is the share of 9,166 selected transition and occlusion–recovery frames with predicted-box IoU above 0.5.The ground-truth comparisons are used only for offline evaluation.Source: B. Tracking Coverage and Recovery · p. 2
  • The framework separates offline continuous-label construction from online confidence-triggered template selection and motion guidance, which do not use motion-capture measurements.The online branch has no access to the offline pose measurements.Source: Figure 2 · p. 3
  • Recovery timing starts at tracking loss and uses the first frame of a five-frame run in which overlap and tracker confidence both exceed 0.5.The run must be confirmed before its first frame is assigned as the recovery time.Source: B. Tracking Coverage and Recovery · p. 3
  • Mean recovery latency averages one first recovery per successfully recovered video, excluding videos without recovery.Timing begins at tracking loss and may include part of the invisible interval.Source: B. Tracking Coverage and Recovery · p. 3

IV. CONTINUOUS GROUND TRUTH AND TEST-SET CONSTRUCTION

The construction pipeline uses motion capture to maintain a physical target reference independently of image visibility, then projects synchronized poses and known geometry into camera images.

IV. CONTINUOUS GROUND TRUTH AND TEST-SET CONSTRUCTION

  • Motion capture serves as an offline measurement instrument for retaining a physically grounded target location when image observations are unreliable.It is used for label construction and is not available to the evaluated trackers.Source: §IV · p. 3

A. Acquisition Setup and Pose Measurement

  • Rigidly mounted marker clusters beneath the camera platform and target vehicle support continuous pose measurement through air–water transitions.Known fixed extrinsics map the measured marker-body poses to camera optical-frame and target-body-frame poses, which are synchronized with images.Source: A. Acquisition Setup and Pose… · p. 3, Figure 3 · p. 4

B. Temporal Synchronization and Coordinate Projection

  • Camera images and motion-capture poses share a monotonic clock and are associated by nearest timestamp before transforming the reference point and target geometry into camera coordinates.The calibrated camera model projects these into the image, transferring measured six-degree-of-freedom pose without using target appearance.Source: B. Temporal Synchronization and… · p. 4

C. Medium-Gated Residual and Geometry-Based Box

A medium-gated residual translates geometry-projected boxes to correct systematic underwater displacement, and reviewers inspect and correct candidate annotations. The resulting evaluation-only set retains continuous reviewed boxes for cross-medium sequences.

C. Medium-Gated Residual and Geometry-Based Box

  • The underwater-gated residual translates the geometry-projected box to correct candidate position, while width and height remain determined by known geometry and measured pose.The shared-translation approximation does not explicitly model differential refractive displacement across the target; the final box follows manual review.Source: C. Medium-Gated Residual and… · p. 4

D. Manual Review and Evaluation Set

  • Reviewers inspect the synchronized image, projected box, target trajectory, and frame-to-frame consistency, then correct misalignment and flag invalid measurements.Projected boxes are annotation candidates, not automatically accepted labels.Source: D. Manual Review and Evaluation Set · p. 4, C. Medium-Gated Residual and… · p. 4
  • The evaluation-only cross-medium set contains 18 videos and 22,346 frames at 60 FPS, focused on stages surrounding air–water transitions.It is not used to train or fine-tune trackers; all compared methods use the same images, first-frame boxes, and reviewed annotations.Source: D. Manual Review and Evaluation Set · p. 4

V. CMRP: RECOVERY-ORIENTED TRACKING

CMRP is an inference-time strategy for multi-template tracking, with confidence-triggered template selection at its center. Reliability-gated motion feedback uses a Kalman filter to accept or reject visual measurements.

V. CMRP: RECOVERY-ORIENTED TRACKING

  • CMRP is an inference-time strategy for multi-template trackers, described using MixFormerV2 without backbone retraining; confidence-triggered template selection is its central design.Motion feedback, search adaptation, and recovery verification provide supporting control.Source: §V · p. 4

A. Reliability-Gated Motion Feedback

  • An eight-dimensional constant-velocity Kalman filter represents target center, scale, and velocity and produces a motion-box prediction.The visual measurement is evaluated using its innovation and Mahalanobis distance.Source: A. Reliability-Gated Motion Feedback · p. 4
  • A visual result is accepted only when confidence, innovation distance, normalized center displacement, and logarithmic scale change jointly pass their gates.The confidence gate tests appearance evidence; the remaining gates check temporal consistency.Source: A. Reliability-Gated Motion Feedback · p. 4

B. Uncertainty-Adaptive Search

Uncertainty-adaptive search expands around the motion prediction as position uncertainty and unreliable-tracking duration increase. At a low-confidence trigger, CMRP selects retained templates using pre-trigger confidence and a Kalman-predicted crop.

B. Uncertainty-Adaptive Search

  • Search is centered on the motion prediction, and its expansion factor increases with position uncertainty and the duration of unreliable tracking.Bounds limit expansion during prolonged unreliable tracking.Source: B. Uncertainty-Adaptive Search · pp. 4–5
  • When local search remains unreliable, wider redetection ranks candidates by visual confidence, motion consistency, and similarity to retained appearance references.The highest-ranked candidate must remain reliable for multiple consecutive frames before stable tracking is restored; localization does not promote it to a new template.Source: B. Uncertainty-Adaptive Search · p. 5

C. Confidence-Triggered Template Update

  • At the first low-confidence frame, CMRP selects the highest-confidence historical template from the preceding window, excluding the trigger frame.The three template sources include the fixed initial template, the window-best pre-trigger template, and a crop at the trigger-frame Kalman-predicted box; the latter is not assumed to be a clean view.Source: C. Confidence-Triggered Template Update · p. 5
  • The initial template anchors identity, while the window-best template supplies a pre-trigger appearance reference and the Kalman crop provides an auxiliary observation.The selected templates are used from the next frame and retained during the same low-confidence episode while Kalman prediction guides position.Source: C. Confidence-Triggered Template Update · p. 5

D. Scale Stability

A weak soft scale constraint limits abnormal box-scale feedback while retaining legitimate variation. The experiments assess projection accuracy and tracking, without giving trackers motion-capture measurements.

D. Scale Stability

  • A weak soft scale constraint uses recent reliable tracking history to suppress abnormal feedback while retaining legitimate scale variation.It addresses implausible box-scale jumps around medium transitions.Source: D. Scale Stability · p. 5

VI. EXPERIMENTS

  • The experiments examine projection accuracy for label construction, cross-medium tracking accuracy and recovery, and CMRP on conventional benchmarks.Trackers receive no motion-capture measurements during these experiments; Table I is titled as motion-capture projection error in pixels.Source: §VI · p. 5, Table I · p. 5

A. Experimental Setup and Metrics

The experiments assess label projection and tracker behavior across domains under shared evaluation protocols. The medium-gated residual reduces projection error, while ordinary template-update policies show domain-dependent AUC effects.

A. Experimental Setup and Metrics

  • Projection validation uses 14,984 control points from 722 images, while tracking comparisons share a fixed MixFormerV2-Base checkpoint and first-frame initialization.Macro Success AUC equally weights sequence-level AUC; accuracy is evaluated on all 18 cross-medium videos and 22,346 frames.Source: A. Experimental Setup and Metrics · p. 5

B. Projection Accuracy of Label Construction

  • The medium-gated MLP residual reduces median projection error by 91.6% and P95 error by 93.3% relative to direct geometric projection.The corrected projections provide annotation candidates for manual review; checkpoint selection uses the same fixed validation set, so independent-set generalization is not claimed.Source: B. Projection Accuracy of Label… · p. 5

C. Controllable Cross-Medium Image Acquisition Platform

D. Template-Update Behavior Across Domains

  • Official updating raises LaSOT AUC by 1.31 points over Static but lowers cross-medium AUC by 0.36; Fixed is lowest in all three domains.These results motivate adapting template-update timing to degraded observations.Source: D. Template-Update Behavior Across… · p. 6

E. Standard Cross-Medium Tracking Accuracy

CMRP improves MixFormerV2's complete-set cross-medium accuracy over Official updating and raises selected-interval coverage while shortening recovery latency. On LaSOT and UAV123, it incurs small AUC decreases.

E. Standard Cross-Medium Tracking Accuracy

F. Tracking Coverage and Recovery

G. Stability on Conventional Tracking Data

H. Towards Robotic Deployment

The section reports preliminary outdoor acquisition tests and embedded execution, alongside cross-medium recovery results and planned validation.

H. Towards Robotic Deployment

VII. CONCLUSION

  • 2.95 points higher Macro Success AUC than MixFormerV2 Official updating, with mean loss-to-recovery latency falling from 55.3 to 49.3 frames among successfully recovered videos.Source: §VII · p. 8
  • The evaluation-only set contains 18 videos and 22,346 frames with continuous physical-position references.Source: §VII · p. 8
  • The reported gains include small AUC decreases on LaSOT and UAV123; future work will broaden evaluation across targets and environments and validate closed-loop formation control.Source: §VII · p. 8
Loading 2610.11096v1…