PaperlayerView exact revision on arXiv

Source-linked AI summary

SiamCorners: Siamese Corner Networks for Visual Tracking

Kai Yang, Zhenyu He, Wenjie Pei, Zikun Zhou, Xin Li, Di Yuan, Haijun Zhang

arXiv:2104.07303v1cs.CV

TL;DR

Anchor-box trackers require choices about box number, scale, and aspect ratio, as well as IoU-based assignment. SiamCorners predicts target boxes as corner pairs, combining modified corner pooling, layer-wise feature fusion, and a penalty for selecting boxes. It reports state-of-the-art results on five tracking datasets and runs at 42 FPS on an RTX 2080 with ResNet-50.

  • Problem

    Anchor-based trackers require hand-crafted choices of anchor number, size, and aspect ratio, along with IoU-based assignment during training.

    Source: §I · pp. 1–2

  • Method

    SiamCorners is an anchor-free Siamese tracker that uses modified corner pooling to predict target-specific corner pairs and aggregates features across network layers.

    Source: §I · p. 2

  • Results

    SiamCorners achieves state-of-the-art results on five tracking datasets and runs at 42 FPS with ResNet-50 on a single NVIDIA RTX 2080 GPU.

    Source: §I · p. 2, §IV · p. 6

  • Takeaways & Limitations

    SiamCorners directly tracks targets as corner pairs without predefined anchor boxes and evaluates its effectiveness and efficiency across five datasets.

    Source: §V · p. 10

Abstract

from arXiv · show

The current Siamese network based on region proposal network (RPN) has attracted great attention in visual tracking due to its excellent accuracy and high efficiency. However, the design of the RPN involves the selection of the number, scale, and aspect ratios of anchor boxes, which will affect the applicability and convenience of the model. Furthermore, these anchor boxes require complicated calculations, such as calculating their intersection-over-union (IoU) with ground truth bounding boxes.Due to the problems related to anchor boxes, we propose a simple yet effective anchor-free tracker (named Siamese corner networks, SiamCorners), which is end-to-end trained offline on large-scale image pairs. Specifically, we introduce a modified corner pooling layer to convert the bounding box estimate of the target into a pair of corner predictions (the bottom-right and the top-left corners). By tracking a target as a pair of corners, we avoid the need to design the anchor boxes. This will make the entire tracking algorithm more flexible and simple than anchorbased trackers. In our network design, we further introduce a layer-wise feature aggregation strategy that enables the corner pooling module to predict multiple corners for a tracking target in deep networks. We then introduce a new penalty term that is used to select an optimal tracking box in these candidate corners. Finally, SiamCorners achieves experimental results that are comparable to the state-of-art tracker while maintaining a high running speed. In particular, SiamCorners achieves a 53.7% AUC on NFS30 and a 61.4% AUC on UAV123, while still running at 42 frames per second (FPS).

I. INTRODUCTION

SiamCorners replaces hand-crafted anchor boxes with an anchor-free tracker that predicts targets as pairs of corners. It combines target-specific corner prediction, multi-layer feature fusion, and a penalty for selecting tracking boxes.

I. INTRODUCTION

  • Anchor-based trackers require many boxes and IoU-based sample assignment, which can create positive-negative imbalance and slow training.The passage notes examples of more than 2k anchors in SiamRPN and more than 3k in SiamRPN++.Source: §I · p. 1
  • Fixed anchor counts, sizes, and aspect ratios require tuning and can hinder tracking targets with large shape changes, especially small targets.Source: §I · p. 2
  • A new penalty term considers adjacent-frame changes in target center, width, and height when selecting tracking boxes.Source: §I · p. 2
  • SiamCorners discards anchor boxes and predicts target boxes as corner pairs, avoiding hand-crafted anchor configurations.The method uses modified corner pooling to integrate target-specific information into corner prediction.Source: §I · p. 2
  • Multi-layer feature fusion combines low-level and high-level information for SiamCorners.Source: §I · p. 2
  • SiamCorners is reported to achieve state-of-the-art results on five tracking datasets, with ablations on UAV123 and LaSOT.The datasets are OTB100, UAV123, LaSOT, NFS30, and TrackingNet.Source: §I · p. 2

II. RELATED WORKS

Prior Siamese trackers use similarity or proposal-based approaches, while anchor-free trackers use points and regression. SiamCorners instead predicts target-specific corner pairs in a Siamese architecture, using features from multiple network layers.

II. RELATED WORKS

  • SiameseFC predicts template-search similarity, CFNet adds correlation filters, and SiamRPN uses anchor-based classification and regression.DaSiamRPN addresses imbalanced training data through a sampling strategy.Source: §II · p. 2
  • SiamCorners uses multiple modules over deep layer-wise features to predict candidate corners, unlike shallow Siamese trackers or anchor-based SiamRPN++.The passage contrasts this layer-wise use of deep features with shallow networks and SiamRPN++’s deeper backbone.Source: §II · p. 3
  • FCAF and Ocean use hand-crafted points, classification to select target locations, and regression to predict distances to box sides; SiamCorners predicts corner pairs without that classification phase.Source: §II · p. 3
  • CornerNet predicts bounding boxes as top-left and bottom-right corners, while related detectors use heatmaps, corner matching, centers, or centripetal shifts.Source: §II · p. 3

A. Overview

  • SiamCorners applies the corner-pair approach to tracking with a Siamese network that integrates target-specific information.The overview uses parallel top-left and bottom-right branches after the Siamese feature extractor.Source: A. Overview · p. 3
  • Four cropped boundary images feed ResNet-50; conv3, conv4, and conv5 features are correlated and passed through corner pooling to produce heatmaps and offsets.The top and left boundaries feed the top-left branch; bottom and right boundaries feed the bottom-right branch.Source: A. Overview · p. 3

B. Tracking Corners Loss

SiamCorners trains corner heatmaps and offsets for top-left and bottom-right target corners. Its training objective uses a radius-based reduction of negative-location penalties and smooth L1 offset loss to address corner localization.

B. Tracking Corners Loss

  • The network predicts top-left and bottom-right corner heatmaps, with a search-image scale factor mapping locations to the heatmaps.Source: B. Tracking Corners Loss · p. 3
  • For each ground-truth corner, negative locations within a radius defined by a minimum IoU of 0.5 receive reduced penalty through an unnormalized Gaussian heatmap.The Gaussian spread parameter is one-third of the radius.Source: B. Tracking Corners Loss · p. 4
  • A variant of focal loss trains corner heatmaps, reducing penalty for predicted corners within the positive location’s radius.Source: B. Tracking Corners Loss · p. 4
  • An offset network adjusts corner locations before remapping from heatmaps to the original image to address network-stride discretization error.The predicted and ground-truth offsets are trained with smooth L1 loss.Source: B. Tracking Corners Loss · p. 4

C. Corner Pooling

SiamCorners adapts corner pooling to Siamese template-search features to predict corner heatmaps and offsets. Its modified pooling uses separate boundary features and pooling branches, unlike CornerNet’s unified-feature design.

C. Corner Pooling

  • The module correlates template and search features at multiple network levels before applying modified corner pooling to predict corner heatmaps and offsets.The feature maps come from conv3, conv4, and conv5 branches.Source: C. Corner Pooling · p. 5
  • For bottom-right corners, bottom and right boundary features are max-pooled along their respective axes, and the two pooling results are summed.Top-left pooling analogously combines horizontal and vertical pooling results.Source: C. Corner Pooling · p. 5
  • Unlike CornerNet’s bottom-right pooling on one unified feature map, SiamCorners pools separate bottom and right boundary features, each with a projection shortcut.The modified corner-pooling structure also uses a residual-block variant and predicts heatmaps and offsets.Source: C. Corner Pooling · p. 5

D. Decoding for Corner Heatmaps and Offsets

Decoding converts top-left and bottom-right heatmaps and offsets into refined corner candidates. The method averages paired corner scores and retains these candidates for later fusion and selection.

D. Decoding for Corner Heatmaps and Offsets

  • After sigmoid activation and non-maximum suppression, the decoder selects the 15 highest-scoring locations from each corner heatmap and uses their offsets to refine coordinates.The selected locations and corresponding offsets form the refined corner candidates.Source: D. Decoding for Corner Heatmaps and… · p. 5
  • Each candidate box receives the average of its top-left and bottom-right corner scores, and the resulting 3N corner sets are passed to multi-layer feature fusion and penalty-based selection.The corner-set representation contains coordinates and a score.Source: D. Decoding for Corner Heatmaps and… · p. 5

E. Multi-level Features Fusion and Corners Selection

SiamCorners fuses corner candidates across network levels and scores them with penalties for changes in target scale, aspect ratio, and motion. It selects the highest-scoring corner pair and interpolates target size for smoother changes.

E. Multi-level Features Fusion and Corners Selection

  • Multi-layer feature fusion maps corner coordinates and scores from each level back to the original image, forming a 3N × 5 candidate set.The fused features draw on low- and high-level representations.Source: E. Multi-level Features Fusion and… · p. 5
  • A scale penalty suppresses candidate boxes whose padded overall scale changes sharply from the previous frame.Scale is computed from box width and height with padding defined as half their sum.Source: E. Multi-level Features Fusion and… · p. 6, Figure 4 · p. 6
  • An aspect-ratio penalty uses the current and previous height-to-width ratios to suppress large ratio changes.The penalty is applied to corner scores.Source: E. Multi-level Features Fusion and… · p. 6
  • A motion-based index ranks candidates by changes in target center, width, and height between adjacent frames, and contributes to their final scores.The score combines this index with the other score terms.Source: E. Multi-level Features Fusion and… · p. 6
  • The highest-scoring corner pair becomes the tracking box, and linear interpolation updates target size so its shape changes smoothly.The procedure is summarized as one-shot tracking: extract features, correlate, pool, decode, fuse, score, and post-process.Source: E. Multi-level Features Fusion and… · p. 6

IV. EXPERIMENTAL RESULTS

The reported implementation runs SiamCorners at 42 FPS with a ResNet-50 backbone on an RTX 2080 GPU. The supplied experimental figure concerns OTB100, but these passages do not report its results.

IV. EXPERIMENTAL RESULTS

  • 42 FPS on a single NVIDIA RTX 2080 GPU with ResNet-50 as the backbone is the reported tracking speed.The method was implemented in Python using PyTorch.Source: §IV · p. 6
  • Figure 5 presents experimental results for methods on the OTB100 dataset.The supplied caption does not report specific comparisons or values.Source: Figure 5 · p. 7
  • The authors state that completed training and testing code will be released.Source: §IV · p. 7

A. Training Details

SiamCorners is trained offline using several tracking and detection datasets, with staged branch-specific and end-to-end optimization.

A. Training Details

  • The network uses a ResNet-50 backbone pretrained on ImageNet and training splits from GOT-10k, LaSOT, COCO, YouTube-BoundingBoxes, ImageNet VID, and ImageNet DET.Template and test images are 127 and 255 pixels; augmentation includes flipping, color jitter, scaling, cropping, brightness, and contrast adjustments.Source: A. Training Details · p. 7
  • Training first optimizes the top-left and bottom-right branches for 5 epochs, then trains the whole network end-to-end for 15 epochs.The learning rate is 0.001 in the first stage and decays from 0.005 to 0.0005 in the second; SGD uses batch size 58, weight decay 0.0001, and momentum 0.9.Source: A. Training Details · p. 7
  • The authors omit the backbone-finetuning operation used in SiamRPN++ during its last 10 epochs, which they say increases training time exponentially.Source: A. Training Details · p. 7

B. Testing Metrics

The evaluation uses success and precision measures, with AUC used to rank trackers; Fig. 6 presents experimental results on UAV123.

B. Testing Metrics

  • Success is the percentage of frames whose overlap exceeds a threshold varied from 0 to 1, and trackers are ranked by the success-plot AUC.Source: B. Testing Metrics · p. 7
  • Precision is the percentage of frames whose predicted-target center lies within a specified distance of the ground-truth center.Source: B. Testing Metrics · p. 7
  • Fig. 6 shows experimental results for methods evaluated on the UAV123 dataset.Source: Figure 6 · p. 7
  • Normalized precision is precision normalized by ground-truth bounding-box size; LaSOT and TrackingNet rankings also use normalized-precision AUC from 0 to 0.5.Source: B. Testing Metrics · p. 7

C. State-of-the-art Comparison

SiamCorners reports competitive or leading results across several tracking benchmarks, with particularly strong comparisons against anchor-based trackers and on challenging UAV123 attributes.

C. State-of-the-art Comparison

  • SiamCorners beats the top anchor-based comparators on OTB100, NFS30, and UAV123, leads LaSOT's success and normalized-precision plots, and is competitive on TrackingNet.Source: C. State-of-the-art Comparison · pp. 7, 9, Table I · p. 9
  • SiamCorners gains 0.6% and 1.3% in success score over CIResNet-22-RPN and DaSiamRPN, respectively, and 2.2% AUC over FCAF on OTB100.It is on par with Ocean and slightly below DIMP, while outperforming ATOM.Source: C. State-of-the-art Comparison · p. 7
  • On UAV123, SiamCorners gains 0.2% in success score and 1.3% in precision score over SiamRPN++; Fig. 6 presents the methods' experimental results.Source: C. State-of-the-art Comparison · p. 7
  • SiamCorners scores 7.8% higher AUC than DaSiamRPN under fast motion and handles the other listed UAV123 challenge factors well.The factors include scale and aspect-ratio change, low resolution, occlusion, out-of-view, viewpoint change, camera motion, and similar objects.Source: C. State-of-the-art Comparison · pp. 8–9, Figure 7 · p. 8
  • On TrackingNet, SiamCorners scores 69.5% AUC, while ATOM leads the compared approaches at 70.3%.The evaluation uses a 511-video test split; Table I compares AUC, precision, and normalized precision.Source: C. State-of-the-art Comparison · p. 9, Table I · p. 9
  • On LaSOT, SiamCorners achieves the best success and normalized-precision plots, exceeding MDNet by 8.0% and 9.3% in those scores, respectively.The benchmark test split contains 280 videos; Fig. 9 reports the experimental results.Source: C. State-of-the-art Comparison · p. 9, Figure 9 · p. 9
  • LaSOT visualizations report SiamCorners performing well on most sequences, while comparison trackers show difficulties with aspect-ratio change, illumination, occlusion, deformation, and scale variation.Fig. 8 shows sequences including bicycle-9, boat-4, cat-3, crocodile-10, giraffe-10, and hand-16.Source: Figure 8 · p. 8, C. State-of-the-art Comparison · p. 9
  • SiamCorners exceeds SiamRPN++ by 4.5% AUC and gains 7 FPS on NFS30.Source: C. State-of-the-art Comparison · p. 9, Table II · p. 9

D. Ablation Study

The ablations examine feature aggregation, offset prediction, corner pooling and penalty choices, and template design. The study also reports component-wise computation time.

D. Ablation Study

  • The ablation study uses the same training datasets and parameter settings across components, with AUC of the success plot as its measure.The component-wise computation-time analysis on UAV123 identifies conv5 as the most time-consuming component and corner pooling as the third most time-consuming.Source: D. Ablation Study · pp. 9–10, Table IV · p. 10
  • 0.614 AUC on UAV123 is achieved by aggregating conv3, conv4, and conv5, 2.7% above the single-layer baseline.Among single-level features, conv4 performs best; combining conv4 and conv5 also improves performance, while other two-branch combinations do not.Source: D. Ablation Study · p. 10, Table III · p. 10
  • The offset operation improves AUC by 1.5% on UAV123 and 3.9% on LaSOT.The authors attribute this improvement to alleviating the impact of network stride.Source: D. Ablation Study · p. 10, Table III · p. 10
  • Modified corner pooling improves results by 2.1% on UAV123 and 2.5% on LaSOT; the proposed penalty function improves them by 1.4% and 1.7%, respectively.The comparison is against original corner pooling and the original penalty function, respectively.Source: D. Ablation Study · p. 10, Table III · p. 10
  • Four respective templates improve performance by 6.3% on UAV123 and 6.9% on LaSOT compared with one unified template.The unified-template setup generates a single feature map as input to corner pooling.Source: D. Ablation Study · p. 10, Table III · p. 10

V. CONCLUSIONS

SiamCorners tracks targets as a pair of corners without multi-scale testing or predefined anchor boxes with different aspect ratios. The authors evaluate it on five datasets and propose further exploration of anchor-free visual tracking.

V. CONCLUSIONS

  • SiamCorners tracks targets as a pair of corners and does not require multi-scale testing or predefined anchor boxes with different aspect ratios.The architecture integrates layer-wise features and uses a penalty function to constrain adjacent-frame changes in bounding-box center, width, and height.Source: §V · p. 10
  • The authors evaluate SiamCorners on OTB100, UAV123, NFS30, LaSOT, and TrackingNet, reporting effectiveness and efficiency.They do not state numerical results in this conclusion passage.Source: §V · p. 10
  • The authors hope this work will lead to further exploration of anchor-free methods in visual tracking.Source: §V · p. 11
Loading 2104.07303v1…