PaperlayerView exact revision on arXiv

Source-linked AI summary

STNReID : Deep Convolutional Networks with Pairwise Spatial Transformer Networks for Partial Person Re-identification

Hao Luo, Xing Fan, Chi Zhang, Wei Jiang

arXiv:1903.07072v2cs.CV

TL;DR

Partial ReID concerns matching people when only part of a person image is available, and few deep-learning studies have addressed matching partial images to holistic ones. STNReID samples an affined patch from a holistic image using pairwise spatial transformers and two-stage training; it reports 66.7% rank-1 accuracy on Partial-ReID and 54.6% on Partial-iLIDS.

  • Problem

    Partial ReID requires matching people from partial images, while few studies have addressed matching partial images with holistic images.

    Source: §I · p. 1

  • Method

    STNReID uses a spatial transformer to sample a corresponding patch from a holistic image and a ReID module to extract features, with two-stage training.

    Source: §I · p. 2

  • Results

    66.7% rank-1 accuracy on Partial-ReID and 54.6% rank-1 accuracy on Partial-iLIDS were reported for STNReID (PM).

    Source: B. Ablation studies of two-stage… · p. 7

  • Takeaways & Limitations

    STNReID can be trained on holistic person datasets without additional labeled partial images.

    Source: §I · p. 2

Abstract

from arXiv · show

Partial person re-identification (ReID) is a challenging task because only partial information of person images is available for matching target persons. Few studies, especially on deep learning, have focused on matching partial person images with holistic person images. This study presents a novel deep partial ReID framework based on pairwise spatial transformer networks (STNReID), which can be trained on existing holistic person datasets. STNReID includes a spatial transformer network (STN) module and a ReID module. The STN module samples an affined image (a semantically corresponding patch) from the holistic image to match the partial image. The ReID module extracts the features of the holistic, partial, and affined images. Competition (or confrontation) is observed between the STN module and the ReID module, and two-stage training is applied to acquire a strong STNReID for partial ReID. Experimental results show that our STNReID obtains 66.7% and 54.6% rank-1 accuracies on partial ReID and partial iLIDS datasets, respectively. These values are at par with those obtained with state-of-the-art methods.

I. INTRODUCTION

Partial ReID must match people when only part of an image is visible, but resizing and matching differently sized images create difficulties for CNN-based methods. STNReID uses pairwise spatial transformers and two-stage training, and is reported to achieve competitive results on two partial-ReID benchmarks.

I. INTRODUCTION

  • Partial ReID must match a partial person image to a holistic image despite missing visual information.Resizing can deform the partial image, while leaving image sizes different creates a matching problem for CNNs.Source: §I · pp. 1–2
  • Existing partial-ReID approaches face trade-offs in computation, alignment, or handling different occlusion directions.Sliding-window search is computationally expensive; feature-map reconstruction limits GPU tensor utilization, while stripe-based methods struggle with vertical occlusions and align images roughly.Source: §I · pp. 1–2
  • STNReID predicts affine transformations from paired holistic and partial images, then samples a corresponding patch from the holistic image.It can be trained on holistic person datasets without additional labeled partial images.Source: §I · p. 2
  • Two-stage training addresses the reported competition between STN and ReID modules to acquire a strong STNReID model.The paper reports that STN performance decreases as ReID-module power increases.Source: §I · p. 2
  • STNReID is reported to achieve competitive outcomes on the Partial-ReID and Partial-iLIDs benchmarks.The framework uses pairwise spatial transformers and a two-stage training process.Source: §I · p. 2

II. RELATED WORKS

The related-work section reviews deep person ReID approaches and the challenges of partial-image matching. It situates STNReID among methods using global and local features, matching patches, and spatial transformers.

A. Deep person ReID

  • Deep person ReID commonly uses representation learning with ID loss or metric learning, including triplet loss; combining ID and triplet losses has attracted attention.Triplet loss pulls positive-pair distances together and pushes negative-pair distances apart.Source: A. Deep person ReID · p. 2
  • Global features can encounter a performance bottleneck, while local-feature approaches use pose, stripes, or grids to represent spatial information.Pose-based methods require additional labeled data for pose estimation; stripe- or grid-based methods can be combined with global features.Source: A. Deep person ReID · p. 2

B. Partial ReID

  • Partial-ReID methods include resizing, patch matching, feature-map reconstruction, and stripe-based alignment, each with stated limitations.Resizing can cause scale deformation; AMC+SWM is costly, and stripe-based SCPNet and VPM handle horizontal but not vertical occlusions well.Source: B. Partial ReID · p. 3

C. Spatial Transformer Networks

STNReID uses pairwise image information to estimate affine transformations and sample a holistic-image patch for comparison with a partial image. Its localization network processes concatenated image tensors and produces transformation parameters.

C. Spatial Transformer Networks

  • Unlike methods that estimate transformations from one image, STNReID predicts affine parameters by analyzing a holistic and partial image pair.The paper positions this approach in relation to STNs, PAN, and DSR.Source: C. Spatial Transformer Networks · p. 3

A. STNReID

  • STNReID has an STN module that crops affined patches from holistic images and a ReID module that extracts global features from holistic, partial, and affined images.The ReID loss combines ID loss and triplet loss.Source: A. STNReID · p. 3, Figure 2 · p. 4

1) STN Module:

  • The localization network receives concatenated holistic and partial image tensors and outputs affine transformation parameters.During training, the partial image is randomly cropped from the holistic image; each image tensor has shape H × W × C.Source: 1) STN Module · p. 3
  • The localization network uses few layers with a large receptive field and estimates six transformation parameters for sampling an affined image.Its stated design balances retaining spatial information with associating relevant patches across the image pair.Source: 1) STN Module · p. 3, Table I · p. 4

2) ReID Module:

The ReID module extracts global features from holistic, partial, and affined images, and produces person-ID predictions during training. Its feature comparison guides the STN, while the total loss combines ReID losses across the three image types with an STN loss.

2) ReID Module:

  • The module extracts global features and predicted person IDs during training, but outputs only global features at inference.ResNet50 is used as the ReID module.Source: 2) ReID Module · p. 4
  • The STN loss compares the global features of partial and affined images, encouraging similarity in feature space.The total objective adds this term to the ReID losses for holistic, partial, and affined images.Source: 2) ReID Module · p. 4
  • The training discussion avoids regressing the affine parameters from ground-truth parameters or original image information because real-world partial and holistic images come from different images.Instead, the sampled patch may have features similar to the partial image even when pose variation is large.Source: 2) ReID Module · p. 4

3) ReID Loss:

The ReID objective combines identity classification with an adaptive weighted triplet loss. The triplet term compares anchor images with positive and negative samples using feature distances, an inter-person margin, and adaptive weights.

3) ReID Loss:

  • The ReID loss combines identity loss with adaptive weighted triplet loss.The identity loss is cross-entropy over predicted person-ID probabilities and one-hot labels.Source: 3) ReID Loss · p. 4
  • Each mini-batch contains K images for each of P identities, with batch size B = P × K.The triplet is defined using an anchor, positive samples of the same identity, and negative samples.Source: 3) ReID Loss · pp. 4–5
  • The triplet loss uses feature distances, an inter-person separation margin, and adaptive positive and negative sample weights.The hinge term is defined as [·]+ = max(0, ·).Source: 3) ReID Loss · p. 5

B. Two-Stage Training

A strong ReID module can make the STN difficult to train because same-identity partial and affined images have similar features. The two-stage approach first trains a strong STN with a weak ReID module, then freezes the STN while acquiring a strong ReID module.

B. Two-Stage Training

  • A strong ReID module can leave the STN with little training signal because same-ID partial and affined images have similar, clustered features.The paper describes the resulting STN performance as poor.Source: B. Two-Stage Training · p. 5
  • Stage one trains the ReID module using ID loss alone without ImageNet-pretrained initialization; stage two freezes the acquired STN and trains ReID with ID loss, triplet loss, and training tricks.This sequencing first acquires a strong STN, then a strong ReID module.Source: B. Two-Stage Training · p. 5
  • Pipeline Mode fine-tunes ReID with the frozen STN, while Merge Mode combines an STN with an already trained ReID model.The paper presents Pipeline Mode as more suitable for partial ReID after fine-tuning, and Merge Mode as an option for extending a strong trained model.Source: B. Two-Stage Training · p. 5
  • Both modes train a strong STN first and then acquire a strong ReID module to obtain a better STNReID for partial ReID.The two-stage mechanism is illustrated in Fig. 3.Source: B. Two-Stage Training · p. 5, Figure 3 · p. 5

IV. EXPERIMENTAL SETTINGS

The experiments train on Market1501 and evaluate on Partial-ReID and Partial-iLIDS, treating partial images as probes and holistic images as galleries. Training uses simulated partial images and a strengthened baseline, with repeated CMC evaluation on randomly split query and gallery sets.

IV. EXPERIMENTAL SETTINGS

  • Market1501 training uses 12,936 images from 751 identities; evaluation uses Partial-ReID and Partial-iLIDS, with partial images as probes and holistic images as galleries.Partial-ReID has 600 images of 60 people; Partial-iLIDS has 238 images of 119 people.Source: §IV · p. 5
  • Evaluation follows the protocol in,: randomly split query and gallery sets, compute CMC ten times, and average the accuracies.Models are trained on Market1501 and evaluated on partial ReID datasets.Source: §IV · p. 5
  • Simulated partial images are created by randomly choosing a crop direction and retaining 20%–60% of the holistic image information.These generated images train STNReID and augment strong partial-ReID baselines.Source: §IV · p. 5
  • The baseline combines adaptive weighted triplet loss, label smoothing, and generated partial-image augmentation.Adaptive weighted triplet loss improves rank-1 accuracy by approximately 3% over hard-example-mining triplet loss, while label smoothing improves it by more than 2%.Source: §IV · p. 5
  • The strengthened baseline achieves 58.2% rank-1 accuracy on Partial-ReID after training only on Market1501.It also achieves 93.1% rank-1 accuracy on Market1501 under the standard evaluation protocol.Source: §IV · p. 6

V. EXPERIMENTAL RESULTS

Experiments show a confrontation between the STN and ReID modules: a strong ReID model can hinder STN learning during end-to-end training. Training the STN in a second stage with a weak ReID model can produce a strong STN module.

A. The ReID module affects the STN module

B. Ablation studies of two-stage training

The two-stage ablation compares merge and pipeline modes using a frozen STN initialized from Ep1. Pipeline mode achieved the strongest reported rank-1 accuracies on both partial-ReID benchmarks.

B. Ablation studies of two-stage training

C. Comparison to the State-of-the-Art

STNReID is compared with partial ReID methods and holistic ReID methods. It reports stronger rank-1 accuracy than DSR on both partial datasets and is described as competitive with SFR and VPM, while its holistic ReID performance is close to the baseline.

C. Comparison to the State-of-the-Art

  • 66.7% and 54.6% rank-1 accuracies for STNReID (PM) on Partial-ReID and Partial-iLIDS, respectively, exceed DSR’s 39.3% and 51.1%.DSR was also evaluated with the authors’ baseline for a fair comparison.Source: C. Comparison to the State-of-the-Art · p. 7, Table IV · p. 8
  • 62.2% and 47.3% rank-1 accuracies for SCPNet evaluated with the authors’ baseline on Partial-ReID and Partial-iLIDS, respectively; its reported original results are 42.6% and 37.0%.The comparison notes that SCPNet’s original baseline differs from the authors’ baseline.Source: C. Comparison to the State-of-the-Art · p. 8, Table IV · p. 8
  • STNReID is described as competitive with SFR and VPM on partial ReID, although both methods achieve better performance on Partial-iLIDS.SFR and VPM use multi-local features, while STNReID uses global features; their input images are larger than STNReID’s.Source: C. Comparison to the State-of-the-Art · p. 8

D. Extended Comparison on Holistic Perons ReID

  • 93.8% rank-1 accuracy and 84.9% mAP on Market1501 are close to the baseline, but STNReID does not outperform the newest holistic methods named in the comparison.The paper notes that the STN module learns no matching knowledge when holistic ReID does not require partial-to-holistic matching.Source: D. Extended Comparison on Holistic… · p. 8, Table V · p. 8

E. Computation Efficiency

The efficiency comparison reports that STNReID’s matching time decreases with batch size, whereas DSR’s costs rise with image size. The conclusion also identifies limitations in negative-pair association and the efficiency of the siamese design.

E. Computation Efficiency

  • 1.899s at N = 1 falls to 0.303s at N = 48 for STNReID on Partial-iLIDS, while DSR takes 0.112s on Partial-ReID and 3.042s on Partial-iLIDS.STNReID’s time cost mainly depends on gallery size; DSR’s feature extraction and sparse reconstruction costs rise with image size.Source: E. Computation Efficiency · pp. 8–9

VI. CONCLUSION, DISADVANTAGES AND FUTURE WORKS

  • The conclusion reports that STN can learn from high-level semantic features and presents the method as potentially useful for partial ReID.It also says experiments demonstrate effectiveness on Partial-ReID and Partial-iLIDS.Source: §VI · p. 9
  • STNReID minimizes L2 distances for positive pairs, but accurate negative-pair association is hard at inference and the siamese network is less efficient than one-stream ReID.The authors identify a simpler, more direct, and more efficient training approach as future work.Source: §VI · p. 9
Loading 1903.07072v2…