PaperlayerView exact revision on arXiv

Source-linked AI summary

Where-and-When to Look: Deep Siamese Attention Networks for Video-based Person Re-identification

Lin Wu, Yang Wang, Junbin Gao, Xue Li

arXiv:1808.01911v2cs.CV

TL;DR

Video-based person re-identification must match pedestrians across disjoint views despite visual variation, while existing approaches do not fully integrate feature learning, metric learning, and video structure. The paper introduces a Siamese attention architecture with spatially guided recurrent processing to jointly learn representations and similarity. Experiments on three datasets report rank-1 accuracies of 61.9%, 77.0%, and 73.5% with KISSME, outperforming benchmark methods.

  • Problem

    Video re-identification is challenging because appearance varies across views, and existing approaches treat feature learning and metric learning independently while underusing spatial and temporal information.

  • Method

    A deep Siamese attention model jointly learns spatiotemporal video representations and similarity by propagating relevant spatial contexts through convolutional gated recurrent units.

  • Results

    The method with KISSME achieves rank-1 accuracy of 61.9%, 77.0%, and 73.5% on iLIDS-VID, PRID2011, and MARS, outperforming benchmark methods.

  • Takeaways & Limitations

    Attention-guided spatial context makes local video features more discriminative and supports aligning dynamic person appearances across space and time.

Abstract

from arXiv · show

Video-based person re-identification (re-id) is a central application in surveillance systems with significant concern in security. Matching persons across disjoint camera views in their video fragments is inherently challenging due to the large visual variations and uncontrolled frame rates. There are two steps crucial to person re-id, namely discriminative feature learning and metric learning. However, existing approaches consider the two steps independently, and they do not make full use of the temporal and spatial information in videos. In this paper, we propose a Siamese attention architecture that jointly learns spatiotemporal video representations and their similarity metrics. The network extracts local convolutional features from regions of each frame, and enhance their discriminative capability by focusing on distinct regions when measuring the similarity with another pedestrian video. The attention mechanism is embedded into spatial gated recurrent units to selectively propagate relevant features and memorize their spatial dependencies through the network. The model essentially learns which parts (\emph{where}) from which frames (\emph{when}) are relevant and distinctive for matching persons and attaches higher importance therein. The proposed Siamese model is end-to-end trainable to jointly learn comparable hidden representations for paired pedestrian videos and their similarity value. Extensive experiments on three benchmark datasets show the effectiveness of each component of the proposed deep network while outperforming state-of-the-art methods.

I. INTRODUCTION

Video-based person re-identification addresses matching pedestrians across disjoint camera views, where appearance ambiguity and viewpoint, illumination, and occlusion create substantial variation. The paper proposes jointly learning spatiotemporal representations and similarity through attention-guided recurrent processing of video regions.

  • Person re-identification matches individuals across disjoint camera views despite visual ambiguity and appearance changes from viewpoints, illumination, and occlusion.
  • Video supplies temporal motion cues and multiple poses or viewpoints that can help distinguish impostors and improve appearance descriptions.
  • Prior video re-id methods separate feature learning from metric learning and incompletely capture spatial correlations within frames.
  • The proposed Siamese architecture jointly learns spatiotemporal video features and similarity by processing paired variable-length videos with shared subnetworks, CNNs, recurrent units, and temporal pooling.
  • Attention selects discriminative local regions and propagates relevant spatial contexts through recurrence, while dynamically pooling features rather than compressing fragments into static representations.
  • The paper reports extensive experiments demonstrating state-of-the-art performance and effectiveness of the proposed components for video-based person re-identification.

II. RELATED WORK

Related work spans image-based appearance and metric learning, multi-shot and video methods, deep representations, and visual attention. The paper positions its approach as an attention-based Siamese model that encodes spatial context while jointly learning video features and similarity.

  • Image-based re-id commonly learns appearance features, distance metrics, or post-ranking refinements from still images.
  • Multi-shot methods exploit multiple images or video fragments to strengthen spatial descriptions and cross-view matching.
  • The proposed model jointly learns spatiotemporal features and a similarity metric for pedestrian sequences by encoding spatial contexts through attention.
  • Deep re-id models learn identity-preserving embeddings, while attention methods add interpretability by identifying task-relevant regions.
  • The CNN encoder and recurrent layers use soft location attention to weight feature regions and propagate spatially informed representations over time.
  • Its architecture uses parameter-sharing convolution-recurrent subnetworks, spatial context encoding, temporal pooling, and cross-entropy optimization to produce a similarity value.

A. Convolutional Layers

The model uses CNN features from local frame regions, then applies deterministic soft attention to select spatially relevant information before recurrent processing.

  • CNN Feature Extraction: CNNs extract local convolutional features from each frame region to form the input representation for pedestrian videos.The architecture uses the last convolutional layer of GoogLeNet pretrained on ImageNet.
  • Deterministic Soft Attention: A recurrent attention model predicts a softmax over K × K spatial locations using the previous hidden state.Each location receives a positive weight representing its relative importance in combining regional feature slices.
  • Deterministic Soft Attention: The location softmax represents the model’s probability that each input region is important, with subsequent attention depending on previously examined regions.Thus, the selected spatial focus changes as the recurrent output sequence advances.
  • Deterministic Soft Attention: Deterministic soft attention computes the expected next input by taking a weighted combination of feature slices from different regions.This produces a soft attention-weighted feature cube rather than sampling a single location.
  • Deterministic Soft Attention: Unlike stochastic hard attention, deterministic soft attention is differentiable and supports end-to-end training with standard back-propagation.Hard attention requires sampling because its location decisions are not differentiable.

C. Deep Attention based Spatial Correlation Encoding

The architecture encodes spatial correlations over time with convolutional GRUs that preserve feature-map structure and propagate information through local neighborhoods.

  • Spatial Correlation Encoding: The recurrent stack propagates attention-weighted feature maps over time while preserving their spatial structure.Hidden activations are defined at grid locations, and zero-padding maintains spatial size across recurrent updates.
  • Output Representation: The final video representation is obtained by average or attention pooling of the top recurrent hidden activations across all time steps.The recurrent function is implemented with gated recurrent units at each layer.
  • Convolutional GRUs: Convolutional GRUs replace fully connected GRUs so convolutional operations model spatiotemporal relationships in video activations.This design performs convolution in both input-to-hidden and hidden-to-hidden transitions.
  • Convolutional GRUs: Fully connected GRUs can be inefficient for 3D convolutional maps because they introduce many parameters and redundancy.The input convolutional maps contain spatial dimensions and channels that a vector-based recurrent unit does not exploit directly.
  • Convolutional GRUs: ConvGRUs exploit local correlations and temporal motion smoothness by replacing fully connected connections with convolutional kernels.Their hidden units use local neighboring states, while shared kernels capture repeated spatial structure.
  • Convolutional GRUs: The hidden recurrent representation can capture faster motion with larger transitional kernels and slower motion with smaller kernels.The convolutional receptive field determines the spatial neighborhood used during recurrent propagation.

D. Temporal Pooling: Attention on Temporal Selection

Temporal pooling prevents recurrent summaries from overemphasizing later frames by weighting selected time steps when constructing the complete sequence representation.

  • Temporal Selection: Temporal pooling is needed because recurrent outputs may favor later time steps even though discriminative regions can appear anywhere in a sequence.The architecture addresses this limitation by combining information across selected frames.
  • Temporal Selection: Soft temporal attention assigns each frame a positive weight α_t that determines its relative importance in the pooled representation.The selection vector reweights frames to focus on a subset of time steps.
  • Temporal Selection: The weighted temporal context is differentiable, allowing the entire model to remain trainable end-to-end with standard back-propagation.After pooling, the representation aggregates information from all time steps and is flattened into a video-level vector.

E. Complexity

Convolutional GRUs reduce recurrent computation and parameter storage by using small convolutional kernels instead of fully connected transitions while retaining local spatial processing.

  • Kernel Structure: ConvGRU input-to-hidden and hidden-to-hidden transitions use convolutional kernels of size k1 × k2 rather than full spatial maps.The kernels are much smaller than the original convolutional-map dimensions H1 × H2.
  • Kernel Structure: Each ConvGRU gate and candidate activation is computed from local neighborhoods on the current input and previous hidden units.The layer conducts six two-dimensional convolutions at each time step, with two convolutions per GRU gate and two for the candidate hidden representation.
  • Computational Complexity: ConvGRU requires O(3k1k2(CxCh + C2_h)) multiplications, substantially less than the O(3TH2_1H2_2(CxCh + C2_h)) required by FC-GRU.The comparison is expressed for a sequence of length T, with convolutional kernels replacing full spatial connectivity.
  • Initialization: The model initializes hidden states with a multilayer perceptron to support faster convergence.The resulting values determine the first softmax location and initial input.

B. Loss Function and Attention Penalty

The model jointly learns sequence representations and similarity using paired similar and dissimilar videos, while an attention penalty regularizes spatial location selection.

  • Loss and penalty: The loss jointly optimizes network parameters with binary cross-entropy and a penalty on the location softmax.The penalty coefficient is λ, and the objective is equivalent to maximizing the conditional log-likelihood of training data.
  • Attention regularization: The attention regularization encourages the model to attend to each frame region at some point during the sequence.This constraint prevents spatial regions from being permanently ignored by forcing their attention weights to sum approximately to one over time.
  • Similarity computation: Similarity is computed from the element-wise inner product of the two subnetworks’ hidden representations, followed by a weighted sum and bias.The learned similarity function is a weighted inner product between sequence representations.
  • Joint representation and metric learning: The Siamese recurrent network combines time-series modeling and metric learning by learning representations whose similarity reflects whether paired videos depict the same person.The two subnetworks share weights and are optimized on similar and dissimilar sequence pairs using cross-entropy loss.

V. EXPERIMENTS

Experiments evaluate the proposed architecture on three video person re-identification datasets with differing camera conditions, sequence lengths, and scale. Training uses sampled subsequences, augmentation, and architecture studies.

  • Datasets: Experiments use iLIDS-VID, PRID 2011, and MARS, covering challenging cross-camera conditions and dataset scales from 600 to 20,000 sequences.iLIDS-VID has lighting, viewpoint, appearance, and background challenges; PRID 2011 has color changes and shadows; MARS is the largest dataset.
  • Experimental protocol: The iLIDS-VID and PRID 2011 datasets are split by persons into 50% training and 50% testing, while MARS uses fixed training and testing identities.The implementation uses one PC with a single NVIDIA GTX 980 GPU.
  • Training protocol: Training randomly samples positive and negative sequence pairs and uses T=20 consecutive frames from arbitrary-length videos.A different subset of 20 frames is randomly selected from each whole video at every epoch.
  • Training protocol: Training augments sequences through cropping, mirroring, and random 2D translations applied consistently across frames.Translations are sampled uniformly within ±0.05 of the frame height and width dimensions.
  • Architecture and optimization: The model uses three stacked recurrent layers with 128, 256, and 256 channels, zero-padded 5×5 convolutions, and spatial pooling between layers.Top-layer hidden representations are average pooled across frames to form the overall sequence representation.

C. Baselines and Evaluation Metric

The evaluation compares convolutional-GRU attention models across kernel configurations and backbone features against fully connected GRU baselines, using rank-1 CMC performance.

  • Baselines: The baselines include VGG-16 and GoogLeNet features passed to fully connected GRUs.These baselines use three GRU layers with 128, 256, and 256 channels and 1×1 kernels.
  • Architecture variants: The proposed variants change recurrent state-to-state kernels among 9×9, 1×1, and 5×5 configurations across the stacked GRU layers.Variants are built with either GoogLeNet or VGG-16 convolutional features.
  • Attention ablation: The OursG (5×5, 5×5) variant removes attention by fixing every attention weight to 1/K^2 = 1/49 for comparison.This isolates the contribution of learned attention while retaining the same configuration.
  • Evaluation metric: Performance is evaluated with average Cumulative Matching Characteristics curves after 10 trials using different train/test splits.Testing ranks gallery sequences using Euclidean distances between probe and gallery features.

D. Experimental Results

Ablation studies show benefits from spatially aware recurrent processing and temporal attention, while pooling and sequence-length analyses characterize the model’s behavior across video inputs.

  • Architecture variations: Attention-based ConvGRU variants consistently outperform VGG-16 and GoogLeNet fully connected GRU baselines.The authors attribute this to convolutional features retaining spatial information about moving patterns.
  • Architecture variations: Larger state-to-state kernels, especially 1×1 followed by 9×9 and 5×5 followed by 5×5, better capture spatiotemporal correlations than 9×9 followed by 1×1.Removing attention from OursG (5×5, 5×5) lowers rank-1 performance.
  • Temporal pooling with attention: Average/attention pooling outperforms max pooling because it reweights frames instead of retaining only the feature values with the largest activations.Figure 6 illustrates higher weights for informative frames and reduced emphasis on less informative frames.
  • Temporal pooling with attention: Fisher vector encoding performs slightly better than average/attention pooling, while the proposed pooling is more efficient and end-to-end trainable.Fisher vectors model higher-order statistics but incur higher computational cost when updating parameters.
  • Sequence length: Increasing either probe or gallery sequence length changes rank-1 matching rates, as summarized by the probe-versus-gallery length matrix in Figure 7.The study varies both lengths from 1 to 128 frames in powers-of-two steps.

4) Attention over Time:

The attention-based Siamese model focuses on distinctive regions across time while outperforming conventional multi-shot representations and benchmark methods. Its strongest reported benchmark results combine learned deep features with metric learning.

  • Attention over Time:: The model attends to distinctive person parts across time, including jackets and bags, despite cluttered backgrounds and occlusions.Attention remains on important regions such as a large black bag across video frames.
  • Feature representations: The proposed deep representation consistently outperforms alternative feature representations in the reported comparison.The paper attributes this to deep networks reconstructing spatiotemporal patterns.
  • Multi-shot methods: Convolutional GRUs with attention selection leverage spatial context and capture temporal input variation for multi-shot person matching.This design is contrasted with low-level feature combinations and S-LSTM-based aggregation.
  • Multi-shot methods: Compared with S-LSTM, the method improves rank-1 accuracy by 8.8, 10.9, and 12.1 points on three datasets, respectively.The comparison is reported for the multi-shot evaluation.
  • State-of-the-art comparison: Combined with KISSME, the method reaches rank-1 accuracies of 61.9%, 77.0%, and 73.5% on iLIDS-VID, PRID2011, and MARS, respectively.These results outperform all benchmark methods in the reported comparison.

H. Failure Examples

The reported failure cases arise when different pedestrians share similar upper-body appearance and walking gait, especially when lower-body evidence is occluded. The paper also identifies viewpoint changes and cross-dataset bias as boundaries for the approach.

  • Failure Examples: The method degenerates when different pedestrians have very similar upper-body appearance and walking gait patterns.A notice sign can occlude the query’s lower body, leaving visually similar upper-body regions to distinguish the candidates.
  • Cross-Dataset Testing: Cross-dataset recognition results are worse than within-dataset results, which the paper attributes to dataset bias.The model is trained on MARS and transferred to iLIDS-VID and PRID2011.
  • Conclusion: The paper reports effectiveness against cluttered backgrounds and occlusions while aligning dynamic appearance spatially and temporally.This conclusion follows experiments on three benchmark datasets.
  • Future Work: The conclusion identifies arbitrary viewpoint changes as a continuing problem for spatial alignment.The authors propose robust body-part models as one direction for addressing pose and viewpoint issues.
Loading 1808.01911v2…