PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-based Person Re-identification with Accumulative Motion Context

Hao Liu, Zequn Jie, Karlekar Jayashree, Meibin Qi, Jianguo Jiang, Shuicheng Yan, Jiashi Feng

arXiv:1701.00193v2cs.CV

TL;DR

Video person re-identification must match people across challenging camera views while exploiting information beyond ambiguous appearance cues. AMOC addresses this gap with an end-to-end two-stream network that learns appearance and motion from raw frames and recurrently accumulates motion context. Across three benchmark datasets, the authors report performance improvements over state-of-the-art methods and conclude that accumulative motion context benefits person matching accuracy.

  • Problem

    Video person re-identification requires robustly matching people across camera views despite appearance ambiguity and challenging conditions.

  • Method

    AMOC jointly learns spatial appearance and motion context from raw video frames using two streams, recurrent accumulation, and end-to-end training.

  • Results

    AMOC outperforms other state-of-the-art video person re-identification methods in most cases across iLIDS-VID, PRID-2011, and MARS, with end-to-end training improving Rank1 by 3.2% on iLIDS-VID and 1.7% on PRID-2011.

  • Takeaways & Limitations

    The experiments support the benefit of accumulating motion context alongside appearance for video person matching.

  • Takeaways & Limitations

    The paper acknowledges potential limitations of the proposed model but does not specify them in the supplied passage.

Abstract

from arXiv · show

Video based person re-identification plays a central role in realistic security and video surveillance. In this paper we propose a novel Accumulative Motion Context (AMOC) network for addressing this important problem, which effectively exploits the long-range motion context for robustly identifying the same person under challenging conditions. Given a video sequence of the same or different persons, the proposed AMOC network jointly learns appearance representation and motion context from a collection of adjacent frames using a two-stream convolutional architecture. Then AMOC accumulates clues from motion context by recurrent aggregation, allowing effective information flow among adjacent frames and capturing dynamic gist of the persons. The architecture of AMOC is end-to-end trainable and thus motion context can be adapted to complement appearance clues under unfavorable conditions (e.g. occlusions). Extensive experiments are conduced on three public benchmark datasets, i.e., the iLIDS-VID, PRID-2011 and MARS datasets, to investigate the performance of AMOC. The experimental results demonstrate that the proposed AMOC network outperforms state-of-the-arts for video-based re-identification significantly and confirm the advantage of exploiting long-range motion context for video based person re-identification, validating our motivation evidently.

I. INTRODUCTION

Video-based person re-identification addresses cross-camera matching under substantial appearance variation. AMOC combines separate appearance and motion processing with recurrent accumulation to learn long-range, task-adapted motion context directly from raw frames.

  • Problem: Cross-camera person re-identification remains challenging because lighting, pose, viewpoint, and appearance vary across camera views.The task supports surveillance applications including people tracking and forensic search.
  • Motivation: Still-image appearance representations are limited by clothing similarity and cross-camera visual ambiguity, motivating the use of person video sequences.Video sequences provide both spatial appearance and temporal information such as gait.
  • Motivation: Single-stream CNNs do not fully exploit temporal information, limiting the discriminative value of learned spatial-temporal features.The paper motivates separating spatial and temporal processing into two streams.
  • AMOC: Unlike offline optical-flow pipelines, AMOC learns motion information end-to-end toward person re-identification rather than relying on task-independent pre-extraction.The paper identifies offline flow extraction as storage-intensive and potentially suboptimal for person re-identification.
  • AMOC: AMOC jointly learns spatial features and motion context from raw person frames, then accumulates both through recurrent processing for long-term information.Motion context is defined as motion between two consecutive frames.
  • Evaluation: Extensive comparisons evaluate the end-to-end two-stream recurrent convolutional network on iLIDS-VID, PRID-2011, and MARS.The authors report better performance than other state-of-the-art video person re-identification methods.

II. RELATED WORK

Prior work spans still-image and video person re-identification, using invariant representations, metric learning, sequence matching, and deep feature or ranking models. AMOC is introduced as an end-to-end architecture that jointly learns spatial appearance and accumulates motion context from raw video frames.

  • Research settings: Person re-identification research is commonly divided into still-image and video-sequence settings.The paper places AMOC within video-based person re-identification.
  • Still-image re-identification: Still-image methods learn invariant features and distance metrics to emphasize inter-person separation and deemphasize intra-person variation.Examples include discriminant localized features, perceptual attributes, LMNN, RankSVM, RDC, and color-texture representations.
  • Video re-identification: Video-based methods exploit multiple frames through sequence matching, conditional random fields, and learned mappings between appearances in sequences.The cited approaches include Dynamic Time Warping and CRF-based label consistency.
  • Deep methods: Deep video re-identification methods use neural networks to learn feature representations and metric or ranking functions from raw images, pairs, or triplets.The paper contrasts these approaches with its proposed joint spatial-motion architecture.
  • AMOC: AMOC addresses video person re-identification by jointly learning spatial appearance and accumulating motion context from raw video frames in an end-to-end architecture.The method description covers motion and spatial networks, recurrent fusion, and two-stream spatial-temporal integration.

A. Architecture Overview

AMOC processes consecutive frames through separate spatial and motion streams, fuses their features, and accumulates motion context over time. Its spatial fusion preserves corresponding locations, while temporal pooling produces one representation for sequences of arbitrary length.

  • A. Architecture Overview: AMOC processes each pair of consecutive frames with spatial and motion streams to learn appearance and temporal motion features.The motion stream predicts motion between frames, while spatial networks extract features from raw frames and motion predictions.
  • A. Architecture Overview: The two streams are fused recurrently to learn discriminative accumulative motion contexts from each time step.The architecture combines spatial appearance with temporal information before accumulating it across the sequence.
  • A. Architecture Overview: Motion networks use six convolutional layers and deconvolutional refinement to predict dense motion maps from concatenated frame pairs.The refinement pathway combines coarse high-level representations with lower-level local information; the final prediction remains at half the input resolution.
  • A. Architecture Overview: Spatial networks contain three convolutional and three max-pooling layers, followed by a fully connected layer, with separate inputs for appearance and motion features.The two spatial networks share the same structure but serve different stream roles.
  • A. Architecture Overview: Fusion aligns feature responses at corresponding spatial locations and can use concatenation, summation, or elementwise maximum.Fusion may occur at different network points, implementing early- or late-fusion; the evaluated methods compare these three operations.

2) Motion Context Accumulation:

AMOC uses recurrent connections to accumulate fused appearance and motion features across arbitrary-length sequences. Temporal pooling supplements recurrent memory by aggregating information from all time steps, reducing sensitivity to which frames dominate the recurrent output.

  • 2) Motion Context Accumulation:: An RNN accumulates fused spatial appearance and motion features across an arbitrarily long sequence.Its feedback connections carry information from the previous time step into the current output.
  • 2) Motion Context Accumulation:: The recurrent output at time t combines the current fused feature f(t) with the previous state r(t−1).The output is represented as o(t), with learned parameters mapping the current feature and previous recurrent state.
  • 2) Motion Context Accumulation:: Temporal pooling aggregates recurrent outputs across all time steps because discriminative frames may occur anywhere in the sequence.Average-pooling or max-pooling combines o(1), o(2), ..., o(T), where T is the sequence length.

3) Multi-task Loss:

AMOC jointly trains classification and contrastive objectives so sequence embeddings support both identity prediction and pairwise discrimination. The contrastive term separates negative pairs beyond a margin while keeping positive pairs close.

  • 3) Multi-task Loss:: The classification branch predicts a person’s identity from the sequence feature vector produced by AMOC.The feature vector combines accumulative spatial appearance and motion-context information.
  • 3) Multi-task Loss:: The contrastive loss penalizes positive pairs that are too far apart and negative pairs closer than margin α.Negative pairs already separated by α receive no contrastive penalty.
  • 3) Multi-task Loss:: AMOC jointly optimizes classification and contrastive losses in an end-to-end multi-task objective.The network is trained using both identity prediction and sequence-pair comparison.
  • 3) Multi-task Loss:: During training, recurrent connections are unrolled and AMOC weights are shared across time steps.The full architecture is optimized from raw video frames through back-propagation-through-time.

1) Pre-training of the Motion Networks:

The motion networks are first initialized by learning to predict optical flow from raw consecutive frames. Multi-scale smooth-L1 supervision provides the pre-training signal before integrating the motion networks into AMOC.

  • 1) Pre-training of the Motion Networks:: Motion networks are pre-trained using pre-extracted optical flow as ground truth, then transferred into the full AMOC framework.This initialization supports the later goal of learning motion context directly from raw consecutive frames.
  • 1) Pre-training of the Motion Networks:: The motion networks produce optical-flow maps at three scales for each pair of person frames.The prediction levels are named Pred1, Pred2, and Pred3.
  • 1) Pre-training of the Motion Networks:: Smooth-L1 losses compare each predicted flow map with the correspondingly downsampled ground-truth optical flow.The three prediction maps are sized at 1/8, 1/4, and 1/2 of the input frame resolution.
  • 1) Pre-training of the Motion Networks:: The multi-scale pre-training objective weights the three prediction levels at 0.01, 0.02, and 0.08, respectively.These weights are assigned across the different scale-level prediction and ground-truth losses.
  • 1) Pre-training of the Motion Networks:: Training resizes frame pairs to 128 × 64 and uses Adam with mini-batches of four frame pairs.The learning rate starts at 1e−4 and is halved every 10k iterations after the first 20k.

2) Training of the Overall Architecture:

After motion pre-training, the overall AMOC architecture is trained end-to-end with fixed embedding, margin, pooling, and learning-rate settings. Data augmentation introduces random translations to address imbalance and overfitting.

  • 2) Training of the Overall Architecture:: End-to-end AMOC training uses margin α = 2, a 128-dimensional embedding, average temporal pooling, and learning rate 1e−3.Average pooling is used unless otherwise specified.
  • 2) Training of the Overall Architecture:: Random 2D translations augment every dataset to increase sequence diversity and reduce data imbalance and overfitting.Translations are sampled within ±5% of the frame width and height around the image center.

A. Datasets

AMOC is evaluated on three video re-identification benchmarks spanning crowded, occluded indoor scenes, cleaner outdoor scenes, and a large-scale dataset. Analytic experiments compare motion representations and test the contribution of motion information.

  • A. Datasets: iLIDS-VID contains 600 sequences from 300 individuals and is challenging because of clothing similarity, lighting and viewpoint changes, clutter, and occlusions.Each person has a sequence pair from two non-overlapping camera views.
  • A. Datasets: PRID-2011 includes 749 people overall, with 400 sequences from 200 identities used under the two-camera evaluation protocol.Compared with iLIDS-VID, it has less crowded outdoor scenes, simpler backgrounds, and rare occlusions.
  • A. Datasets: MARS contains 1,261 identities and around 20,000 video sequences, making it the largest benchmark among the three datasets described.Its provided training and test split contains 631 and 630 identities, respectively.
  • A. Datasets: Experiments use repeated train/test splits for iLIDS-VID and PRID-2011, while MARS uses its fixed split of 631 training and 630 test identities.The smaller datasets are evaluated over 10 different splits and averaged.
  • 1) Effect of Different Motion Information :: EpicFlow improves motion extraction robustness relative to LK-Flow, particularly for motion boundaries, occlusions, and large displacements.The analysis reports improved performance when EpicFlow replaces LK-Flow in baseline and AMOC variants.
  • 1) Effect of Different Motion Information :: Removing AMOC’s temporal stream produces the worst performance on both iLIDS-VID and PRID-2011 among the compared motion settings.This ablation supports the usefulness of motion information for video-based person re-identification.

2) Effect of Spatial Fusion Method and Location:

The experiments examine how AMOC’s fusion, sequence length, temporal pooling, contrastive-loss margin, and RNN embedding size affect re-identification performance. Longer sequences improve performance, while average pooling and selected parameter settings provide better results.

  • Effect of Spatial Fusion Method and Location: Concatenation fusion outperforms sum and max fusion methods, with performance also depending on the spatial fusion location.The experiments compare rank1, rank5, rank10, and rank20 recognition rates on iLIDS-VID and PRID-2011.
  • Effect of Sequence Length and Temporal Pooling Methods: Increasing both probe and gallery sequence lengths improves performance because more appearance and motion information becomes available.The sequence lengths are varied from 1 to 128 on iLIDS-VID, and AMOC achieves higher rank1 rates than the baseline under different settings.
  • Effect of Sequence Length and Temporal Pooling Methods: Average pooling performs better than max pooling for aggregating AMOC’s recurrent spatial-temporal features.Average pooling weights all time steps, whereas max pooling uses only the feature value at the step with the largest activation.
  • Effect of Other Parameter Settings: 68.7% rank1 recognition is achieved with contrastive-loss margin 2, while performance drops considerably when the margin increases from 5 to 10.Rank1 performance remains stable for margins below 5, with margin 2 reported as the best setting.
  • Effect of Other Parameter Settings: RNN embedding size 128 produces the highest rank1 recognition rate, while sizes 64 and 256 cause slight declines.The tested embedding sizes are 64, 128, 256, 512, and 1024.

D. Comparison with State-of-the-Art Methods

AMOC is compared with state-of-the-art methods on video re-identification benchmarks, including iLIDS-VID and PRID-2011. End-to-end training further improves recognition, and the two-stream design outperforms competing spatial-temporal approaches in the reported comparisons.

  • Comparison with State-of-the-Art Methods: The evaluation compares Rank1, Rank5, Rank10, and Rank20 recognition rates across fusion methods on iLIDS-VID and PRID-2011.The comparison table reports these four CMC metrics for the evaluated methods.
  • Comparison with State-of-the-Art Methods: AMOC’s end-to-end experiments use a motion network pretrained only on iLIDS-VID across the three benchmark datasets.The paper states that the motion network has good generalization ability and is reused for experiments on iLIDS-VID, PRID-2011, and MARS.
  • Results on iLIDS-VID and PRID-2011: End-to-end AMOC improves Rank1 recognition by 3.2% on iLIDS-VID and 1.7% on PRID-2011 over the non-end-to-end version.The comparison uses CMC results reported for both datasets.
  • Results on iLIDS-VID and PRID-2011: AMOC outperforms compared state-of-the-art methods on iLIDS-VID and PRID-2011, including methods using spatial-temporal features.The paper attributes the improvement to separately processing spatial appearance and motion context before spatial fusion.

2) Results on MARS:

On MARS, AMOC is evaluated against established descriptors and metric-learning methods, achieving stronger Rank1 performance than the previous best reported method while retaining high mAP. The discussion also identifies background motion as a potential source of irrelevant flow information, especially with detection or tracking errors.

  • Dataset and comparisons: MARS is a large-scale, realistic dataset with complex environments and natural detection/tracking errors.It contains four times as many identities and thirty times as many tracklets as iLIDS-VID and PRID-2011, respectively.
  • Performance on MARS: 3% higher Rank1 accuracy than IDE+XQDA establishes AMOC above the current best method reported for MARS.AMOC achieves 52.9% mAP, although its Rank5 performance is slightly lower than IDE+XQDA.
  • End-to-end learning: AMOC learns motion and appearance features jointly with feature accumulation in an end-to-end process.This differs from compared methods that separate descriptor extraction and metric learning.
  • Overall conclusion: Across the reported experiments, AMOC is comparable to other methods in complex multi-camera re-identification while benefiting from joint appearance and motion accumulation.The paper concludes that its experimental results outperform other state-of-the-art methods in most cases.
  • Limitations: A potential limitation is that learned flow may contain redundant background motion unrelated to person identification.This issue may be more likely in datasets such as MARS, where detector or tracker errors introduce non-related motion information; suppressing it is left for future work.
Loading 1701.00193v2…