PaperlayerView exact revision on arXiv

Source-linked AI summary

Person Re-Identification by Unsupervised Video Matching

Xiaolong Ma, Xiatian Zhu, Shaogang Gong, Xudong Xie, Jianming Hu, Kin-Man Lam, Yisheng Zhong

arXiv:1611.08512v2cs.CV

TL;DR

Conventional DTW assumes sequence endpoints align and uses every element, although surveillance videos can contain incomplete or noisy observations. The paper presents STPS and TS-DTW, including a multidimensional extension, for selective matching of video sequences; evaluations report advantages over a range of contemporary methods.

  • Problem

    Conventional DTW assumes aligned sequence starts and ends and uses all elements regardless of quality, despite noisy or incomplete surveillance observations.

    Source: §2 · p. 3, §5.1 · p. 6

  • Method

    The framework combines the Spatio-Temporal Pyramid Sequence representation with TS-DTW and its multidimensional extension for selective matching of incomplete, noisy image sequences.

    Source: §4.3 · p. 6, §5.3 · p. 8, §9 · p. 14

  • Results

    The authors report advantages over contemporary gait-recognition, temporal-sequence-matching, supervised, and unsupervised ReID methods across closed- and open-world scenarios.

    Source: §9 · p. 14

  • Takeaways & Limitations

    The framework applies video-based ReID to image sequences that are inherently incomplete and noisy.

    Source: §9 · p. 14

    Future work includes further suppressing background-distraction noise beyond time-shift-based selection and exploiting additional information such as semantic text.

    Source: §9 · p. 14

Abstract

from arXiv · show

Most existing person re-identification (ReID) methods rely only on the spatial appearance information from either one or multiple person images, whilst ignore the space-time cues readily available in video or image-sequence data. Moreover, they often assume the availability of exhaustively labelled cross-view pairwise data for every camera pair, making them non-scalable to ReID applications in real-world large scale camera networks. In this work, we introduce a novel video based person ReID method capable of accurately matching people across views from arbitrary unaligned image-sequences without any labelled pairwise data. Specifically, we introduce a new space-time person representation by encoding multiple granularities of spatio-temporal dynamics in form of time series. Moreover, a Time Shift Dynamic Time Warping (TS-DTW) model is derived for performing automatically alignment whilst achieving data selection and matching between inherently inaccurate and incomplete sequences in a unified way. We further extend the TS-DTW model for accommodating multiple feature-sequences of an image-sequence in order to fuse information from different descriptions. Crucially, this model does not require pairwise labelled training data (i.e. unsupervised) therefore readily scalable to large scale camera networks of arbitrary camera pairs without the need for exhaustive data annotation for every camera pair. We show the effectiveness and advantages of the proposed method by extensive comparisons with related state-of-the-art approaches using two benchmarking ReID datasets, PRID2011 and iLIDS-VID.

1. Introduction

The introduction motivates video-based ReID as a way to use space-time information where appearance alone is ambiguous, and proposes unsupervised matching for unaligned, noisy sequences. It presents a sequential representation and selective matching model, evaluated against existing approaches on two benchmark datasets.

1. Introduction

  • Appearance similarity between different people and cross-view changes in the same person make appearance-based ReID unreliable.The introduction identifies viewpoint, pose, illumination, occlusion, and background clutter as sources of variation.Source: §1 · p. 1
  • Unregulated surveillance videos have arbitrary start and end phases, variable walking cycles, and corrupted frames, challenging holistic matching and explicit phase or occlusion detection.The paper seeks to avoid explicit walking-phase detection and frame-level occlusion estimation.Source: §1 · p. 2
  • The proposed approach is unsupervised, avoiding exhaustive cross-view pair labels for each camera pair.The introduction contrasts it with supervised DVR.Source: §1 · p. 2
  • The method represents videos as time series of space-time dynamics and uses TS-DTW and MDTS-DTW to align sequences, select matched segments, and fuse descriptions.The matching process varies the mutual time shift and matches partial segments.Source: §1 · p. 2
  • Evaluations on PRID2011 and iLIDS-VID compare the approach with sequence-matching, gait-recognition, and state-of-the-art ReID methods in closed- and open-world settings.Comparators include SDALF, eSDC, DVR, RDL, and XQDA.Source: §1 · p. 2

2. Related Work

Related work spans gait recognition, temporal sequence matching, and image-based ReID. The paper positions its approach against methods whose alignment, segmentation, segmentation-quality, or supervision assumptions are difficult to meet in surveillance video.

2. Related Work

  • Gait methods often assume aligned sequences and controlled, uncluttered settings, while segmentation and occlusion-based approaches depend on accurate body-part processing.The cited constraints are problematic for public-space ReID videos with clutter and occlusion.Source: §2 · pp. 2–3
  • Conventional DTW assumes aligned sequence endpoints and uses all observations; unregulated videos violate these assumptions and may contain noisy or missing frames.Periodic walking sequences can also have multiple possible starting elements, while spotting requires a threshold unavailable in this ReID problem.Source: §2 · p. 3
  • Multi-shot ReID work largely aggregates or models spatial appearance, whereas this paper explores space-time information from available videos.Prior approaches include interest-point aggregation, manifold structures, image sets, and local regions or patches.Source: §2 · p. 3
  • Prior video ReID methods include gait-based features and supervised DVR, which uses only a few local fragments and requires matched people for each camera pair.The paper describes those requirements as limiting scalability across large camera networks.Source: §2 · p. 3
  • Unlike vector-based action representations, the paper represents person videos as temporal sequences that support selective matching of unregulated videos.Its temporal-pyramid design preserves dynamics at different sequence-element lengths, adds spatial pyramids at each granularity, and avoids requiring segmented videos.Source: §2 · p. 4

3. Approach Overview

The approach treats videos as sequences of localized space-time dynamics and seeks to handle noisy, unaligned surveillance clips without supervision. Its framework combines multi-scale representation with alignment and adaptive selection, and the authors report greater scalability than supervised DVR.

3. Approach Overview

  • Localized space-time sequences represent motion dynamics and support flexible temporal alignment and unsupervised selective matching.The approach targets videos with random occlusions, arbitrary duration, uncertain start and end phases, and uncontrolled background clutter.Source: §3 · p. 4
  • The method extracts multi-scale spatio-temporal structure, automatically aligns sequence pairs, and adaptively selects informative observations from noisy videos.The overview frames these as responses to extracting walking dynamics and suppressing occlusion and clutter.Source: §3 · p. 4
  • Compared with DVR, the approach uses richer space-time cues and requires no matched cross-view pairs for each camera pair, which the authors say makes it more scalable.It also considers fusion across multiple feature sequences describing different aspects of video data.Source: §3 · p. 4

4. Structured Video Representation

The section converts variable-length videos into ordered localized slice sequences, then enriches them with temporal and spatial pyramids and HOG3D descriptors.

4.1. Video Sequentialisation

  • Localized slice sequences preserve temporal order and provide many short segments for robust alignment and flexible selection of informative video content.Unlike isolated fragments, the representation retains sequence order for sequence matching.Source: §4.1 · pp. 4–5

4.2. Temporal Pyramid

  • Varying slice lengths form temporal-pyramid levels that capture localized temporal dynamics at multiple granularities as sequences.The approach segments videos at different element lengths rather than using temporal subsampling.Source: §4.2 · p. 5

4.3. Spatial Pyramid

  • Each slice is divided into 2×5 body-part cells, then subdivided into a 40-cell spatial pyramid for finer cues.The representation combines these spatial levels with the temporal pyramid as the Spatio-Temporal Pyramidal Sequence (STPS).Source: §4.3 · p. 6

4.4. Localised Space-Time Descriptor

  • HOG3D represents each slice using concatenated space-time gradient histograms, with 50% overlap between adjacent cells and histogram equalisation.The resulting descriptors form a feature-sequence for each slice-sequence.Source: §4.4 · p. 6

5. Unsupervised Video Matching

The section develops TS-DTW to address DTW’s fixed endpoint alignment and use of all sequence elements, then extends it to fuse multiple feature-sequences. Time shifts support partial alignment and selection, while the fusion strategies differ in their alignment assumptions and scope.

5.1. Conventional DTW

  • Conventional DTW fixes the first and last elements as aligned and uses every sequence element, making distances sensitive to noisy observations.Its warp path is constrained to start at (1, 1) and end at the two sequence endpoints.Source: §5.1 · p. 6

5.2. Time Shift Driven Alignment and Selective Matching

  • TS-DTW shifts one sequence across the other, applies conventional DTW to each overlapping partial pair, then selects the best local match.The method varies ∆t across the sequences and combines local matching distances by selecting the best-matched result.Source: §5.2 · pp. 6–7, Figure 6 · p. 7
  • The shifts allow automatic selection of starting and ending poses and of well-aligned partial segments, potentially suppressing noisy elements without supervision.The two selection capabilities are described as complementary for incomplete, noisy surveillance videos.Source: §5.2 · p. 7

5.3. Generalisation to the Multi-Dimensional Setting

  • Dependent fusion jointly aligns dimensions with the same sequence length, restricting fusion to spatial divisions within a temporal-pyramid level.It assumes strongly correlated dimensions with identical warping paths and combines spatially divided sequences extracted from the same slice-sequence.Source: §5.3 · p. 8
  • Independent fusion combines dimension-specific matching distances, allowing different sequence structures and fusion across temporal-pyramid levels.The paper names the dependent and independent generalisations MDTS-DTWD and MDTS-DTWI, respectively.Source: §5.3 · p. 8

5.4. Model Complexity

  • TS-DTW’s repeated matching across time shifts supports short-sequence processing, and independent shifts can be parallelised.The cited analysis reports feature sequences typically shorter than 25 elements on PRID2011 and 40 on iLIDS-VID; standard DTW has O(|Xp(∆t)|^2) complexity, while fast variants have O(|Xp(∆t)|).Source: §5.4 · p. 8

6. Person Re-Identification

For a probe video, the method computes space-time matching distances to gallery videos and ranks them in ascending distance order. Those distances can also be fused with spatial-appearance distances after per-method normalization.

Person re-identification

  • Gallery videos are ranked by ascending space-time matching distance, with the rank-1 video treated as the most likely identity match.Distances are computed using TS-DTW or either multi-dimensional TS-DTW variant.Source: §6 · pp. 8–9

Combination with the spatial appearance methods

  • Space-time matching distances can be fused with spatial-appearance distances after each method’s per-probe distances are normalized to [0, 1].The method weights are set to 1 for generality rather than cross-validated, since cross-validation requires pairwise labelled data.Source: §6 · p. 9

7. Experimental Settings

The experiments compare video ReID on PRID2011 and iLIDS-VID against gait, sequence-matching, and person-ReID baselines. They evaluate both closed-world and open-world settings using repeated partitions and established matching and detection metrics.

7.1. Datasets

  • PRID2011 contains 400 sequences from 200 people across two outdoor views; iLIDS-VID contains 600 sequences from 300 people across two indoor views and has more occlusion and background clutter.Sequence lengths vary from 5 to 675 frames for PRID2011 and 22 to 192 frames for iLIDS-VID.Source: §7.1 · p. 9

7.2. Baseline Methods

  • The baseline set spans gait recognition, dynamic sequence matching, and person ReID methods, including supervised and unsupervised approaches.The comparisons include DTW variants, appearance-based methods, and the supervised video model DVR.Source: §7.2 · pp. 9–10

7.3. Person ReID Scenarios

  • Closed-world evaluation assumes every probe identity appears in the gallery, whereas open-world evaluation removes one third of test identities from the gallery as imposters.The model does not use the training partition because it is unsupervised.Source: §7.3 · p. 10

7.4. Evaluation Metrics

  • Closed-world comparisons use Cumulated Matching Characteristics curves, while open-world evaluation measures detection and identification rates alongside false accept rates.The open-world metrics assess detection and matching over decision thresholds and ranking positions.Source: §7.4 · pp. 10–11

7.5. Implementation Details

  • Both scenarios are evaluated over 10 random train/test partitions per dataset, with results averaged across folds.Implementation choices include a two-level temporal pyramid, a two-level spatial pyramid, and L1 distance between sequence elements.Source: §7.5 · p. 11

8. Experimental Results

Across the closed-world comparisons, MDTS-DTWI outperforms the evaluated gait, sequence-matching, and unsupervised ReID alternatives, while achieving competitive results against supervised DVR. The evaluations also report benefits from combining appearance and space-time methods and assess performance in the open-world setting.

8.1. Evaluation on Our Proposed Approach

  • Fusing temporal and spatial pyramid features improves matching, and combining temporal granularities further increases accuracy with MDTS-DTWI.Fine spatial division performs similarly on PRID2011 but better on iLIDS-VID; the two temporal pyramid levels produce similar results when considered separately.Source: Table 1 · p. 11, §8.1 · p. 11
  • Matching one probe video against the gallery takes an average of 5.26 seconds on PRID2011 and 9.50 seconds on iLIDS-VID.The reported average pairwise sequence matching time is around 0.06 second.Source: §8.1 · pp. 11–12

8.2.1. Comparing Gait Recognition and Temporal Sequence Matching Methods

  • MDTS-DTWI outperforms gait-recognition and dynamic sequence-matching alternatives on both datasets; the authors attribute this to selective matching of aligned, best-matched sequence parts.Conventional sequence matching can be affected by variable lengths, arbitrary starting and ending phases, noisy frames, and missing-data handling.Source: Table 2 · p. 12, §8.2.1 · p. 12

8.2.2. Comparing Person ReID Methods

  • MDTS-DTWI surpasses SS-/MS-ColLBP on both datasets and XQDA/KCVDCA on iLIDS-VID, while producing competitive results with supervised DVR.DVR uses space-time and colour information but requires exhaustive pairwise labels; MDTS-DTWI uses dynamic space-time cues without those labels.Source: Table 4 · p. 12, §8.2.2 · p. 13
  • Combining appearance and space-time methods improves matching accuracy, with the best unsupervised combinations differing by dataset in the closed-world evaluation.eSDC+Ours performs best on iLIDS-VID and RDL+Ours on PRID2011.Source: Table 5 · p. 13, §8.2.2 · p. 13

8.3. Evaluation on Open-World Person ReID

  • Fusing appearance and space-time methods improves recognition scores across FARs, and RDL+Ours is best on both datasets among the unsupervised combinations.The best combinations with truth-match labels are XQDA+DVR on PRID2011 and KCVDCA+Ours or KCVDCA+DVR on iLIDS-VID.Source: §8.3 · pp. 13–14
  • In open-world evaluation, the proposed method has the best PRID2011 DIR scores at FAR rates of 10% or less; on iLIDS-VID, DVR ranks first, followed by the proposed method and KCVDCA.DIR is evaluated at rank 1 over four FAR values.Source: §8.3 · p. 13, Table 6 · p. 14

9. Conclusion and Future Work

The conclusion presents STPS as a video representation and TS-DTW/MDTS-DTW as selective matching models for noisy, incomplete cross-view sequences. The authors identify further noise suppression and additional information sources as future work.

9. Conclusion

  • STPS encodes space-time information, while TS-DTW and MDTS-DTW selectively match incomplete and noisy sequences from disjoint camera views without exhaustive cross-view pairwise annotation.The framework is reported to complement spatial appearance-based ReID approaches.Source: §9 · p. 14

9. Future Work

  • Future work includes adding selection schemes beyond time shift to further suppress background-distraction noise and using extra information such as semantic text as matching constraints.Source: §9 · p. 14
Loading 1611.08512v2…