Source-linked AI summary
Video-based Person Re-identification with Spatial and Temporal Memory Networks
Chanho Eom, Geon Lee, Junghyup Lee, Bumsub Ham
TL;DR
Video-based person reID must handle spatial clutter and temporal distractions that obscure identity cues. STMN uses spatial and temporal memories to refine frame features and aggregate them, and achieves state-of-the-art performance on standard benchmarks.
Problem
Video reID retrieves identities across cameras, but background clutter and misaligned people or partial occlusions across frames distract from identity cues.
Source: §1 · p. 1Method
STMN uses spatial memory to suppress recurring scene details, temporal memory to focus aggregation on discriminative frames, and memory spread loss to encourage use of all memory items.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors report that ablation studies demonstrate the effectiveness of each STMN component.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Video-based person re-identification (reID) aims to retrieve person videos with the same identity as a query person across multiple cameras. Spatial and temporal distractors in person videos, such as background clutter and partial occlusions over frames, respectively, make this task much more challenging than image-based person reID. We observe that spatial distractors appear consistently in a particular location, and temporal distractors show several patterns, e.g., partial occlusions occur in the first few frames, where such patterns provide informative cues for predicting which frames to focus on (i.e., temporal attentions). Based on this, we introduce a novel Spatial and Temporal Memory Networks (STMN). The spatial memory stores features for spatial distractors that frequently emerge across video frames, while the temporal memory saves attentions which are optimized for typical temporal patterns in person videos. We leverage the spatial and temporal memories to refine frame-level person representations and to aggregate the refined frame-level features into a sequence-level person representation, respectively, effectively handling spatial and temporal distractors in person videos. We also introduce a memory spread loss preventing our model from addressing particular items only in the memories. Experimental results on standard benchmarks, including MARS, DukeMTMC-VideoReID, and LS-VID, demonstrate the effectiveness of our method.
1. Introduction
Video reID must contend with recurring spatial clutter and temporal distractions; STMN uses separate memories to suppress and prioritize these signals, with reported state-of-the-art benchmark performance.
1. Introduction
- Video reID must handle background clutter and temporal distractors such as misaligned people and partial occlusions across frames.Prior attention methods may miss sequence-wide context, while co-attention can focus on distractors shared across frames and produce incorrect representations.Source: §1 · pp. 1–2
- STMN uses spatial memory to store recurring scene distractors and temporal memory to store attentions optimized for recurring video patterns.The memories refine frame representations and aggregate them into a sequence representation focused on discriminative frames.Source: §1 · p. 2
- Memory spread loss encourages the model to use all memory items rather than repeatedly accessing only a few.The paper reports joint use of the memories and this loss as part of STMN.Source: §1 · p. 2
- STMN achieves state-of-the-art performance on standard video reID benchmarks.The authors also report ablation studies validating the method's components.Source: §1 · p. 2
2. Related Work
Prior video reID work uses attention, co-attention, and recurrent networks to handle spatial and temporal distractors, while STMN stores recurring scene details and temporal attentions in memories.
2. Related Work
- Attention methods [22] [18] [5] assign frame-level attentions without considering the whole sequence, while co-attention methods [16] [20] [40] model relations across frames.Shared backgrounds or occlusions can be propagated by co-attention and interfere with retrieval.Source: §2 · p. 2
- RNN-based methods [39] use previous-frame context for selective aggregation, but STMN uses RNNs to encode temporal patterns for memory access instead.The passage notes that direct RNN aggregation may be suboptimal because temporal context across whole frames is only considered at the last time step.Source: §2 · p. 2
- Earlier work overlooks recurring scene details and temporal patterns in surveillance videos, which STMN stores in spatial and temporal memories, respectively.The memories are used to produce person representations robust to spatial and temporal distractors.Source: §2 · p. 2
- Prior memory networks include approaches requiring extra supervision and lacking end-to-end training [37], while soft addressing [32] and key-value memories [25] address access and readout differently.STMN uses memories to record distracting scene features and temporal attentions, and adds a spread loss against repeatedly accessing only particular items.Source: §2 · p. 3
3. Approach
STMN combines spatial and temporal memories to suppress recurring scene distractors and use sequence patterns to focus frame aggregation. Its encoder, memory-access modules, and memory-spread and identification losses are trained end-to-end.
3.1. Overview
- STMN uses an encoder, spatial memory, and temporal memory, trained end-to-end with memory-spread, triplet, and cross-entropy terms.The spatial memory refines frame representations; the temporal memory aggregates the refined features into a sequence representation.Source: §3.1 · p. 3
3.2. Network architecture
- The spatial memory stores recurring scene-detail features and uses frame-level queries to retrieve them, refining person representations to suppress distracting information.Its keys access values that encode scene details; the retrieved features are used in the representation refinement.Source: §3.2 · pp. 3–4
- The temporal memory stores temporal-pattern prototypes and corresponding attentions; sequence query maps access these attentions to weight refined frame features.The query maps are pooled and processed by an LSTM to encode temporal context before memory access.Source: §3.2 · p. 4
- Unlike approaches that choose frames using person representations, STMN decouples temporal-context encoding from those representations.This design is intended to avoid encoding temporal context in representations that should distinguish identities.Source: §3.2 · p. 4
- Memory spread penalizes repeated access to a particular memory item, encouraging use of all items, while identification losses train discriminative person representations.The identification terms combine cross-entropy and batch-hard triplet losses, also applied to frame-level representations.Source: §3.2 · p. 5
4. Experiments
The experiments evaluate STMN on three video re-identification benchmarks, analyze its components and memory behavior, and compare it with prior methods. On the RRS setting, STMN sets a new state of the art on all three benchmarks.
4.1. Implementation details
- The evaluation uses the standard protocols and reports rank-1 accuracy and mAP on MARS, DukeV, and LS-VID, with PRID and iLIDS-VID excluded because they have few sequences from only two cameras.The final model uses spatial and temporal memory sizes M = 10 and N = 5, margin α = 0.3, and sequence length L = 6.Source: §4.1 · pp. 5–6
4.2. Discussion
- Ablations show that spatial and temporal memories improve the baseline and complement each other, with larger gains on LS-VID than on DukeV.The paper attributes the smaller DukeV gains to fewer distractors and a baseline already at 95% rank-1 accuracy; Table 1 compares variants across the three datasets.Source: §4.2 · p. 6, Table 1 · p. 6
- The memory spread loss consistently improves performance by encouraging use of all memory items rather than repeatedly accessing only a few.Matching-probability maps show that the loss supports diverse outputs depending on frame-level scene details and sequence-level temporal context.Source: §4.2 · p. 6, Figure 5 · p. 6
- Spatial-memory keys retrieve frames with similar scene details, and refinement suppresses background features so retrieval is less affected by clutter.The visual comparisons show differences before and after refinement and compare top-10 retrievals using original versus refined frame-level features.Source: §4.2 · pp. 6–7, Figure 6 · p. 6, Figure 7 · p. 6
- Temporal-memory keys represent recurring sequence patterns and their values store attentions; the memory lowers attention on frames with misalignment or occlusion.For sequences with fewer temporal distractors, it generates similar attentions across frames, behaving like temporal average pooling.Source: §4.2 · pp. 7–8, Figure 10 · p. 7
- On MARS, replacing the temporal memory with a two-layer MLP causes 1.3/1.5 (R-1/mAP) performance drops because it often focuses on only a few frames.The reported pattern remains when the MLP size increases.Source: §4.2 · p. 7
4.3. Comparison with the state of the art
- The paper compares RRS, which samples one frame from each of L chunks, with an all-frames strategy that averages representations from multiple sequences.Using all frames improves results for STMN and reproduced TCLNet and MGH, but increases evaluation time and search cost as video length grows.Source: §4.3 · p. 8
- On the RRS setting, STMN sets a new state of the art on MARS, DukeV, and LS-VID, surpassing some prior methods evaluated with all frames.The all-frames strategy gives performance gains for STMN and other methods but runs up to 35 times slower than RRS on LS-VID.Source: Table 2 · p. 8, §4.3 · p. 8
5. Conclusion
STMN uses spatial and temporal memories to refine frame-level representations and aggregate them into a sequence representation. The paper reports state-of-the-art performance on standard video re-identification benchmarks and effectiveness for each component.
5. Conclusion
- STMN extracts person representations robust to spatial and temporal distractors in videos.The conclusion describes spatial and temporal memories that refine frame-level representations and aggregate them into a sequence-level representation focused on discriminative frames.Source: §5 · p. 8
- The memory spread loss prevents certain memory items from remaining redundant.The paper also describes it as preventing the model from relying repeatedly on only some items.Source: §5 · p. 8
- STMN achieves state-of-the-art performance on standard video-based re-identification benchmarks, with component effectiveness supported by an extensive ablation study.Source: §5 · p. 8