PaperlayerView exact revision on arXiv

Source-linked AI summary

Occluded Gait Recognition with Mixture of Experts: An Action Detection Perspective

Panjian Huang, Yunjie Peng, Saihui Hou, Chunshui Cao, Xu Liu, Zhiqiang He, Yongzhen Huang

arXiv:2609.18432v1cs.CV

TL;DR

Real-world occlusions make gait recognition difficult by removing or corrupting information and misaligning visible body regions. GaitMoE treats gait as a composition of actions and uses temporal and action experts, alongside joint training with identity labels. The authors report that it outperforms state-of-the-art methods across four gait benchmarks.

  • Problem

    Real-world occlusions introduce missing and noisy information and position and scale misalignment, while existing studies largely overlook these conditions.

    Source: §1 · p. 2

  • Method

    GaitMoE treats gait sequences as compositions of actions, using temporal experts to form anchors and action experts to integrate similar anchors into proposals, trained jointly with gait recognition using ID labels.

    Source: §1 · p. 2, §3.3 · p. 7

  • Results

    GaitMoE significantly outperforms other state-of-the-art methods on OccGait, OccCASIA-B, Gait3D, and GREW.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The paper presents sequence context as a way to infer occluded body regions and integrate holistic and occluded actions, and introduces OccGait with varied occlusion scenarios and type annotations.

    Source: §1 · pp. 2–3

Abstract

from arXiv · show

Extensive occlusions in real-world scenarios pose challenges to gait recognition due to missing and noisy information, as well as body misalignment in position and scale. We argue that rich dynamic contextual information within a gait sequence inherently possesses occlusion-solving traits: 1) Adjacent frames with gait continuity allow holistic body regions to infer occluded body regions; 2) Gait cycles allow information integration between holistic actions and occluded actions. Therefore, we introduce an action detection perspective where a gait sequence is regarded as a composition of actions. To detect accurate actions under complex occlusion scenarios, we propose an Action Detection Based Mixture of Experts (GaitMoE), consisting of Mixture of Temporal Experts (MTE) and Mixture of Action Experts (MAE). MTE adaptively constructs action anchors by temporal experts and MAE adaptively constructs action proposals from action anchors by action experts. Especially, action detection as a proxy task with gait recognition is an end-to-end joint training only with ID labels. In addition, due to the lack of a unified occluded benchmark, we construct a pioneering Occluded Gait database (OccGait), containing rich occlusion scenarios and annotations of occlusion types. Extensive experiments on OccGait, OccCASIA-B,Gait3D and GREW demonstrate the superior performance of GaitMoE.OccGait is available at https://github.com/BNU-IVC/OccGait.

1 Introduction

Real-world occlusions create missing and noisy information as well as position and scale misalignment in gait representations. The paper treats gait as a composition of actions and proposes GaitMoE to use sequence context for recognition under occlusion.

1 Introduction

  • Occlusion causes missing and noisy information and position and scale misalignment, complicating fine-grained feature matching.Discarding occluded frames can remove discriminative regions, while retaining them can cause erroneous feature extraction and matching.Source: §1 · p. 2
  • Gait continuity lets holistic frames provide cues about occluded regions, while gait-cycle repetition lets similar holistic and occluded actions be integrated.The paper frames gait sequences as compositions of actions and uses this action-detection perspective for recognition.Source: §1 · pp. 2–3
  • GaitMoE uses temporal experts to form action anchors and action experts to integrate similar anchors into proposals used as identification features.The experts specialize in body regions and temporal granularity; action proposals serve identification rather than action localization and classification.Source: §1 · p. 3, Figure 2 · p. 3
  • OccGait provides four occlusion conditions and explicit occlusion-type annotations for each gait sequence.Its conditions are none, carrying, crowd, and static occlusion.Source: §1 · p. 3
  • GaitMoE significantly outperforms other state-of-the-art methods on OccGait, OccCASIA-B, Gait3D, and GREW.The experiments assess controlled occlusions as well as complex covariates, including uncertain occlusions.Source: §1 · p. 4

2 Related Work

The related work reviews gait-recognition representations, occlusion-specific datasets and architectures, and mixture-of-experts models. It positions GaitMoE as an extension of expert specialization to fine-grained actions in occluded gait recognition.

2.1 Gait Recognition

  • Appearance-based gait methods use silhouette templates or frame sequences, while model-based methods represent body structure with skeletons or meshes.The review notes that most approaches neglect substantial real-world occlusion.Source: §2.1 · p. 4

2.2 Occluded Gait Recognition

  • Earlier occluded-gait datasets and architectures vary in occlusion coverage and availability, with prior methods using reconstruction-based and reconstruction-free approaches.Reported dataset limitations include single occlusion scenarios, small occlusion regions, or lack of public availability.Source: §2.2 · pp. 4–5
  • GaitMoE extracts frame features, forms temporal action anchors, and uses part-wise action experts to combine anchors into identification features.Horizontal partitioning precedes the action experts, whose proposals are concatenated for identification.Source: Figure 3 · p. 5

2.3 Mixture of Experts

  • Mixture-of-experts models use router weights to aggregate specialized experts, and this work applies that design to representative actions in occluded gait recognition.The review describes the approach as extending expert specialization to fine-grained actions.Source: §2.3 · p. 5

3 Methodology

GaitMoE treats a gait sequence as a composition of actions, using temporal experts to form action anchors and action experts to integrate them into proposals. It trains action detection jointly with gait recognition using identity labels.

3.1 Mixture of Temporal Experts

  • Mixture of Temporal Experts (MTE) forms action anchors from adjacent frames using dilated temporal experts, while preserving channel segments to select cleaner regions.Experts use different dilation ratios to capture temporal context; residual learning is embedded within MTE.Source: §3.1 · p. 6

3.2 Mixture of Action Experts

  • Mixture of Action Experts (MAE) uses learnable action prototypes to score anchors across time and weight similar anchors into action proposals.Softmax scores weight the corresponding values, and concatenated proposals feed identification layers.Source: §3.2 · p. 7

3.3 Joint Loss

  • GaitMoE trains end-to-end using identity labels, with triplet and cross-entropy losses constraining each part independently.The joint objective combines the two losses with a balancing hyperparameter.Source: §3.3 · pp. 7–8

4 The OccGait Benchmark

OccGait was collected to support qualitative and quantitative analysis of gait recognition under different occlusion types. It includes four occlusion situations and uses a subject split and Rank 1 evaluation protocol.

4 The OccGait Benchmark

  • OccGait contains 101 subjects, four occlusion types, eight camera views, and over 80,000 sequences.The four situations are no occlusion, carrying, crowd, and static occlusion.Source: §4 · p. 8

4.1 Data Collection and Pre-processing

  • Collection used three cameras and walking routes to obtain eight views, with four normal-walking repetitions and two repetitions per occlusion type.The collection includes carrying, crowd, and static occlusions, alongside normal walking.Source: §4.1 · p. 8
  • Carrying scenarios obscure the upper body or torso and lower body, crowd scenarios range from partial to complete occlusion, and static obstacles hinder the lower body.MaskFormer extracts silhouettes from the original RGB data for input.Source: §4.1 · p. 9

4.2 Evaluation Protocol

  • The evaluation assigns 51 odd-numbered subjects to training and 50 even-numbered subjects to testing, then measures Rank 1 under occlusions and viewing angles.NM01 and NM02 from each test subject form the gallery.Source: §4.2 · p. 9

5 Experiments

Across controlled-occlusion and wild gait benchmarks, GaitMoE outperforms state-of-the-art methods. The experiments examine diverse datasets, the MTE and MAE components, and accuracy–efficiency trade-offs; the conclusion notes redundancy among some experts.

5.1 Datasets

  • Evaluation spans OccGait’s NO, CA, CR, and ST conditions, OccCASIA-B’s four occlusion tests and three walking conditions, and the wild datasets Gait3D and GREW.Gait3D contains 3,000 subjects and 25,309 sequences; GREW contains 26,345 subjects and 128,671 sequences.Source: §5.1 · pp. 9–10, Table 1 · p. 10, Table 2 · p. 11

5.3 Main Results

  • GaitMoE outperforms state-of-the-art methods across tested occlusion types on OccGait and OccCASIA-B, while GaitMoE-T and GaitMoE-B achieve the highest performance on Gait3D and GREW.On OccCASIA-B, the paper reports margins over GaitBase of 12.0% in NO and 16.8% in DO under the cloth-changing condition.Source: Table 1 · p. 10, Table 3 · p. 11, Table 2 · p. 11

5.2 Implementation Details

  • Inputs are resized to 64 × 44, and GaitMoE is trained end-to-end with ID labels using dataset-specific backbones, frame sampling, and optimization schedules.Training samples 30 frames per gait sequence; inference uses all frames.Source: §5.2 · p. 11

5.4 Ablation Study

  • MTE and MAE each improve dynamic information extraction, while their combination significantly improves over the baseline; the study selects 16 action experts to balance redundancy against insufficient action coverage.The paper also reports that smaller S or larger K degrades original information, while larger S or smaller K restricts dynamic information.Source: Table 4 · p. 12, Table 5 · p. 12, §5.4 · p. 13
  • GaitMoE achieves state-of-the-art accuracy without substantially increasing computational cost, whereas DyGait demands significant computation and GaitBase is more efficient but less accurate.The conclusion identifies redundancy and similarity among some experts as a limitation and points to expert-selection optimization as future work.Source: §5.4 · p. 14, Figure 7 · p. 14, §5.5 · p. 14

5.5 Conclusion and Limitations

  • The conclusion identifies redundancy and similarity among some action-anchor experts and proposes exploring expert-selection optimization.Source: §5.5 · p. 14

Related papers

Loading 2609.18432v1…