PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Remains, What Changes: Identity Anchored Composed Gait Retrieval

Jingchen Fei, Zengbin Wang, Yukun Liu, Muyi Sun, Shibiao Xu, Man Zhang

arXiv:2608.26632v1cs.CV

TL;DR

Existing gait retrieval is largely limited to rigid visual matching, while natural-language interactive retrieval lacks suitable gait-language data and must preserve biometric identity during semantic changes. The paper introduces CoGR, constructs two VLM-annotated benchmarks, and proposes ComposeGait with identity-anchored Q-Former composition. ComposeGait achieves the best reported R@1 among compared methods on both benchmarks, establishing a strong CoGR baseline.

  • Problem

    Existing gait retrieval lacks natural-language interaction and gait-language training data, while generic composed retrieval can change attributes without preserving biometric identity.

  • Method

    The paper introduces CoGR, constructs Language-Augmented CCPG and CASIA-B with a VLM pipeline, and uses ComposeGait to inject sample-specific PIA identity tokens into both branches of a shared Q-Former.

  • Results

    ComposeGait achieves the best R@1 among compared methods on Language-Augmented CCPG and Language-Augmented CASIA-B.

  • Takeaways & Limitations

    The datasets and ComposeGait establish a strong baseline for identity-preserving, language-guided compositional gait retrieval.

  • Takeaways & Limitations

    Future work must extend CoGR to large-scale in-the-wild data, open-vocabulary descriptions, flexible annotation, and unconstrained identity-aware retrieval.

Abstract

from arXiv · show

Gait recognition has achieved remarkable progress, yet existing methods remain confined to rigid visual matching and often overlook the potential of natural language instructions for interactive retrieval. In this paper, we introduce Composed Gait Retrieval (CoGR), a novel task that retrieves a target gait sequence based on a reference sequence and a natural language modification query. To address the absence of existing datasets for this task, we design an automated annotation pipeline powered by large vision-language models (VLMs) to construct the first gait-language datasets: Language-Augmented CCPG and Language-Augmented CASIA-B. Building on this, we propose ComposeGait, an identity-anchored composition framework designed to prevent the identity drift that arises when generic composed retrieval follows the instruction but returns the wrong person. Its Part-aware Identity Adapter (PIA) aggregates multi-frame, part-aware identity evidence into a sample-specific ID token. We inject the ID tokens into both branches of a shared Q-Former to preserve identity, while excluding the ID-token outputs from the final retrieval embeddings. Joint identity and task-adapted composed-retrieval objectives optimize this space end to end. We evaluate ComposeGait on both benchmarks and show that it achieves the best R@1 among the compared methods, reaching 72.38% on Language-Augmented CCPG and 83.61% on Language-Augmented CASIA-B. These results establish ComposeGait as a strong baseline for CoGR. The datasets and code will be made publicly available.

1 Introduction

The paper introduces CoGR for retrieving gait sequences that preserve identity while following natural-language modifications, and builds datasets and ComposeGait to address identity drift. ComposeGait anchors identity with sample-specific tokens while adapting retrieval to instruction-conditioned gait matching.

  • Existing composed-retrieval methods can follow requested attribute changes while returning the wrong subject because gait identity and semantics are entangled.
  • CoGR retrieves a target gait sequence that preserves reference identity while satisfying a natural-language modification instruction.
  • An automated VLM-based pipeline converts discrete gait conditions into natural-language instructions, creating Language-Augmented CCPG and Language-Augmented CASIA-B.
  • ComposeGait uses a Part-aware Identity Adapter to project part-aware gait evidence into sample-specific ID tokens injected into a shared Q-Former.
  • ComposeGait injects identity tokens into both query and target branches while excluding their outputs from retrieval embeddings and jointly optimizing identity and CoGR objectives.
  • Extensive experiments on two language-augmented benchmarks evaluate retrieval effectiveness, identity preservation, and instruction satisfaction, establishing ComposeGait as a strong CoGR baseline.

2 Related Work

Related work spans gait recognition, composed image and video retrieval, and person-level search, but CoGR combines instruction following with identity-preserving gait-tracklet retrieval. The paper also addresses the lack of natural-language gait annotations through automated VLM-based construction.

  • Gait Recognition: Gait recognition methods use silhouettes, set-based representations, semantic parsing, skeletons, and other modalities to identify people under varied conditions.
  • Gait Recognition: CASIA-B and OU-MVLP provide controlled multi-view benchmarks, while Gait3D, GREW, CCPG, CCGR, and SUSTech1K introduce outdoor or cross-covariate variation.
  • Gait-Language Alignment: Existing gait datasets encode conditions as discrete labels, so the paper uses an automated VLM pipeline to generate fine-grained textual descriptions.
  • Composed Image and Video Retrieval: Composed image retrieval combines a reference image with modification text, using supervised joint representations, feature fusion, and Q-Former-based alignment.
  • Composed Image and Video Retrieval: Static-image person retrieval does not aggregate multi-frame gait evidence or condition both query and target encoding on sample-specific identity information.
  • Composed Image and Video Retrieval: CoGR lies at the intersection of composed retrieval and gait recognition: valid targets must follow instructions, preserve biometric identity, and retain unspecified conditions.

3 Methodology

The methodology defines CoGR as identity-preserving composed retrieval, constructs language annotations for gait conditions, and implements ComposeGait with identity-conditioned shared Q-Formers and relevance-aware supervision.

  • Paradigm Definition: CoGR ranks gallery gait sequences by the inner product between a reference–text query embedding and each candidate target embedding.The reference sequence and modification instruction form fr, while each gallery sequence forms ft; both embeddings are L2-normalized.
  • Paradigm Definition: CoGR relevance preserves subject identity while applying requested condition changes and retaining unspecified conditions.The target condition is defined by updating only components specified by the modification instruction.
  • Language-Augmented Dataset Construction: The annotation pipeline converts gait tracklets into language-guided retrieval data through attribute extraction, static description assembly, and dynamic triplet generation.It uses VLM-filled semantic slots and dataset-specific templates, including viewpoint changes for CASIA-B but not CCPG.
  • ComposeGait: ComposeGait uses a frozen ViT-G, PIA, a projected sample-specific ID token, and a shared Q-Former to encode both composed queries and targets.PIA extracts hierarchical, part-aware identity evidence with multi-layer fusion, temporal max pooling, and horizontal pooling before token projection.
  • ComposeGait: The composed-query branch receives reference visual tokens, modification text, and a reference ID token, whereas the target branch receives target visual tokens and its ID token.The branches share Q-Former weights, placing their outputs in a common embedding space while conditioning attention on identity.
  • Joint Identity and Composition Learning: Only original query-token outputs form the normalized retrieval embeddings, while ID-token outputs guide attention without entering those embeddings directly.Identity and hard-triplet losses supervise PIA jointly with a CoGR contrastive loss that supports multiple relevant positives and excludes ambiguous same-identity targets.

4 Experiments

Experiments evaluate ComposeGait on two language-augmented benchmarks using strict retrieval, instruction-satisfaction, and identity-preservation metrics. ComposeGait achieves strong top-ranked retrieval and ablations support the importance of identity anchoring and ID-token content.

  • Experimental Setup: Language-Augmented CCPG contains 50,000 triplets, while Language-Augmented CASIA-B contains 64,506 triplets spanning viewpoints and appearance conditions.Training and evaluation identities are disjoint, and original gallery definitions are preserved.
  • Experimental Setup: R@K requires identity preservation, unchanged unspecified covariates, and satisfaction of the requested change, while SC-R@K isolates instruction satisfaction.CASIA-B additionally reports viewpoint-only, attribute-only, and composite recall, alongside ID R@1 for identity preservation.
  • Comparison with Prior Methods: 72.38% R@1 on CCPG surpasses FAFA by 3.56 pp, while ComposeGait obtains the best ID R@1 of 76.56%.FAFA performs better at R@5 and R@10, but ComposeGait has stronger top-ranked matching under joint identity and instruction constraints.
  • Comparison with Prior Methods: 83.61% R@1 on CASIA-B exceeds SPRC by 17.15 pp, with 96.56% viewpoint-only recall and 78.36% composite recall.Conventional supervised and zero-shot CIR methods reach at most 22.34% R@1 under severe cross-view variation.
  • Architectural Ablation Study: With shared Q-Formers, PIA raises R@1 from 61.47% to 67.74% and ID R@1 from 65.31% to 72.63%.Without sharing, gains shrink to 0.81 pp and 1.69 pp, while PIA adds 0.03 billion parameters in both settings.
  • Controlled ID-Token Interventions: Randomizing both ID tokens lowers R@1 to 68.39%, ID R@1 to 72.17%, and mAP to 65.24%, below the corresponding Zero ID results.Shuffling the query-side identity produces the largest degradation: 64.27% R@1, 67.83% ID R@1, and 62.23% mAP.
  • Qualitative Results: Qualitative examples show instruction-following appearance and viewpoint changes while preserving reference identity across CCPG and CASIA-B.CASIA-B examples include drastic side-to-frontal perspective shifts combined with appearance modifications.

5 Conclusion

The paper introduces CoGR, its VLM-generated language benchmarks, and ComposeGait for identity-anchored retrieval. It reports strong R@1 performance while identifying broader, open-vocabulary and unconstrained settings as future directions.

  • 5 Conclusion: ComposeGait mitigates identity drift by generating sample-specific ID tokens from part-aware, multi-frame gait features and injecting them into both branches of a shared Q-Former.The framework is evaluated as an identity-anchored composition method for CoGR.
  • 5 Conclusion: Future work will extend CoGR to large-scale in-the-wild datasets, open-vocabulary descriptions, flexible annotation strategies, and unconstrained identity-aware retrieval.These directions define the stated scope beyond the current benchmarks.
  • 5 Conclusion: An automated VLM pipeline constructs Language-Augmented CCPG and Language-Augmented CASIA-B by extracting visual attributes and synthesizing compositional instructions.CASIA-B prompts cover clothing, carried accessories, and controlled viewpoint shifts.
  • 5 Conclusion: The pipeline constrains generated language with placeholder-based templates, JSON-only outputs, controlled vocabulary, and dataset-specific appearance and viewpoint prompts.CCPG prompts also mask invisible shoe regions and handle occlusion without describing shoes.
  • 5 Conclusion: The annotation design keeps queries visually grounded and structurally consistent at scale, omitting viewpoint prompts for CCPG because its camera identifiers lack user-facing semantic meaning.CASIA-B includes viewpoint-change templates, while both datasets share appearance-change templates.

B Manual Review of the Datasets

The synthesized gait-language datasets were manually reviewed through random sampling by independent evaluators, with more than 92% of reviewed annotations satisfying all criteria.

  • Review protocol: 500 triplets from each dataset were randomly sampled, yielding 1,000 reviewed CASIA-B and CCPG triplets.Each triplet included source and target gait sequences, descriptions, and a modification instruction.
  • Review protocol: Three independent evaluators assessed description fidelity and whether instructions accurately captured source-to-target changes.A triplet was fully correct only when all evaluators agreed that every textual component was accurate.
  • Review results: More than 92% of reviewed annotations satisfied the evaluation criteria, indicating generally reliable automated annotations.The remaining errors motivate quality control for automatically generated annotations.

C.1 Bilateral ID-Token Injection

Bilateral ID-token conditioning is consistently stronger than conditioning either retrieval branch alone, improving both strict retrieval and identity-preserving retrieval.

  • Ablation results: Bilateral injection improves R@1 by 2.37 pp and ID R@1 by 2.60 pp over composed-query-only injection.It also exceeds target-only injection by 1.26 pp in R@1 and 1.47 pp in ID R@1.
  • Ablation results: Target-only injection itself improves R@1 by 1.11 pp and ID R@1 by 1.13 pp over composed-query-only injection.This suggests that anchoring gallery representations is particularly useful.
  • Interpretation: The best performance requires conditioning both shared-Q-Former branches on the identity evidence of their respective inputs.This places the composed query and gallery candidates in a mutually identity-aware embedding space.

C.2 Comparison of Contrastive Objectives

The contrastive-loss comparison evaluates single-positive, multi-positive, and filtered-denominator objectives for CoGR, where same-identity condition-mismatched targets require special treatment.

  • Objective design: CoGR defines P(i) as all targets satisfying the relevance criterion and excludes same-identity, condition-mismatched targets A(i) from the denominator D(i).The setup compares objectives under a common notation while keeping the remaining experimental setup unchanged.
  • Objective design: InfoNCE uses one designated positive, SupConLoss uses all relevant positives but retains the full batch, and CoGR removes A(i) from the denominator.All objectives are averaged over batch queries.
  • Ablation results: SupConLoss improves over InfoNCE by 0.29 pp in R@1 and by 1.31, 0.23, and 0.04 pp in attribute-only, viewpoint-only, and composite SC-R@1.These gains support using all in-batch targets satisfying the CoGR relevance criterion as positives.
  • Ablation results: The CoGR loss further improves viewpoint-only SC-R@1 by 1.94 pp over SupConLoss, while its R@1 gain is 0.01 pp and composite SC-R@1 gain is 0.03 pp.The improvement is concentrated in change-specific recall, especially viewpoint compliance.

D Qualitative Results and Failure Analysis

Qualitative examples show successful retrievals that satisfy requested attribute, viewpoint, and composite changes while preserving the subject identity.

  • Visualization format: Visualizations include the instruction, query, ground truth, and top-5 results for jointly inspecting identity preservation and condition consistency.Blue and orange borders mark query and ground truth, while green and red borders follow evaluation relevance labels.
  • Successful cases: Figures 5 and 6 provide successful CCPG and CASIA-B examples covering isolated and multi-condition changes.Each task family includes two examples spanning attribute, viewpoint, and composite condition changes.
  • Successful cases: Top-ranked results in successful cases follow the requested condition without replacing the subject identity.The examples include clothing changes, accessory removal, viewpoint shifts, and combined appearance-viewpoint constraints.

D.2 Failure Cases

Failure cases expose identity drift, ranking errors, condition leakage, and sensitivity to illumination-confounded color prompts. These errors motivate stronger identity-conditioned scoring and preservation of unmodified conditions.

  • Identity drift occurs when visually plausible candidates from other subjects match attribute, viewpoint, or composite semantics more strongly.This can happen even when the retrieved candidate satisfies much of the requested modification.
  • Some valid targets remain within the top five, but final ranking insufficiently weights available identity evidence.The failure is therefore sometimes a ranking error rather than complete retrieval failure.
  • The second viewpoint example preserves identity and the requested back view but changes the unmodified bag-carrying condition from bg to nm.This illustrates condition leakage under an instruction that modifies viewpoint without requesting a bag change.

Ethical Statement

The study uses publicly available CCPG and CASIA-B data with automatically generated annotations and derived retrieval splits. It acknowledges misuse, representation, and deployment risks, and recommends restricted research use with safeguards and bias evaluation.

  • The study uses publicly available CCPG and CASIA-B datasets without collecting new human-subject data or adding identity annotations.It follows the original datasets' licenses and intended research use.
  • The language-augmented datasets contain automatically generated textual annotations, retrieval triplets, and splits derived from the original datasets.
  • Composed gait retrieval could be misused for non-consensual identification, persistent tracking, or large-scale surveillance.
  • Performance may vary across demographic groups, clothing conditions, physical abilities, and capture environments unevenly represented in the source datasets.
  • Real-world deployment should require authorization, privacy safeguards, human oversight, and careful evaluation of demographic and operational biases.The annotations, retrieval splits, code, and models are recommended for legitimate research use under the original datasets' licenses and access policies.
Loading 2608.26632v1…