PaperlayerView exact revision on arXiv

Source-linked AI summary

A Generative AI Integrated Multimodal Framework for Low-Latency Multi-Camera Person Re-Identification

Leon Fernando, C Dombawala, P. Hettigoda, Vanodhya G. Warnasooriya, Ishara Neranjana, Rashmika Nawaratne

arXiv:2609.14419v1cs.CVcs.AIeess.IV

TL;DR

Person re-identification is difficult under changing camera conditions and variable cue availability, while many multimodal systems process every modality for every query. The paper proposes a cost-aware cascade that adds semantic or facial evidence selectively, resolving 60.7% of DukeMTMC-reID queries and 68.4% of Market-1501 queries without semantic enhancement.

  • Problem

    Person re-identification must match identities across cameras despite image variation and unreliable cues, while always-on multimodal systems add computational overhead and inference latency.

    Source: §I · p. 1

  • Method

    The framework starts with visual retrieval, selectively adds VLM-generated semantic embeddings for ambiguous queries, and optionally uses facial features with reliability-aware fusion.

    Source: §I · p. 1, §III · pp. 3–4

  • Results

    60.7% of DukeMTMC-reID queries and 68.4% of Market-1501 queries are resolved without semantic enhancement; estimated average DukeMTMC-reID query latency is approximately 900 ms.

    Source: 2) Latency and Efficiency Analysis · p. 5

  • Takeaways & Limitations

    The reported results support selective activation of expensive modalities for ambiguous queries while maintaining competitive retrieval performance.

    Source: §VI · p. 6

Abstract

from arXiv · show

Person re-identification (ReID) is essential for multi-camera surveillance and tracking, yet remains difficult due to viewpoint and illumination changes, occlusion, background clutter, and low resolution imagery. We propose a generative AI integrated multimodal ReID framework designed explicitly for robustness under missing cues and low latency deployment. The key idea is a cost aware early-exit cascade that prioritizes inexpensive, high confidence evidence and only triggers expensive modalities for ambiguous cases. Our system integrates (i) global visual embeddings from segmented person regions, (ii) automatically generated fine grained semantic attribute descriptions generated by vision-language models (VLMs), and (iii) optional facial embeddings when face observations are reliable. To optimize the balance between accuracy and latency, we use a cost aware early-exit cascade instead of fusing all modalities. Specifically, we first inspect the top-k retrieval results to determine whether the query is unambiguous. If the best match is clearly separated from the remaining candidates, we stop early and return the result to minimize latency; in ambiguous cases, we keep multiple hypotheses and invoke additional modalities (face/semantic) with adaptive reliability weighting to refine the decision. We report person re-identification performance using mAP and Rank-1 accuracy on the Market-1501 and DukeMTMC-reID benchmarks. The proposed adaptive early-exit cascade resolves 60.7% of DukeMTMC-reID queries and 68.4% of Market-1501 queries without invoking semantic reasoning, reducing computational overhead while maintaining competitive retrieval performance.

I. INTRODUCTION

Person re-identification must match identities across cameras despite changing image conditions, unreliable cues, and computational limits. This work proposes selective multimodal inference, against a related-work landscape that includes semantic, visual, facial, and staged retrieval approaches.

I. INTRODUCTION

  • ReID is challenged by viewpoint, illumination, occlusion, blur, clutter, low resolution, and limited computation, all of which complicate real-world deployment.The task supports surveillance, forensic retrieval, and multi-camera tracking.Source: §I · p. 1
  • Existing multimodal systems often process every modality for every query, adding latency even when a match could be resolved with inexpensive evidence.Face and semantic-cue availability can also vary across queries.Source: §I · p. 1
  • The proposed framework selectively activates semantic embeddings and facial features after inexpensive visual cues, with fusion weights adjusted to modality availability and confidence.Semantic embeddings are generated by vision-language models.Source: §I · p. 1

II. RELATED WORK

  • Vision-language methods align visual and textual representations, while CLIP-based and caption-based approaches incorporate semantic information into ReID, [3], [4].Related work also discusses domain adaptation and persistent CLIP limitations under domain shift,.Source: §II · p. 2
  • Prior work combines OSNet visual features, semantic cues, facial embeddings, staged retrieval, and temporal information, but commonly uses fixed inference or uniform frame processing,,,,.The paper positions its approach as combining vision-language reasoning, visual features, and facial cues within an adaptive early-exit cascade.Source: §II · p. 2

III. METHODOLOGY

The method builds person representations from segmented detections using visual, semantic, and quality-gated facial features. Retrieval starts with visual evidence and selectively adds modalities for uncertain matches, with quality filtering and temporal aggregation supporting the pipeline.

III. METHODOLOGY

  • The pipeline detects and segments people, extracts multimodal features, performs adaptive cascade retrieval, and applies temporal aggregation under a cost-aware strategy.Segmentation masks isolate foreground content to reduce background interference before feature extraction.Source: §III · pp. 2–3
  • OSNet x1.0 extracts visual features, while Qwen3-VL-4B-Instruct generates fine-grained attribute descriptions encoded by all-MiniLM-L6v2.The semantic branch captures clothing and contextual appearance cues for conditions including viewpoint variation, occlusion, and low resolution.Source: §III · p. 3
  • When available, face regions detected by MTCNN are evaluated for resolution, visibility, and detection confidence before InceptionResNetV1 extracts embeddings from reliable faces.The quality gate avoids using low-quality facial inputs.Source: §III · p. 3
  • Retrieval first ranks visual top-k candidates by cosine similarity and exits when confidence exceeds a threshold; ambiguous cases trigger semantic refinement and optional face verification.Final predictions use adaptive fusion based on modality availability and reliability.Source: §III · pp. 3–4
  • A quality-aware filter suppresses ambiguous matches using measures such as inter-class similarity margins, while video frames are filtered by detection confidence and motion consistency before aggregation.Remaining frame embeddings are combined by mean pooling or exponential moving average into track-level representations.Source: §III · p. 4
  • The gallery-construction and cascade-search algorithms together implement selective modality activation and early-exit retrieval for balancing accuracy and computational efficiency.The pipeline also includes efficient temporal aggregation for real-time multi-camera deployment.Source: §III · p. 4

1) Datasets and Implementation Details:

The evaluation uses DukeMTMC-reID and Market-1501, with implementation details reported for the GPU-enabled system.

2) Datasets and Implementation Details:

  • DukeMTMC-reID contains 17,661 gallery images across 1,110 identities and 2,228 queries across 702 identities, captured from 8 camera views.The dataset counts are reported as the evaluation setup.Source: 1) Datasets and Implementation Details · p. 4
  • Market-1501 contains 13,115 gallery images and 3,368 queries, with 750 identities represented in each set across 6 cameras.The dataset counts are reported as the evaluation setup.Source: 1) Datasets and Implementation Details · p. 4
  • The PyTorch implementation ran on an NVIDIA RTX 5070 Ti Laptop GPU with 12 GB VRAM and CUDA 12.8, at approximately 84% GPU utilization.The reported software version is PyTorch 2.11.0+cu128.Source: 1) Datasets and Implementation Details · p. 4

2) Evaluation Metrics:

The evaluation section names standard ranked-retrieval metrics and frames the comparison around accuracy and inference cost. The proposed design prioritizes reducing computation and latency while preserving acceptable retrieval performance.

2) Evaluation Metrics:

  • The evaluation uses CMC at Rank-1, Rank-5, and Rank-10, alongside mean Average Precision (mAP), to assess retrieval accuracy.These metrics cover multiple ranking conditions.Source: 2) Evaluation Metrics · p. 4

1) Overall Performance:

  • The comparison covers CNN-, ViT-, and LVLM-based ReID methods, and the paper notes that LVLM methods incur overhead by invoking large VLMs for every query.The supplied table passage identifies the comparison scope but provides no cell values.Source: 1) Overall Performance · p. 4, Table I · p. 4
  • CNN- and ViT-based methods are described as relying heavily on supervised, dataset-specific features, limiting generalization without extensive retraining.The passage reports this as a limitation of those approaches.Source: 1) Overall Performance · p. 4
  • The proposed design uses pretrained models and prompt-optimized semantic representations, targeting lower computation and inference time while preserving acceptable accuracy rather than outperforming all methods in accuracy.Its adaptive cascade avoids activating expensive modalities such as VLMs for every query.Source: 1) Overall Performance · p. 4

2) Latency and Efficiency Analysis:

The adaptive cascade resolves many queries before expensive semantic processing, reducing query latency. Retrieval performance is stable across the tested threshold settings.

2) Latency and Efficiency Analysis:

  • Detection + Crop, Cascade Retrieval, and Vector DB Update run for every query, forming the lightweight baseline computational cost.Table II presents component-wise latency analysis.Source: 2) Latency and Efficiency Analysis · p. 4, Table II · p. 5
  • Semantic extraction takes approximately 2.1 s and is activated for 31.6% of Market-1501 and 39.3% of DukeMTMC-reID queries.The remaining queries use the lightweight visual branch, reducing average inference latency versus an always-on multimodal pipeline.Source: 2) Latency and Efficiency Analysis · p. 4
  • 60.7% of DukeMTMC-reID and 68.4% of Market-1501 queries exit before semantic enhancement; DukeMTMC-reID averages approximately 900 ms.Table III reports the cascade path distribution across both datasets.Source: Table III · p. 5, 2) Latency and Efficiency Analysis · p. 5
  • Across threshold configurations, DukeMTMC-reID Rank-1 accuracy ranges from 90.1% to 90.6% and mAP from 81.6% to 82.2%.The tested operating thresholds were EARLY_EXIT_THRESHOLD=0.75, MARGIN_THRESHOLD=0.01, STD_THRESHOLD=0.15, and STAGE_1_TOP_K=20.Source: 2) Latency and Efficiency Analysis · p. 5

C. Qualitative Analysis

The examples compare descriptions and matching scores for the same identity across two camera views. Semantic similarity remains high despite moderate visual similarity.

C. Qualitative Analysis

  • The two descriptions vary in wording, such as “sweatshirt” versus “jacket,” but both capture color, clothing type, and overall appearance.The descriptions are for images of the same identity captured under different viewpoints.Source: C. Qualitative Analysis · p. 5
  • Fig. 2(a) and Fig. 2(b) show the same identity across different camera views.The examples are described as samples of that identity captured under different viewpoints.Source: Figure 2 · p. 5, C. Qualitative Analysis · p. 5
  • Image (a) has visual similarity 0.80, semantic similarity 0.97, and overall matching score 0.86; image (b) scores 0.78, 0.91, and 0.83, respectively.The passage reports that higher semantic scores compensate for moderate visual similarity in both examples.Source: C. Qualitative Analysis · p. 5

V. ABLATION STUDIES

The ablation passages contrast a ReID-specific visual extractor with broad semantic representations and describe how the semantic branch captures person attributes.

A. Visual Feature Extractor

  • OSNet x1.0 is used for discriminative ReID features after CLIP’s semantic representations were found insufficiently granular for precise identity distinctions.The authors report that CLIP can collapse distinct identities into similar semantic clusters; OSNet’s multiscale architecture is intended to decouple identity from environmental noise.Source: A. Visual Feature Extractor · p. 5

B. Semantic Feature Extractor

  • The VLM semantic branch supplies complementary identity cues, with prompts targeting upper- and lower-body clothing, footwear, accessories, and distinctive traits.The stated use cases include poor facial visibility or low sensor resolution.Source: B. Semantic Feature Extractor · pp. 5–6

C. Inference Efficiency Analysis

The framework balances retrieval accuracy and computational efficiency through an adaptive early-exit cascade. The reported results show that most queries avoid semantic reasoning, while the conclusion identifies further evaluation and efficiency work.

C. Inference Efficiency Analysis

  • Qwen3-VL-4B achieves the lowest reported semantic inference latency at 2.12 s while maintaining high discriminative accuracy; gemma-4-E2B-it takes 34.23 s.The comparison weighs descriptive richness against computational overhead.Source: C. Inference Efficiency Analysis · p. 6

VI. CONCLUSION

  • The cascade resolves over 60% of queries without semantic reasoning, with average query latency of approximately 900 ms while maintaining competitive retrieval performance.The conclusion attributes this to selectively activating expensive modalities for ambiguous queries.Source: §VI · p. 6
  • Future work is directed toward large-scale real-world evaluation, faster semantic inference, and integrating emerging foundation and transformer-based ReID models.The conclusion describes the framework as an efficient and scalable approach to real-time multi-camera ReID through adaptive multimodal inference.Source: §VI · p. 6
Loading 2609.14419v1…