PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic-to-Real Text-Based Person Anomaly Search via Multi-Model Retrieval Fusion and Cross-Encoder Reranking

Eugene Denteh, Blessing Agyei Kyem, Andrews Danyo, Joshua Kofi Asamoah, Richard Dzinyela, Solomon Owusu-Ansah, Dickson G. Asiedu, Armstrong Aboah

arXiv:2610.31930v1cs.CV

TL;DR

Text-based person anomaly search must match appearance and behavior across a synthetic-training and real-surveillance gap with visually similar distractors. The paper adapts and fuses Qwen3-VL and SigLIP2 retrieval with cross-encoder reranking, achieving 94.7815% mAP and 90.8493% R@1 in AI City Challenge 2026 Track 4.

  • Problem

    Person anomaly search requires retrieving images that match both the person and behavior in a query, despite domain differences, near-duplicate frames, and plausible distractors.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Method

    The framework adapts Qwen3-VL on synthetic descriptions, fuses six checkpoints with frozen SigLIP2 scores, and reranks top candidates using a cross-encoder.

    Source: §1 · p. 2

  • Results

    94.7815% mAP and 90.8493% R@1; the final system placed eighth on the reported public leaderboard.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The results support combining complementary retrieval representations with focused candidate refinement for this task setting.

    Source: §5 · p. 11

    Image-level retrieval does not explicitly model temporal evidence, and near-duplicate frames remain difficult.

    Source: §4.6 · p. 10

Abstract

from arXiv · show

Text-based person anomaly search requires a model to retrieve individuals from a large image gallery using natural-language descriptions of both appearance and behavior. This task is challenging because model training relies on synthetic image-text pairs, whereas evaluation is performed on real surveillance images containing domain differences, near-duplicate frames, and many visually similar distractors. We present a multi-model retrieval and reranking framework designed to improve synthetic-to-real transfer. The framework first adapts Qwen3-VL-Embedding-8B to synthetic person descriptions using bidirectional contrastive learning, explicit negative sampling, false-negative masking, and domain randomization. It then combines the normalized scores of six complementary Qwen3-VL checkpoints with those of a frozen SigLIP2 model using four-view gallery encoding. The fused retrieval scores identify the strongest candidates, which are jointly evaluated with the query using Qwen3-VL-Reranker-8B. The cross-encoder and retrieval scores are subsequently blended to produce the final ranking. We also construct a distractor-matched validation protocol entirely from synthetic data to select checkpoint weights, fusion coefficients, candidate depths, and reranking parameters without accessing the official test images. The final system achieves 94.7815% mAP and 90.8493% R@1 under the official AI City Challenge 2026 Track 4 test setting, with a reported eighth-place finish on the public leaderboard.

1 Introduction

Text-based person anomaly search must match both appearance and behavior despite synthetic-to-real domain differences and visually similar distractors. The paper integrates adapted and complementary retrieval models with reranking, reporting an eighth-place public-leaderboard finish.

1 Introduction

  • A valid match must satisfy clothing and physical-attribute constraints while depicting the described action, making similar people performing different actions difficult distractors.Domain differences, near-duplicate frames, and many plausible distractors further challenge ranking.Source: §1 · pp. 1–2
  • The framework adapts Qwen3-VL on synthetic descriptions, fuses six checkpoints with frozen SigLIP2, then blends cross-encoder reranking with retrieval scores.Synthetic validation selects fusion and reranking settings; the results support this task-specific integration, not universal model complementarity.Source: §1 · p. 2
  • 94.7815% mAP and 90.8493% R@1; the system placed eighth on the reported public leaderboard.The evaluation was under the AI City Challenge 2026 Track 4 setting.Source: §1 · p. 2

2 Related Work

Related work spans shared-space vision-language retrieval, person-specific alignment, behavior understanding, synthetic-domain adaptation, candidate refinement, and model selection under domain shift.

2.1 Vision-Language Pretraining

  • CLIP [9] and ALIGN [10] establish shared-space retrieval, while SigLIP [11] and BLIP-2 [12] illustrate alternative training and component-reuse strategies.These approaches motivate retaining general vision-language knowledge while adding behavior-sensitive matching.Source: §2.1 · pp. 2–3

2.2 Text-Based Person Retrieval and Anomaly Search

  • Person retrieval methods refine local or fine-grained image-text alignment [14] [15] [16] [19] [20] [21], while Beyond Walking [1] and AnomalyLMM [2] address behavior-sensitive PAB retrieval.RaSa [22] and RDE [23] address imperfect image-text correspondence; these works provide foundations for appearance-sensitive retrieval.Source: §2.2 · p. 3

2.3 Behavior Understanding and Multimodal Transportation Models

  • Transportation studies examine temporal, multi-view, and language-based behavior cues [4] [24] [25] [26] [27] [28] [29], but do not directly establish person-anomaly retrieval performance.Temporal or multi-view evidence is not evaluated by this paper’s image-gallery protocol.Source: §2.3 · p. 3

2.4 Synthetic Data, Visual Robustness, and Domain Adaptation

  • Domain adaptation work ranges from domain randomization [30] and synthetic-image refinement [31] to camera-style augmentation [32] and synthetic pedestrian data [33] [34] [35] [36] [37].These studies motivate synthetic-to-real transfer; the paper’s validation protocol separately addresses configuration selection before real-gallery evaluation.Source: §2.4 · p. 3

2.5 Domain-Specific Adaptation, Fusion, and Candidate Refinement

  • Parameter interpolation and model soups [49] [50] combine models in parameter space, unlike this paper’s score fusion; prior retrieval work also explores reranking and rank fusion [51] [52].The study combines standardized scores with query-image cross-encoder refinement.Source: §2.5 · p. 4

2.6 Negative Sampling and Model Selection under Domain Shift

  • Hard-negative sampling [54] and debiased contrastive learning [55] motivate jointly handling challenging negatives and false negatives, while DomainBed [56] highlights model-selection criteria.The paper’s distractor-matched synthetic validation is designed to expose ranking errors before real-gallery evaluation.Source: §2.6 · p. 4

3.1 Problem Formulation

The task ranks real-gallery images against a natural-language query that describes both a person and behavior. Synthetic training and difficult distractors make this ranking problem especially challenging.

3.1 Problem Formulation

  • Relevant gallery images must rank above distractors by matching both the person and behavior described in the query.Queries can describe clothing, physical attributes, routine actions, or anomalous activities.Source: §3.1 · p. 4
  • Distractors may show the right person performing another action, a similarly dressed person, or a near-duplicate frame, so partial matches are insufficient.The model must satisfy appearance and behavioral constraints jointly.Source: §3.1 · p. 4
  • Synthetic training images and real surveillance galleries differ in illumination, compression, background complexity, image quality, and visual artifacts.This domain gap makes direct application of standard retrieval approaches suboptimal.Source: §3.1 · p. 4
  • The proposed pipeline adapts Qwen3-VL on synthetic data, fuses its checkpoints with frozen SigLIP2, and reranks the strongest candidates with a query-image cross-encoder.The overview retains the top 100 retrieval candidates, reranks the top 50, and uses distractor-matched synthetic validation for selection.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2 Embedding Representation Learning

The embedding stage learns a shared representation of person appearance and behavior from synthetic PAB image-text samples, using bidirectional contrastive training and varied augmentations.

3.2 Embedding Representation Learning

  • Synthetic PAB training samples pair images and text descriptions with identity and behavior labels.Qwen3-VL-Embedding-8B is adapted using LoRA while its visual encoder remains frozen and multimodal alignment layers remain trainable.Source: §3.2 · p. 5
  • Bidirectional contrastive learning uses four explicit negatives per sample and masks candidates identified as possible false negatives.The image-to-text loss reverses the text-to-image formulation, and the two losses are averaged to align matched pairs while separating different identities or behaviors.Source: §3.2 · p. 5
  • Domain randomization varies image conditions while preserving identity and behavior labels, encouraging learning of target semantics rather than synthetic visual patterns.Variations include illumination, contrast, color, blur, noise, shadows, compression, crop scale, and orientation.Source: §3.2 · p. 5
  • Different augmentation settings and random seeds produce six complementary Qwen3-VL checkpoints for weighted ensemble fusion.The checkpoints capture different perspectives on synthetic-to-real transfer.Source: §3.2 · p. 6

3.3 Multi-Model Retrieval Fusion

The retrieval stage standardizes scores from six Qwen3-VL checkpoints and a frozen SigLIP2 branch, then fuses them to rank gallery images. Checkpoint and fusion weights are selected using distractor-matched validation.

3.3.1 Score Standardization:

  • All model scores are standardized before fusion because checkpoints and model families produce scores on different scales.The standardization uses the gallery mean and standard deviation, with epsilon preventing division by zero.Source: §3.3.1 · p. 6

3.3.2 Qwen3-VL Checkpoint Ensemble:

  • Each of six Qwen3-VL checkpoints averages scores from the original gallery image and its horizontal reflection before weighted fusion.Checkpoint weights are selected by validation.Source: §3.3.2 · p. 6

3.3.3 Checkpoint Ensemble Strategy:

  • Different augmentations and random seeds yield complementary checkpoint representations, which are weighted to balance their contributions.The checkpoints share the PAB training dataset and may capture different appearance details or behavioral nuances.Source: §3.3.3 · p. 6

3.3.4 SigLIP2 Four-View Encoding:

  • The frozen SigLIP2 branch averages and renormalizes embeddings from the full image, its horizontal flip, and upper and lower 60% crops.The complete query is processed without paraphrasing.Source: §3.3.4 · p. 6

3.3.5 Cross-Model Score Fusion:

  • The standardized six-checkpoint Qwen3-VL ensemble score is fused with SigLIP2 using a validation-selected SigLIP2 contribution of β = 0.30.The fused retrieval score ranks the gallery and supplies candidates for reranking.Source: §3.3.5 · p. 6

3.4 Candidate Reranking

Candidate reranking applies detailed query-image comparison to the strongest retrieval candidates. The top 50 are reordered using a blend of retrieval and cross-encoder scores, while ranks 51–100 retain their retrieval order.

3.4 Candidate Reranking

  • Detailed comparison is limited to the strongest candidates, addressing similar appearance, behavior, or scene context while reducing computational cost.The top 100 retrieval-scored images form the pool; the top 50 are processed by the reranker.Source: §3.4 · p. 6
  • The cross-encoder jointly encodes each query and candidate image to score agreement in appearance, action, body state, and scene context.This allows direct visual-textual interaction that dual encoders cannot capture.Source: §3.4 · p. 7
  • The top 50 are reordered by a blend of retrieval and cross-encoder scores standardized over those candidates, with α = 0.70 selected by validation.Candidates ranked 51–100 retain their original retrieval ranking.Source: §3.4 · p. 7

3.5 Distractor-Matched Validation

The paper uses distractor-matched synthetic validation to make model and pipeline selection more informative under the difficulty of the evaluation gallery. The official challenge test images are not accessed during training or validation.

3.5 Distractor-Matched Validation

  • Conventional synthetic validation has fewer challenging distractors and nearly saturated scores, so the protocol is designed to surface ranking errors found on real data.Source: §3.5 · p. 7
  • Validation uses 1,978 held-out queries and 34,795 synthetic distractors, including 15% that share query identity but depict different behaviors.These negatives test distinguishing actions rather than relying solely on identity or clothing.Source: §3.5 · p. 7
  • Synthetic validation selects checkpoint weights, fusion coefficients, candidate and reranking depths, while the official challenge test images remain unseen.Selected parameters include the checkpoint combination, β, candidate pool depth, reranking depth, and α.Source: §3.5 · p. 7

3.6 Implementation Details

Implementation uses a 6,000-step AdamW training run for Qwen3-VL and fixes the retrieval and reranking configuration after synthetic validation.

3.6 Implementation Details

  • Qwen3-VL is trained for 6,000 steps with AdamW, a cosine schedule, weight decay of 0.1, and a primary learning rate of 1 × 10−7.Continuation runs use 5 × 10−8; the contrastive temperature is 0.015, texts are limited to 512 tokens, and images use 32 to 64 visual tokens.Source: §3.6 · p. 7
  • The fixed retrieval configuration combines pretrained, unadapted SigLIP2 with the Qwen3-VL ensemble at β = 0.30 and reranks 50 of 100 candidates using α = 0.70.Checkpoint weights, fusion coefficients, candidate depths, and reranking parameters remain fixed after validation.Source: §3.6 · p. 7

4 Results and Discussion

The results evaluate the framework under the official AI City Challenge 2026 Track 4 setting, then examine retrieval, training, reranking, validation, qualitative examples, and limitations.

4.1 Official Challenge Results

  • 94.7815% mAP and 90.8493% R@1 are the final scores; the submission ranked eighth on the reported public leaderboard.The final result improves on Qwen3-VL reranking alone by 0.3229 mAP points and 0.5055 R@1 points; the transferred CMP consensus also outperformed its individual configurations.Source: §4.1 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8

4.2 Retrieval Representation and Fusion Analysis

  • Before reranking, six-checkpoint weighting raised Qwen3-VL mAP to 92.5510%, while adding frozen SigLIP2 improved mAP by 0.5423 points and R@1 by 0.7273 points.The fused retrieval system remained the strongest retrieval-only configuration, although standalone SigLIP2 was below the Qwen3-VL ensemble.Source: §4.2 · p. 8

4.3.1 Embedding-Stage Training Strategies

  • Direct query-paraphrase augmentation caused the largest reduction; retaining original descriptions with moderate hard negatives was preferred over identity-based mining or watermark augmentation.Combining original and paraphrased queries slightly exceeded baseline, while iterative identity-based mining reduced performance.Source: §4.3.1 · p. 9, Table 4 · p. 9

4.3.2 Reranked Training Strategies

  • Neither reranked training alternative improved on the baseline: the real-data mixture remained close, while identity-based continuation reduced performance more.Source: §4.3.2 · p. 9, Table 5 · p. 10

4.3.3 Candidate-Reranking Strategies

  • The zero-shot cross-encoder improved mAP by 1.9076 points, whereas LoRA adaptation and the 30B reranker performed slightly worse; reranker weighting alone did not match the complete system.The strongest Qwen3-VL-only result at approximately α = 0.75 remained 0.2499 mAP points below the Qwen3-VL–SigLIP2 system.Source: §4.3.3 · p. 9, Table 6 · p. 10

4.4 Effect of Distractor-Matched Validation

  • Distractor-matched validation reduced mAP to 97.1995% and R@1 to 95.3488% from nearly saturated conventional validation, exposing top-rank ordering challenges.Because the protocols use different queries and gallery construction, the numerical difference is diagnostic; the matched protocol more closely reproduces the target task's large-gallery, high-distractor structure.Source: §4.4 · p. 9, Table 7 · p. 11

4.5 Qualitative Analysis

  • Across three examples, fusion placed the annotated image in the top three and reranking promoted each match to Rank 1, including behavior- and scene-sensitive distinctions.A query with nearly identical frames illustrates the remaining difficulty of exact frame-level retrieval.Source: §4.5 · p. 10, Figure 3 · p. 11

4.6 Limitations

  • Evaluation covers one challenge setting; comparisons lack uncertainty estimates, costs are unquantified, and reproduction details are incomplete.Near-duplicate frames remain difficult, image-level retrieval does not explicitly model temporal evidence, and the validation protocols differ in queries and gallery construction.Source: §4.6 · p. 10

5 Conclusion

The conclusion presents a multi-stage retrieval and reranking framework for synthetic-to-real person anomaly search. It reports strong results on AI City Challenge 2026 Track 4 and identifies temporal information and distillation as future directions.

5 Conclusion

  • 94.7815% mAP and 90.8493% R@1, improving the Qwen3-VL retrieval baseline by 2.2305 and 3.4883 percentage points, respectively, with a reported eighth-place leaderboard finish.Results are under the official AI City Challenge 2026 Track 4 test setting.Source: §5 · p. 11
  • The framework combines a Qwen3-VL retrieval ensemble with SigLIP2 scores, then reranks the strongest gallery candidates with a cross-encoder.The design preserves efficient global retrieval while improving fine-grained query-candidate agreement.Source: §5 · p. 11
  • The results support complementary retrieval representations and focused candidate reranking over increasing training complexity or reranker capacity alone.Source: §5 · p. 11
  • Future work will investigate adjacent-frame temporal information and model distillation to improve near-duplicate discrimination while reducing inference cost.Source: §5 · p. 11

Related papers

Loading 2610.31930v1…