Source-linked AI summary
Action-Aligned Retrieval with Pairwise Multimodal Reranking for Text-Based Person Anomaly Search
Thanh-Khoi Nguyen, Thanh-Nhan Vo, Trong-Thuan Nguyen, Minh-Triet Tran
TL;DR
Text-based person anomaly search must distinguish individuals through fine-grained, context-dependent behavior rather than appearance alone. ActPair combines action-aligned VLM learning, dual-view retrieval, and efficient pairwise reranking, achieving the best performance among compared baselines on PAB and improving cross-dataset transfer to RSTPReid.
Problem
Text-based person anomaly search requires behavior-sensitive representations because visually similar individuals may differ in fine-grained actions and context.
Method
ActPair uses action-aligned multi-task VLM learning, parallel late-fusion retrieval from original and rewritten queries, and pivot-promote pairwise multimodal reranking.
Results
ActPair achieves the best performance among compared published baselines on the original PAB test and substantially improves over same-source CMP in cross-dataset transfer to RSTPReid.
Takeaways & Limitations
The framework supports context-grounded retrieval beyond pedestrian anomaly search, including distinguishing similar actions in sports.
Takeaways & Limitations
Constructing the action and scene vocabularies currently requires dataset-specific annotations, limiting scalability in unannotated domains.
Abstract
from arXiv · showhide
Text-based person anomaly search requires distinguishing individuals based on fine-grained, context-dependent behaviors rather than mere appearance. Existing methods struggle to capture these context-conditioned actions, frequently relying on isolated skeletal geometry, discarding raw query details during reformulation, or utilizing absolute pointwise scoring for multimodal verification. To address these limitations, we propose \textbf{ActPair}, a unified three-stage coarse-to-fine framework that combines action-aligned retrieval with pairwise multimodal reranking to bridge the pose-semantic gap. First, we fine-tune a vision-language model (VLM) with an action-aligned multi-task objective that encourages the representations to encode action-discriminative semantics. Second, we perform parallel late-fusion retrieval using the original query and a large language model (LLM)-generated context-grounded rewrite, retaining complementary details from both semantic views. Finally, we propose an efficient off-the-shelf reranking module that leverages a pivot-promote algorithm to perform direct pairwise visual comparisons, mitigating residual spatial and compositional ambiguities without the prohibitive inference costs of exhaustive evaluation. Extensive experiments demonstrate that our framework achieves the best results among the compared methods on the Pedestrian Anomaly Behavior (PAB) public test and transfers effectively to an unseen, non-anomaly-specific dataset.
1 Introduction
Text-based person anomaly search must distinguish visually similar individuals through fine-grained, context-dependent actions. ActPair addresses this with action-aligned learning, dual-view retrieval, and efficient pairwise multimodal reranking.
- Person anomaly search focuses on behaviors that distinguish visually similar individuals and depend on surrounding scene context.
- Existing approaches may rely on ambiguous isolated skeletal geometry, replace original queries during reformulation, or use absolute multimodal scores.
- ActPair fine-tunes a VLM with contrastive alignment, action-tag supervision, and cross-modal consistency to encode context-conditioned action semantics.
- Parallel late-fusion retrieval combines the original query with an LLM-generated context-grounded rewrite, preserving complementary semantic details.
- Pairwise multimodal reranking uses direct candidate comparisons through a pivot-promote strategy instead of absolute scoring.
2 Related Work
Prior work increasingly models relations, pose, and query context for fine-grained retrieval, but anomaly search still faces a gap between skeletal structure and action semantics. ActPair combines explicit action supervision with parallel query views and pairwise verification to address these challenges.
- Conventional person retrieval emphasizes identity-related appearance, whereas anomaly search distinguishes visually similar pedestrians through behavior.
- Pose-aware methods capture structural variation, but similar skeletal configurations can correspond to different actions.
- Action semantics depend on interacting objects and scene context, motivating explicit action supervision and cross-modal action consistency.
- Pairwise ranking avoids independently calibrated relevance scores, although exhaustive comparisons impose quadratic multimodal inference cost.
- ActPair retains the original caption while adding a single action-scene-grounded rewrite, preserving instance-specific details alongside behavioral context.
3 Proposed Method
ActPair addresses text-based person anomaly search by aligning representations with context-conditioned actions and combining dual-view retrieval with targeted pairwise reranking. Its three-stage pipeline builds a high-recall candidate pool, then resolves residual spatial and compositional ambiguities efficiently.
- 3.1 Problem Formulation: Text-based person anomaly search retrieves a pedestrian image matching the appearance and behavior described by a natural-language query.The ranking function targets the ground-truth image depicting the specified normal or anomalous behavior.
- Overview: ActPair uses action-aligned representation learning, dual-view retrieval, and pairwise multimodal reranking in a unified coarse-to-fine pipeline.The stages are shown as action-aligned learning, retrieval with original and rewritten queries, and pivot-promote reranking.
- 3.2 Offline Action and Scene Tag Extraction: The method constructs a fixed action-and-scene vocabulary by retaining frequent raw tags, canonicalizing them with an LLM, and using the resulting mappings for supervision and rewriting.Top-K tags are retained because the raw tag distribution is heavy-tailed; canonicalization reduces synonymy and surface variation.
- 3.3 Multi-Task Representation Learning: The VLM is fine-tuned with native SigLIP2 alignment plus action supervision and symmetric cross-modal consistency to emphasize action-discriminative evidence.The action objective applies classification to image and text embeddings and encourages their action predictions to agree, reducing reliance on spurious background correlations.
- 3.4 Query Rewriting and Parallel Late-Fusion Retrieval: ActPair rewrites the original query with LLM-selected action and scene tags, retrieves independently from both views, unions their top-K candidates, and averages their scores.The fused pool is sorted and reduced to a top-M subset before multimodal reranking, with M < K.
- 3.5 Pairwise Multimodal Reranking: Pairwise reranking directly compares candidate images with the query and uses pivot-promote to target the top-ranked subset instead of assigning absolute scores to every pair.The comparator is used within a pivot-promote procedure, avoiding a complete pairwise ordering while addressing residual ambiguities.
4 Experiment
ActPair is evaluated on PAB and RSTPReid using standard cross-modal retrieval metrics, with results covering in-domain performance, zero-shot transfer, and qualitative stage-wise behavior.
- Main Results on PAB: ActPair achieves 88.62% R@1 and 93.97% mAP on the original PAB test, outperforming the compared published baselines.Appearance-oriented methods reach at most 44.41% mAP in the training-free setting, while X-VLM reaches 83.96% mAP.
- Main Results on PAB: ActPair improves over CMP by 3.69 percentage points in R@1 and 2.31 points in mAP under PAB fine-tuning.It also exceeds SSDC by 1.41 points in R@1 and 1.10 points in mAP, with the largest gains at rank 1 and in overall ranking quality.
- Main Results on RSTPReid: 55.25% Recall@1, 69.05% Recall@5, and 73.75% Recall@10 are obtained on zero-shot transfer from PAB to RSTPReid with reranking.CMP reaches only 29.15% Recall@1 under identical transfer conditions.
- Main Results on RSTPReid: ActPair substantially improves over CMP on RSTPReid without using RSTPReid annotations or domain adaptation.The evaluation targets a predominantly appearance-based, non-anomaly-specific dataset, supporting transfer beyond the training domain.
- Qualitative Analysis: In qualitative retrieval, dual-view fusion improves candidate discovery and pairwise reranking promotes the ground-truth image, while complex spatial and action-state constraints remain a failure case.One example moves from rank 57 to 6 to 1; another starts at rank 1 but is later demoted.
5 Ablation Study
The ablations show that explicit action supervision strengthens retrieval more than pose supervision, while pairwise reranking improves ranking quality over pointwise scoring with similar comparison cost.
- Action versus Pose Supervision: 72.14% Recall@1, 92.77% Recall@5, and 95.50% Recall@10 are achieved with explicit action-tag supervision using the SigLIP2 backbone.This controlled comparison keeps the backbone and training configuration fixed when isolating the supervision signal.
- Action versus Pose Supervision: +4.19% Recall@1 favors action-based over pose-based supervision, with a 95% CI of [+2.38, +6.07].Pose supervision reaches 67.95% Recall@1, whereas explicit action supervision reaches 72.14%.
- Pointwise versus Pairwise Reranking: 84.88% Recall@1 and 90.14% mAP are achieved by pairwise reranking, exceeding pointwise scoring by +4.75% and +2.44%, respectively.Both variants use the identical Qwen3.5-9B model and candidate pool.
- Pointwise versus Pairwise Reranking: 9.09 model calls per query are used by pairwise reranking versus 10.00 calls for the pointwise implementation.The comparison uses the top-10 configuration.
6 Conclusion
ActPair is a three-stage framework for text-based person anomaly search that combines action-aligned representation learning, context-grounded query fusion, and training-free pairwise multimodal reranking. It achieves the best performance among compared published baselines on the original PAB test and improves cross-dataset transfer to RSTPReid, while requiring dataset-specific annotations for its canonical vocabularies.
- Framework: ActPair combines action-aligned representation learning, context-grounded query fusion, and training-free pairwise multimodal reranking in a three-stage framework.The framework is designed to bridge the pose-semantic gap in text-based person anomaly search.
- Results: ActPair achieves the best performance among the compared published baselines on the original PAB test.
- Results: ActPair substantially improves over the same-source CMP baseline when transferring across datasets to RSTPReid.
- Limitations and Future Work: Constructing the canonical Tact and Tscene vocabularies requires dataset-specific annotations, limiting scalability in unannotated domains.Future work proposes an unsupervised MLLM pipeline to parse raw captions into action and scene clusters.