PaperlayerView exact revision on arXiv

Source-linked AI summary

GradCO: Behavior-Aware Gradient-Conflict Offloading for Text-Based Person Anomaly Search

Zhiheng Fu, Zixu Li, Zhiwei Chen, Jiale Huang, Fangxu Liu, Jinhe Bi, Yupeng Hu

arXiv:2610.37267v1cs.CVcs.IR

TL;DR

TPAS must retrieve pedestrians using both appearance and behavior, but hard negatives can create gradient conflict and strong appearance cues can dominate aggregation. GradCO combines expert streams, pair-conditioned offload anchors, and query-conditioned aggregation; experiments report improved retrieval and robustness across five TPAS/TIPR benchmarks.

  • Problem

    TPAS requires retrieving a pedestrian using both identity-related appearance and described behavior, which conventional appearance-focused retrieval often overlooks.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    GradCO combines B-MoE appearance and behavior streams, COL pair-conditioned offload anchors, and HIA query-conditioned aggregation of expert similarities and features.

    Source: §1 · p. 2

  • Results

    Experiments on five TPAS/TIPR benchmarks demonstrate improved retrieval, corruption robustness, and out-of-distribution generalization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TPAS retrieval is framed as matching both pedestrian identity and the specified behavior, not appearance alone.

    Source: §3.1 · p. 3

    Behavior retrieval remains challenging with highly similar candidates, visual clutter, or complex multi-person interactions.

    Source: §4.8 · p. 22

Abstract

from arXiv · show

Text-based Person Anomaly Search (TPAS) retrieves a target pedestrian using natural-language descriptions of appearance and normal or anomalous behavior. Unlike conventional text-based person retrieval, TPAS must preserve identity cues while distinguishing fine-grained behaviors. It presents two challenges: same-identity, different-behavior hard negatives cause gradient conflict during granular alignment, while strong appearance cues can obscure weak behavioral evidence during aggregation. We propose GradCO (behavior-aware Gradient-Conflict Offloading), comprising three complementary components. Behavior-Aware Mixture-of-Experts (B-MoE) forms appearance, normal-behavior, and anomalous-behavior streams whose roles are learned from retrieval objectives without expert-specific semantic labels. Conflict Offload Learning (COL) uses pair-conditioned offload anchors in similarity computation to reduce the conflicting-gradient burden on the main text representations. Hierarchical Intent Alignment (HIA) aggregates expert similarities and features using query-conditioned weights. Experiments on five TPAS/TIPR benchmarks show GradCO significantly surpasses SOTA methods and validate our design.

1. Introduction

TPAS retrieves pedestrians using descriptions of both appearance and normal or abnormal behavior, but identity-matched behavior mismatches create gradient conflict and strong appearance cues can dominate aggregation. GradCO combines learned expert streams, pair-conditioned gradient offloading, and query-conditioned aggregation to address these challenges.

1. Introduction

  • TPAS retrieves a pedestrian matching both identity and described behavior, extending retrieval beyond static appearance descriptions.The introduction notes that conventional person retrieval often overlooks dynamic behavior, particularly abnormal behavior relevant to safety-critical scenarios.Source: §1 · pp. 1–2
  • Same-identity, different-behavior hard negatives create competing identity and behavior signals, while global aggregation can let appearance overwhelm behavior and produce intent mismatch.The conflict arises under the joint retrieval objective; context-agnostic global features also cannot adapt to query intent.Source: §1 · p. 2
  • GradCO combines B-MoE streams learned without expert-specific semantic labels, COL pair-conditioned offload anchors, and HIA text-conditioned aggregation of expert similarities and features.HIA addresses signal domination and intent mismatch through query-conditioned weights.Source: §1 · p. 2
  • Experiments on five TPAS/TIPR benchmarks demonstrate improved retrieval, corruption robustness, and out-of-distribution generalization.The introduction reports ablation and visualization results supporting these findings.Source: §1 · p. 2

2. Related Work

Prior work addresses text-to-image person retrieval and video anomaly detection, but their joint alignment remains unresolved. GradCO extends mixture-of-experts to TPAS by redirecting conflicting optimization signals and making expert aggregation responsive to text intent.

2. Related Work

  • TIPR and video anomaly detection generally optimize identity matching and behavior recognition separately, leaving their joint alignment unresolved in TPAS.GradCO targets a joint representation that preserves identity-consistent appearance cues while retaining behavior-sensitive evidence.Source: §2 · p. 3
  • Existing mixture-of-experts methods span large-model optimization, multitask learning, transfer learning, and ReID, where they model modality variation, biometric-factor fusion, or domain generalization.The cited ReID approaches focus on these settings rather than the TPAS challenges described here.Source: §2 · p. 3
  • GradCO extends MoE to TPAS by redirecting conflicting optimization signals during granular alignment and making expert aggregation responsive to text intent.The stated aim is to let identity consistency and behavior sensitivity coexist under TPAS hard negatives.Source: §2 · p. 3

3. Methodology

GradCO formulates TPAS as retrieving images matching a pedestrian's identity and described behavior, then uses three jointly trained components to model appearance and behavior cues. B-MoE forms expert streams, COL offloads conflict-bearing gradients, and HIA aggregates similarities and features according to query intent.

3.1. Problem Formulation

  • TPAS retrieves database images matching both the pedestrian's identity and the specified normal or abnormal behavior in a text query.The formulation learns text and image encoders that map their inputs into a unified embedding space.Source: §3.1 · p. 3

3.2. Behavior-Aware Mixture-of-Experts (B-MoE)

  • B-MoE constructs parallel appearance, normal-behavior, and abnormal-behavior streams whose input-dependent roles emerge from retrieval objectives rather than semantic labels.The streams provide representations for subsequent cross-modal alignment.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Conflict Offload Learning (COL)

  • COL predicts pair-conditioned offload anchors to modify contrastive-learning gradient paths and provide an offload path for conflicts from same-identity, different-behavior hard negatives.The text-side anchor absorbs part of the conflicting alignment pressure while preserving the text feature's semantic content.Source: §3.3 · p. 5
  • COL's direction-diversity regularizer encourages pair-conditioned anchors associated with one text query to maintain distinct directions.It uses cosine similarity between normalized anchors to discourage collapse into similar directions.Source: §3.3 · p. 5

3.4. Hierarchical Intent Alignment (HIA)

  • HIA predicts query-conditioned weights from text expert features to aggregate expert similarities and features, with the same weights used for image-text matching representations.The weighted representations are fused by BERT's last six layers to produce a matching logit.Source: §3.4 · pp. 5–6

4. Experiments

The experiments evaluate GradCO on TPAS and conventional TIPR benchmarks, with dataset protocols and implementation details defining the evaluation scope.

4.1.1. Datasets.

  • GradCO is evaluated on PAB for TPAS, Multi-Weather corruption conditions, UCC out-of-distribution transfer, and conventional TIPR datasets including CUHK-PEDES, ICFG-PEDES, RSTPReID, and UFineBench.UFineBench includes UFine6926 and UFine3C; UCC is derived from UCF-Crime.Source: §4.1.1 · pp. 6–7
  • PAB contains 1,015,583 image-text pairs, with 1,013,605 synthetic training pairs and 1,978 real-world test pairs.Its test descriptions are generated with Qwen2-VL and checked through human quality control.Source: §4.1.1 · p. 6

4.1.2. Implementation details.

  • GradCO uses X2VLM encoder weights, AdamW, batch size 22, weight decay 0.01, four B-MoE sub-experts, and direction-diversity weight γ=0.5 for 10 epochs.Experiments run on one NVIDIA V100 GPU with 32GB memory.Source: §4.1.2 · p. 8
  • TPAS is evaluated with Recall@K for K∈{1,5,10} and mAP; TIPR uses R-Avg, the arithmetic mean of R@1, R@5, and R@10.Multi-Weather reports R@1 and mAP for each condition and their arithmetic means.Source: §4.1.2 · p. 8

4.2. Performance Comparison

GradCO is compared with existing methods on TPAS retrieval, weather robustness, OOD transfer, and conventional TIPR datasets. It leads on most reported comparisons, with gains across the evaluated TPAS settings and four of five TIPR sets.

4.2.1. TPAS task

  • At 0.1M gallery size, GradCO raises PAB R@1 from 83.06 to 85.39 and mAP from 90.41 to 92.13 over CMP; at 1M, gains are 1.67 and 1.22 points.It achieves the highest R@1, R@5, and mAP in both settings and ties CMP at 1M R@10.Source: §4.2.1 · p. 8
  • GradCO records mean Multi-Weather R@1 of 72.2 and mAP of 80.67, versus CMP’s 67.12 and 77.44, and leads both metrics in all ten conditions.The reported comparison includes the challenging Dark+Rain condition.Source: §4.2.1 · p. 8
  • On UCC with the 1M gallery, GradCO improves over CMP from 55.23 to 62.53 in R@1 and from 44.35 to 52.01 in mAP.The passage reports gains of 7.30 and 7.66 percentage points, respectively.Source: §4.2.1 · p. 8

4.2.2. TIPR task

  • GradCO scores highest on four TIPR sets; on UFine3C it reaches 82.01 versus CFAM’s 74.63, while Bi-IRRA scores 95.18 on UFine6926 versus GradCO’s 95.13.The UFine3C comparison is the largest improvement reported for TIPR.Source: §4.2.2 · pp. 8–9

4.3. Efficiency Evaluation Analysis

GradCO improves reported retrieval performance over CMP but uses more parameters and training memory and has higher per-sample inference latency. The reported latency is 2.3 ms per sample, which the authors describe as supporting real-time retrieval.

Efficiency and performance

  • GradCO has 247.36M parameters versus CMP’s 226.13M and uses 18,413 MiB of GPU memory versus 14,575 MiB.The parameter increase is mainly associated with B-MoE’s separate expert streams, while the higher training memory is partly attributed to COL’s pair-conditioned offload anchors.Source: §4.3 · p. 9
  • At inference, GradCO takes 2.3 ms per sample versus CMP’s 1.5 ms, an absolute increase of 0.8 ms (53.3%).The passage attributes most of this overhead to HIA’s text-conditioned gating.Source: §4.3 · p. 9
  • Multi-Weather-Avg rises from 67.12% to 72.19% and UCC (OOD)-Avg from 68.30% to 76.08% alongside the higher latency.The authors state that the 2.3 ms per-sample latency indicates support for real-time retrieval.Source: §4.3 · p. 9

4.4. Ablation Studies

Ablations show that GradCO’s three components contribute complementary benefits: B-MoE uses multiple expert streams, COL offloads conflict-bearing signals, and HIA aggregates expert information according to intent.

B-MoE

  • Removing any B-MoE stream reduces PAB-Avg and Multi-Weather-Avg, while removing image-side experts, text-side experts, or B-MoE lowers UCC-Avg.The reported UCC-Avg values are 62.95, 63.13, and 62.60, respectively, for D#4, D#5, and D#6.Source: §4.4 · p. 9

COL

  • Removing the offload-anchor path reduces UCC-Avg to 55.43, while removing directional diversity regularization reduces it to 70.58.The larger drop without anchors identifies the pair-conditioned anchor as the primary mechanism for mitigating same-identity, different-behavior gradient conflicts.Source: §4.4 · pp. 9–10, Figure 4 · p. 10

HIA

  • Removing similarity aggregation reduces PAB-Avg by 4.69 points, while removing matching reduces PAB-Avg, Multi-Weather-Avg, and UCC-Avg to 83.70, 47.66, and 56.86.Removing intent weighting and feature aggregation primarily affects PAB and Multi-Weather; removing the image-text contrastive objective causes performance to nearly collapse.Source: §4.4 · p. 10, Figure 4 · p. 10

4.5. Sensitivity Analysis

GradCO’s performance first rises and then falls as the number of B-MoE sub-experts or direction-diversity weight changes; the selected settings are K=4 and γ=0.5.

4.5. Sensitivity Analysis

  • K=4 and γ=0.5 provide the best reported settings; performance first increases and then decreases as either hyperparameter varies.K≥5 may over-partition features, while larger γ over-constrains anchor directions and smaller γ provides insufficient regularization.Source: §4.5 · p. 10

4.6. Mechanism Analysis

Mechanism analyses report stronger retrieval on PAB’s normal and anomaly subsets, alongside evidence that COL’s anchors evolve to address alignment conflicts and HIA adapts expert weighting to query semantics.

4.6.1. Performance Analysis on PAB Test Subsets (Normal vs. Anomaly)

  • GradCO reaches 86.55% R@1 on PAB-Normal and 86.65% R@1 with 92.94% mAP on PAB-Anomaly.CMP reaches 83.22% R@1 on PAB-Anomaly; the analysis attributes its decline to aggregation distortion that can overwhelm subtle behavior cues.Source: §4.6.1 · p. 11, Table 6 · p. 11

4.6.2. The Evolution Analysis of Offload Anchors Δ

  • COL anchor norms rise as training focuses on hard negatives, with the anomaly stream largest at approximately 9.0, versus approximately 7.5 for appearance and 7.0 for normal behavior.Anchor similarity to the main text-image semantic difference stays near zero; the anomaly stream has an approximately −0.1 offset, consistent with stream-dependent corrections distinct from the main alignment direction.Source: §4.6.2 · pp. 11–12, Figure 6 · p. 11, Figure 7 · p. 11

4.6.3. How COL Resolves Identity–Behavior Conflicts

  • COL reduces gradient conflicts, stabilizes optimization, and improves retrieval separation; without anchors, hard-negative similarity is generally lower after correction, while directional regularization preserves anchor diversity.Pair-specific anchors provide an additional optimization pathway for conflict-sensitive information while preserving the original feature pathway for identity-related representation.Source: Figure 9 · p. 12, §4.6.3 · pp. 13–14

4.6.4. How COL Improves Feature Alignment and Embedding Geometry

  • Full COL reduces mean positive-pair distance to 1.078, compared with 1.206 without COL and 1.238 without L_dir, while retaining more balanced angular coverage and semantic-group separation.Removing L_dir produces a more concentrated embedding distribution.Source: Figure 10 · p. 13, §4.6.4 · p. 15

4.6.5. How Does HIA Improve Query-level Retrieval Consistency?

  • HIA improves query-level retrieval consistency by dynamically combining complementary expert representations rather than using fixed expert contributions.Individual experts have different strengths across queries, while HIA adjusts their contributions according to query characteristics.Source: Figure 11 · p. 14, §4.6.5 · pp. 15–16

4.6.6. How Does HIA Adapt Expert Weights to Query Semantics?

  • Normal queries assign relatively larger weights to normal-behavior experts, while anomaly queries increase abnormal-behavior expert contributions and retain distinct appearance weighting.Behavior-description changes induce coordinated shifts in normal- and abnormal-behavior weights, whereas appearance masking mainly affects the corresponding routing response with limited behavior-gate variation.Source: §4.6.6 · p. 16, Figure 12 · p. 16

4.7. Qualitative Results

Across the evaluated TPAS benchmarks, GradCO shows the strongest displayed performance, while remaining competitive on TIPR benchmarks. Qualitative visualizations show sample-dependent expert preferences and clearer identity-level feature organization under hard-negative conflicts.

4.7.1. Visualization of B-MoE Expert Gating Weights

  • B-MoE assigns sample-dependent routing preferences: selected abnormal examples favor the anomaly stream, while controlled or skillful normal examples favor the normal-behavior stream.Case 341 assigns 0.594 to the anomaly stream; the Rubik’s cube example assigns 0.529 to the normal-behavior stream and 0.137 to the anomaly stream, while appearance remains active in displayed examples.Source: §4.7.1 · pp. 17–18, Figure 14 · p. 18

4.7.2. Visualization of Hard-negative Feature Separation

  • GradCO’s hard-negative similarity matrices show higher within-identity similarity and reduced cross-identity similarity, forming a clearer block-like structure than the diffuse X2VLM baseline.For example, ID1_ActA and ID1_ActB retain similarity 0.86, while some ID1–ID9 pairs fall to 0.44; the qualitative comparison does not isolate B-MoE’s and COL’s individual effects.Source: §4.7.2 · pp. 18–19, Figure 15 · p. 19

4.7.3. Multi-dimensional Performance Comparison

  • GradCO leads the displayed PAB, Multi-Weather, and OOD TPAS metrics; on UCC OOD it improves over CMP (1M) by 7.30 percentage points in R@1 and 7.66 in mAP.On TIPR, it achieves the best displayed averages on CUHK-PEDES, ICFG-PEDES, and RSTPReID and leads UFine3C, while Bi-IRRA remains higher on UFine6926 R-Avg.Source: Figure 16 · p. 19, §4.7.3 · p. 20

4.8. Case Study

On representative PAB queries, GradCO often ranks targets ahead of CMP when identity and behavior cues conflict, including cases where CMP selects a behavior-mismatched image. It also misses Rank-1 in some difficult cases, though it ranks the target above CMP in the cited failures.

4.8. Case Study

  • GradCO ranks the target first in the swing and race cases, while CMP ranks a behavior-mismatched image first in the swing case; in the football case, GradCO ranks a hard negative first and the target second, while CMP ranks the target fifth.The swing query combines appearance cues with the behavior cue “falling down.”Source: §4.8 · p. 20
  • Across nine further cases, GradCO ranks the target first, including examples where it retrieves a same person with the queried behavior rather than CMP’s behavior-mismatched result.Examples include the tilting-gate and handstand cases; in the lamppost case it distinguishes a collision moment from a visually similar walking case.Source: §4.8 · pp. 20–22, Figure 18 · p. 21
  • In the gymnasium failure, GradCO ranks a same-identity, behavior-mismatched sample first and the target second, while CMP ranks the target fifth or lower.Along with the football case, this indicates that GradCO can rank the target above CMP even when it misses Rank-1.Source: §4.8 · p. 22

5. Conclusion

The paper identifies gradient conflict and aggregation distortion as challenges in TPAS decoupling paradigms. It proposes GradCO to address them and reports experiments on five TPAS/TIPR benchmarks.

5. Conclusion

  • The paper identifies gradient conflict and aggregation distortion as two challenges of existing TPAS decoupling paradigms.Source: §5 · p. 22
  • GradCO combines appearance–behavior decoupling, Conflict Offload Learning to alleviate hard-negative-induced optimization conflicts, and Hierarchical Intent Alignment for intent-aware feature aggregation.Source: §5 · p. 22
  • Experiments on five TPAS/TIPR benchmark datasets validate the effectiveness and robustness of GradCO.Source: §5 · p. 22
Loading 2610.37267v1…