Source-linked AI summary
An End-to-End Transformer Model for Crowd Localization
Dingkang Liang, Wei Xu, Xiang Bai
TL;DR
Crowd localization needs accurate head positions, but existing detection- and map-based methods rely on pseudo boxes, complex representations, or post-processing. CLTR performs end-to-end point-set regression with transformer features, trainable queries, and context-aware KMO matching. Experiments on five datasets demonstrate effectiveness, with state-of-the-art or highly competitive localization performance and reported best results on several datasets.
Problem
Existing crowd-localization methods rely on pseudo-bounding boxes or map representations with complicated post-processing, while L1 matching can produce ambiguous point assignments.
Method
CLTR formulates crowd localization as end-to-end point-set regression using CNN features, transformer encoder-decoder layers, trainable embeddings, and a KMO-based Hungarian matcher with nearby-head context.
Results
Experiments on five challenge datasets demonstrate effectiveness, and CLTR achieves state-of-the-art or highly competitive localization performance using a single-scale, low-resolution feature map.
Takeaways & Limitations
CLTR provides a simple regression-based perspective for crowd localization while removing additional pre-processing and post-processing.
Takeaways & Limitations
Fixed-size 256 × 256 crops may fail on extremely large heads, while confused background regions can produce unsatisfied localization.
Abstract
from arXiv · showhide
Crowd localization, predicting head positions, is a more practical and high-level task than simply counting. Existing methods employ pseudo-bounding boxes or pre-designed localization maps, relying on complex post-processing to obtain the head positions. In this paper, we propose an elegant, end-to-end Crowd Localization Transformer named CLTR that solves the task in the regression-based paradigm. The proposed method views the crowd localization as a direct set prediction problem, taking extracted features and trainable embeddings as input of the transformer-decoder. To reduce the ambiguous points and generate more reasonable matching results, we introduce a KMO-based Hungarian matcher, which adopts the nearby context as the auxiliary matching cost. Extensive experiments conducted on five datasets in various data settings show the effectiveness of our method. In particular, the proposed method achieves the best localization performance on the NWPU-Crowd, UCF-QNRF, and ShanghaiTech Part A datasets.
1 Introduction
Crowd localization seeks head-center points, but detection- and map-based pipelines rely on pseudo boxes, complex representations, or non-differentiable post-processing. CLTR instead formulates localization as end-to-end point-set regression and adds context-aware matching to address ambiguous assignments.
- Crowd localization predicts each head’s center point, using point-level annotations because bounding-box annotation is expensive in dense scenes.
- Detection-based methods initialize pseudo ground-truth boxes and use NMS, while map-based methods require complex multi-scale representations and non-differentiable location extraction.
- Regression-based methods directly predict coordinates without pseudo boxes, generated maps, NMS, find-maxima, or complex multi-scale fusion.
- L1-based Hungarian matching produces ambiguous point-to-ground-truth assignments because it lacks context and dense heads have similar textures and confidence scores.
- CLTR treats localization as point-set prediction and uses KMO-based Hungarian matching with nearby-head context as an auxiliary cost.
- Using a single-scale, low-resolution 1/32 input feature map, CLTR achieves state-of-the-art or highly competitive localization performance.
2 Related Works
Prior crowd-localization work largely uses detection or map representations, while regression-based approaches remain limited. CLTR’s transformer design uses trainable instance queries to predict points directly, avoiding massive fixed proposal sets.
- Detection-based methods generate pseudo bounding boxes from nearest-neighbor distances and commonly apply NMS to filter predictions.
- Map-based methods derive head locations from density-map maxima, but overlapping responses in extremely dense regions make locations indistinguishable.
- CLTR extracts CNN features, processes them with transformer encoder and decoder layers, and decouples decoded features into point coordinates and confidence scores.
- Compared with P2PNet’s large fixed proposal set and preprocessing, CLTR uses a few trainable instance queries for a more unified regression framework.
3 Our Method
CLTR directly predicts head points through an end-to-end CNN–Transformer architecture, while KMO-based Hungarian matching uses nearby-head context to improve assignments. Training combines one-to-one matching with point regression and classification losses.
- 3 Our Method: CLTR directly predicts all head instances using a CNN backbone, Transformer encoder-decoder, point regression head, and classification head.The backbone uses a single-scale feature map, which is flattened with positional embeddings before encoding.
- 3 Our Method: Trainable head queries interact with encoded image features through decoder cross-attention to produce point coordinates and confidence scores.The decoder uses conditional cross-attention, combining trainable embeddings with content queries.
- 3.3 KMO-based Matcher: After one-to-one matching, unmatched predictions are treated as background and the model optimizes point regression alongside classification.The final loss is L = L_cls + λL_loc, with λ set to 2.5; ground-truth coordinates are normalized to [0, 1].
- 3.3 KMO-based Matcher: The L1-based Hungarian matcher uses point-coordinate distance and prediction confidence, but can produce unsatisfactory assignments without context.The paper describes a case where a prediction matches a less appropriate ground-truth point because the matcher lacks contextual information.
- 3.3 KMO-based Matcher: KMO-based Hungarian matching adds nearby-head context as an auxiliary cost to reduce ambiguous assignments and preserve point-set geometric relationships.KMO uses average distances to neighboring ground-truth and predicted points, helping resolve conflicts when multiple ground truths favor one prediction.
- 3.3 KMO-based Matcher: The matching procedure is used only during training, while point predictions are produced directly by the network.This separates context-aware assignment from inference-time prediction.
4 Experiments
Experiments evaluate CLTR across five public crowd datasets using localization and counting metrics. The setup includes dataset-specific thresholds and standard point-based evaluation, while counting is incidental to predicted-point localization.
- Implementation Details: The implementation uses a ResNet50 backbone, six encoder layers, six decoder layers, and 500 instance queries.Training uses Adam with a learning rate of 1e-4 plus random cropping, scaling, and horizontal flipping.
- Datasets: CLTR is evaluated on five public datasets, including NWPU-Crowd, JHU-Crowd++, UCF-QNRF, and ShanghaiTech Parts A and B.NWPU-Crowd provides both point-level and box-level annotations, while the other datasets differ in scale and density.
- Evaluation Metrics: Counting is treated as an incidental task, with the total count equal to the number of predicted points.MAE and MSE compare predicted and ground-truth counts across images.
- Evaluation Metrics: Localization is measured with Precision, Recall, and F1-measure using distance thresholds between predicted points and ground-truth points.Thresholds are dataset-specific: box-derived for NWPU-Crowd, fixed values for ShanghaiTech, and a range from 1 to 100 pixels for UCF-QNRF.
5 Results and Analysis
Across localization, counting, visualizations, ablations, and complexity analyses, CLTR performs strongly across varied crowd scenes while exposing matching and computational trade-offs.
- Crowd Localization: CLTR outperforms GL and AutoScale by at least 5.8% and 2.5% in F1-measure on NWPU-Crowd validation and test sets, respectively.It remains competitive with TopoCount despite using only point-level annotations rather than box-level labels.
- Crowd Localization: CLTR achieves the best Average Precision, Average Recall, and F1-measure on UCF-QNRF, and exceeds TopoCount by 2.1% F1-measure on strict ShanghaiTech Part A.The ShanghaiTech comparison remains favorable under the less strict σ = 8 setting.
- Crowd Counting: CLTR achieves state-of-the-art or highly competitive counting performance using a single-scale, low-resolution feature map, including first-place MAE and MSE on NWPU-Crowd test.It also leads MAE and MSE on ShanghaiTech Part A and Part B, leads MSE on UCF-QNRF, and improves JHU-Crowd++ MSE by 18.9 over GL.
- Visualizations: Qualitative examples show robust localization across negative, sparse, extremely dense, and dark NWPU-Crowd scenes.The visualizations specifically highlight resistance to dense fake humans and performance in congested or dark images.
- Ablation Studies: On UCF-QNRF, six transformer layers and 500 queries yield the best reported configuration, while 700 or 300 queries worsen MAE from 85.8 to 86.8 or 89.9.The authors hypothesize that too few queries lose heads, whereas too many generate massive negative samples.
- Ablation Studies: Removing KMO increases UCF-QNRF counting MAE from 85.8 to 91.3, while different nearest-neighbor counts consistently improve over baseline and k = 4 performs best.A fixed-radius alternative is rejected because dynamic KNN computation makes training time unacceptable.
- Computational Statistics: CLTR has the largest parameter count but the smallest MACs, and speeding up the model remains future work.The complexity comparison uses 1024 × 768 inputs on a 3090 GPU.
- Limitations: Fixed-size 256 × 256 crops may fail on extremely large heads, while confusing background regions can produce unsatisfactory localization.The paper suggests image resizing for large heads and additional modalities such as thermal images for confusing regions.
6 Conclusion
The paper presents CLTR as an end-to-end crowd localization framework using a regression-based paradigm and one-to-one matching. Its KMO-based Hungarian matcher uses context as an auxiliary matching cost, and experiments on five challenge datasets demonstrate effectiveness.
- CLTR frames crowd localization as an end-to-end regression-based task.
- The training process uses one-to-one matching between predictions and targets.
- The KMO-based Hungarian matcher incorporates context information as an auxiliary matching cost.
- Experiments on five challenge datasets demonstrate the effectiveness of the proposed methods.