Source-linked AI summary
Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks
Weihua Chen, Xianzhe Xu, Jian Jia, Hao luo, Yaohua Wang, Fan Wang, Rong Jin, Xiuyu Sun
TL;DR
Human-centric tasks need general representations learned from abundant unlabeled human images, but different tasks demand different balances of semantic and appearance information. SOLIDER uses human priors for pseudo semantic supervision and a semantic controller for adjustable representations, and verifies the resulting representation across six downstream tasks. The paper scopes its main concerns to 2D image-level tasks.
Problem
Learning a general representation from massive unlabeled human images is challenging because downstream human-centric tasks require different ratios of semantic and appearance information.
Method
SOLIDER uses human-image prior knowledge to generate pseudo semantic labels and a conditional semantic controller that adjusts representation semantics through λ ∈[0, 1].
Results
SOLIDER representations are verified on six downstream human-centric visual tasks.
Takeaways & Limitations
A single pretrained model can be adjusted to produce representations with different semantic-information ratios for downstream task requirements.
Takeaways & Limitations
The paper focuses on 2D image-level feature representations, excluding 3D and sequential-data tasks from its main scope.
Abstract
from arXiv · showhide
Human-centric visual tasks have attracted increasing research attention due to their widespread applications. In this paper, we aim to learn a general human representation from massive unlabeled human images which can benefit downstream human-centric tasks to the maximum extent. We call this method SOLIDER, a Semantic cOntrollable seLf-supervIseD lEaRning framework. Unlike the existing self-supervised learning methods, prior knowledge from human images is utilized in SOLIDER to build pseudo semantic labels and import more semantic information into the learned representation. Meanwhile, we note that different downstream tasks always require different ratios of semantic information and appearance information. For example, human parsing requires more semantic information, while person re-identification needs more appearance information for identification purpose. So a single learned representation cannot fit for all requirements. To solve this problem, SOLIDER introduces a conditional network with a semantic controller. After the model is trained, users can send values to the controller to produce representations with different ratios of semantic information, which can fit different needs of downstream tasks. Finally, SOLIDER is verified on six downstream human-centric visual tasks. It outperforms state of the arts and builds new baselines for these tasks. The code is released in https://github.com/tinyvision/SOLIDER.
1. Introduction
SOLIDER targets the challenge of learning broadly useful human representations from massive unlabeled data while addressing the differing semantic and appearance needs of downstream tasks. It introduces human-prior semantic supervision and a controllable representation, then evaluates the framework across six human-centric tasks.
- Motivation: Massive unlabeled human-image collections create an opportunity, but learning representations that benefit diverse downstream tasks remains challenging.LUPerson contains approximately 4.18M images, compared with approximately 1M in ImageNet.
- Motivation: Existing self-supervised methods learn general image representations but lack designs specifically targeting human-centric visual tasks.
- Motivation: DINO representations cluster different parts of the same person by appearance continuity, even when those parts have different semantic meanings.This can produce sub-optimal results on tasks such as human parsing and pedestrian detection because semantic information is lacking.
- Approach: SOLIDER uses human-image prior knowledge to produce pseudo semantic labels and adds token-level semantic supervision to learn stronger semantic information.
- Approach: Its semantic controller adjusts the representation's semantic-information ratio through an input value, allowing different downstream tasks to receive task-suitable representations.
- Evaluation: The learned general human representation is evaluated on six downstream human-centric visual tasks.
2. Related Work
Related work covers self-supervised representation learning, clustering-based semantic discovery, and human-centric visual tasks. The paper positions SOLIDER as a human-focused self-supervised approach using token-level semantic information rather than supervised training or appearance-dominated clustering alone.
- Self-supervised learning: Self-supervised learning uses unlabeled data for representation learning through contrastive learning and masked image modeling.
- Self-supervised learning: Contrastive methods bring augmented views of the same image closer while distinguishing each image from others; MoCo uses a momentum encoder to improve training.
- Self-supervised learning: Masked image modeling methods predict discrete or masked-image tokens, but they do not explicitly identify semantic information from images for supervision.
- Semantic clustering: DeepCluster performs image-level clustering, while pixel-level extensions attempt semantic labeling but still face representation-learning problems.
- Human-centric tasks: Human-centric vision includes tasks such as person re-identification, attribute recognition, person search, detection, parsing, and pose estimation.
- Human-centric tasks: The paper focuses on 2D image-level feature representations and excludes tasks involving 3D or sequential data from its main scope.
- Human-centric representation learning: TCFormer addresses human-centric tasks with supervision, whereas SOLIDER is motivated by exploiting massive public unlabeled data.
3. The SOLIDER
SOLIDER combines semantic self-supervision from human-image priors with a semantic controller that adjusts representations for downstream task needs.
- 3.1. Semantic Supervision from Human Prior: DINO provides appearance-focused representations, but clustering them groups tokens by visual appearance without reliably identifying semantic parts.The paper illustrates appearance-driven confusion between visually similar clothing regions.
- 3.1. Semantic Supervision from Human Prior: Human-image layout priors assign pseudo semantic labels according to token positions, such as upper body at the top and shoes at the bottom.The method exploits the recurring upright arrangement of people in unlabeled images.
- 3.1. Semantic Supervision from Human Prior: Foreground-background clustering removes background tokens before semantic clustering, producing upper-body, lower-body, and shoe parts.Foreground tokens are selected by larger vector magnitudes, after which clustering is performed only on foreground tokens.
- 3.1. Semantic Supervision from Human Prior: The semantic pretext task applies token-level classification to backbone feature maps and also trains on randomly masked images.Feature maps are flattened into token vectors, and the semantic head predicts clustered part probabilities.
- 3.2. Semantic Controller: Downstream tasks require different semantic-to-appearance ratios, motivating controllable representations rather than one fixed pretrained model.Re-identification favors appearance, whereas pedestrian detection and human parsing emphasize semantic information; task tokens offer limited discrete control.
- 3.2. Semantic Controller: SOLIDER uses a continuous controller value λ ∈[0, 1] to modulate feature maps and produce representations with the requested semantic-information ratio.The controller encodes λ into weights and biases applied after Swin blocks; λ is randomly sampled during training, with B(p = 0.5) performing best among tested distributions.
4. Experiments
SOLIDER is evaluated as a pre-trained representation across six human-centric tasks, with analyses examining semantic clustering, controllable semantic–appearance balance, and comparisons against baselines and state-of-the-art methods. The experiments show that semantic supervision and controller-based adjustment improve broad downstream performance while task-specific ratios matter.
- Experimental Setup: SOLIDER is evaluated on six human-centric visual tasks using LUPerson's 4.18M unlabeled human images and commonly used downstream datasets.The tasks are person re-identification, attribute recognition, person search, pedestrian detection, human parsing, and pose estimation.
- Semantic Clustering Analysis: Semantic clustering makes features with the same semantic meaning closer, even when their appearances differ.Before SOLIDER, features primarily group by appearance; for example, blue shorts are closer to a blue shirt than to black shorts despite the latter sharing the same semantic meaning.
- Semantic Controller Analysis: Increasing λ enlarges intra-image distances and reduces inter-image distances, indicating a shift toward semantic information in the representation.The controller experiment defines these distances so that larger intra-image and smaller inter-image distances correspond to stronger semantic dominance.
- Semantic Controller Analysis: Increasing λ improves pedestrian detection but worsens person re-identification, so large λ suits detection while small λ better fits re-identification.The observed trade-off supports using the controller to match representation composition to downstream task requirements.
- Ablation Study: Semantic clustering improves most downstream tasks, while adding the controller further improves results especially for person re-identification and person search.Re-identification gains over DINO remain limited because appearance is already the key clue and is well learned by DINO.
- Ablation Study: 74.9 average performance with SOLIDER exceeds DINO LUP1M's 74.0 across five non-re-identification tasks, a further 0.9 improvement consistent across all tasks.DINO LUP1M itself averages 1.4 points above supervised ImageNet, partly because its street-view training images resemble re-identification data.
5. Conclusion
SOLIDER uses human-image prior knowledge to learn more semantic representations and adjusts their semantic–appearance balance through a controller. Its representations are evaluated across six human-centric visual tasks.
- SOLIDER uses prior knowledge from human images to train representations with more semantic information.
- A semantic controller adjusts the pretrained model using an input value to produce representations with different semantic-information ratios.
- SOLIDER representations are verified on six human-centric visual tasks.