Source-linked AI summary
Learning Attributes Equals Multi-Source Domain Generalization
Chuang Gan, Tianbao Yang, Boqing Gong
TL;DR
Robust attribute detection across categories, especially unseen ones, remains underexplored despite attributes’ broad utility. The paper treats categories as related source domains, learns category-invariant and attribute-discriminative representations, and reports gains across four datasets and three tasks, with benefits for attribute detection, zero-shot recognition, and image retrieval.
Problem
Robust attribute detection across categories, including previously unseen ones, remains insufficiently addressed by existing work.
Method
The paper casts attribute detection as multi-source domain generalization and learns a kernel projection that distills category-invariant and attribute-discriminative signals for standard classifiers.
Results
Experiments on four datasets and three tasks report improved attribute quality and benefits for zero-shot recognition and image retrieval, with results significantly better than competitive baselines.
Takeaways & Limitations
Treating categories as domains provides a representation for attribute detectors that generalize across categories and supports succeeding applications.
Takeaways & Limitations
The approach models marginal distributions Py(x) rather than the joint distributions Py(x, a), which the paper leaves for future work.
Abstract
from arXiv · showhide
Attributes possess appealing properties and benefit many computer vision problems, such as object recognition, learning with humans in the loop, and image retrieval. Whereas the existing work mainly pursues utilizing attributes for various computer vision problems, we contend that the most basic problem---how to accurately and robustly detect attributes from images---has been left under explored. Especially, the existing work rarely explicitly tackles the need that attribute detectors should generalize well across different categories, including those previously unseen. Noting that this is analogous to the objective of multi-source domain generalization, if we treat each category as a domain, we provide a novel perspective to attribute detection and propose to gear the techniques in multi-source domain generalization for the purpose of learning cross-category generalizable attribute detectors. We validate our understanding and approach with extensive experiments on four challenging datasets and three different problems.
1. Introduction
Attributes support tasks including recognition, zero-shot learning, and image/video search, but robust detection across seen and unseen categories remains underexplored. The paper reframes attribute detection as multi-source domain generalization and learns attribute-oriented representations for cross-category detectors.
- Attributes are human-nameable and machine-detectable concepts used to describe instances, support zero-shot learning, and enable image/video search.
- Existing work rarely explicitly addresses whether attribute detectors generalize across categories, including previously unseen ones.For example, a four-legged detector should recognize pandas despite training on horses, cows, zebras, and pigs.
- Object-recognition features may be suboptimal for attributes because attributes occupy a middle semantic level, span objects, and cross object-category boundaries.The attribute “bush” can be shared by elephant, sheep, and giraffe, while one object can have multiple distinctive attributes.
- The paper casts attribute detection as multi-source domain generalization, treating categories as domains to learn detectors that generalize to seen and unseen classes.This perspective aligns cross-category attribute detection with extracting knowledge from related source domains for unseen domains.
- The approach distills category-invariant and attribute-discriminative signals through a kernel projection into features usable by standard classifiers such as SVMs.The projection operates on kernels derived from image features, category labels, and attribute labels.
2. Related work and background
The paper connects attribute detection with domain adaptation and generalization, building on methods that reduce distributional differences across related domains. Its background formalizes distributional variance and its empirical kernel estimate.
- The paper unifies attribute detection with domain adaptation and generalization as two previously separate research areas.
- Earlier attribute-detection work modeled attribute correlations, localized part-related attributes, or studied attribute-category relationships, while newer work applied deep models.
- Prior methods generally did not explicitly model cross-category attribute generalization, and category-specific feature selection offered only limited gain.
- Attributes describe instances, compose categories, transfer knowledge between them, and reveal structure useful for regulating visual-recognition models.
- Multi-source domain generalization learns from several related source domains so knowledge applies to different, especially unseen, domains.The paper adapts this objective to learning attribute detectors that generalize across categories.
- Distributional variance measures differences among domain distributions through distances between their kernel mean maps and their pooled mean.The empirical estimate uses a centered kernel matrix K and coefficient matrix Q: bV(Y) = tr(KQ).
3. Attribute detection
The paper formulates attribute detection over category-labeled image distributions and targets detectors that remain accurate across seen and unseen categories. It models categories as related domains sampled from a common distribution.
- The formal problem uses image features, category labels, and binary attribute annotations, with the approach extendable to multi-way and continuous-valued attributes.
- Because an attribute can appear across many categories, the objective is to learn accurate and robust detectors that perform well on previously unseen classes.
- The paper introduces attribute detection as a domain-generalization problem by treating each category as a domain.
- Category domains are distributions over image representations and attribute labels, assumed related and sampled from a common distribution P.These relationships motivate representations whose detectors work on both seen and unseen classes.
4. Approach
The approach learns kernel-based representations that combine category invariance with attribute discriminativeness. UDICA reduces category distributional mismatches, while kernel alignment incorporates correlated attribute labels for cross-category attribute detection.
- Representation learning: The method transforms image features into new representations that preserve category-invariant and attribute-discriminative signals for training attribute detectors.The transformed features can be used with linear classifiers, while the transformed kernel supports kernelized classifiers.
- UDICA: UDICA minimizes distributional variance across categories so the learned projection can generalize to unseen classes.The method expects perfectly matched domains when the variance is zero.
- UDICA: The UDICA objective combines data variance maximization, category distributional-variance minimization, and transformation regularization.The numerator represents data variance, while the denominator combines distributional variance with regularization over B.
- Centered kernel alignment: Kernel alignment models supervised multi-attribute information while preserving correlations among attribute labels.The attribute kernel L is computed directly from attribute labels and serves as the target for the transformed kernel.
- KDICA: KDICA integrates centered kernel alignment with UDICA, balancing data variance against supervised attribute labeling information through γ.The alignment term tracks attribute discriminativeness, while UDICA facilitates cross-category generalization.
- Optimization: The resulting optimization is solved as a generalized eigen-decomposition, with leading eigenvectors defining the learned projection.The number of retained eigenvectors is selected by cross-validation.
5. Experiment
The experiments evaluate attribute detection and downstream zero-shot learning and image retrieval across four datasets, comparing the proposed representations with established baselines. Results show consistent gains in attribute prediction and downstream recognition, while retrieval improvements depend on the framework and query type.
- Experimental scope: Experiments cover attribute detection on four datasets, zero-shot learning on three datasets, and multi-attribute image retrieval on AWA.Attribute detection and retrieval use averaged AUC, while zero-shot learning uses classification accuracy.
- Baselines: The evaluation compares the proposed approaches with DAP, IAP, ALE, and HAP, using identical input features and classifiers for direct comparison with DAP.Results reported from original papers are supplemented by runs using authors’ released source code, marked by ‘*’.
- Attribute prediction: KDICA substantially improves DAP for attributes appearing across visually different categories, whereas gains are limited for trivial or already category-shared attributes.DAP already detects “yellow” at nearly 100% accuracy, while “swim” is shared by visually similar categories.
- Zero-shot learning: The proposed zero-shot learning solution outperforms other state-of-the-art methods on AWA, CUB, and UCF101, especially DAP.KDICA’s advantage over UDICA is significant on AWA, and downstream improvements can exceed those observed for attribute prediction.
- Attribute prediction: UDICA and KDICA outperform all baselines on all four datasets, with UDICA gaining 6.3% over DAP on AWA and 5.4% on CUB.KDICA is slightly better than UDICA, and improves over DAP for 71 of 85 AWA attributes and 272 of 312 CUB attributes.
- Image retrieval: Attribute-oriented representations improve late-fusion image retrieval across single-attribute, pair, and triplet queries, while TagProp gains are significant mainly for single-attribute queries.Under TagProp, improvements for attribute pairs and triples are marginal.
6. Conclusion
The paper develops an attribute-oriented feature representation by casting attribute detection as multi-source domain generalization. Experiments on four datasets and three tasks show benefits for attribute quality and downstream applications.
- The proposed representation casts attribute detection as multi-source domain generalization and supports off-the-shelf classifiers.
- The learned attribute detectors generalize across object-category boundaries to unseen classes.
- Experiments on four datasets and three tasks show improved attribute quality and benefits for zero-shot recognition and image retrieval.