PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Recognizing Unseen Categories in Unseen Domains

Massimiliano Mancini, Zeynep Akata, Elisa Ricci, Barbara Caputo

arXiv:2007.12256v2cs.CV

TL;DR

Deep visual recognition can degrade on images from classes and domains unseen during training, while ZSL and DG have generally been treated separately. CuMix uses curriculum-based mixing of source images and features to simulate both shifts; the paper reports effectiveness across ZSL, DG, and their joint setting, with state-of-the-art performance in almost all task settings.

  • Problem

    ZSL and DG address semantic and domain shifts separately, leaving the joint task of recognizing unseen categories in unseen domains unresolved.

    Source: §1 · pp. 1, 3

  • Method

    CuMix simulates unseen categories and domains during training by mixing available source classes and domains at image and feature levels, with a curriculum of increasingly complex samples.

    Source: §1 · p. 3, §3.2 · p. 6

  • Results

    CuMix achieves state-of-the-art performance in almost all settings across ZSL, DG, and ZSL+DG tasks.

    Source: §1 · pp. 3–4, §5 · p. 14

  • Takeaways & Limitations

    The paper demonstrates that one holistic method can be applied to ZSL, DG, and the joint task of recognizing unseen concepts in unseen domains.

    Source: §1 · p. 4, §5 · p. 14

    The evaluated joint setting holds out both the test classes and domains, with Ys ∩ Yu = ∅ and Ds ∩ Du = ∅.

    Source: §3.1 · pp. 5–6

Abstract

from arXiv · show

Current deep visual recognition systems suffer from severe performance degradation when they encounter new images from classes and scenarios unseen during training. Hence, the core challenge of Zero-Shot Learning (ZSL) is to cope with the semantic-shift whereas the main challenge of Domain Adaptation and Domain Generalization (DG) is the domain-shift. While historically ZSL and DG tasks are tackled in isolation, this work develops with the ambitious goal of solving them jointly,i.e. by recognizing unseen visual concepts in unseen domains. We presentCuMix (CurriculumMixup for recognizing unseen categories in unseen domains), a holistic algorithm to tackle ZSL, DG and ZSL+DG. The key idea of CuMix is to simulate the test-time domain and semantic shift using images and features from unseen domains and categories generated by mixing up the multiple source domains and categories available during training. Moreover, a curriculum-based mixing policy is devised to generate increasingly complex training samples. Results on standard SL and DG datasets and on ZSL+DG using the DomainNet benchmark demonstrate the effectiveness of our approach.

1 Introduction

Deep visual models can fail when test images come from both unseen classes and unseen domains, a joint setting that separate ZSL and DG approaches do not resolve. CuMix addresses this setting by mixing training samples across classes and domains, at image and feature levels, with increasingly challenging mixtures.

1 Introduction

  • The joint problem matters because real-world training sets cannot capture the full variability of visual concepts and their acquisition conditions.Prior approaches separately consider recognizing familiar categories in a new domain or new categories in familiar domains; the new-category, new-domain case remains unclear.Source: §1 · pp. 2–3
  • The joint task asks a model trained on multiple domains and categories to recognize unseen concepts in unseen domains, using semantic side information.Unlike standard ZSL or DG considered separately, both the test label set and test domain set are unseen.Source: Figure 1 · p. 2, §1 · p. 3
  • ZSL+DG removes guarantees available in the separate tasks: class-domain disentanglement may not hold for unseen classes, and image-to-attribute mappings may not generalize across domains.These challenges arise because ZSL+DG combines semantic shift with domain shift.Source: §1 · p. 3
  • CuMix simulates test-time semantic and domain shifts by mixing samples from different training categories and domains into partially novel examples.The paper also applies mixing at image and feature levels and uses a curriculum to generate increasingly complex samples.Source: §1 · p. 3
  • CuMix reports state-of-the-art performance in ZSL and DG benchmarks and effectiveness in the joint ZSL+DG setting.The method simulates new domains and categories by mixing available training domains and classes at both image and feature levels, with a curriculum that increases difficulty.Source: §1 · pp. 3–4

2 Related Works

Prior DG work includes invariant representations, domain-specific architectures, specialized training procedures, and synthetic augmentation. CuMix is closest to augmentation methods but uses mixup to simulate both semantic and domain shifts; related ZSL and DA work does not address DG without target data.

2 Related Works

  • DG methods span invariant representations, domain-agnostic architectures, specialized optimization or training, and data or feature augmentation.Examples include alignment losses, meta-learning, episodic training, and adversarial augmentation.Source: §2 · p. 4
  • CuMix belongs to synthetic augmentation approaches but uses mixup to perturb features and simulate test-time semantic and domain shifts.The authors distinguish this use from prior mixup work for domain adaptation, which reinforced domain-discriminator judgments.Source: §2 · p. 4
  • Prior ZSL methods learn image-to-semantic mappings, while ZSL-plus-DA approaches address domain shift but do not cover DG without target data.The cited related work includes transductive ZSL and DA and cross-domain image retrieval.Source: §2 · p. 5

3 Method

CuMix jointly formulates zero-shot learning and domain generalization as recognizing unseen classes in unseen domains. It uses curriculum mixup to simulate both shifts during training by interpolating available images, features, and labels.

3.1 Problem Formulation

  • ZSL+DG requires predicting unseen classes in unseen domains, unlike standard ZSL or DG, which share test domains or label sets with training, respectively.The formulation assumes multiple source domains with different distributions and disjoint train/test class and domain sets.Source: §3.1 · pp. 5–6

3.2 Simulating Unseen Domains and Concepts through Mixup

  • CuMix interpolates available samples to simulate novel domains and semantic concepts, avoiding explicit generation of unseen-domain and unseen-class images.Its triplets contain two samples from the same domain and one from a different domain, enabling intra-domain or cross-domain mixes.Source: §3.2 · pp. 6–8
  • Image-level and feature-level mixup train the feature extractor and semantic projection to handle samples and features from mixed domains and classes.A further loss enforces semantic consistency of mixed features in the embedding space, making the classifier and projection robust to mixed-domain, mixed-semantic features.Source: §3.2 · p. 8
  • A curriculum progressively increases mixing complexity: single-domain classification, same-domain mixes, then cross-domain mixes.The transition is scheduled over training epochs, with the final phase using cross-domain mixing to simulate test-time domain shift.Source: §3.2 · pp. 8–9
  • The final objective combines the standard classification loss with image-level and feature-level mixup losses in the semantic embedding space.The mixup-loss weights are hyperparameters; cross-entropy is used, although any ZSL objective can be applied.Source: §3.2 · p. 9

4 Experimental results

Experiments evaluate CuMix on standard ZSL and DG benchmarks and the new ZSL+DG setting. CuMix outperforms EpiFCR+SPNet by almost 1% on average in ZSL+DG, with stronger gains over the ZSL-only baseline on infograph and quickdraw.

4.1 Datasets and implementation details

  • CuMix is evaluated on CUB, SUN, AWA, and FLO for ZSL, PACS for DG, and DomainNet for ZSL+DG; DomainNet uses 245 seen training classes, 55 validation classes, and 45 unseen test classes.DomainNet results average per-class accuracy on a held-out domain; real photos are excluded because the backbone was pretrained on ImageNet.Source: §4.1 · pp. 10–11

4.2 Results

  • CuMix performs at or above state-of-the-art on the ZSL benchmarks except AWA, and consistently outperforms state-of-the-art on fine-grained CUB, SUN, and FLO.The authors suggest ImageNet may represent test-class information for AWA, a non-fine-grained dataset.Source: §4.2 · pp. 11–12
  • On PACS, CuMix achieves results comparable to Epi-FCR and improves over CrossGrad, using mixed source samples or features to simulate novel domains.CuMix uses one domain-agnostic architecture, unlike Epi-FCR’s multiple domain-specific architectures.Source: §4.2 · pp. 12–13, Table 1 · p. 12
  • Feature-level mixing improves over the baseline, while image-level mixing can reduce performance; combining both improves results in almost all domains.In sketches, feature-level mixing gains 2% versus baseline, while image-level mixing drops 10%; the curriculum further boosts difficult cases and stabilizes training.Source: §4.2 · p. 13, Table 2 · p. 13
  • CuMix beats EpiFCR+SPNet by almost 1% on average in ZSL+DG, and exceeds the ZSL-only baseline by more than 1.5% on infograph and quickdraw.It also outperforms SPNet with standard mixup in every scenario; sketch is the exception to its advantage over EpiFCR+SPNet.Source: Table 3 · p. 14, §4.2 · p. 14

5 Conclusions

The paper introduces ZSL+DG, where models must recognize unseen concepts in unseen domains, and proposes CuMix to simulate these shifts by mixing source domains and classes. Experiments report state-of-the-art performance in almost all settings across the three tasks.

5 Conclusions

  • CuMix simulates inputs and features from unseen domains and categories by mixing available source domains and classes at image and feature levels.Source: §5 · p. 14
  • The paper proposes ZSL+DG as recognition of unseen concepts in unseen domains.Source: §5 · p. 14
  • Experiments on public benchmarks show state-of-the-art performance in almost all settings across ZSL, DG, and ZSL+DG.Source: §5 · p. 14

A Hyperparameter choices

The appendix reports dataset-specific training choices for ZSL, DG, and ZSL+DG, including backbone configurations and optimization settings.

ZSL

  • For ZSL, CuMix uses ResNet-101 ImageNet features without end-to-end training, identity f, and a fully connected g for feature-level mixing.Source: A Hyperparameter choices · p. 18
  • ZSL training uses dataset-specific epoch counts and mixing parameters, with batch size 128 and SGD at initial learning rate 0.1.Source: A Hyperparameter choices · p. 18

DG

  • For DG, the model uses an ImageNet-pretrained ResNet-18 as f, identity g, and a learned fully connected classifier, following training protocols.Source: A Hyperparameter choices · p. 18

ZSL+DG

  • For ZSL+DG, experiments use an ImageNet-pretrained ResNet-50, SGD with momentum, and eight training epochs, with CuMix parameters varying by scenario.Source: A Hyperparameter choices · p. 19

B ZSL+DG: analysis of additional baselines

CuMix outperforms the additional ZSL and DG baselines evaluated for DomainNet ZSL+DG, while some baseline combinations underperform their standalone counterparts. The section also explains how the combined EpiFCR and ZSL baselines were adapted.

B ZSL+DG: analysis of additional baselines

C ZSL+DG: ablation study

In the two-source Real-Painting setting, the ablations favor CuMix’s mixing choices and curriculum order over the tested alternatives.

C ZSL+DG: ablation study

  • CuMix improves average performance over SPNet by 2% and over SPNet + Epi-FCR by 1.1% in the two-source setting.Source: C ZSL+DG: ablation study · p. 20
  • Without curriculum, CuMix outperforms standard image-level mixup by more than 2%, while feature-and-image mixing without curriculum remains below CuMix.Source: C ZSL+DG: ablation study · p. 20
  • Reversing CuMix’s curriculum order lowers performance; the reported comparison favors semantic mixing within domains before inter-domain mixing.Source: C ZSL+DG: ablation study · p. 20

D ZSL results

In ZSL scenarios, the cross-entropy-only baseline performs below most methods except on FLO, while adding CuMix’s mixing strategy boosts performance in all scenarios and achieves state-of-the-art results in most.

D ZSL results

  • Adding CuMix’s mixing strategy boosts performance in all ZSL scenarios and achieves state-of-the-art results in most.The results are reported in Table 6.Source: D ZSL results · p. 21
  • The cross-entropy-only baseline performs below most ZSL methods in all scenarios except FLO.Source: D ZSL results · p. 21
  • The comparison excludes methods that generate features of unseen classes because they can augment any ZSL method, outside this work’s scope.The authors note that such methods can improve both baselines and CuMix.Source: D ZSL results · p. 21

Related papers

Loading 2007.12256v2…