PaperlayerView exact revision on arXiv

Source-linked AI summary

Invertible Zero-Shot Recognition Flows

Yuming Shen, Jie Qin, Lei Huang

arXiv:2007.04873v1cs.LGcs.CVstat.ML

TL;DR

Existing GAN- and VAE-based ZSL methods face challenges with ZSL-oriented regularization, sample quality, and seen–unseen bias. IZF uses an invertible flow to factor semantic and non-semantic embeddings and generate samples, with training strategies that target the bias. The paper reports consistent benchmark superiority over state-of-the-art baselines.

  • Problem

    Existing GAN- and VAE-based ZSL methods face difficulty with ZSL-oriented regularization and generation quality, while generative methods also remain challenged by seen–unseen bias.

    Source: §1 · p. 2

  • Method

    IZF factorizes visual features into semantic and non-semantic embeddings with an invertible flow, generates samples in reverse, and uses bidirectional objectives to centralize prototypes and diverge seen–unseen distributions.

    Source: §1 · p. 3

  • Results

    IZF consistently outperformed state-of-the-art baselines on benchmark datasets.

    Source: §7 · p. 14

  • Takeaways & Limitations

    The reported superiority covers both generalized and classic ZSL settings.

    Source: §1 · p. 3

    Large values of the negative MMD loss may confuse some generative models into producing unrealistic samples.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Deep generative models have been successfully applied to Zero-Shot Learning (ZSL) recently. However, the underlying drawbacks of GANs and VAEs (e.g., the hardness of training with ZSL-oriented regularizers and the limited generation quality) hinder the existing generative ZSL models from fully bypassing the seen-unseen bias. To tackle the above limitations, for the first time, this work incorporates a new family of generative models (i.e., flow-based models) into ZSL. The proposed Invertible Zero-shot Flow (IZF) learns factorized data embeddings (i.e., the semantic factors and the non-semantic ones) with the forward pass of an invertible flow network, while the reverse pass generates data samples. This procedure theoretically extends conventional generative flows to a factorized conditional scheme. To explicitly solve the bias problem, our model enlarges the seen-unseen distributional discrepancy based on negative sample-based distance measurement. Notably, IZF works flexibly with either a naive Bayesian classifier or a held-out trainable one for zero-shot recognition. Experiments on widely-adopted ZSL benchmarks demonstrate the significant performance gain of IZF over existing methods, in both classic and generalized settings.

1 Introduction

Existing generative ZSL methods face limitations in regularization, sample quality, and seen–unseen bias. IZF applies invertible flows to ZSL, factorizing semantic and non-semantic representations and using bidirectional training strategies to address these challenges.

1 Introduction

  • GANs are difficult to train with additional ZSL-oriented regularizers, VAEs/CVAEs use stochastic approximate optimization, and generative ZSL models may retain seen–unseen bias.The paper notes these approaches can address one another's drawbacks only partially.Source: §1 · p. 2
  • Flow-based models offer a shared invertible network for encoding and decoding, with the forward pass acting as an encoder and flows extendable to conditional generation.The authors motivate flows as a way to use semantic knowledge in the forward pass while generating unseen data in reverse.Source: §1 · p. 2
  • IZF introduces a factorized conditional flow for ZSL, with forward-pass semantic and non-semantic factors and reverse-pass conditional generation.The semantic factors can be regularized with category-wise similarity knowledge.Source: §1 · p. 2
  • IZF's bidirectional training centralizes seen-class prototypes and diverges synthesized-unseen from real-seen distributions to address classification stability and bias.The contribution description identifies sample-based seen–unseen distribution discrepancy as the bias mechanism.Source: §1 · p. 3
  • IZF is presented as the first generative flow model for ZSL, supports exact density estimation, and is reported superior to existing methods in GZSL and CZSL experiments.The contribution statement also describes its bias mechanism as measuring and diversifying sample-based seen–unseen discrepancy.Source: §1 · p. 3

2 Related Work

Related work situates ZSL among projection and shared-feature approaches, then describes generative methods that synthesize unseen examples for classification. It also distinguishes practical deep flows from prior literally invertible ZSL projections that cannot generate samples.

2 Related Work

  • ZSL methods inject semantic knowledge through visual-to-semantic or semantic-to-visual projections, or through shared features and multimodal functions.Generative approaches commonly synthesize unseen data for a subsequent classification stage.Source: §2 · p. 3
  • Deep generative ZSL commonly uses GANs, CVAEs, or mixtures, while flow research developed practical deep invertible networks through coupling layers.The passage also notes conditional, discrete, image-generation, adversarial, and speech-synthesis flow variants.Source: §2 · p. 3
  • Prior literally invertible ZSL projections cannot generate samples for held-out-classifier GZSL and are linear, limiting capacity and large-scale training efficiency.The passage contrasts these methods with the generative and inverse-training benefits discussed for IZF.Source: §2 · p. 4

3 Preliminaries: Generative Flows and INNs

Generative flows estimate data density by mapping data through an invertible transformation to a latent space and accounting for the Jacobian determinant. Coupling layers provide an invertible transformation whose Jacobian is efficiently computed.

3 Preliminaries: Generative Flows and INNs

  • The change-of-variables formula estimates data density from the latent prior and the Jacobian determinant of an invertible transformation.The transformation maps data x to a latent z, and the corresponding generative process uses the inverse to map z back to data.Source: §3 · p. 4
  • A flow stacks invertible functions, using f as the forward data-to-latent pass and f^-1 as the reverse latent-to-data generation pass.The passage describes a zero-mean, unit-variance Gaussian as a usually adequate latent prior.Source: §3 · p. 4
  • Coupling layers split inputs into two partitions and transform one partition using the other, with a conveniently computed log-Jacobian determinant.The transformation uses element-wise operations and neural networks s and t, followed by permutations to build compact transformations.Source: §3 · p. 4

4 Formulation: Factorized Conditional Flow

IZF uses an invertible flow to map visual features into factorized semantic and non-semantic representations, then reverses that mapping for conditional generation. Its factorized conditional scheme uses semantic knowledge to support zero-shot generalization without an auxiliary condition component.

4 Formulation: Factorized Conditional Flow

  • The conditional flow maps visual data and labels through a shared invertible network, using its forward pass for visual-to-semantic projection and its reverse pass for semantic-to-visual projection.The formulation defines seen and unseen visual and semantic spaces, and shares parameters between the two passes.Source: §4 · pp. 4–5, Figure 2 · p. 5

4.1 Forward Pass: Factorizing the Semantics

  • IZF’s forward pass factors each visual representation into a predicted semantic component and a low-level non-semantic component independent of both the semantic component and the label.The factorization is modeled with a full likelihood for training samples, unlike the cited empirical factorization approach.Source: §4.1 · pp. 5–6
  • Factored priors place the predicted semantic component around the class embedding and assign the non-semantic component a standard Gaussian prior.The semantic prior is reduced to an L2 norm against the corresponding class embedding.Source: §4.1 · p. 6
  • The factorized conditional scheme requires no auxiliary condition encoder and uses class-semantic similarity to support smooth seen-to-unseen generalization.The factorization also allows conditional generation by feeding a category embedding to the reverse pass.Source: §4.1 · p. 6

4.2 Reverse Pass: Conditional Sample Generation

  • The reverse pass generates samples conditionally from semantic category embeddings, allowing IZF to synthesize unseen visual data.Unlike approaches that use generated unseen samples only for a held-out classifier, IZF also uses them to measure seen-unseen distributional overlap.Source: §4.2 · p. 6
  • IZF’s reverse pass additionally uses generated unseen samples to measure distributional overlap between synthesized unseen and real seen data.This use of generated data is distinct from approaches that feed synthesized unseen samples only to a held-out classifier.Source: §4.2 · p. 6

4.3 Network Structure

  • The network comprises five permutation-coupling blocks, each combining a coupling layer with a fixed channel-wise permutation.The built-in coupling networks use fully connected layers with leaky ReLU activation.Source: §4.3 · pp. 6–7

5 Training with the Merits of Generative Flow

IZF trains its invertible flow with forward-pass likelihood and reverse-pass objectives for classification prototypes and seen-unseen separation. These objectives support two recognition strategies, while the bias-separation loss carries a noted risk of unrealistic synthesis when made too large.

5 Training with the Merits of Generative Flow

  • Bidirectional training adds reverse-pass objectives to forward-pass flow training because the reverse pass is used for zero-shot classification and seen-unseen generalization.The training setup uses only seen visual features, with the flow loss based on analytic log-likelihood maximization.Source: §5 · p. 7, §5.1 · p. 7

5.1 Learning to Decode by Encoding

  • The forward-pass likelihood objective on seen training samples also supports unconditional generation of seen data through IZF’s invertibility.The authors describe this as an explicit, simpler objective than GAN/VAE-based ZSL models use to fulfill the same functionality.Source: §5.1 · p. 7

5.2 Centralizing Classification Prototypes

  • The centralizing loss trains reverse-pass class prototypes to lie near the visual means of their corresponding seen classes for naive Bayesian classification.The prototypes are generated by applying the reverse pass to the class semantic embedding paired with a zero non-semantic component.Source: §5.2 · p. 7

5.3 Measuring the Seen-Unseen Bias

  • LiMMD increases the discrepancy between synthesized unseen and real seen distributions to address bias, using a negative two-sample MMD measurement.The objective is inductive: it uses seen visual samples and synthesized unseen samples, while unseen visual samples remain unavailable.Source: §5.3 · p. 8
  • Large LiMMD values may confuse some generative models into producing unrealistic samples in order to favor the loss value.The authors identify this as a possible side effect of increasing the seen-unseen discrepancy.Source: §5.3 · p. 9

5.4 Overall Objective and Training

  • The combined objective balances its loss terms with three hyperparameters and is differentiable for direct optimization with stochastic gradient descent.The section introduces the overall IZF learning objective before describing this optimization.Source: §5.4 · p. 9

5.5 Zero-Shot Recognition with IZF

  • IZF supports naive Bayesian classification by comparing test samples with reverse-generated prototypes, or a held-out classifier trained on generated unseen data.The held-out classifier is a single-layer fully connected network trained with softmax cross-entropy; test labels span unseen classes in CZSL and seen plus unseen classes in GZSL.Source: §5.5 · p. 9

6 Experiments

Experiments test IZF on toy and real-data ZSL tasks, compare it with existing methods, and examine its components, hyperparameters, and unseen-class behavior. The reported results favor IZF overall, while analyses identify conditions that can impair its performance.

6.1 Implementation Details

  • The implementation uses FrEIA invertible networks, coupling layers with fully connected subnetworks, and Adam with a 5 × 10^-4 learning rate and batch size 256.The reported comparison settings use λ1 = 2, λ2 = 1, and λ3 = 0.1.Source: §6.1 · p. 9

6.2 Toy Experiments: Illustrative Analysis

  • In a four-class simulation, IZF synthesizes unseen samples reflecting their relationships to seen classes, while GAN+LiMMD training fails and CVAE+LiMMD yields poor-quality samples.Without LiMMD, IZF's unseen samples cluster near seen classes; an excessively large LiMMD weight produces unreasonable samples, while positive MMD increases bias.Source: §6.2 · pp. 10–11, Figure 4 · p. 10

6.4 Comparison with the State-of-the-Arts

  • IZF-Softmax generally outperforms compared methods in GZSL, with balanced seen/unseen accuracy and significantly higher unseen accuracy; IZF-NBC also performs well.The authors attribute the higher unseen accuracy to LiMMD's effectiveness in addressing seen-unseen bias.Source: Table 1 · p. 11, §6.4 · p. 12

6.3 Real Data Experimental Settings

  • The real-data evaluation uses five ZSL benchmarks, ResNet-101 image features, and per-class accuracy measures, including harmonic mean for GZSL.The benchmarks include AwA1, AwA2, CUB, SUN, and aPY; experiments use the PS train-test setting.Source: §6.3 · pp. 11–12

6.5 Component Analysis

  • On AwA1, component tests find that removing regularization harms performance, positive MMD worsens unseen accuracy, and the Gaussian-kernel variant is on par with IZF.The CVAE+LC+LiMMD baseline does not work well with the distance-based classifier; the component-analysis table reports NBC and Softmax results.Source: §6.5 · p. 13, Table 3 · p. 13

6.6 Hyper-Parameters

  • A large λ1 supports conditional generation, but excessive λ3 overfits unreasonable samples and can prevent LFlow from functioning; IZF reaches its best performance in about 20 epochs.Deeper networks are less consequential for IZF-Softmax than IZF-NBC, but may produce infinite gradients.Source: §6.6 · pp. 13–14, Figure 5 · p. 13

6.7 Discriminability on Unseen Classes

  • On AwA1, IZF generates distinguishable unseen samples according to their semantic meanings despite having no direct regularization on unseen classes.The paper also reports an AwA1 CZSL confusion matrix and a t-SNE visualization of synthesized unseen samples.Source: §6.7 · p. 14, Figure 6 · p. 14

7 Conclusion

The conclusion presents IZF as a flow-based ZSL model that maps between visual and semantic spaces and addresses seen-unseen bias. It reports consistent outperformance of state-of-the-art baselines on benchmark datasets.

7 Conclusion

  • IZF uses an invertible flow for bidirectional mapping between visual and semantic spaces, with semantic information factored out during the forward pass.The model regularizes reverse-pass classification prototypes and uses kernel-based discrepancy between generated unseen and real seen data to address bias.Source: §7 · p. 14
  • Reverse-pass classification prototypes are regularized to stabilize distance-based classification.Source: §7 · p. 14
  • IZF consistently outperformed state-of-the-art baselines on benchmark datasets.Source: §7 · p. 14
Loading 2007.04873v1…