PaperlayerView exact revision on arXiv

Source-linked AI summary

Nonparametric Variational Auto-encoders for Hierarchical Representation Learning

Prasoon Goyal, Zhiting Hu, Xiaodan Liang, Chenyu Wang, Eric Xing

arXiv:1703.07027v2cs.LGstat.ML

TL;DR

Fixed, simple priors in many VAEs may fail to represent rich semantics and complex hierarchical data structure. This paper combines VAEs with tree-structured Bayesian nonparametric priors and jointly learns the neural parameters and priors. In video experiments, the model outperforms other models on classification and retrieval and produces interpretable hierarchies.

  • Problem

    Many VAEs use a fixed simple latent prior, while unsupervised sequential data can contain complex semantic hierarchies that call for compact instance codes and a corpus-level representation.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    The model combines VAEs with tree-structured Bayesian nonparametric priors and jointly estimates neural and prior parameters using variational inference.

    Source: §1 · pp. 1–2, §4.2.1 · p. 5

  • Results

    In video experiments, the model outperforms other models on classification and retrieval, and qualitative hierarchy visualizations show rich interpretable structure.

    Source: §1 · p. 2, §7 · p. 8, §6.5 · p. 8

  • Takeaways & Limitations

    The learned coarse-to-fine video hierarchies capture interpretable structure and may support downstream tasks such as retrieval, summarization, and captioning.

    Source: §3.1 · p. 3, §6.5 · p. 8

    For evaluation, frames must be assigned to paths created during training, although standard variational inference can assign unseen frames to new paths; class labels are used only for evaluation.

    Source: §6.3 · pp. 7–8

Abstract

from arXiv · show

The recently developed variational autoencoders (VAEs) have proved to be an effective confluence of the rich representational power of neural networks with Bayesian methods. However, most work on VAEs use a rather simple prior over the latent variables such as standard normal distribution, thereby restricting its applications to relatively simple phenomena. In this work, we propose hierarchical nonparametric variational autoencoders, which combines tree-structured Bayesian nonparametric priors with VAEs, to enable infinite flexibility of the latent representation space. Both the neural parameters and Bayesian priors are learned jointly using tailored variational inference. The resulting model induces a hierarchical structure of latent semantic concepts underlying the data corpus, and infers accurate representations of data instances. We apply our model in video representation learning. Our method is able to discover highly interpretable activity hierarchies, and obtain improved clustering accuracy and generalization capacity based on the learned rich representations.

1. Introduction

VAEs often use a fixed, simple latent prior, which can lose rich data semantics; hierarchical nonparametric VAEs pair tree-structured Bayesian priors with neural representations. The model learns neural parameters and priors jointly and, in video experiments, yields interpretable activity hierarchies and stronger downstream performance than VAEs with parametric priors.

1. Introduction

  • VAEs commonly use a standard-normal prior, and mapping data to a fixed single-mode prior can yield overly simplified representations that lose rich semantics.This concern is especially relevant to unsupervised data with complex hidden structures, such as video activities with temporal dependencies and hierarchical relationships.Source: §1 · p. 1
  • Hierarchical nonparametric VAEs combine Bayesian nonparametric priors with VAEs, learning neural parameters and priors jointly for data-calibrated model capacity.The nCRP supplies potentially infinitely deep and branching trees; instances receive distributions over tree paths and corresponding latent priors.Source: §1 · pp. 1–2
  • The work presents joint learning of tree-structured Bayesian nonparametric priors and VAE neural models as a unified framework.It also claims the first VAE extension to learn latent-space priors from data and the first integration of neural networks for efficient generation and inference in Dirichlet process models.Source: §1 · p. 2
  • The video application reports superior classification and retrieval performance over VAEs with parametric priors, better test-set reconstruction generalization, and interpretable activity hierarchies.Videos are modeled as mixtures of tree paths, with frame attachments organizing activity topics from abstract to concrete.Source: §1 · p. 2

2. Related Work

Prior VAE work commonly regularizes latent codes toward simple priors, while Bayesian nonparametric methods offer flexible clustering and hierarchical structure. This work combines those approaches for flexible instance priors and corpus-level structured representations.

2. Related Work

  • Vanilla VAEs use a standard-normal prior for closed-form optimization, while adversarial autoencoders [13] allow richer priors; this work instead combines VAEs with Bayesian nonparametrics.Other VAE-graphical-model combinations target disentangled semantics or graphical-model potentials, whereas this work addresses hierarchical nonparametric inference.Source: §2 · p. 2
  • The nCRP assigns instances paths through an unbounded tree, producing hierarchical clusters from abstract high-level semantics to concrete low-level content.Figure 2 depicts a sample nCRP tree and its tree-based stick-breaking construction.Source: §2 · p. 2, Figure 2 · p. 2
  • Other Bayesian nonparametric priors include Dirichlet-process mixtures with unbounded centers and Gaussian processes, often used with neural networks for supervised tasks.This work targets unsupervised representation learning with hierarchical Dirichlet nonparametrics.Source: §2 · pp. 2–3

3. Preliminaries

The preliminaries frame unsupervised sequential representation learning as recovering compact instance codes and a coarse-to-fine corpus hierarchy. They introduce the nCRP as a tree prior over paths with unbounded depth and width.

3. Preliminaries

  • The preliminaries set the paper's focus on unsupervised hierarchical representation learning for sequential data.Table 1 consolidates notation used in the paper.Source: §3 · p. 3, Table 1 · p. 3

3.1. Problem Description

  • Given unlabeled sequences, the goal is to learn compact representations for instances and capture corpus-level structure, with coarse concepts above fine-grained children.For videos, activities such as playing basketball can sit above finer activities such as running and shooting; these hierarchies may support retrieval, summarization, and captioning.Source: §3.1 · p. 3

3.2. Nested Chinese Restaurant Process

  • The nCRP recursively applies Dirichlet processes at each tree node, defining a distribution over paths in an infinitely wide and deep tree.Its tree-based stick-breaking construction starts with the root and repeats the process over child nodes; Figure 2 illustrates this construction.Source: §3.2 · p. 3

4. Hierarchical Nonparametric Variational Autoencoders

The model replaces a fixed latent-space prior with a tree-structured nCRP prior and jointly learns it with VAE parameters. Variational inference and alternating updates support a dynamically adaptable hierarchy of latent codes.

4.2. Parameter Learning

  • The nCRP prior and VAE neural parameters are jointly optimized through alternating neural-network and variational-inference updates.Neural updates optimize the encoder and decoder, while nCRP updates refine the prior parameters.Source: §4 · p. 4, §4.2 · pp. 4–5, §4.2.3 · p. 6

4.1. Generative Model

  • Each sequence draws a distribution over tree paths, and each data element samples a path whose node parameter generates its latent code.The tree has infinite depth and branching, with each node parameter depending on its parent.Source: §4.1 · p. 4

4.2.1 Variational Inference

  • The variational family assigns Gaussian node parameters, Beta edge variables, and multinomial path assignments, with inference restricted to a nested truncated tree.The nested truncation preserves configurations represented by smaller trees.Source: §4.2.1 · p. 5
  • Leaf-node means are weighted averages of associated latent codes, while internal-node means average child and parent nodes, with larger child datasets exerting greater implicit influence.Leaf-node variance decreases when more points are associated with that node.Source: §4.2.1 · p. 5

4.2.2 Neural Network Parameter Updates

  • Encoder and decoder parameters are updated by gradient steps using backpropagation and a learning rate.The encoder parameters are φ, and θNN denotes the decoder-network parameters.Source: §4.2.2 · pp. 5–6

4.2.3 Joint Training

  • Alternating updates let increasingly accurate latent codes inform the hierarchy, while the improving hierarchy guides more semantically meaningful codes.The procedure alternates several neural-network steps with several nCRP-parameter steps.Source: §4.2.3 · p. 6

4.3. Dynamically Adapting the Tree Structure

  • The tree grows by splitting a leaf whose weighted radius exceeds R into K children, and prunes leaves whose data fraction falls below F.R, K, and F are set using the validation set; unary internal nodes are also replaced by their sole child.Source: §4.3 · p. 6

5. Video Hierarchical Representation Learning

The section applies the model to unlabeled video sequences to build hierarchies from fine-grained to coarse-grained activities. It describes frame-feature inputs and a procedure for assigning trained latent codes to hierarchy paths.

5. Video Hierarchical Representation Learning

  • The target hierarchy places fine-grained activities at leaves and increasingly coarse-grained activities higher in the tree.The section illustrates this with sports, specific sports, and activities such as diving or backstroke.Source: §5 · p. 6
  • Videos are represented as sampled frame sequences, with each frame passed through a pretrained CNN to obtain sequence-element features.The framework can also use raw frames or sequences of subshots.Source: §5 · p. 6
  • After joint training, each frame feature is encoded and assigned to the path whose node parameter αp is closest to its latent code.Applying this to all frames associates each hierarchy node with representative frames of its activity.Source: §5 · p. 6

6. Experiments

Experiments evaluate reconstruction, classification, retrieval, and learned hierarchies on TRECVID MED 2011. The model achieves higher test-set log likelihood, the highest overall classification accuracy and retrieval F-1, and produces interpretable activity hierarchies.

6. Experiments

  • Because no data are labeled at both coarse and fine activity levels, quantitative evaluation uses classification and retrieval, while hierarchy interpretability is assessed qualitatively.The hierarchy is constructed unsupervised; class labels are used only for evaluation.Source: §6 · p. 6, §6.3 · p. 8

6.1. Experimental Settings

  • The evaluation uses labeled MED 2011 videos, with 1,241 for training, 138 for validation, and 1,169 for testing.Frames are sampled every five seconds and represented by 4096-dimensional VGG-16 features; the encoder maps inputs to 48-dimensional latent parameters.Source: §6.1 · pp. 6–7, Figure 4 · p. 7

6.2. Test Set Reconstruction

  • VAE-nCRP obtains higher test-set log likelihood than VAE-StdNormal.The comparison reports the sum of log likelihood over all test frames, averaged across three independent runs.Source: §6.2 · p. 7, Table 2 · p. 7

6.3. Video Classification

  • VAE-nCRP achieves the highest overall classification accuracy and outperforms the baselines on 8 of 15 classes.Accuracy is averaged over three independent runs.Source: Table 3 · p. 7, §6.3 · p. 8

6.4. Video Retrieval

  • VAE-nCRP achieves the highest overall video-retrieval F-1 score and outperforms the baselines on 8 of 15 classes.F-1 incorporates false-positive and false-negative rates.Source: §6.4 · p. 8, Table 4 · p. 8

6.5. Qualitative Analysis

  • The learned hierarchy refines vehicle-related frames into cross-country and repair activities, then separates repair into bike and car-wheel repair.The authors present this as evidence of meaningful, interpretable hierarchical patterns in the video data.Source: §6.5 · p. 8, Figure 5 · p. 8

7. Conclusions

The paper combines an nCRP prior with VAEs and jointly optimizes neural and nCRP parameters to learn hierarchical latent representations. On video data, it reports better performance on classification and retrieval than other models, alongside interpretable learned structure.

7. Conclusions

  • The framework combines an nCRP prior with VAEs and jointly optimizes variational updates for neural and nCRP parameters.The resulting hierarchy places more abstract concepts higher and less abstract concepts lower.Source: §7 · p. 8
  • On video data, the model outperforms other models on video classification and video retrieval.Source: §7 · p. 8
  • Visualized learned hierarchies capture rich, interpretable structure in the data.Source: §7 · p. 8

Related papers

Loading 1703.07027v2…