PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Exponential Families

Rajesh Ranganath, Linpeng Tang, Laurent Charlin, David M. Blei

arXiv:1411.2581v1stat.MLcs.LG

TL;DR

The paper develops deep exponential families (DEFs), hierarchical latent-variable models with generic variational inference, and applies them to text and pairwise recommendation data. DEFs improve predictive performance over shallow and baseline models while producing interpretable hierarchical structures.

  • Problem

    Existing latent-variable models need flexible ways to represent hierarchical dependencies and support applications across text and pairwise data.

  • Method

    The paper develops DEFs whose latent layers are linked through exponential-family distributions, uses generic variational inference, and combines two DEFs for pairwise observations.

  • Results

    DEFs outperform baselines on text and collaborative filtering, while deeper models generally improve predictive performance and reveal interpretable semantic structures.

  • Takeaways & Limitations

    DEFs provide a flexible framework for modeling hierarchical latent relationships across text and collaborative filtering with improved prediction and exploratory structure.

  • Takeaways & Limitations

    Text evaluation uses a held-out perplexity metric rather than predictive-distribution perplexity, which requires conditioning on only part of each document and costly sampling.

Abstract

from arXiv · show

We describe \textit{deep exponential families} (DEFs), a class of latent variable models that are inspired by the hidden structures used in deep neural networks. DEFs capture a hierarchy of dependencies between latent variables, and are easily generalized to many settings through exponential families. We perform inference using recent "black box" variational inference techniques. We then evaluate various DEFs on text and combine multiple DEFs into a model for pairwise recommendation data. In an extensive study, we show that going beyond one layer improves predictions for DEFs. We demonstrate that DEFs find interesting exploratory structure in large data sets, and give better predictive performance than state-of-the-art models.

1 Introduction

Deep exponential families (DEFs) model observations through cascades of latent variables, combining hierarchical representations with exponential-family flexibility. The paper develops these models for text and pairwise recommendation data, reporting richer exploratory structure and improved predictive performance.

  • Model motivation: DEFs generate observations through cascades of latent variables, with each layer governed by the previous layer’s variables and shared weights.The hierarchy is designed to reflect deep unsupervised feature learning while remaining probabilistic.
  • Document modeling: For documents, latent topic activations explain term counts, while higher-level super-topics organize related topics through shared weights.This creates compositional, coarse-to-fine representations of document content.
  • Exploratory structure: A three-level DEF trained on 166K New York Times articles discovered government-related structure whose super-topics corresponded to the judiciary, legislative, and executive branches.The model organized topics into interpretable higher-level concepts such as Government and Politics.
  • Generality: DEFs support multiple latent-variable families and observation models, including Bernoulli, Gamma, Gaussian, real-valued, binary, and multinomial settings.The choice of weight prior corresponds to regularization, while the observation model corresponds to a loss type.
  • Pairwise recommendation: The double DEF combines separate user and item DEFs, with each pairwise observation determined by the lowest-level representations of both.This embeds hierarchical latent representations into a factorization model for recommendation data.
  • Reported outcomes: Across text and collaborative filtering, the study reports that deeper DEF variants provide better predictive performance and more interesting exploratory structure than shallow counterparts.The paper positions DEFs as flexible models for exploring many solutions to modern data-analysis problems.

2 Deep exponential families

A DEF chains exponential-family distributions into a hierarchy in which higher-layer draws govern the natural parameters of lower layers. Its link functions, weights, and observation likelihood determine how latent variables generate data.

  • Exponential-family foundation: Exponential families are specified through sufficient statistics, a base measure, natural parameters, and a log-normalizer.The expected sufficient statistics equal the gradient of the log-normalizer.
  • Hierarchical construction: DEFs chain exponential families so that a draw from one layer controls the natural parameters of the next.This construction creates a hierarchy of latent variables with potentially different exponential-family types across layers.
  • Latent-variable structure: Each data point has L hidden layers, while shared weight layers connect variables across adjacent levels and encode dependencies among all variables in the higher layer.The hierarchy can be interpreted as a stochastic feed-forward structure or a random-effects model.
  • Link functions: The link function maps inner products to natural parameters, and the resulting log-normalizer transformation introduces nonlinearity into expected sufficient statistics.This provides a connection between DEFs and neural-network activation mechanisms.
  • Observation model: The likelihood is separated from the DEF so the latent hierarchy can be composed with other models; for count data, the paper uses a Poisson observation likelihood.The same separation supports embedding DEFs in pairwise-data models.
  • Illustration: Figure 1 visualizes hierarchical groupings in a three-layer topic model, with top words displayed for each topic and arrows marking the groupings.The example uses 166K New York Times articles.
  • Document example: In the document example, Poisson observations represent term counts, while successive weight layers correspond to topics, super-topics, and concepts.The weights place positive mass on groups of terms and give the hierarchy compositional and sharing semantics.

3 Examples

DEF examples instantiate the framework with different latent-variable families and link functions, including sigmoid belief networks, Poisson models, and sparse gamma models. These choices support binary, count, and positive-valued representations with distinct modeling behavior.

  • Examples: DEFs can use Bernoulli, Poisson, or gamma latent layers, yielding sigmoid belief networks, latent Poisson models, and sparse gamma models.The paper presents these as three principal examples of the framework.
  • Sparse gamma DEF: Sparse gamma DEFs use gamma-distributed layers with fixed shape parameters, while layer components control expected activations through the link function.Gamma weights preserve positivity, matching the positive expectation of gamma variables.
  • Sparse gamma DEF: When the gamma shape is below 1, sparse gamma distributions concentrate mass near zero and behave like soft spike-and-slab priors for feature selection and discovery.Compared with Poisson draws, sparse gamma draws tend toward either near-zero or very large values when the expected value is high.
  • Sigmoid belief network: A Bernoulli DEF with identity links recovers the sigmoid belief network, whose latent-feature means are determined by higher-layer features and weights through the logistic function.The model uses real-valued weights with a factorized normal prior and may include intercepts.
  • Poisson DEF: A Poisson DEF uses latent Poisson levels and a log link, modeling count-valued observations and representing higher-level activations as counts such as super-topic occurrences.With positive weights, the mean activation is a linear combination of the preceding layer’s weights; Gaussian weights and a log-softmax link allow negative relations.

4 Related Work

DEFs extend directed stochastic feed-forward latent-variable models and connect to several existing probabilistic architectures. Their untied weights and hierarchical structure distinguish them from narrower or shallower related models.

  • Stochastic feed-forward models: DEFs belong to stochastic feed-forward belief networks, alongside latent Gaussian models and sigmoid belief networks with layerwise dependencies.Earlier work focused mainly on specific members of this broader class.
  • Directed versus undirected models: Unlike undirected deep probabilistic models such as RBMs, directed DEFs support explaining away, making latent variables dependent after conditioning on observations.RBMs are described as two-layer undirected models connecting one latent layer to observations.
  • Exponential-family models: DEFs form a broader class than exponential family harmoniums and RBMs because their weights are not tied.An infinite DEF with tied weights is equivalent to an exponential family harmonium.
  • Hierarchical latent-variable models: Single-layer DEFs encompass models such as exponential family PCA and multinomial PCA, while latent-tree work uses more specific distributions such as the Dirichlet.This places DEFs within the broader literature on hierarchical latent-variable models and Bayesian factor analysis.

5 Inference

Posterior inference in DEFs is difficult because exact computations are intractable, so the paper optimizes a variational approximation. Black box variational inference supplies stochastic gradients applicable to general DEFs.

  • Inference challenge: The central computational challenge is posterior inference, because the intractable partition function makes exact posterior computations unavailable.The paper develops general variational methods instead of relying on greedy layer-wise learning for a specific network.
  • Variational objective: Variational inference minimizes divergence to the posterior by optimizing an evidence lower bound on log p(x).The approximation uses latent variables associated with observations and shared latent weights.
  • Variational family: The mean-field variational family factorizes latent variables and uses exponential-family approximations matching the model families for latent variables and weights.For the Poisson example, both q(z) and q(W) are fully factorized.
  • Black box inference: General DEF expectations lack simple analytic forms, so black box variational inference uses stochastic optimization and noisy unbiased gradients.This avoids analytically computing the objective function and its expectations.
  • Gradient computation: Gradient estimation uses Monte Carlo samples to evaluate likelihood, score-function, and Markov-blanket terms, with likelihood computation parallelized across samples.The primary computational cost is evaluating the likelihood and score function on sampled variables.
  • Optimization: RMSProp scales noisy gradients using an online average of squared gradients to accommodate varying length scales and noise during stochastic optimization.The paper presents this optimizer in its algorithm sketch and appendix.

6 Experiments

The experiments evaluate DEF variants across text modeling and pairwise recommendation, varying depth, latent distributions, weight priors, and model composition. DEFs generally outperform baselines, with deeper and sparse gamma models showing the strongest overall performance.

  • Experimental design: The evaluation reports 28 DEF instances varying one to three layers, latent distributions, weight distributions, and a two-DEF model for pairwise data.The study uses a Poisson observational model for the single-DEF experiments.
  • Evaluation: Held-out perplexity is a computationally efficient evaluation metric but differs from perplexity computed from the predictive distribution p(x*| x).The predictive-distribution metric can require demanding sampling procedures, limiting evaluation to small test sets for some models.
  • Text modeling: DEFs outperform LDA and DocNADE on both Science and New York Times text datasets.The text evaluation uses held-out perplexity on 1,000 documents, with 10% of each document observed and 90% held out.
  • Text modeling: Moving beyond one layer generally improves text-modeling performance, while gamma latent variables produce similar or better results when stacked.Gamma-distributed weights also outperform normally distributed weights, whereas deeper sigmoid DEFs are more difficult to train and perform poorly.
  • Matrix factorization: The double DEF composes two DEFs to model pairwise observations, assigning hierarchical latent representations to both users and items.The experiments cover 50K Netflix users and 17.7K movies, plus 18K ArXiv users and 20K documents.
  • Matrix factorization: Two-layer double DEFs improve over shallow DEFs, and all DEFs outperform Gaussian matrix factorization on Netflix and ArXiv recommendation data.The advantage of deeper models is especially clear in ranking performance for low-activity users.

7 Discussion

The paper presents deep exponential families as hierarchical latent-variable models for compositional semantics and reports improved predictive power and interpretable semantic structures in text modeling and collaborative filtering.

  • Discussion: DEFs describe hierarchical latent-variable relationships to capture compositional semantics in data.The paper applies them to text modeling and collaborative filtering.
  • Discussion: DEF instantiations achieve improved predictive power and interpretable semantic structures in text modeling and collaborative filtering.The conclusion summarizes the reported benefits across both application areas.

Appendix

The appendix specifies BBVI for DEFs, including sampling, score-function gradients, parameter updates, variational families, and experimental convergence settings.

  • Algorithm 2 BBVI for DEFs: BBVI initializes variational parameters, samples a data point and parallel latent-variable realizations, then evaluates model and variational log probabilities.The algorithm samples latent variables and weights layer by layer before computing gradient terms.
  • Algorithm 2 BBVI for DEFs: The algorithm updates layer-specific variational parameters using mean Monte Carlo gradient estimates until validation-likelihood change falls below 0.01.Separate updates are given for latent-variable parameters and weight parameters.
  • Properties of q: The variational approximations use Poisson, gamma, Bernoulli, and normal families, with score functions detailed for these distributions.Gamma parameters use shape and scale, while Bernoulli approximations use a natural parameterization.
  • Parameterizations of Variational Distributions: Positive constraints are enforced by transforming unconstrained variables with log(1 + exp(x)), with truncation used when necessary for numerical stability.Gradients of transformed parameters are obtained through the chain rule.
  • Optimization: Optimization uses gradient ascent on the ELBO with noisy BBVI gradients and RMSProp-style diagonal preconditioning.The fixed step scalar is 0.2, and the running-gradient window size is 10.
  • Hyperparameters and Convergence: Experiments use 10,000 iterations, with validation likelihood reported as stable or converged by that point.The stated priors include Gamma shape and rate 0.3, a Poisson prior rate of 0.1, and a sigmoid belief-network prior of 0.1.
Loading 1411.2581v1…