PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian Inference with Posterior Regularization and applications to Infinite Latent SVMs

Jun Zhu, Ning Chen, Eric P. Xing

arXiv:1210.1766v3cs.LGcs.AIstat.MEstat.ML

TL;DR

Existing Bayesian methods often encode domain knowledge through priors, while direct regularization of post-data posteriors offers a more flexible alternative. The paper develops RegBayes, characterizes it with convex analysis, and instantiates it in iLSVM and MT-iLSVM; empirical results appear to combine advantages of Bayesian nonparametrics and large-margin learning.

  • Problem

    Existing Bayesian and nonparametric Bayesian models primarily incorporate domain knowledge through priors, motivating a more direct and flexible way to regularize post-data posterior distributions.

  • Method

    RegBayes formulates posterior inference with regularization or constraints on post-data distributions, provides a convex-analysis representation theorem, and instantiates large-margin iLSVM and MT-iLSVM models with nonparametric priors.

  • Results

    Empirical studies on several real datasets appear to show that the proposed models inherit merits from both Bayesian nonparametrics and large-margin learning.

  • Takeaways & Limitations

    The paper supplies a framework connecting posterior regularization with Bayesian nonparametrics and large-margin learning while automatically resolving latent dimensionality from data.

  • Takeaways & Limitations

    RegBayes may require additional assumptions for tractability, which can make the feasible space non-convex and may lead to inconsistent posteriors.

Abstract

from arXiv · show

Existing Bayesian models, especially nonparametric Bayesian methods, rely on specially conceived priors to incorporate domain knowledge for discovering improved latent representations. While priors can affect posterior distributions through Bayes' rule, imposing posterior regularization is arguably more direct and in some cases more natural and general. In this paper, we present regularized Bayesian inference (RegBayes), a novel computational framework that performs posterior inference with a regularization term on the desired post-data posterior distribution under an information theoretical formulation. RegBayes is more flexible than the procedure that elicits expert knowledge via priors, and it covers both directed Bayesian networks and undirected Markov networks whose Bayesian formulation results in hybrid chain graph models. When the regularization is induced from a linear operator on the posterior distributions, such as the expectation operator, we present a general convex-analysis theorem to characterize the solution of RegBayes. Furthermore, we present two concrete examples of RegBayes, infinite latent support vector machines (iLSVM) and multi-task infinite latent support vector machines (MT-iLSVM), which explore the large-margin idea in combination with a nonparametric Bayesian model for discovering predictive latent features for classification and multi-task learning, respectively. We present efficient inference methods and report empirical studies on several benchmark datasets, which appear to demonstrate the merits inherited from both large-margin learning and Bayesian nonparametrics. Such results were not available until now, and contribute to push forward the interface between these two important subfields, which have been largely treated as isolated in the community.

1. Introduction

The paper introduces RegBayes as a flexible framework for directly regularizing post-data posterior distributions, complementing prior-based domain knowledge. It instantiates this framework in iLSVM and MT-iLSVM, combining Bayesian nonparametrics with large-margin learning for classification and multi-task learning.

  • Motivation: Nonparametric Bayesian models typically encode domain knowledge through priors that indirectly influence posterior distributions via Bayes’ rule.These models allow complexity to grow with observed data, including through Gaussian, Dirichlet, and Beta processes.
  • RegBayes: RegBayes directly regularizes the desired post-data posterior through an information-theoretical optimization framework.Its regularization can be expressed using posterior constraints and convex penalty functions.
  • RegBayes: RegBayes applies to both directed Bayesian networks and undirected Markov networks, with the latter yielding hybrid chain graphical models.The framework supports both parametric and nonparametric Bayesian inference.
  • RegBayes: The framework provides extra flexibility for post-data posterior inference and integrates Bayesian nonparametrics with large-margin learning.The paper presents this integration as a way to combine complementary advantages of the two approaches.
  • Applications: iLSVM and MT-iLSVM use large-margin posterior regularization with IBP-based priors to learn unbounded latent features for classification and multi-task learning.Their regularized inference problems can be solved with an iterative procedure using existing convex optimization techniques.

2. Related Work

Related work includes expectation-based posterior regularization, convex-duality analyses of relaxed constraints, and large-margin Bayesian models. The paper positions RegBayes as a broader formalism extending these lines to parametric and nonparametric Bayesian inference.

  • Expectation constraints: Expectation regularization has been used in semi-supervised and weakly labeled learning to regularize discriminative model distributions with side information.Generalized expectation criteria commonly penalize discrepancies between empirical and model expectations.
  • Convex duality: Prior convex-duality work studied generalized maximum entropy and expectation constraints, but some analyses were limited to KL-divergence or finite-dimensional observation spaces.These results provide related theoretical foundations for posterior-constraint formulations.
  • Large-margin Bayesian models: Large-margin posterior regularization generalizes maximum entropy discrimination and extends earlier max-margin nonparametric Bayesian models.The paper’s iLSVM and MT-iLSVM extend infinite SVM ideas from latent classes to infinite latent feature models.

3. Regularized Bayesian Inference

Regularized Bayesian inference (RegBayes) reformulates Bayes’ rule as convex optimization over post-data distributions, then adds knowledge- or data-driven posterior constraints. Its representation theorem characterizes solutions under convex expectation regularization, while the framework applies to directed and undirected Bayesian models and can be more flexible than standard Bayesian conditionalization.

  • Variational formulation of Bayes’ theorem: Bayes’ rule is equivalent to minimizing the KL divergence between a candidate post-data distribution and the Bayesian posterior.The candidate distribution is constrained to be a valid density, and the variational objective differs from KL divergence to the posterior only by the constant log p(D).
  • Regularized Bayesian inference: RegBayes generalizes this variational problem by imposing additional knowledge- or data-driven constraints on the post-data distribution.The additional constraints replace the standard normality-only formulation and can be expressed through a regularization term in the master equation.
  • Expectation constraints: Hard constraints correspond to indicator penalties and produce one feasible subspace, whereas convex soft penalties allow multiple feasible subspaces with different complexity costs.The penalty U determines the cost of soft constraints; in classification models it can correspond to a surrogate loss such as hinge loss.
  • Scope and model classes: The framework covers directed Bayesian networks, undirected Markov networks, and empirical Bayesian inference, although undirected models require handling a more challenging hybrid chain graph.The challenge for undirected models arises from normalization factors in the chain graph produced by Bayesian inference.
  • Representation theorem: For convex lower-semicontinuous regularization induced by an expectation operator, the RegBayes posterior has an exponential-tilting form with coefficients obtained from a dual optimization problem.The solution introduces an extra factor exp(⟨φ, ψ(M;D)⟩−Λφ) relative to the Bayesian form, where Λφ is the log-partition function.
  • Interpretation: RegBayes can be more flexible than standard Bayesian inference because its posterior may not be obtainable through Bayesian conditionalization with any prior and likelihood.When the relevant joint measure is finite, an implicit prior and likelihood can exist; otherwise, no such representation is available.
  • Limitations: Practical tractability may require additional assumptions that make the feasible space non-convex, so the convex-analysis guarantees require caution.Mean-field assumptions are identified as one source of a non-convex feasible space.

4. Infinite Latent Support Vector Machines

The paper combines IBP-based nonparametric latent feature models with large-margin classification, using RegBayes to infer predictive latent representations for single- and multi-task learning. iLSVM uses posterior expectations over latent features and weights, while inference relies on tractable approximations and finite truncation.

  • iLSVM and MT-iLSVM combine IBP-based infinite latent feature models with large-margin classifiers for classification and multi-task learning.The models define latent feature dimensions nonparametrically rather than fixing them in advance.
  • The IBP prior permits an unbounded number of latent features while yielding finitely many active features for finite datasets.The prior gives zero mass to matrices with infinitely many nonzero entries, although the number of nonzero columns remains unbounded.
  • The effective discriminant function averages the latent discriminant function over the posterior distributions of latent features and feature weights.Variables absent from the feature map, such as likelihood variables W, are marginalized in the expectation.
  • The classification penalty supports hinge loss when κ is 1 and squared ℓ2-loss when κ is 2.The cost function measures the cost of predicting an example as an incorrect label, while the margin compares the true label against alternatives.
  • Training minimizes KL divergence to the joint Bayesian model while penalizing soft large-margin constraint violations.The joint model combines an IBP prior for Z with Gaussian priors for η and W and a linear-Gaussian likelihood.
  • Inference can use truncated mean-field constraints, while alternative approximate methods include MCMC-based posterior inference with iterative dual-parameter estimation.Mean-field factorization breaks dependencies in the desired posterior and introduces additional constraints; truncation bounds the possible feature count by K.

5. Experiments

Experiments evaluate iLSVM and MT-iLSVM on classification and multi-task datasets. The results indicate that joint large-margin latent-feature learning performs competitively or better than decoupled alternatives while learning a suitable latent dimensionality.

  • Classification: iLSVM is evaluated on TRECVID2003 and Flickr image datasets against MMH, EFH+SVM, and IBP+SVM.The datasets contain five video categories and thirteen animal-image categories, respectively.
  • Classification: iLSVM achieves comparable performance with nearly optimal MMH without pre-specifying latent-feature dimensionality, and outperforms IBP+SVM and EFH+SVM.Among decoupled methods, IBP+SVM performs worse than EFH+SVM on TRECVID but better on Flickr.
  • Latent features: About 45 latent features are active on average for TRECVID, while larger cross-class standard deviations identify more discriminative features.Features 26 and 34 are described as less discriminative than many others.
  • Latent features: On Flickr, feature activity decreases as feature index increases, and many discovered features are semantically interpretable.Examples include squirrel, whale, and hawk features; two whale features capture different backgrounds.
  • Multi-task learning: MT-iLSVM is evaluated on Scene, Yeast, and School multi-task datasets using classification performance or explained variance.Scene and Yeast treat each label assignment as a binary task, while School predicts exam scores across schools.
  • Multi-task learning: MT-iLSVM outperforms tested existing methods and the decoupled MT-IBP+SVM, while concatenating original inputs provides only a slight additional boost for MT-iLSVM.MT-iLSVM needs about 50 latent features for sufficiently good and robust performance.
  • Sensitivity analysis: MT-iLSVM is insensitive to α and C on Yeast, stable for C between 0.3 and 1 on School, and reaches state-of-the-art performance with about 70% of School training data.Performance and running time generally increase with training size, and mean-field inference is described as efficient.

6. Conclusions and Discussions

The paper introduces RegBayes and develops large-margin nonparametric Bayesian models for classification and multi-task learning. Experiments and discussion position the framework as flexible, while noting that harder extensions remain open.

  • RegBayes performs post-data posterior inference with rich regularization or constraints on desired posterior distributions.
  • The proposed large-margin nonparametric Bayesian models learn predictive latent features while automatically resolving latent dimensionality from data.
  • The empirical results on several real datasets appear to combine merits from Bayesian nonparametrics and large-margin learning.
  • Future work includes broader posterior constraints, additional nonparametric Bayesian applications, and systematic investigation of undirected-model inference.

Appendix A: Generalization Beyond Bayesian Networks

The appendix extends Bayesian inference beyond standard Bayesian networks to unknown parameters and chain graphs. It shows how regularized inference can be formulated using variational objectives and convex optimization, although joint optimization may remain nonconvex.

  • Generalization beyond Bayesian networks: The generalized formulation covers directed, undirected, and hybrid chain-graph latent-variable models.
  • Unknown parameters: Empirical Bayesian inference incorporates unknown model parameters through constrained or unconstrained regularized formulations.
  • Unknown parameters: For fixed parameters, the objective is convex in q(M), but joint optimization is generally nonconvex and can be addressed with EM toward a local optimum.
  • Bayesian special case: Without posterior regularization, the optimal q(M) equals the Bayesian posterior and the optimal parameters are the maximum-likelihood estimate.
  • Chain graphs: The generalized objective can be expressed through KL divergence involving the joint distribution, including undirected MRF models and hybrid chain graphs.

Appendix B: MedLDA—A RegBayes Model with Finite Latent Features

MedLDA is interpreted as a RegBayes model that combines a finite topic latent space with large-margin posterior constraints. Variational inference and convex duality provide practical routes for approximate optimization.

  • MedLDA as RegBayes: MedLDA is a max-margin supervised topic model extending LDA for supervised learning.
  • Model structure: Its latent representation uses topics as features, with model parameters and document-level topic assignments defining the posterior.
  • Posterior constraints: MedLDA imposes expectation-based posterior constraints derived from the large-margin principle and training prediction quality.
  • Posterior constraints: The constrained objective is equivalent to minimizing an ε-insensitive loss.
  • Approximate inference: Variational MedLDA replaces the true posterior with an auxiliary distribution and the negative log-likelihood with an upper bound, with optional mean-field assumptions for tractability.
  • Approximate inference: Convex conjugates and duality support approximate algorithms for handling the large-margin constraints.

Appendix C.1: Proof of Theorem 6

The proof establishes the dual characterization of RegBayes when regularization is induced by an expectation operator. It uses KL-divergence structure and Fenchel duality to show equality of primal and dual optima.

  • The expectation operator’s adjoint is used to connect posterior expectations with the dual representation.
  • The primal KL objective differs from KL(q(M)∥p(M|D)) only by the constant −log p(D).
  • The convex conjugate of the KL divergence and absorption of the normalization constant yield the stated dual objective.
  • Fenchel weak duality gives d ≤ t, while appropriate regularity conditions yield strong duality with t = d.
  • The dual optimum is attained, and deriving the infimum produces the optimal posterior distribution q.

Appendix C.2: Proof of Lemma 9

The proof evaluates 0(µ) by separating the case µ < 0 and then combines the resulting expressions to establish the claim.

  • The proof begins from the definition 0(µ) = supx∈R(xµ − C max(0, x)) and considers µ < 0.
  • The cases are combined to prove the stated claim.

Appendix C.3: Proof of Lemma 10

The proof establishes Lemma 10 by analyzing the conjugate through separate sign cases for the parameters and a structured subset G1.

  • The proof first shows that all µi must satisfy µi ≥ 0 for g∗ to be finite.
  • When some µj < 0, the relevant conjugate value is infinite.
  • The proof combines these results to establish the claim.

Appendix C.4: Proof of Lemma 11

The proof derives the conjugate by evaluating two infima under the constraint |µ| ≤ C, with positivity of ǫ used in one equality.

  • The conjugate is obtained from its definition by analyzing separate first and second infimum terms.
  • The derivation uses α = |µ| when ǫ is positive, while |µ| ≤ C follows from α + β = C and β ≥ 0.

Appendix C.5: Proof of Lemma 12

The appendix derives inference procedures for iLSVM and MT-iLSVM using variational bounds, convex duality, and SVM solvers. The procedures iteratively update latent-variable distributions and solve dual problems until convergence criteria are met.

  • Algorithm derivations: The inference algorithms for MT-iLSVM and iLSVM are outlined in Algorithms 2 and 3.
  • Variational bounds: Variational parameters are optimized on simplex constraints to obtain tight lower bounds, with normalization factors ensuring valid distributions.
  • Variational bounds: The KL-divergence is decomposed across latent variables, and an upper bound is used within the iterative inference procedure.
  • Distribution updates: The procedures iteratively infer q(ν), q(Z), q(W), and q(η), while updating model-specific variational quantities.
  • Convergence: The algorithms stop when the relative change in the objective falls below a threshold or a maximum iteration count is reached.
  • Dual optimization: Convex duality produces dual problems that are solved efficiently with binary or multi-class SVM learners.
  • Testing: For testing data, the large-margin constraint term is omitted because such constraints are absent.
Loading 1210.1766v3…