PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiplexNet: Towards Fully Satisfied Logical Constraints in Neural Networks

Nicholas Hoernle, Rafael Michael Karampatsis, Vaishak Belle, Kobi Gal

arXiv:2111.01564v1cs.LGcs.SC

TL;DR

Neural networks need ways to use expert knowledge without domain-specific engineering, while loss-based constraints do not guarantee satisfaction. MultiplexNet encodes knowledge as DNF output transformations selected by a latent Categorical variable, and experiments report efficient learning with guaranteed constraint compliance across density estimation and classification tasks.

  • Problem

    Neural-network training needs to incorporate expert constraints without specialized engineering while guaranteeing that outputs satisfy them, particularly in safety-critical settings.

  • Method

    MultiplexNet represents domain knowledge as DNF, adds one output transformation per term, and uses a latent Categorical variable to select the transformation optimizing the data loss.

  • Results

    Across density estimation and supervised or unsupervised classification experiments, MultiplexNet learned effectively with fewer samples in some cases and guaranteed 100% constraint satisfaction.

  • Takeaways & Limitations

    The approach provides a general way to restrict neural-network outputs to expert-specified domains while using domain knowledge to support data-efficient training.

  • Takeaways & Limitations

    The method requires experts to express valid constraints precisely and requires DNF formulas, which can have exponentially large representations.

Abstract

from arXiv · show

We propose a novel way to incorporate expert knowledge into the training of deep neural networks. Many approaches encode domain constraints directly into the network architecture, requiring non-trivial or domain-specific engineering. In contrast, our approach, called MultiplexNet, represents domain knowledge as a logical formula in disjunctive normal form (DNF) which is easy to encode and to elicit from human experts. It introduces a Categorical latent variable that learns to choose which constraint term optimizes the error function of the network and it compiles the constraints directly into the output of existing learning algorithms. We demonstrate the efficacy of this approach empirically on several classical deep learning tasks, such as density estimation and classification in both supervised and unsupervised settings where prior knowledge about the domains was expressed as logical constraints. Our results show that the MultiplexNet approach learned to approximate unknown distributions well, often requiring fewer data samples than the alternative approaches. In some cases, MultiplexNet finds better solutions than the baselines; or solutions that could not be achieved with the alternative approaches. Our contribution is in encoding domain knowledge in a way that facilitates inference that is shown to be both efficient and general; and critically, our approach guarantees 100% constraint satisfaction in a network's output.

Introduction

MultiplexNet addresses the challenge of integrating expert-specified constraints into neural-network training without domain-specific engineering or merely trading constraint satisfaction against data fit. It represents constraints as DNF transformations in the output layer, uses a latent Categorical selector, and targets guaranteed satisfaction alongside improved data efficiency.

  • Introduction: Loss-based approaches combine data and constraint penalties but cannot guarantee satisfaction, limiting their suitability for safety-critical domains.They may still be useful for reducing data requirements when domain knowledge structures training.
  • Introduction: MultiplexNet compiles DNF domain constraints into an existing network’s output layer and uses a latent Categorical variable to select the loss-optimizing constraint term.This design represents arbitrarily complex constraints while guaranteeing that network outputs satisfy them.
  • Introduction: MultiplexNet’s density-estimation experiment targets learning an unknown distribution from fewer samples while producing only constraint-satisfying samples.The experiment addresses both data efficiency and guaranteed constraint satisfaction.
  • Introduction: On MNIST, the paper uses structured arithmetic relations in a label-free training setting, while CIFAR100 uses super-class knowledge to improve super-class classification accuracy.The experiments cover both unsupervised structured learning and supervised image classification.
  • Introduction: On CIFAR100, the approach uses knowledge that images within a super class are related and increases classification accuracy at the super-class level.
  • Problem Specification: The problem specification assumes data from an unknown process p*(x) and a logical formula Φ describing its feasible, non-zero domain, then seeks a constrained parametric approximation p_θ(x).For safety-critical use, every sample from p_θ(x) should satisfy Φ without requiring network reparameterization engineering.

Related Work

Prior methods typically incorporate logical knowledge through additive losses, fuzzy logic, probabilistic model counting, or specialized network embeddings. MultiplexNet instead compiles constraints into outputs, guarantees satisfaction, and supports quantifier-free linear arithmetic over real-valued variables.

  • Related Work: DL2 and Semantic Loss add constraint terms to the network loss, whereas MultiplexNet compiles constraints directly into outputs so every output satisfies them.This distinction is especially important for safety-critical domains.
  • Related Work: MultiplexNet supports quantifier-free linear arithmetic over the rationals, combining real-valued variables with + and ≥ under logical operators ¬, ∨, and ∧.This is more expressive than Semantic Loss’s Boolean-variable formulation and probabilistic unlike DL2’s fuzzy-logic framework.
  • Related Work: Iterative rule knowledge distillation balances first-order-logic satisfaction with classification accuracy but cannot express rules constraining real-valued outputs.It uses a student–teacher projection scheme and soft logic for gradient estimation.
  • Related Work: Xsat solves related SMT satisfiability problems but is not differentiable, while ProbLog-based neural predicates embed networks into logic and do not clearly handle real-valued arithmetic constraints.These methods address adjacent logical-inference or integration settings rather than the same differentiable output-constraint problem.
  • Related Work: Program-synthesis methods produce programs satisfying specifications, but they do not focus on using constraints to aid neural-network training while ensuring full satisfaction.Other knowledge-guided methods likewise append loss terms or constrain latent variables rather than the generative model’s output space.
  • Related Work: Post-hoc verification checks whether a network respects predefined restrictions, whereas MultiplexNet aims to guarantee compliance during network operation.The distinction is between validating outputs after construction and constraining the output space directly.

Incorporating Domain Constraints into Model Design

MultiplexNet compiles logical domain constraints into neural-network outputs through satisfiability-preserving transformations, DNF branches, and a latent categorical selector. The resulting architecture supports constrained generative and discriminative modeling while guaranteeing that outputs satisfy the specified formula.

  • Satisfiability as Reparameterisation: For a non-negative activation g, a linear transformation f can reparameterize an unconstrained output so that f(g(˜x)) satisfies a specified inequality.More complex conjunctions are built by composing these transformations; examples include bounded intervals and relational constraints.
  • Satisfiability as Reparameterisation: Each DNF term receives a separate output transformation, allowing conjunctions and disjunctions to be compiled into the network’s output layer.Conjunctions restrict the representable space, while disjunctions create branches covering alternative satisfying terms.
  • Satisfiability as Reparameterisation: MultiplexNet supports domain knowledge expressed as Boolean formulas over linear inequalities after conversion to disjunctive normal form.The paper proposes using an off-the-shelf solver such as Z3 to provide the DNF logical input.
  • Satisfiability as Reparameterisation: Every MultiplexNet branch satisfies its assigned DNF term, so any selected output satisfies the full logical formula by construction.The stated lemma formalizes this guarantee for quantifier-free first-order formulas in DNF.
  • MultiplexNet as a Latent Variable Problem: The approach uses a latent Categorical variable k to select among constraint transformations hk associated with the DNF terms.A variational lower bound is introduced for this latent variable, and the experiments marginalize it because the categorical dimensionality is small.
  • MultiplexNet as a Latent Variable Problem: The latent-variable optimization note acknowledges that alternative estimators for the categorical variable remain worth exploring.The experiments marginalize the variable, while the paper explicitly notes that other alternatives should also be investigated.
  • MultiplexNet as a Latent Variable Problem: The method is presented as applicable to both density estimation and structured classification, covering generative and discriminative learning settings.This is intended to demonstrate applicability across distinct neural-network tasks.
  • MultiplexNet as a Latent Variable Problem: Figure 1 illustrates the latent-density setting: the model must represent an unknown distribution while restricting samples to regions defined by known constraints.The red boxes depict the domain constraints imposed on the simulated density.

Experiments

MultiplexNet integrates logical domain knowledge into density estimation, label-free structured learning, and hierarchical classification. Across these experiments, it improves data efficiency or task performance while enforcing the specified constraints through its output construction.

  • Experiments: Across three domains, MultiplexNet uses domain knowledge to improve learning with limited data, support label-free inference, and improve super-class classification while enforcing constraints.The experiments cover synthetic density estimation, unsupervised MNIST, and hierarchical CIFAR100 classification.
  • Synthetic Data: The MultiplexNet loss outperforms both baselines across all tested training-set sizes, while generated samples satisfy the constraints by construction.The comparison uses N ∈ {100, 250, 500, 1000}; the DL2-VAE baseline appends a loss term but does not guarantee constraint satisfaction.
  • MNIST - Label-free Structured Learning: The label-free MNIST model learns class-conditional representations and reaches 97.5 ± 0.3 accuracy on inferred labels in held-out data.The categorical variable selects among 100 combinations satisfying the digit-sum structure, and the reported accuracy uses the top 5 runs selected by validation ELBO.
  • Hierarchical Domain Knowledge on CIFAR100: The CIFAR100 formulation encodes semantic grouping by requiring each super-class prediction to meet a minimum probability threshold α.The constraint forces a low-entropy prediction at the super-class level by comparing within-group logits against labels from other super-classes.
  • Hierarchical Domain Knowledge on CIFAR100: MultiplexNet slightly improves super-class accuracy over the vanilla models and always satisfies the constraints, while not surpassing Vanilla ResNet on class accuracy.The hierarchical baseline also achieves 100% constraint satisfaction through bespoke engineering; DL2 improves class accuracy but has limited impact on super-class accuracy and constraint satisfaction.

Limitations and Discussion

The approach is limited by how precisely experts can specify valid domain constraints and by the representational demands of DNF formulas. Guaranteed constraint enforcement can also be vulnerable in unforeseen edge cases where violating a constraint might avoid a worse outcome.

  • Specification of domain knowledge: Experts must express valid domain knowledge precisely as logical formulas, which may be difficult when conceptual constraints do not map robustly to observable inputs.For image classification, statements such as dogs having four legs or fish being in water may be brittle because relevant features can be absent or atypical.
  • DNF representation: DNF representation can require exponentially many terms, forcing MultiplexNet to choose among an impractically large number of alternatives.
  • Safety implications: Safety guarantees may be problematic in unforeseen edge cases where an agent should violate a predefined constraint to avoid a more undesirable outcome.The authors argue that experts should still define operational boundaries and explicitly test known worst-case scenarios.

Conclusions and Future Work

The paper presents MultiplexNet as a general output-layer method for encoding expressive logical knowledge, guaranteeing constraint compliance while improving data efficiency across experiments. It identifies downstream robustness to adversarial attacks as future work.

  • Conclusions: MultiplexNet guarantees 100% compliance with domain constraints, unlike alternative approaches that append a constraint loss.The approach also allows arithmetic operators in constraints, extending beyond propositional logic.
  • Conclusions: Experiments show improved data efficiency, reducing the training data required for good performance.
  • Future Work: Future work will explore using the framework for downstream tasks such as robustness to adversarial attacks.

Appendix A: Additional Experimental Details

The appendix documents reproducible evaluation procedures, including fixed test sets, validation-based model selection, and publicly released code and data. Hardware and dataset licensing details are also specified.

  • Reproducibility: Code and experimental data are publicly available, while the MNIST and CIFAR100 datasets are distributed under Creative Commons licenses.The appendix also identifies the DL2 baseline repository and the hardware used for image and synthetic-data experiments.
  • Evaluation protocol: All experiments used train, validation, and test splits, with test sets held constant across experimental conditions for fair comparisons.Validation data were extracted from training data, using 10% for validation when model selection was performed.

Synthetic Data

The synthetic-data experiment applies MultiplexNet to constrained density estimation with a variational autoencoder. MultiplexNet reconstructs inputs within the constraints and learns the data manifold more effectively than the vanilla VAE, although prior sampling can produce unsupported modes.

  • Experimental setup: The VAE uses an isotropic Gaussian prior, likelihood, and posterior, with a fixed precision parameter controlling reconstruction quality.
  • MultiplexNet loss: MultiplexNet transforms unconstrained decoder outputs into constrained terms for each DNF component and assigns probabilities to those terms before forming the final loss.The method evaluates the loss separately for each constrained transformation and combines these terms through the learned categorical selection.
  • Samples from the Posterior: MultiplexNet samples strictly satisfy the constraints, whereas the baseline VAE fails to capture the constraint boundaries in posterior reconstructions.
  • Samples from the Prior: The vanilla VAE fails to capture many complexities of the data distribution, while MultiplexNet prior samples include two vertical modes containing no real data.The authors attribute these unsupported modes to random sampling from the latent categorical variable and note that a trainable categorical prior could address them.
  • Network architecture: The default encoder and decoder are single-hidden-layer feedforward networks with 15 latent dimensions, 50 hidden units, and ReLU activations.

MNIST - Label-free Structured Learning

The MNIST experiment applies MultiplexNet to label-free structured inference, using a VAE formulation with 100 possible structured terms selected by a categorical variable. Comparisons with a vanilla VAE examine posterior and prior samples across training-set sizes.

  • MNIST - Label-free Structured Learning: The structured MNIST inference considers 100 possible terms for (i, j, k1, k2), with a categorical variable selecting which term is present.The likelihood formulation combines two output terms with the selected categorical probability.
  • MNIST - Label-free Structured Learning: Posterior samples compare vanilla and MultiplexNet VAE reconstructions across different training-data sizes.The posterior samples represent attempts to reconstruct the input.
  • MNIST - Label-free Structured Learning: Prior samples show that the vanilla VAE fails to capture many complexities of the data distribution, while MultiplexNet produces samples through random categorical-term selection.This random selection creates samples in two vertical modes that contain no real data; a trainable categorical prior is proposed as an easy fix.
  • MNIST - Label-free Structured Learning: The vanilla VAE captures label clustering but cannot infer class labels correctly without the dataset’s structural information.

Hierarchical Domain Knowledge on CIFAR100

The CIFAR100 construction expresses hierarchical superclass knowledge as DNF-compatible constraints over class outputs. The resulting MultiplexNet loss combines constrained outputs, categorical-term probabilities, and cross-entropy while using a Wide ResNet backbone.

  • Hierarchical Domain Knowledge on CIFAR100: The CIFAR100 implementation encodes superclass constraints in DNF, including alternatives that make the people superclass highly likely while making other superclasses unlikely.The displayed logical form uses terms such as p_people < ϵ and p_trees > 1 − ϵ, with analogous alternatives for other superclasses.
  • Hierarchical Domain Knowledge on CIFAR100: The constraints operate over class probabilities on a simplex, allowing a single superclass constraint to imply low probability for all other superclasses.The normalization constant Z converts exponentiated class outputs into a valid probability distribution.
  • Hierarchical Domain Knowledge on CIFAR100: For the people superclass, the complementary right-hand side contains 95 labels from the other superclasses.
  • Hierarchical Domain Knowledge on CIFAR100: The final MultiplexNet loss uses constrained outputs for the 20 superclasses, categorical selection probabilities π_k, and standard cross-entropy.
  • Hierarchical Domain Knowledge on CIFAR100: The CIFAR100 experiments use a Wide ResNet 28-10 under all experimental conditions.

Appendix B: MultiplexNet Architecture Overview

MultiplexNet appends constraint handling to an existing network by transforming one unconstrained output into multiple outputs, each satisfying a DNF term. A latent categorical variable selects among these terms, and marginalization yields the training objective.

  • Appendix B: MultiplexNet Architecture Overview: Each DNF term φ_k receives a transformation h_k that maps real-valued network outputs into values satisfying that term.Softplus-based transformations handle inequality constraints, while equality constraints can directly set the output to a specified value.
  • Appendix B: MultiplexNet Architecture Overview: The architecture converts the unconstrained output x′ into K constrained outputs x_k, each guaranteed to satisfy the overall constraint formula.
  • Appendix B: MultiplexNet Architecture Overview: A latent categorical variable selects the active constraint term, and marginalizing it produces the specified loss function.
  • Appendix B: MultiplexNet Architecture Overview: The framework is designed as an appendable layer for existing learning schemes rather than a domain-specific network architecture.
Loading 2111.01564v1…