Augmenting Neural Networks with First-order Logic

1 Introduction

• Large datasets are often needed for end-to-end neural training, but annotated data may be difficult and expensive to curate for new tasks, domains, or languages.The introduction notes that typical machine-translation training may require millions of sentence pairs.Source: §1 · p. 1
• The motivating proposal is to use first-order rules, such as preferring alignments marked similar by an external resource, to provide an inductive bias that can reduce data dependence.The paper asks whether declarative rules can integrate with end-to-end training, ease data needs, and compare with pre-trained representations.Source: §1 · pp. 1–2
• The framework systematically converts rules into differentiable computation graphs, allowing logic to guide training and prediction without extensive redesign or additional trainable parameters.It is designed to admit soft constraints in case rules are incorrect.Source: §1 · p. 2
• The framework improves existing neural designs across machine comprehension, natural language inference, and text chunking, especially when training examples are limited.The evaluations use off-the-shelf task models and logical constraints on observed neurons across different data sizes.Source: §1 · p. 2

2 Problem Setup

• Neural networks are directed acyclic graphs of neurons and weighted edges, and named neurons can carry task-related semantics.Predicates are associated with named nodes; the paper uses uppercase letters for predicates and lowercase letters for their nodes.Source: §2 · p. 2
• The rules considered are implications whose antecedent may combine literals conjunctively or disjunctively and whose consequent is a single literal.These rules include Horn clauses and generalizations.Source: §2 · p. 2
• Rule integration must map predicates to graph nodes, encode non-differentiable logic for gradient-based training, and prevent user-defined rules from creating graph cycles.The paper treats a named neuron's post-activation value as the degree of truth of its predicate and compiles conditional statements into differentiable statements.Source: §2 · pp. 2–3
• An implication is cyclic when a consequent node lies upstream of an antecedent node; otherwise it is acyclic.A node is upstream of another when a directed path connects them.Source: §2 · p. 3
• Some cyclic statements can be rewritten as equivalent acyclic contrapositives, such as B1 → A1 and ¬A1 → ¬B1.Figure 2 contrasts an example cyclic statement with an acyclic one.Source: §2 · p. 3, Figure 2 · p. 3

3.1 Constraints Beget Distance Functions

• For a rule with a conjunctive or disjunctive antecedent and single-literal consequent, a differentiable distance function adjusts the downstream neuron's computation.The ideal distance indicates whether the antecedent holds, but because that indicator is not differentiable, the framework uses smooth surrogates inspired by probabilistic soft logic and the Łukasiewicz T-norm and T-conorm.Source: §3.1 · p. 3
• Negated antecedents substitute 1 − z_i for the corresponding neuron state, while negated consequents negate the distance function; ρ controls constraint strength.Large ρ makes the distance dominate the downstream pre-activation, whereas small ρ leaves more influence to the network parameters.Source: §3.1 · p. 4

3.2 General Boolean Antecedents

• General Boolean antecedents are converted using auxiliary neurons defined by biconditional constraints, allowing the framework to reuse its distance functions while maintaining an acyclic subgraph.The contrapositive may not help when it produces a complex right-hand side that the framework cannot yet compile.Source: §3 · p. 3, §3.2 · p. 4

3 A Framework for Augmenting Neural Networks with Constraints

• The workflow converts antecedents to conjunctive or disjunctive normal form, builds distance functions and constrained or auxiliary layers, then trains and runs inference on the augmented network.The augmentation adds no trainable parameters and is designed for implementation with modern neural-network libraries.Source: §3.2 · pp. 4–5, §3.3 · p. 5

3.3 Discussion

• Conflicting constraints on one downstream neuron are left to the base network to reconcile, with summing constrained pre-activation scores serving as a practical heuristic.Conjunctive consequents can be decomposed into independent constraints; handling more complex consequents remains future work.Source: §3.3 · p. 5

4 Experiments

• Across machine comprehension, natural language inference, and text chunking, augmentation is especially effective with small training sets, while SNLI models only match baseline with the full dataset.The text-chunking results also show that combining local augmentation with global inference performs best with full data.Source: §4.1 · p. 6, §4.2 · p. 7, §4.3 · p. 8

4.1 Machine Comprehension

• In machine comprehension, R1 and R2 improve BiDAF without extra trainable parameters, with larger gains on smaller datasets.Table 2 reports average span F1 over three runs on the official development set.Source: §4.1 · p. 6, Table 2 · p. 6

How does it compare to pretrained encoders?

• With ELMo, augmented models still outperform baseline; R1 consistently beats R2, which also exceeds baseline.The comparison is reported in Table 2.Source: How does it compare to pretrained… · p. 6, Table 2 · p. 6

4.2 Natural Language Inference

• On SNLI, N1 performs strongly with ≤2% data, N2 works better with larger sets, and augmented models are only on par with baseline on full data.Table 3 reports average accuracy on the SNLI test set over three runs.Source: §4.2 · p. 7, Table 3 · p. 7
• ≤10% data can still benefit from noisy N3, but pairing it with N2 degrades performance from 5% onward as N2 amplifies its noise.N3 alone performs worse than baseline, and the authors report finding counterexamples to that rule during preliminary analysis.Source: §4.2 · p. 7

4.3 Text Chunking

• ≤40% data: CRF underperforms baseline while constrained models remain strong; with full data, locally augmented CRF performs best.The authors describe local augmentation and global inference as orthogonal methods.Source: §4.3 · p. 8, Table 4 · p. 8

5 Related Work and Discussion

• The framework uses first-order logic to augment a given architecture, unlike KBANN and CILP++, which construct networks from logical rules [23].It retains the original network design and augments interpretable layers rather than using the teacher-student procedure described by Hu et al. (2016) [24].Source: §5 · p. 8
• Related methods add logical rules to loss functions to constrain labels, promote agreement, or reduce prediction inconsistencies [25, 26].Other approaches derive symbolic-knowledge losses or integrate indirect supervision expressed through probabilistic logic [26].Source: §5 · p. 8
• The framework’s logic-as-template approach resembles Markov Logic Networks, while its constraint penalties resemble Constrained Conditional Models [27].Unlike structured prediction approaches, structured inference is not mandatory [28].Source: §5 · pp. 8–9
• Related attention augmentation uses word relatedness as an extra input feature to attention neurons; this work is related to neural-symbolic learning, which seeks to integrate neural networks with symbolic knowledge.Source: §5 · pp. 8–9

6 Conclusions

• The framework introduces logical statements as constraints in neural networks to guide learning and prediction.It converts first-order logic into differentiable components without extra learnable parameters or extensive redesign.Source: §6 · p. 9
• Experiments explore the framework’s flexibility with different constraints across diverse tasks.Source: §6 · p. 9
• Neural models can benefit from external knowledge during learning and prediction, especially with limited training data.Source: §6 · p. 9

A Appendices

• The appendix documents model setups, hyperparameters, and data splits for machine comprehension, natural language inference, and text chunking.Source: A Appendices · p. 11
• All three tasks use Adam for training.Source: A Appendices · p. 11
• All three tasks use 300-dimensional GloVe embeddings trained on 840B tokens.Source: A Appendices · p. 11

A.1 Machine Comprehension

• SQuAD v1.1 has 87,599 training instances and 10,570 development examples; sampled training data is split 9/1 for training and development.The original development set is reserved for testing because the official test set is hidden and evaluating the many models there is impractical.Source: §A.1 · p. 11
• The BiDAF model is trained for 20 epochs with learning rate 0.001, dropout 0.2, and 100 hidden units per BiLSTM direction.Source: §A.1 · p. 11
• ELMo models are trained for 25 epochs with learning rate 0.0002, using the other hyperparameters from Peters et al. (2018).ELMo embeddings are neither pre-tuned nor post-tuned; the best development-split model is selected, and ρ is grid-searched over {1, 2, 4, 8, 16}.Source: §A.1 · p. 11

• SNLI contains 549,367 training, 9,842 development, and 9,824 test examples.Source: §A.2 · p. 11
• For each training-data percentage, validation examples are sampled at the same proportion from the original development set, with a minimum of 1,000 examples.Source: §A.2 · p. 11
• The BiLSTM DAtt model is trained for 100 epochs with learning rate 0.0001 and dropout 0.2, and its development-best checkpoint is evaluated.The scaling factors ρ are grid-searched over {0.5, 1, 2, 4, 8, 16} without extensive tuning.Source: §A.2 · p. 11

• CoNLL2000 has 8,936 training examples and 2,012 test examples, with performance reported on the original full test set.Source: §A.3 · p. 11
• Training and development examples are sampled from the original training set and split into 9/1 folds.Source: §A.3 · p. 11
• The BiLSTM encoder uses hidden size 100 per direction and dropout probability 0.5 before the final linear layer.Source: §A.3 · p. 11
• Models train for 100 epochs at learning rate 0.0001, with development-best checkpoint selection and manually grid-searched ρ values.The search uses {1, 2, 4, 8, 16, 32, 64} without extensive tuning.Source: §A.3 · p. 11