Source-linked AI summary
Harnessing Deep Neural Networks with Logic Rules
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, Eric Xing
TL;DR
Deep neural networks require substantial labeled data and can be difficult to interpret or guide with human knowledge. The paper introduces iterative rule knowledge distillation to transfer declarative first-order logic into general neural-network parameters. Across sentiment analysis and named entity recognition, a few intuitive rules improve base networks and yield better or comparable performance to state-of-the-art systems.
Problem
Deep neural networks rely heavily on labeled data and can produce uninterpretable results, while human intentions and structured knowledge are difficult to encode directly.
Method
The framework uses iterative rule knowledge distillation, with a posterior-regularized teacher and student network, to transfer first-order logic information into general neural-network parameters.
Results
With one or two intuitive rules, the framework improves CNN sentiment analysis and RNN named entity recognition, achieving better or comparable performance to state-of-the-art models.
Takeaways & Limitations
The framework provides a general way to incorporate human knowledge and intentions into CNNs and RNNs across classification and sequence-learning applications.
Takeaways & Limitations
The evaluation focuses on sentiment analysis and named entity recognition, while application to other domains such as vision remains future work.
Abstract
from arXiv · showhide
Combining deep neural networks with structured logic rules is desirable to harness flexibility and reduce uninterpretability of the neural models. We propose a general framework capable of enhancing various types of neural networks (e.g., CNNs and RNNs) with declarative first-order logic rules. Specifically, we develop an iterative distillation method that transfers the structured information of logic rules into the weights of neural networks. We deploy the framework on a CNN for sentiment analysis, and an RNN for named entity recognition. With a few highly intuitive rules, we obtain substantial improvements and achieve state-of-the-art or comparable results to previous best-performing systems.
1 Introduction
The paper addresses data dependence, limited interpretability, and difficulty encoding human intention in deep neural networks by integrating declarative logic rules through a general framework. An iterative distillation procedure transfers structured rule information into neural networks, improving CNN and RNN applications with only a few intuitive rules.
- Deep neural networks can achieve high predictive accuracy but often depend on large labeled datasets and produce uninterpretable or counter-intuitive results.
- Logic rules provide a declarative way to express high-level cognition and structured knowledge for guiding neural models.
- The framework enhances general neural architectures, including CNNs and RNNs, with logic-rule knowledge across tasks.
- Iterative rule knowledge distillation transfers structured information from logic rules into network parameters while learning from labeled instances.
- A few intuitive rules improve CNN sentiment analysis and RNN named entity recognition, reaching better or comparable performance to state-of-the-art models.
- The authors characterize the framework as a first principled integration of logic rules with general workhorse deep neural networks.
2 Related Work
Prior approaches combined structured knowledge with neural or probabilistic models, but often relied on specialized architectures, feature-label representations, or indirect integration. The proposed approach instead distills declarative first-order logic into parameters of general neural networks.
- Earlier neural-symbolic systems constructed network architectures from rule sets for reasoning and knowledge acquisition.
- Feature augmentation incorporated domain knowledge into neural inputs but remained limited to instance-label supervision and could not naturally encode many structural relations.
- Other frameworks encoded structured constraints in latent-variable models, but were not directly applicable to neural networks or could yield inferior empirical performance.
- The proposed method differs by iteratively transferring rich declarative first-order logic knowledge into parameters of general neural networks.
- The authors report that the approach strongly outperforms an extensive range of ad-hoc and general integration methods.
3 Method
The framework iteratively projects neural predictions into a rule-regularized space and distills those structured predictions back into network parameters. It applies architecture-agnostic teacher-student optimization while balancing rule imitation with supervised labels.
- Framework: The framework applies posterior regularization to construct a rule-regularized teacher from a neural network, then iteratively updates the student to emulate it.The formulation keeps the teacher close to the student with KL-divergence while enforcing rule expectations, slack variables, and confidence-weighted constraints.
- Framework: The method is designed for general neural architectures, including CNNs and RNNs, rather than a task-specific network form.The paper frames the approach as learning simultaneously from labeled instances and first-order logic rules.
- Logic representation: Soft logic represents continuous truth values in [0, 1] and supports flexible encoding and stable optimization of first-order rules.Conjunction operators are approximated differently: & acts as a selection operator, whereas ∧ acts as an averaging operator.
- Iterative distillation: Each student update balances prediction of true labels with imitation of the teacher’s soft outputs.The imitation parameter controls the relative importance of the two objectives, while the teacher’s soft predictions transfer rule knowledge into the student weights.
- Teacher construction: The teacher-construction problem is convex, efficiently solvable in dual form, and has a closed-form solution without significant computational overhead.Rule confidence controls how strongly predictions violating constraints are downweighted, while C regularizes slackness.
- Optimization procedure: At each iteration, Algorithm 1 samples a minibatch, constructs the teacher, updates the student, and repeats until convergence.Soft predictions can be computed by direct enumeration for factored constraints and dynamic programming for additional dependencies such as NER transitions.
4 Applications
The framework is demonstrated on CNN-based sentiment classification and RNN-based named entity recognition using linguistically motivated logic rules. The applications target contrastive sentiment structure and valid entity-label dependencies, including list-based correspondence.
- Architectures: The applications pair a sentence-level CNN for sentiment analysis with a bidirectional LSTM recurrent network for named entity recognition.The sentiment model uses a sentence representation, fully connected layer, and softmax; the NER architecture includes bidirectional LSTM sequence tagging with character and word features.
- Scope: The method is presented as reusable across classification and sequence-learning applications through declarative first-order logic rules.The experiments use the same general framework while designing task-specific, linguistically motivated rules.
- Sentiment classification: Sentiment rules address A-but-B sentences by requiring the whole-sentence sentiment to agree with the clause following “but.”The rule targets contrastive constructions because the post-“but” clause generally dominates the sentence’s sentiment.
- Named entity recognition: NER rules encode valid successive-label constraints that the base network’s largely independent tagging decisions can violate.The approach uses logic rules instead of adding CRF parameters, with hard constraints assigned infinite confidence.
- Named entity recognition: List-based NER rules connect corresponding positions across lists, making counterpart entities likely to receive the same category.For example, “Barcelona” is classified as an organization when its counterpart “Juventus” is an organization.
5 Experiments
Across sentiment classification and named entity recognition, the framework improves neural baselines by integrating logic rules, including under limited-label settings. Its teacher and distilled student models achieve strong benchmark performance, while unlabeled data provides further gains in semi-supervised learning.
- Overall evaluation: Substantial improvements were obtained on sentiment classification and named entity recognition using simple rules integrated with CNN and RNN models.The evaluation covers public benchmarks and compares the rule-enhanced models with base networks and prior systems.
- Sentiment classification: Rows 1–3 of Table 1 show strong accuracy boosts over the base CNN across SST2, MR, and CR.The teacher network q further improves over the distilled student network p.
- Sentiment classification: On MR and CR, the sentiment model outperforms all listed baselines, while MVCNN is slightly better on SST2.MVCNN uses diverse pretrained embeddings and a larger neural architecture than the reported model.
- Rule-integration comparisons: The iterative distillation models outperform alternative rule-integration methods on SST2, while training one neural parameter set instead of two separate sets.The distilled student also substantially outperforms the base CNN and projection-based alternatives.
- Semi-supervised learning: With 5% labeled data, Rule-q improves accuracy over the base by 2.6%, compared with 2.3% at 10% and 2.0% at 30%.Adding unlabeled instances for semi-supervised learning further improves accuracy.
- Named entity recognition: On CoNLL-2003 NER, transition rules give the joint teacher model a 1.56 F1-score improvement, and adding the list rule brings performance close to the best systems.The rule-enhanced models do not introduce extra model parameters for these constraints.
6 Discussion and Future Work
The framework combines deep neural networks with first-order logic rules through iterative distillation, improving base networks on sentiment analysis and named entity recognition. The authors identify extensions to probabilistic knowledge representations, broader domains, and automated rule learning as future directions.
- The framework transfers structured first-order logic information into neural-network weights through iterative distillation.A teacher network constructed with posterior regularization guides the transfer.
- With a few intuitive rules, the framework significantly improves base networks on sentiment analysis and named entity recognition.
- Future work: The soft-logic formulation can be extended to probabilistic models for expressing structured distributions and performing inference and reasoning.
- Future work: The authors plan to explore richer knowledge representations and apply the approach to other domains such as vision tasks.
- Future work: Future work includes automatically learning rule confidence and deriving new rules from data.
A.1 Solving Problem Eq.(3), Section 3.3
The appendix derives a closed-form solution for the logic-rule constraint optimization problem. The derivation uses stationarity, monotonicity, and substitutions from intermediate equations to obtain q as in Eq.(4).
- The derivation is adapted from prior work on logic-rule constraint settings and reformulated to produce a closed-form solution.
- Stationarity of the Lagrangian yields µ_l,gl = C − η_l,gl.
- Because Z_η monotonically decreases as η increases and η_l,gl ≤ C, the derivation establishes the stated constraint condition.
- Substituting Eqs.(A.6) and (A.8) into Eq.(A.4) gives the solution for q in Eq.(4).
A.2 Identifying Lists for NER
The NER list-identification method uses high-precision pattern matching to find structured lists and filter their items for likely named entities. It handles both intra-sentence and inter-sentence lists, while better detection remains a possible improvement.
- The method retrieves numbered or hyphen-marked lists containing at least three items.It targets patterns such as “1. ... 2. ... 3. ...” and “- ... - ... - ...”.
- Item text must use capitalized words, and each punctuation-delimited block may contain no more than three words.
- The method detects both intra-sentence and inter-sentence lists in documents.
- The authors report that the patterns effectively identify true lists, while a better list detector could further improve NER results.