PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing Differentiable Fuzzy Logic Operators

Emile van Krieken, Erman Acar, Frank van Harmelen

arXiv:2002.06100v2cs.AIcs.LGcs.LO

TL;DR

The paper asks which fuzzy-logic operators have suitable theoretical and practical behavior in gradient-based learning. It analyzes these operators within Differentiable Fuzzy Logics, introduces sigmoidal implications, and evaluates them for semi-supervised learning. Many popular operators are unsuitable, implication gradients are imbalanced, and DFL improves over a supervised baseline but remains behind state-of-the-art methods.

  • Problem

    The paper addresses limited understanding of which fuzzy-logic operators have convenient properties when used in gradient descent for Differentiable Fuzzy Logics.

  • Method

    The authors analyze fuzzy operators theoretically and empirically, introduce sigmoidal implications, and compare operator configurations in semi-supervised learning.

  • Results

    Many popular operators are highly unsuitable for differentiable learning, while DFL significantly improves on a supervised baseline in semi-supervised learning.

  • Takeaways & Limitations

    Effective DFL performance requires operator combinations chosen for learning behavior, even when those combinations no longer satisfy usual logical laws.

  • Takeaways & Limitations

    DFL improves over a supervised baseline but is not competitive with state-of-the-art methods such as Ladder Networks.

Abstract

from arXiv · show

The AI community is increasingly putting its attention towards combining symbolic and neural approaches, as it is often argued that the strengths and weaknesses of these approaches are complementary. One recent trend in the literature are weakly supervised learning techniques that employ operators from fuzzy logics. In particular, these use prior background knowledge described in such logics to help the training of a neural network from unlabeled and noisy data. By interpreting logical symbols using neural networks, this background knowledge can be added to regular loss functions, hence making reasoning a part of learning. We study, both formally and empirically, how a large collection of logical operators from the fuzzy logic literature behave in a differentiable learning setting. We find that many of these operators, including some of the most well-known, are highly unsuitable in this setting. A further finding concerns the treatment of implication in these fuzzy logics, and shows a strong imbalance between gradients driven by the antecedent and the consequent of the implication. Furthermore, we introduce a new family of fuzzy implications (called sigmoidal implications) to tackle this phenomenon. Finally, we empirically show that it is possible to use Differentiable Fuzzy Logics for semi-supervised learning, and compare how different operators behave in practice. We find that, to achieve the largest performance improvement over a supervised baseline, we have to resort to non-standard combinations of logical operators which perform well in learning, but no longer satisfy the usual logical laws.

1. Introduction

The paper studies Differentiable Fuzzy Logics as a way to combine symbolic background knowledge with gradient-based neural learning. It analyzes how fuzzy operators affect learning and introduces sigmoidal implications alongside empirical operator comparisons.

  • Approach: Differentiable Fuzzy Logics translate fuzzy logical formulas into differentiable losses optimized jointly with neural-model parameters.Logical symbols are interpreted by a deep learning model, allowing background knowledge to influence gradient descent.
  • Experiments: Experiments compare fuzzy-logic operators in a semi-supervised learning setting and support recommendations for operator selection.The paper frames these methods as weakly supervised learning that can use unlabeled or noisy data with background knowledge.
  • Operator analysis: The paper theoretically and empirically analyzes operators for quantification, conjunction, disjunction, and implication in gradient-based learning.The analysis focuses on how operator derivatives shape corrections when predictions conflict with background knowledge.
  • New operators: The authors introduce sigmoidal implications using insights from their analysis of fuzzy-implication behavior.This new family is intended to address an implication-related gradient phenomenon identified in the study.

2. Differentiable Logics

Differentiable Logics use background knowledge to supervise neural predictions when exact scene descriptions are unavailable. In DFL, fuzzy truth values and operators become differentiable losses that can be optimized with gradient descent.

  • Differentiable Logics: Differentiable Logics construct differentiable losses from logical formulas whose truth values can be inferred using background knowledge.This allows unlabeled, poorly labeled, or partially labeled data to contribute to learning.
  • Motivation: The motivating scenario has an agent describe images while a supervisor corrects descriptions using a background knowledge base.The supervisor lacks exact image descriptions but can identify inconsistencies with known concepts and relations.
  • Running example: In the running example, confidence values represent whether objects satisfy predicates such as chair(o1).The image contains two objects, o1 and o2, and the agent assigns each observation a value in [0, 1].
  • Running example: A rule stating that chair parts are cushions or armrests can increase belief in cushion(o2) when its predicted confidence exceeds armRest(o2).The correction follows because confident chair and partOf antecedents require one of the consequent predicates to hold.
  • DFL formulation: DFL interprets ground-atom truth values in [0, 1] and translates fuzzy logical expressions into differentiable functions.Predicates, functions, and constants are interpreted using a deep learning model.

3. Background

The background defines fuzzy truth values and the operator families used to generalize logical connectives and quantifiers. It distinguishes t-norms, t-conorms, aggregators, S-implications, and R-implications by their semantic constructions.

  • Fuzzy logic: Fuzzy logic assigns truth values in [0, 1] to model vagueness and imprecise concept classification.Predicate fuzzy logics extend propositional fuzzy logics with universal and existential quantification.
  • Connectives: T-norms generalize conjunction, while t-conorms generalize disjunction through De Morgan duality.T-conorms are formed from a t-norm T using S(a, b) = 1 − T(1 − a, 1 − b).
  • Operator families: The paper surveys common aggregation operators and presents the t-norm, t-conorm, and implication families in reference tables.The tables organize operators alongside their relevant properties or construction sources.
  • Quantification: Universal and existential quantifiers are generalized with increasing aggregation operators A and E.These operators map sequences of truth values to [0, 1] and can be recursively constructed from t-norms or t-conorms.
  • Implications: S-implications generalize material implication through a t-conorm, whereas R-implications are constructed from t-norms using a supremum.The two classes provide alternative semantic constructions for fuzzy implication.

4. Differentiable Fuzzy Logics

Differentiable Fuzzy Logics represents objects as vectors, predicates as neural functions, and formulas as differentiable truth values optimized through gradient descent. Quantifier computation may use finite-domain enumeration or sampled batches, while operator choices determine the resulting learning signals.

  • DFL structures: DFL represents domain objects as d-dimensional vectors and predicates as neural functions mapping object vectors to truth values in [0, 1].Trainable parameters θ define embedded interpretations of predicate symbols.
  • Formula semantics: DFL evaluates fuzzy formulas compositionally using negation, t-norms, t-conorms, implications, and aggregation operators.The valuation function assigns each formula a differentiable truth value from atomic predicates through logical connectives and quantifiers.
  • Relaxing quantifiers: Quantifiers can be approximated by sampling batches when domains are infinite or too large for full evaluation.This relaxation is computationally practical but loses soundness because different batches can produce different formula truth values.
  • Optimization: Gradient descent minimizes the DFL loss by backpropagating through formula valuations to ground-atom truth values and neural parameters.The chain rule updates the embedded interpretation using a learning rate ε.
  • Gradient behavior: Operator choices produce different gradient signals, illustrated by unequal derivatives for predicates and relations in the worked example.The reported derivatives include 0.0029 versus 0.7662 for cushion and -0.4978 versus -0.1103 for selected partOf atoms.

5. Derivatives of Operators

Differentiable learning requires fuzzy operators to provide useful gradients across their input domains. Vanishing, single-passing, and compounded small derivatives can deprive DFL of effective learning signals.

  • Operator choice: Operator choice determines both the inferences and the optimization landscape in DFL, because different operators yield different derivatives and smoothness.Some operator sets make the fuzzy maximum satisfiability problem easier to optimize than others.
  • Gradient attenuation: Long chains of conjunctions can multiply partial derivatives below one until the resulting gradient approaches zero.The product t-norm is given as an example where this compounding effect can occur.
  • Vanishing derivatives: Drastic-product-derived operators have vanishing derivatives almost everywhere and are unsuitable for neural truth values in (0, 1).Their derivatives are nonvanishing only when inputs are exactly 0 or 1.
  • Single-passing operators: Single-passing operators provide a nonzero derivative for at most one input, making forward computation inefficient and limiting feedback.This property also applies compositionally to compositions of single-passing functions and occurs with Gödel-based operators.
  • Gradient requirements: A usable logical operator should have nonvanishing derivatives for most inputs and ideally should not be single-passing.Otherwise it contributes little or only narrowly targeted information to the learning signal.

6. Aggregation

Aggregation operators differ sharply in how broadly they propagate gradients across quantified inputs. Minimum, Łukasiewicz, Yager, and nilpotent aggregators often lose gradients as the number of inputs grows, whereas unbounded generalized-mean and logarithmic product variants offer more promising learning behavior.

  • Minimum and maximum: Minimum aggregation is single-passing, inefficient, and unable to handle exceptions well because only the lowest-truth input receives a gradient.Maximum aggregation similarly focuses on one highest-truth input, which can miss multiple valid existential witnesses.
  • Łukasiewicz aggregation: 1/n! of inputs have nonvanishing Łukasiewicz universal-aggregator derivatives, so most inputs provide no gradient in larger domains.The derivative is active only when the sum of inputs exceeds n − 1, requiring all inputs to be sufficiently high.
  • Łukasiewicz aggregation: The Łukasiewicz existential aggregator is useful only on a fraction of its domain, because learning requires all inputs to be close to 0.Its nonvanishing derivatives occur for only a restricted region of inputs.
  • Yager aggregation: For Yager aggregation, the nonvanishing-derivative region expands with p, from 0 at p = 0 to 1 at p = ∞, but remains small for low p or high n.For p = 2, the fraction has the closed form π^(n/2)/(2^n·Γ(n/2 + 1)).
  • Alternative aggregators: Removing Yager bounds yields generalized-mean variants with nonvanishing derivatives everywhere, while logarithmic product aggregation corresponds to cross-entropy and can handle outliers.These alternatives connect fuzzy aggregation to classical machine-learning losses, although generalized-mean behavior depends on p.
  • Nilpotent minimum: The nilpotent minimum aggregator has nonvanishing derivatives on 1/2^(n−1) of inputs and therefore vanishes on almost every input as n grows.This behavior makes it unsuitable for larger quantified inputs in learning contexts.
  • Summary: Overall, bounded universal aggregators that vanish with many inputs scale poorly, whereas existential aggregation offers more options because only one true input is required.The summary identifies product-derived and generalized-mean variants as promising practical alternatives.

7. Conjunction and Disjunction

The paper compares fuzzy conjunction and disjunction operators by examining their derivatives and optimization behavior. Operators differ in gradient strength, vanishing regions, brittleness, and how updates allocate responsibility between inputs.

  • Gödel operators: Gödel t-norms and t-conorms have derivatives with magnitude nearly always 1, avoiding exploding or vanishing gradients from repeated chain-rule applications.Their derivatives are undefined when the inputs are equal, and gradient ascent finds a global optimum for the running formula.
  • Lukasiewicz operators: Lukasiewicz derivatives vanish on as much as half the domain, although nonzero derivatives are large and avoid gradient attenuation.In the running example, gradient descent reaches a global optimum from about 83.5% of random initializations.
  • Yager operators: Yager t-norms and t-conorms have vanishing derivatives over a non-negligible domain region, with behavior approaching Gödel operators as p increases.Their derivatives are undefined at specific boundary points, requiring care to prevent numerical issues.
  • Product operators: Product t-norm updates scale each input by the other, producing low gradients when both inputs are small, whereas the product t-conorm has a more intuitive update direction.The paper interprets the t-conorm as increasing one input when the other is not true.
  • Summary: Overall, Gödel operators are strong but brittle, Lukasiewicz and Yager operators have substantial vanishing regions, and product operators mix unsuitable and intuitive update behavior.The Yager t-norm gives larger derivatives to lower truth values, while the product t-norm depends on the other input.

8. Implication

The paper analyzes how fuzzy implications distribute gradient corrections between antecedents and consequents. It finds severe imbalance and identifies sigmoidal implications as a smoothing strategy, while many popular implications remain unsuitable for differentiable learning.

  • Challenges of implication: A false implication can be corrected through modus ponens, modus tollens, distrust, or exception, creating multiple competing update choices.For a non-black raven, the system may alter the consequent, antecedent, both observations, or accept an exception.
  • Challenges of implication: The implication exhibits class-imbalance-like behavior because non-black non-ravens vastly outnumber positive or antecedent-matching cases.This makes modus tollens cases statistically more common than modus ponens cases.
  • Theoretical properties: R-implication derivatives vanish whenever the antecedent is no greater than the consequent, covering at least half the domain.S-implications are left-neutral and contrapositive differentiable symmetric, while R-implications are not generally contrapositive differentiable symmetric.
  • Gödel-based implications: Gödel implication increases the consequent when the antecedent exceeds it but never changes the antecedent, preventing modus tollens corrections.This makes it poorly performing in practice, including when both antecedent and consequent are low.
  • Sigmoidal implications: Sigmoidal implications transform existing implications with a sigmoid and translation, preserving boundary conditions while smoothing gradients at high values of s.The Reichenbach-sigmoidal implication remains contrapositive differentiable symmetric and has small gradients in all corners.
  • Summary: Popular Gödel, Yager R-, and Goguen implications are unsuitable for differentiable learning, while other operators may suffer from non-smoothness.The conclusion follows from theoretical analysis of several implication families under material-implication challenges.

9. Experimental setup

The experiments evaluate Differentiable Fuzzy Logic operators on structured MNIST tasks and a semi-supervised learning setup. The methodology combines DFL losses with supervised objectives and measures the direction and magnitude of operator-driven updates.

  • Experimental design: The experiments use MNIST to inspect how different fuzzy operators behave, rather than to establish state-of-the-art semi-supervised performance.The dataset is split into 1% labeled and 99% unlabeled examples.
  • Evaluation metrics: Consequent and antecedent magnitudes quantify summed partial derivatives, while consequent ratio measures the consequent share of total update magnitude.Correctly updated ratios measure the fraction of updates moving true consequents or negated antecedents in the right direction.
  • Tasks and formulas: The study evaluates universally quantified formulas involving digit predicates, a learned same relation, and a sum9 relation.The same task tests implication and universal aggregation, while sum9 tests existential quantification, conjunction, and disjunction.
  • Model: The neural model uses convolutional and fully connected layers for digit classification and a bilinear network with a sigmoid output for same predictions.The bilinear network uses a hidden size of k = 50.
  • Optimization: The loss combines a weighted DFL term with supervised cross-entropy and binary cross-entropy objectives.Training uses ADAM with learning rate 0.001, and negative examples for the binary task are undersampled because they are more numerous.
  • Configurations: The experimental results include symmetric configurations using S-implications or R-implications, with wDFL = 1 except for the product t-norm configuration, where wDFL = 10.The table reports results for the same problem across these configurations.

10. Results

Operator choices strongly affect DFL learning: aggregation functions, t-norms, implications, and optimizer settings produce markedly different results. The best-performing configurations improve over supervised learning but require task-dependent, sometimes asymmetric operator combinations.

  • Implications: S-implications generally outperformed R-implications, with the sigmoidal-Reichenbach implication reaching 97.3% accuracy using vanilla SGD.The authors hypothesize that R-implication derivatives vanish when a ≤ c.
  • Symmetric configurations: The product t-norm was the strongest symmetric choice, and it achieved the highest consequent and antecedent correctly updated ratios.Its performance was attributed partly to the effectiveness of the log-product aggregator.
  • Aggregation: For universal aggregation, log-product performed best, while generalized means were strongest for existential aggregation, with p = 1.5 performing best.Single-passing and broadly vanishing aggregators generally performed poorly in aggregation experiments.
  • Additional experiments: 97.7% accuracy was reached on the sum9 problem with vanilla SGD, while combining formulas from both problems achieved 98.0%.The combined-formula configuration was the best accuracy reported in these experiments.
  • Analysis: Correct antecedent updates correlated positively with accuracy in the sum9 analysis, while larger consequent ratios were associated with lower update correctness.The findings motivate balancing updates to antecedents and consequents.
  • Limitations: DFL improved over the supervised baseline but remained below Ladder Networks, which achieved 98.9% with 100 labeled pictures and 99.2% with 1000.This comparison defines the reported performance boundary against a state-of-the-art method.

11. Related Work

Related work places DFL within neuro-symbolic, statistical-relational, fuzzy-neural, probabilistic, and differentiable reasoning approaches. Prior systems differ in where logic is applied, which operators they use, and whether they differentiate through knowledge-base satisfaction.

  • Statistical Relational Learning: DFL belongs to Statistical Relational Learning, which combines reasoning under uncertainty with learning relational structures such as graphs.This positions DFL within a broader family of relational learning models.
  • Neuro-symbolic systems: Logic Tensor Networks interpret predicates with neural models and apply them to weakly supervised scene-graph parsing and reinforcement-learning transfer.These systems are presented as special cases of DFL.
  • Regularization methods: Semantic-based regularization applies DFL to kernel machines, while posterior regularization projects model outputs into a rule-regularized subspace rather than differentiating knowledge-base satisfaction.The approaches differ in how logical consistency influences learning or prediction.
  • Differentiable reasoning: Other differentiable reasoning systems use fuzzy operators for image generation, query proving, inductive logic programming, and entity-embedding search.The cited systems use varied operator choices, including product, Gödel, and t-norm-based constructions.
  • Neural architectures: Logical Neural Networks apply logical operators at the neuron level, whereas DFL applies them at a higher level by defining the loss function.Logical Neural Networks are therefore described as orthogonal to DFL.
  • Probabilistic logic: Probabilistic logic systems interpret predicates probabilistically and may require compilation techniques because inference can be exponential in grounding size.DeepProbLog supports automatic differentiation through neural predicates.

12. Discussion

The discussion identifies deployment challenges for DFL, including hyperparameter sensitivity, imbalanced implication updates, ambiguous disjunction choices, and tension between task performance and logical properties. It also cautions that fuzzy truth values are not probabilities in probabilistic classification settings.

  • Deployment challenges: DFL losses combine multiple formula objectives, creating varied loss landscapes and requiring substantial hyperparameter tuning.The paper notes that jointly learning relative formula weights as a possible response is outside this discussion.
  • Implication imbalance: Most well-performing implications produced far more derivatives for negated antecedents than consequents, and this imbalance may increase on more complex problems.Removing antecedent derivatives is not presented as a solution because those updates are usually correct.
  • Disjunction: Disjunction operators must choose among possible classes, so class-frequency imbalances can affect which alternative receives updates.The discussion illustrates this with walking objects that could be classified as humans or animals.
  • Logical properties versus performance: The best task-performing configuration combines product and Yager t-norms, whereas product is the only viable symmetric choice in the reported experiments.Symmetry preserves equal truth values for classically equivalent formulas, making analysis easier.
  • Interpretation of truth values: Fuzzy truth values model vagueness rather than probabilistic uncertainty, even though the paper applies them to a classification task with probabilistic origins.This creates an interpretive boundary for the reported truth values.

13. Conclusion

The analysis finds major differences among Differentiable Fuzzy Logic operators, with many unsuitable for gradient-based learning. It identifies promising operator choices, implication-gradient imbalance, and a performance trade-off between empirical gains and standard logical laws.

  • Many fuzzy-logic operators, including popular ones, are highly unsuitable for differentiable learning.
  • Log-product and RMSE aggregators connect naturally to fuzzy logic and machine learning while handling outliers.
  • The G¨odel t-norm and t-conorm are simple choices, while Yager t-norm and product t-conorm have intuitive derivatives.
  • Implication gradients are imbalanced: useful MNIST inferences often reduce antecedents, while consequent gradients can increase false consequents.
  • The product t-norm works across all operator choices, while product t-conorm, Reichenbach implication, and log-product aggregation have intuitive or effective behavior.
  • The largest supervised-baseline improvements require non-symmetric operator combinations that no longer satisfy usual algebraic relations.

Appendix D.2. Sigmoidal Functions

The appendix constructs sigmoidal transformations of fuzzy operators using a sigmoid and affine parameters, then analyzes the resulting properties. The transformation preserves ordering and key boundary behavior, but has restricted neutrality properties.

  • Sigmoidal operators transform a symmetric increasing function through a parameterized sigmoid, with affine parameters restoring boundary values.
  • The affine parameters are chosen so the transformed operator maps its domain minimum to 0 and maximum to 1.
  • The sigmoidal transformation preserves strict ordering and equality of the original operator’s outputs.
  • For every fuzzy implication, the transformed function is also a fuzzy implication and preserves the original implication’s 0 and 1 outputs.
  • Sigmoidal implications satisfy left-neutrality only when the original implication is left-neutral and the sigmoid parameter s approaches 0.
  • Contrapositive symmetry and the identity principle are preserved under the sigmoidal transformation.

Appendix E.1. Proof

The proof connects Differentiable Product Fuzzy Logic with probabilistic inference through factor graphs and belief propagation. The equivalence holds under restricted formula structure, while repeated ground atoms create loops and incorrect messages.

  • The proof represents formulas as Bayesian-network and factor-graph structures whose nodes encode subformulas, atoms, and logical operations.
  • Belief-propagation messages recursively reproduce fuzzy truth values for atoms, negations, and conjunctions under product semantics.
  • The proof uses approximation equality because loopy belief propagation does not necessarily satisfy the displayed equality.
  • When each ground atom appears only once, Differentiable Product Fuzzy Logic computes the same probability as Semantic Loss.
  • The equivalence relies on a tree factor graph; repeated ground atoms create loops that produce incorrect messages and probabilities.

Appendix F.3. Reichenbach-Sigmoidal Implication

The experiments evaluate the Reichenbach-sigmoidal implication with fixed conjunction, aggregation, and training settings. A sigmoid parameter of 9 gives the strongest reported accuracy, while the best gradient-related values do not coincide with the best accuracy.

  • With Yager conjunction p = 2, log-product aggregation, and DFL weight 10, sigmoid parameter s = 9 achieves 97.3% accuracy.
  • The Reichenbach-sigmoidal implication with s = 9 outperforms the other tested implications, including with vanilla SGD.
  • Increasing s raises cons% because the antecedent derivative near a = 0, c = 0 becomes low.
  • The parameter value with the highest cucons% and cuant% is not the one producing the best accuracy.
  • Setting b0 to −1/2 gives the highest accuracy and cucons% in the reported parameter comparison.
  • Removing the formula defining the same predicate causes the largest performance drop among the tested formula removals.
Loading 2002.06100v2…