PaperlayerView exact revision on arXiv

Source-linked AI summary

\L{}ukasiewicz Neural Networks Extended: Residual Architectures and Crystallization Strategies for Interpretable Rule Extraction

Carlos Leandro

arXiv:2609.33028v1cs.LGcs.AI

TL;DR

Prior Łukasiewicz neural networks were limited by probabilistic crystallization and theoretical guarantees for shallow architectures. This paper introduces residual ŁNNs and analyzes three crystallization strategies; its reported results show distinct trade-offs, including reliable crystallization with STE and a clinically interpretable five-variable heart-disease formula.

  • Problem

    Existing Łukasiewicz neural networks face probabilistic crystallization and guarantees established for shallow architectures, limiting the framework’s reach.

    Source: §1 · p. 2

  • Method

    The paper extends ŁNNs with residual connections that guarantee the symbolic correspondence at merge neurons and analyzes corrected LM, STE, and proximal regularization.

    Source: §1 · p. 2, §2.3 · p. 5

  • Results

    No single strategy dominates: LM with residual connections recovers exact formulas on structured problems, STE crystallizes most reliably across benchmarks, and Proximal identifies clinically meaningful features but collapses on dense datasets.

    Source: §8 · p. 19

  • Takeaways & Limitations

    The five-variable Heart Disease formula demonstrates that the framework can produce transparent, verifiable diagnostic rules from real medical data.

    Source: §1 · p. 3, §8 · p. 19

    LM has quadratic parameter complexity, Proximal can collapse to constant-output solutions, and ŁNN expressive power is bounded by McNaughton functions.

    Source: §7.2 · p. 18

Abstract

from arXiv · show

A feed-forward neural network whose weights are integers and whose activation is the truncated identity implements, neuron by neuron, the connectives of \L{}ukasiewicz many-valued logic. This exact correspondence --- established theoretically by Castro and Trillas and developed into a training algorithm by Leandro --- enables \emph{symbolic knowledge extraction}: training produces not a black-box model but a logical formula. Two obstacles have limited the approach to shallow architectures and small datasets: crystallization (forcing weights to integers) succeeds only probabilistically under the original Levenberg--Marquardt training scheme, and the theoretical guarantees break down as networks grow deeper. This paper addresses both obstacles. First, we prove that \emph{residual connections} (skip connections of the kind used in ResNets) extend \L{}ukasiewicz neural networks to arbitrary depth while preserving the symbolic correspondence \emph{at merge neurons} by construction: merge neurons in a \L{}ukasiewicz residual block automatically satisfy the neuron-classification proposition, regardless of the inner layer weights; inner-layer neurons are trained toward representability by the crystallization strategy. Second, we analyse three crystallization strategies --- Levenberg--Marquardt (corrected), straight-through estimation (STE), and proximal regularization --- characterizing their theoretical guarantees, failure modes, and interpretability trade-offs.

1 Introduction

Łukasiewicz neural networks support exact symbolic extraction, but prior methods were limited by unreliable crystallization and guarantees established only for shallow architectures. This paper extends the framework with residual connections and analyzes three crystallization strategies.

1 Introduction

  • The paper motivates interpretable-by-construction models because post-hoc explanations can approximate decisions and be systematically wrong in ways invisible to the explainer.The concern is particularly salient for high-stakes diagnosis, where unauditable reasoning may make a high-performing model clinically useless.Source: §1 · pp. 1–2
  • Exact neuron-to-connective correspondence enables symbolic extraction, unlike classical two-valued rule extraction, which requires an approximation step.The correspondence builds on Castro and Trillas and Leandro [7].Source: §1 · p. 2
  • The original LM scheme crystallizes with high probability on small, low-dimensional datasets but fails systematically on high-dimensional real-world problems.The reported cause is a conditioning bug that produces updates of ∥∆w∥≈47 when ∥w∥≈0.1, causing divergence before convergence.Source: §1 · p. 2
  • Residual connections extend Łukasiewicz networks to deeper architectures, with merge neurons satisfying the symbolic guarantee by construction.The paper also analyzes corrected LM, STE, and proximal regularization as crystallization strategies.Source: §1 · p. 2
  • The paper demonstrates exact formula extraction, including a clinically meaningful five-variable heart-disease rule, and reports that Proximal identifies ca as the strongest clinical predictor across restarts.The introduction also describes a six-benchmark comparison of the three strategies using 10 trials per dataset and Wilcoxon signed-rank tests.Source: §1 · pp. 2–3

2 Background

Łukasiewicz logic uses truth values in [0, 1], and its piecewise-linear connectives correspond exactly to truncated-identity neural computations when neuron weights and biases meet the classification conditions. Rewriting algebra supports formula extraction when neurons are not directly representable.

2.1 Łukasiewicz many-valued logic

  • Łukasiewicz conjunction and disjunction are many-valued operations on [0, 1] that recover Boolean conjunction and disjunction on {0, 1}.The logic’s connectives are piecewise linear with integer coefficients, and its finite subdomains S_n are closed under all four operations.Source: §2.1 · p. 3

2.2 Łukasiewicz neural networks

  • A neuron with the Proposition 2 weight and bias configurations computes a Łukasiewicz conjunction or disjunction, enabling a network of such neurons to encode a formula exactly [7].Any Łukasiewicz formula can conversely be encoded as a network by mapping each connective to a neuron.Source: §2.2 · p. 4

2.3 Rewriting algebra and symbolic extraction

  • For neurons not directly covered by Proposition 2, rewriting algebra tests two-neuron decompositions and uses the highest-λ candidate as the best logical approximation when exact representability fails.Extraction proceeds bottom-up, recording directly representable formulas and otherwise applying Rule R to choose the best-λ candidate.Source: §2.3 · pp. 4–5

2.4 Relationship to Leandro (2009)

  • Proposition 2 and the rewriting algebra are reproduced from Leandro [7], while the paper identifies Sections 3–5 as new.Table 1 is described as delineating the earlier and present contributions.Source: §2.4 · p. 5, Table 1 · p. 5

3 Residual Łukasiewicz Networks

Residual connections address the shallow-architecture restriction by imposing a structure on merge neurons that guarantees their symbolic correspondence regardless of inner-layer weights. Formula extraction handles the merge first and can then extract the inner layers.

3.1 Motivation

  • The original ŁNN framework is restricted to shallow feed-forward networks, while residual connections offer a route to deeper architectures with a structurally constrained merge neuron.The residual architecture adds the layer input to the residual-path output; its merge structure guarantees Proposition 2 without additional training restrictions.Source: §3.1 · p. 5

3.2 The Łukasiewicz residual block

  • Each residual merge neuron combines the residual-path output and the shortcut input with weights +1, leaving its bias as the only free merge parameter.The bias is trained and crystallized alongside the inner-layer weights.Source: §3.2 · p. 6

3.3 Symbolic guarantee: Proposition 2 by construction

  • Merge neurons satisfy Proposition 2 by construction: bias −1 yields conjunction and bias 0 yields disjunction, without constraints on inner-layer weights.Thus merge operations remain exact even if inner neurons are not directly representable and need rewriting algebra for approximation.Source: §3.3 · pp. 6–7

3.4 Formula extraction from residual networks

  • Residual-network extraction first reads the merge formula using Theorem 4, then extracts inner-layer formulas using Proposition 2 or rewriting algebra.On MONK-3, the extracted formula uses a conjunction merge and an inner disjunction and achieves 97.2% classification accuracy, correctly capturing the ground-truth rule.Source: §3.4 · p. 7

4 Crystallization Strategies

The paper compares smooth crystallization, corrected Levenberg–Marquardt, STE, and proximal regularization, with distinct guarantees and interpretability trade-offs. Corrected LM remains probabilistic, while STE and Proximal guarantee integer weights; Proximal can fail on dense problems.

4 Crystallization Strategies

  • Proposition 2 requires weights in {−1, 0, +1}; crystallization measures how closely network weights approach integers, with the network considered crystallized when ∆(N) < ε.The paper uses ε = 10^-3 throughout.Source: §4 · p. 7

4.1 Smooth crystallization

  • A progressive smooth schedule increases the deformation strength from Υ2 through Υ16 near training’s end, then clamps and rounds weights while rounding biases without clamping.This delays rounding to avoid prematurely destroying gradient information while driving quasi-integer weights toward targets.Source: §4.1 · p. 7

4.2 Levenberg–Marquardt training

  • The corrected LM update uses identity damping, avoiding the original scaling bug that caused ill-conditioning and large updates; crystallization remains probabilistic.Corrected LM converges rapidly near solutions and achieves crystallization in a mean of 6.2 iterations on MONK-3.Source: §4.2 · p. 8

4.3 Straight-Through Estimation

  • STE computes with ternary-quantized weights while updating continuous shadow weights, and guarantees crystallization but may require rewriting algebra for formula extraction.The quantization partitions weights into −1, 0, and +1 regions at −1/3 and +1/3.Source: §4.3 · pp. 8–9

4.4 Proximal regularization

  • Proximal regularization combines sparsity and ternary attraction, guarantees crystallization, and can extract a five-feature Heart Disease formula from 22 available features.On high-dimensional dense problems, its ℓ1 penalty can eliminate discriminative weights and collapse the network to a constant-output classifier, with F1 ≈0.Source: §4.4 · p. 9

4.5 Theoretical comparison

  • LM with residual connections guarantees exact Proposition 2 merge neurons, whereas STE and Proximal crystallize reliably but require rewriting algebra for formula extraction.The training and extraction workflow summarizes the common process across the strategies.Source: §4.5 · p. 10, Table 2 · p. 10, §4.6 · p. 10

5 Experimental Evaluation

Across six UCI benchmarks and synthetic truth tables, the evaluation compares crystallization reliability, predictive accuracy, formula quality, and training time. Results vary by strategy: STE often leads in accuracy, LM-Res crystallizes inconsistently across datasets but yields the only fully representable formula, and Proximal can produce sparse clinical rules yet often degenerates.

5.1 Protocol

  • The protocol evaluates six UCI benchmarks with 10 independent trials per method, selecting hyperparameters by validation-split grid search and reporting test results.Reported metrics include mean accuracy and F1, standard deviations, crystallization rate, and wall-clock training time.Source: §5.1 · pp. 10, 12

5.2 Truth table reconstruction

  • On synthetic truth tables, LM exactly recovers f1 in 1/10 trials and f2 in 5/10; STE and Proximal crystallize in all f1 trials but have moderate mean F1.For f2, Proximal produces constant-zero output in all 10 trials, while STE reaches F1 = 0.545.Source: §5.2 · p. 12

5.3 Crystallization reliability on real datasets

  • LM-Res crystallizes in 10/10 MONK-3 trials and 9/10 Heart Disease trials, but in 0/10 Breast Cancer trials; STE and Proximal crystallize in 10/10 trials on every dataset.On Heart Disease, baseline LM without residual connections crystallizes in 0/10 trials, compared with 9/10 for LM-Res.Source: §5.3 · p. 12, Table 4 · p. 13

5.4 Classification accuracy

  • STE leads on Heart Disease, MONK-1, and MONK-2, while LM-Res leads on MONK-3; the methods show no significant accuracy difference on Mushroom.On MONK-3, LM-Res exceeds STE by 8.2 percentage points and Proximal by 24.2 percentage points, with both gaps statistically significant.Source: §5.4 · p. 12, Table 4 · p. 13
  • Proximal collapses to degenerate solutions on five of six datasets, while MONK-1 is the non-degenerate exception.The collapse persists across a wide grid search of regularization strengths; MONK-1 has F1 = 0.147.Source: §5.4 · p. 13

5.5 Formula quality and interpretability

  • LM-Res on MONK-3 is the only method producing a fully representable formula; its extracted formula reaches 97.2% accuracy.The formula correctly classifies all but 2.8% of the test set, corresponding to erroneous labels introduced by the 5% noise.Source: §5.5 · p. 13
  • A Proximal Heart Disease trial extracts a sparse rule using five of 22 input features, and ca is selected in all 10 crystallized trials.The held-out formula achieves accuracy 85.0%, sensitivity 87.5%, specificity 83.3%, and F1 = 0.824.Source: §5.5 · pp. 14–15, Table 6 · p. 15

5.6 Training efficiency

  • LM-Res is fastest on four of six datasets; on MONK-3 it takes 0.033 s versus 0.260 s for STE and 0.338 s for Proximal.Proximal is fastest on Mushroom and Breast Cancer, where early stopping terminates quickly at the cost of solution quality.Source: §5.6 · pp. 15–16, Table 7 · p. 15

6 Related Work

Related work contrasts post-hoc rule approximation with ŁNNs’ exact formula reading, and distinguishes ŁNNs from fuzzy, quantized, and other neuro-symbolic approaches. The key distinction is that crystallized ŁNN weights constitute the formula directly.

6.1 Rule extraction from neural networks

  • Classical rule-extraction methods train continuous networks and then approximate rules, whereas ŁNNs can provide an exact formula when neurons satisfy Proposition 2.When rewriting is needed, the approximation is bounded and measurable by λ-similarity.Source: §6.1 · p. 16

6.2 Fuzzy and many-valued neural systems

  • ANFIS learns continuous membership functions, while Amato et al. establish rational-weight representation without an integer-weight training algorithm or its guarantees.The present work supplies an integer-weight training algorithm in three variants and corresponding theoretical guarantees.Source: §6.2 · p. 16

6.3 Weight quantization

  • Weight quantization targets computational efficiency, not logical interpretation; the proximal ternary attraction term instead targets logically meaningful weights.The ternary target set is {−1, 0, +1}.Source: §6.3 · p. 16

6.4 Neuro-symbolic AI

  • Unlike IBM LNNs and Logic Tensor Networks, ŁNNs crystallize to a discrete formula; differentiable ILP learns clauses without a neural architecture, while DeepProbLog models uncertainty probabilistically.IBM LNNs and LTNs retain continuous weights; ŁNN weights constitute the formula directly after crystallization.Source: §6.4 · p. 17

7 Discussion

The discussion frames interpretability by construction as a way to verify a model through its formula rather than an approximation. It also identifies theoretical and practical limits, including open questions about expressivity and residual-block effects on inner-layer crystallization.

7.1 Interpretability by construction versus post-hoc

  • When fully crystallized neurons satisfy Proposition 2, the formula is the model and can be checked algebraically; clinical experts can audit rules from the formula alone.Unlike decision trees, Łukasiewicz formulas evaluate normalized continuous features directly rather than thresholding them.Source: §7.1 · pp. 17–18

7.2 Limitations

  • LM Jacobian inversion costs O(p^2), while Proximal training can collapse to constant outputs; feature preselection mitigates collapse but adds a training step.Source: §7.2 · p. 18
  • ŁNN expressivity is bounded by McNaughton functions, and whether bounded-depth networks approximate every such function to arbitrary accuracy remains open.Source: §7.2 · p. 18
  • Extending residual blocks to first-order logic remains open, and experiments cover binary classification while compatible multiclass output encoding is left for future work.Source: §7.2 · p. 18

7.3 Future directions

  • Whether finitely many Łukasiewicz layers approximate any McNaughton function to arbitrary precision remains open; other residuated-lattice logics may admit neural encodings.Source: §7.3 · p. 18
  • Theorem 4 guarantees correct residual merge neurons, but the effect of residual-block count on inner-layer crystallization remains empirical.Source: §7.3 · p. 19

8 Conclusion

The conclusion presents Łukasiewicz neural networks as an exact neuron-connective framework extended to residual architectures, with merge-neuron guarantees and analyzed crystallization strategies. Experiments show strategy-specific trade-offs and a clinically interpretable five-variable Heart Disease formula.

8 Conclusion

  • Residual-block merge neurons satisfy Proposition 2 by construction regardless of inner-layer training, while the paper analyzes three crystallization strategies and corrects a critical LM implementation bug.The analysis characterizes conditions under which each strategy succeeds or fails.Source: §8 · p. 19
  • No crystallization strategy dominates: LM with residual connections recovers exact structured-problem formulas and converges in under ten MONK-3 iterations, STE is most reliable across benchmarks, and Proximal collapses on dense datasets.Proximal also crystallizes consistently and identifies clinically meaningful features; the five-variable Heart Disease formula (18) has direct clinical interpretation.Source: §8 · p. 19
  • The authors conclude that Łukasiewicz neural networks provide a mature framework for interpretable neuro-symbolic learning, integer-weight training strategies, and deeper residual architectures.The framework rests on an exact correspondence between integer-weight neurons and logical connectives.Source: §8 · p. 19
Loading 2609.33028v1…