PaperlayerView exact revision on arXiv

Source-linked AI summary

CRISP: A Framework for Clause-Reconstructed Interpretable NeuroSymbolic Propositions

Alex Chan, Shafi Muhtasim Chowdhury, Ekin Can Erkuş, Ole-Christoffer Granmo, Alex Yakovlev, Rishad Shafik

arXiv:2610.02431v1cs.LG

TL;DR

Neural decisions are difficult to audit when evidence is encoded in hidden activations, and CRISP reconstructs a binary teacher’s LLAV as neuron-wise Tsetlin Machine clauses over Booleanized inputs. Across the reported scope, one-bit quartile binning offers the strongest fidelity-cost balance most clearly on SVHN, while reconstruction remains a representation-level audit rather than full teacher transparency.

  • Problem

    CRISP addresses whether binary neural teachers’ internal firing patterns can be approximated by symbolic clauses, a question motivated by the difficulty of auditing decisions encoded in hidden activations.

    Source: §I · p. 1

  • Method

    CRISP sign-binarizes a teacher’s last-layer activations and trains one ITM per hidden coordinate to reconstruct those bits as clauses over Booleanized inputs.

    Source: §I · p. 1, §II · p. 2, C. Input Booleanization · p. 3

  • Results

    Across the reported scope, one-bit quartile binning gives the strongest fidelity-cost balance, most clearly on SVHN; the tested BNN loses no teacher-head accuracy from LLAV sign-binarization alone.

    Source: §V · p. 8

  • Takeaways & Limitations

    CRISP makes the teacher’s final hidden interface inspectable as clauses, but is a representation-level reconstruction method rather than a claim of full teacher transparency.

    Source: §I · p. 2, E. Interpretability Scope and… · p. 4

    Experiments use one training run per configuration, precluding multiseed repetition and formal uncertainty estimation; the 0.04-point CIFAR10 quartile-over-binary advantage is within run-to-run variation and is not claimed as meaningful.

    Source: E. Limitations · p. 7

Abstract

from arXiv · show

Deep neural networks achieve high accuracy through layered numerical transformations, yet their decisions remain difficult to audit because decision evidence is encoded in hidden activations rather than explicit rules. This paper introduces CRISP, a framework that reconstructs the last-layer activation vector (LLAV) of binary neural teachers as Tsetlin Machine (TM) clauses. CRISP sign-binarizes the teacher's penultimate pre-logit activations, and assigns one Individual TM (ITM) to each LLAV neuron. Each reconstructed hidden bit is represented by propositional clauses over Booleanized input features, which gives a direct symbolic trace from named input thresholds to a named teacher neuron. CRISP is evaluated on MNIST, KMNIST, FashionMNIST (FMNIST), SVHN, and CIFAR10 using a BinaryConnect convolutional neural network (BCCNN) teacher and a fully binary neural network (BNN) teacher, with an additional study on binary thresholding, thermometer encoding, and quartile binning at multiple bit depths. The results show that LLAV sign-binarization does not reduce teacher-head accuracy in the tested BNN setting, while ITM reconstruction error is the main limiting factor. Quartile one-bit Booleanization gives the strongest reconstruction fidelity on SVHN at 87.52% test fidelity and is competitive on CIFAR10, and the reconstructed LLAV preserves 78.41% teacher-head accuracy on FMNIST. Pooled clause-evidence visualizations show that the learned ITM literals concentrate on the object region in centered benchmarks. CRISP therefore provides a clause-level route for inspecting the final hidden representation of binary neural teachers.

I. INTRODUCTION

CRISP addresses the difficulty of auditing neural decisions encoded in hidden activations by reconstructing the teacher’s last-layer activation vector as inspectable symbolic clauses. The paper evaluates neuron-wise reconstruction across binary teacher architectures, image benchmarks, and input Booleanization schemes.

I. INTRODUCTION

  • CRISP assigns one Individual Tsetlin Machine (ITM) to each teacher hidden neuron, reconstructing LLAV bits as clauses over Booleanized input features.This preserves neuron identity and exposes literals supporting or opposing each reconstructed firing state.Source: §I · p. 1
  • CRISP tests whether symbolic clauses can approximate binary neural networks’ internal firing patterns and asks how Booleanization and teacher architecture affect reconstruction fidelity.The research questions also ask whether successive neural transformations leave pre-logit activations reconstructable.Source: §I · p. 1
  • Experiments use BNN and BCCNN teachers across MNIST, KMNIST, FMNIST, SVHN, and CIFAR10, with encoding comparisons on SVHN and CIFAR10.The Booleanization study compares binary thresholding, thermometer encoding, and quartile binning at multiple bit depths.Source: §I · pp. 1–2
  • The framework is intended as a representation-level audit tool that exposes the teacher’s final hidden interface rather than replacing its classifier head.It targets the LLAV because that is the representation consumed by the binary classifier head.Source: §I · p. 2
  • CRISP evaluates both clause-level reconstruction and whether reconstructed hidden states retain class-relevant information when passed through the teacher’s own head.Fixed ITM clauses can be inspected across samples, classes, and model variants, while the injection test assesses representational sufficiency.Source: §I · p. 2

II. CRISP: PROPOSED FRAMEWORK

CRISP trains a binary neural teacher, extracts and sign-binarizes its LLAV, Booleanizes the original input, and fits one ITM per LLAV coordinate. The resulting clause-based surrogate preserves the correspondence between hidden neurons and symbolic reconstructions.

II. CRISP: PROPOSED FRAMEWORK

  • CRISP’s two-stage workflow first trains a binary teacher and extracts its sign-binarized LLAV, then trains one ITM per neuron from Booleanized inputs.Each reconstructed bit is represented by inspectable clauses over input features.Source: §II · p. 2
  • The teacher is frozen after training; each ITM independently learns one LLAV coordinate, and the bank’s outputs form the reconstructed hidden vector.The reported 256-neuron configuration therefore comprises 256 binary reconstruction tasks.Source: §II · p. 2
  • At inference, Booleanized inputs pass through the ITM bank to produce a reconstructed LLAV that can be inspected directly.The framework reconstructs the hidden representation rather than introducing a separately trained classifier.Source: §II · p. 2

A. Relation to Prior Work

  • CRISP differs from attribution, local-surrogate, concept-based, and visualization approaches by making a named hidden coordinate a persistent symbolic object reusable across samples.Unlike output-distribution distillation, it reconstructs the hard sign-binarized pre-logit LLAV neuron by neuron; related symbolic regression targets final hidden activations.Source: A. Relation to Prior Work · p. 2

B. Problem Setting and Teacher Extraction

The teacher’s penultimate pre-logit activations are sign-binarized coordinate-wise to form the LLAV, which the ITM bank seeks to reconstruct. Reconstruction fidelity measures agreement with those teacher bits, while an injection test separately assesses their usefulness for classification.

B. Problem Setting and Teacher Extraction

  • Each LLAV bit is formed by thresholding its penultimate pre-activation at zero, yielding a binary hidden state of 256 coordinates in the reported experiments.The teacher’s convolutional, pooling, and fully connected transformations precede this coordinate-wise sign operation.Source: B. Problem Setting and Teacher… · p. 3
  • Reconstruction fidelity averages per-neuron agreement between ITM outputs and teacher LLAV bits, whereas the injection test separately evaluates classification usability.This distinguishes symbolic reproduction of hidden activations from whether those activations retain class-relevant information.Source: B. Problem Setting and Teacher… · p. 3
  • In the tested BNN setting, sign-binarization does not reduce teacher-head accuracy because the final binary output layer applies a sign operation before classification.The experiment therefore treats the LLAV sign pattern as the operative interface to reconstruct.Source: B. Problem Setting and Teacher… · p. 3

C. Input Booleanization

CRISP Booleanizes input features because TM clauses operate on Boolean literals. The encoding determines the retained input information and the clause vocabulary, with threshold, thermometer, and quartile schemes evaluated at multiple bit depths.

C. Input Booleanization

  • CRISP compares binary thresholding, thermometer encoding, and quartile binning at bit depths of 1, 2, and 4 because ITM clauses require Boolean inputs.The chosen encoding determines both the information retained from each feature and the clauses the ITM can form.Source: C. Input Booleanization · p. 3
  • Thermometer encoding represents ordinal thresholds with literals, but its top bit is always zero, leaving nbits−1 informative bits.Each bit tests whether the quantized level exceeds a threshold.Source: C. Input Booleanization · p. 3
  • Quartile binning uses empirical feature quantiles rather than uniform intensity levels, with thresholds fitted on training data and reused unchanged for validation and test data.For one-bit binning, the threshold is the training-set median.Source: C. Input Booleanization · pp. 3–4

D. ITM Bank Training and Inference

CRISP trains one Individual Tsetlin Machine (ITM) per teacher hidden neuron to reconstruct that neuron’s LLAV bit from Booleanized input. Each output can be traced to active propositional clauses over input literals.

E. Interpretability Scope and Computational Cost

CRISP is a representation-level reconstruction method: the ITM bank is symbolic, but the teacher remains a learned neural model. Its clauses and pooled literal evidence provide an inspection path over named input features.

  • CRISP reconstructs the teacher’s LLAV rather than making the full teacher transparent; the teacher remains a learned neural model.Source: E. Interpretability Scope and… · p. 4
  • Booleanization gives each input bit a known feature-threshold meaning, so clause literals can be inspected as statements about named features rather than anonymous indices.Source: E. Interpretability Scope and… · p. 4
  • Pooled clause evidence maps surviving literals across all ITMs back to input coordinates, distinguishing clause support or opposition from literal presence or negation.Under thermometer encoding, the always-false top threshold bit’s negation is excluded from pooled maps.Source: E. Interpretability Scope and… · p. 4
  • Training has an offline cost whose dominant term comes from evaluating clause literals across the ITM bank; the work is parallel across independently trained LLAV coordinates.The modular inference and inspection design allows individual reconstructed bits to be evaluated, inspected, or replaced without retraining the bank.Source: E. Interpretability Scope and… · p. 4

III. EXPERIMENTAL SETUP

The experiments use five ten-class image benchmarks and two binary teacher variants. A fixed training protocol establishes baseline reconstruction and separately compares input Booleanization strategies on SVHN and CIFAR10.

A. Data

  • Five benchmarks span grayscale digits, handwritten characters, clothing, street-view digits, and color objects, with dataset sizes and splits summarized in Table I.Source: A. Data · p. 4, Table I · p. 4

B. Teacher Architectures

  • Both teachers use six binary convolutional blocks, three max-pooling stages, four binary fully connected blocks, and a binary output layer; the penultimate layer yields a 256-dimensional pre-activation vector sign-binarized as the LLAV.Source: B. Teacher Architectures · p. 4
  • The BCCNN binarizes weights but retains ReLU activations, whereas the BNN binarizes both weights and hidden activations, separating weight-only from fully binary hidden computation.Source: B. Teacher Architectures · p. 4

C. Training Protocol

  • The protocol first evaluates baseline 8-bit thermometer encoding across five datasets, then tests binary thresholding, thermometer encoding, and quartile binning on SVHN and CIFAR10.ITMs train for five epochs with fixed hyperparameters, and data-dependent thresholds are fitted on the training split only.Source: C. Training Protocol · p. 5

D. Evaluation Metrics

CRISP evaluates held-out reconstruction fidelity separately from teacher classification accuracy, and its baseline results show that reconstruction fidelity does not track teacher accuracy monotonically. The results also vary with dataset difficulty and include a retrained BNN/CIFAR10 teacher.

D. Evaluation Metrics

  • Reconstruction fidelity is held-out mean per-neuron bit accuracy, while ITM mean accuracy averages per-neuron training accuracy and is used only for selection.Teacher accuracy contextualizes representation informativeness; the injection test also reports Cohen’s κ and MCC.Source: D. Evaluation Metrics · p. 5

A. Baseline Performances Across Datasets

  • The BNN/CIFAR10 teacher is a retrained instance because the original peer-review run was unrecoverable.Source: A. Baseline Performances Across Datasets · p. 5
  • Held-out reconstruction fidelity exceeds majority-class baselines on most settings, with the low-accuracy BNN/SVHN teacher near its majority baseline.The cited majority baselines range roughly from 61% to 74%, with a mean of 65%.Source: A. Baseline Performances Across Datasets · p. 5
  • Across comparable datasets, BCCNN teacher accuracy is higher, but BNN reconstruction fidelity is comparable or higher on FMNIST and SVHN despite lower or similar task accuracy.The results distinguish recoverability of the hidden code from the teacher’s classification quality.Source: A. Baseline Performances Across Datasets · pp. 5–6, Table II · p. 5
  • MNIST-family datasets generally show higher reconstruction than colour-image settings; KMNIST has the lowest fidelity relative to teacher accuracy and loses the most head accuracy in the injection test.The reported discussion attributes the broader pattern to class evidence being more or less aligned with local pixel-threshold clauses.Source: A. Baseline Performances Across Datasets · p. 5

B. Effect of LLAV Binarization and Reconstruction Error

In the tested BNN setting, sign-binarizing the LLAV does not reduce teacher-head accuracy, while ITM reconstruction does. The observed accuracy loss is attributed to imperfect reconstruction of the LLAV sign pattern.

B. Effect of LLAV Binarization and Reconstruction Error

  • The three-way injection test used the BNN and 8-bit thermometer inputs on MNIST-family datasets near the teacher’s accuracy ceiling; lower-accuracy SVHN and CIFAR10 settings remain untested.Source: B. Effect of LLAV Binarization and… · p. 6
  • Sign-binarization incurs no accuracy cost in the reported rows, whereas ITM reconstruction reduces accuracy across all reported datasets, most on KMNIST.The teacher head applies a sign operation before its linear transform, discarding magnitude information at this interface.Source: B. Effect of LLAV Binarization and… · p. 6
  • A flipped LLAV coordinate changes each class logit by a fixed signed amount, so accumulated errors across 256 bits can change the class margin and cause misclassification.Source: B. Effect of LLAV Binarization and… · p. 6

C. Effect of Input Booleanization

One-bit Booleanization performs as well as or better than wider encodings in the reported reconstruction study, while clause-evidence pooling yields spatially coherent maps over image regions. Literal counts require care: thermometer encoding can inflate negated-literal counts for background pixels.

C. Effect of Input Booleanization

  • 87.52% test fidelity is the strongest one-bit SVHN result, from quartile binning; CIFAR10 one-bit schemes are effectively tied at 92.12% versus 92.08%.Across one-bit schemes, fidelity ranges from 85.70% to 87.52% on SVHN and from 92.04% to 92.12% on CIFAR10.Source: C. Effect of Input Booleanization · p. 6
  • Wider thermometer and quartile encodings usually lower fidelity than their one-bit settings, as added dimensions enlarge the clause search space under a fixed training budget.Binary thresholding is tested at one bit; thermometer and quartile encodings are tested at one, two, and four bits.Source: C. Effect of Input Booleanization · p. 6

D. Explainable Symbolic Evidence from ITMs

  • Pooled evidence from all 256 ITMs forms spatially coherent maps despite flat pixel-threshold inputs and no explicit spatial prior.The figure maps positive and negated literals to image coordinates, with color intensity denoting pooled literal count within each row.Source: Figure 2 · p. 7, D. Explainable Symbolic Evidence from… · p. 7

C. Effect of Input Booleanization

  • On SVHN, four-bit quartile inference takes 1056.01s versus 275.65s at one bit, while richer Booleanization does not automatically improve fidelity.The paper identifies one-bit quartile binning as the best reported fidelity-cost trade-off.Source: C. Effect of Input Booleanization · p. 7

D. Explainable Symbolic Evidence from ITMs

  • For the MNIST “0,” positive literals fill both the interior hole and exterior background, while FMNIST evidence follows the T-shirt silhouette.Source: D. Explainable Symbolic Evidence from… · p. 7, Figure 2 · p. 7
  • Thermometer encoding can inflate negated-literal counts for near-zero background pixels, so positive and negative map magnitudes do not directly measure relative importance.Foreground pixels activate a narrower threshold band; negative-polarity clauses contribute less evidence in the two displayed samples.Source: D. Explainable Symbolic Evidence from… · p. 7

E. Limitations

Within the reported scope, reconstruction is harder on complex colour datasets, and one-bit quartile binning gives the strongest fidelity-cost balance. Single-run experiments and substantial CIFAR10 training time limit uncertainty assessment; proposed extensions include broader layers and domains.

E. Limitations

  • All configurations use one training run; a full CIFAR10 run for the 256-ITM bank exceeds 24 hours, precluding multiseed repetition and formal uncertainty estimates.Source: E. Limitations · p. 7

V. CONCLUSION

  • One-bit quartile binning gives the best fidelity-cost balance, reconstruction is harder on complex colour datasets, and teacher architecture affects task accuracy and reconstructability independently.The reported conclusion identifies the quartile result most clearly on SVHN.Source: §V · p. 8

VI. FUTURE WORK

  • Proposed extensions include intermediate-layer reconstruction, satisfiability-based guarantees, larger colour benchmarks, and applications to tabular, sensor, biomedical, and time-series data.Source: §VI · p. 8
Loading 2610.02431v1…