PaperlayerView exact revision on arXiv

Source-linked AI summary

Soft Ontology Priors over Latent Concept Posteriors for Weakly Supervised Neuro-Symbolic Learning

Tengwei Song, Robert Hoehndorf

arXiv:2610.30391v1cs.AIcs.LG

TL;DR

Task-label-only neuro-symbolic learning can leave concept mappings unidentifiable because shortcuts preserve task labels. This paper adds a soft ontology prior to the latent-concept objective and compiles it with task constraints; experiments report improved concept recovery and shortcut resolution, while identifying reliability limits for targeted axioms.

  • Problem

    Task-label-only NeSy predictors can reach near-perfect accuracy with shortcut concept mappings, leaving recovered concepts unidentifiable and unsafe for downstream reuse.

    Source: §1 · p. 1

  • Method

    The method combines a soft axiom-level prior over latent concepts with hard task structure in a normalised compiled-circuit objective.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    On BDD-OIA-6c, Ours gives the best non-Oracle raw concept F1, with a small but significant ten-seed gap; targeted ontology information also resolves most MNIST-Add shortcut modes.

    Source: §5.2 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    Task accuracy alone remains insensitive to concept recovery, and the reported gains require ontology information that resolves ambiguity left by task-label supervision.

    Source: §1 · p. 2, §6 · p. 10

    Targeted shortcut axioms can bias the decoder, and fixed weights with one global β cannot identify unreliable or incomplete knowledge in this label-only setting.

    Source: §6 · p. 10, §5.4 · p. 10

Abstract

from arXiv · show

Neuro-symbolic (NeSy) predictors that supervise only the task label suffer from reasoning shortcuts: distinct concept mappings achieve the same maximum likelihood, leaving the recovered concepts unidentifiable and unsafe for downstream reuse. We attack this failure by injecting a soft ontology prior over the latent concepts into the training objective and compiling it together with the task constraint into a smooth, decomposable probabilistic circuit. Under independent Bernoulli perception, the normalisers of the resulting structured concept posterior reduce to weighted model counts that are linear in the compiled circuit size, because the neural leaf weights fold into the circuit before evaluation. The soft compilation strategy places expert axioms in the circuit's weights rather than its support (task-structural axioms guaranteed by the data-generating process stay hard), so noisy expert axioms only redistribute prior mass and cannot empty the support. We characterise how the prior reweights label-restricted shortcut mass through differences in ontology violation energy, show that soft priors incur bounded log-density perturbations under bounded axiom noise, and prove that the hard-support counterpart can be made degenerate by a single corrupted axiom. We validate these predictions on an audited six-concept view of the real-world BDD-OIA driving benchmark and on controlled NeSy tasks.

1. Introduction

Task-label-only NeSy models can achieve high accuracy through concept mappings that satisfy the symbolic program for the wrong reasons, leaving recovered concepts unreliable for reuse. This paper adds a soft axiom-level ontology prior to the training objective, preserving the factorised perception model while avoiding the support fragility of hard ontology constraints.

1. Introduction

  • Task-label-only NeSy predictors can learn label-equivalent concept shortcuts that are invisible at the label level and undermine out-of-distribution generalisation and safety.The recovered human-readable concepts are intended for reuse, verification, and explanation, but shortcuts undermine that promise.Source: §1 · p. 1
  • The approach differs from perception-side interventions by injecting domain knowledge into the loss surface rather than changing the perception model.The cited interventions use ensembling or a masked diffusion posterior, but leave the training objective unchanged.Source: §1 · pp. 1–2
  • The paper modifies the training objective with a soft prior over latent concepts while leaving factorised perception unchanged.Expert axioms downweight worlds rather than deleting them, so noisy or incomplete ontology knowledge does not impose a hard support restriction.Source: §1 · p. 2
  • The contributions combine a structured concept posterior, a linear-time normalised WMC objective, shortcut reweighting, and an analysis of corruption sensitivity.The paper reports experiments on real-world and controlled benchmarks, with gains requiring ontology information that resolves ambiguity left by task-label supervision.Source: §1 · p. 2

2. Related Work

Prior work addresses shortcuts through perception architectures, tractable circuits, or output constraints, while this paper places a soft ontology prior over latent concepts inside the training objective. Its distinction is to impose structured constraints through an exact normalised likelihood rather than moment regularisation or hard output consistency.

Reasoning shortcuts in neuro-symbolic learning

  • BEARS and NeSy-DM target perception-side shortcuts, whereas this work folds a soft ontology prior into the weighted circuit and leaves factorised perception untouched.BEARS uses a KL-repulsion ensemble; NeSy-DM uses a masked discrete diffusion model capable of concept correlations.Source: §2 · p. 2

Probabilistic circuits for symbolic constraints

  • Probabilistic circuits support linear-time weighted model counting, and SPL uses them to compile task constraints for exact likelihood training.Lazzari et al. (2026) compile ontologies to guarantee output consistency in supervised multi-label classification, unlike this paper's latent-concept setting.Source: §2 · pp. 2–3

Posterior regularisation and constrained inference

  • Posterior regularisation penalises posterior moments, while this method compiles constraints into a weighted circuit and enforces them in an exact normalised likelihood.The circuit formulation avoids moment matching and obtains exact gradients through leaf reweighting.Source: §2 · p. 3

Constraint-driven neural learning

  • Other constraint-driven approaches use symbolic knowledge as output-level constraints or regularisers, while this paper uses a prior over latent concepts.The compiler targets a restricted EL-style language with small Boolean and cardinality macros.Source: §2 · p. 3

3. Preliminaries and Problem Setting

The setting is a factorised neural concept predictor whose symbolic program maps latent concepts to observed task labels; label-only likelihood can leave distinct concept mappings equivalent. The paper uses compiled probabilistic circuits to express task constraints and ontology information over these latent concepts.

Latent-concept NeSy predictors

  • A factorised Bernoulli perception model maps inputs to latent concepts, and a fixed symbolic program maps those concepts to the observed label.Training by semantic loss sums the perception probability over concept assignments consistent with the label.Source: §3 · p. 3
  • Reasoning shortcuts are non-identity deterministic concept maps that preserve labels on the support of the data-generating distribution.The factorised optimum can concentrate on deterministic Bernoullis, with gradient descent selecting among disconnected vertex flats.Source: §3 · p. 3

Probabilistic circuits and the EL ontology fragment

  • Smooth, decomposable, deterministic probabilistic circuits support linear-time WMC, and SPL uses them to compile task constraints for semantic loss.The ontology language includes EL-style inclusions and small Boolean and cardinality macros such as SubClass, Implies, Disjoint, and ExactlyOne.Source: §3 · pp. 3–4
  • Figure 1 contrasts hard support removal with soft edge reweighting and shows a forward/backward pass using leaf reweighting and two LogWMC evaluations.The caption identifies the illustrated Disjoint(r, g) axiom.Source: Figure 1 · p. 4

Worked two-concept example

  • For the red/green light example, the hard Disjoint axiom removes the joint-activation world, whereas the soft version retains it with factor e^-βλ.The example's stop label is defined by r; a further hard axiom excluding (0, 0) can make a label-restricted partition zero, while the soft variant only rescales a violating world.Source: §3 · p. 4

4. Circuit-Regularized Concept Posteriors

The method separates hard task-structural axioms from soft ontology penalties, then combines the ontology, task constraint, and neural evidence in a normalised structured concept posterior. Compiled circuits make its partition functions tractable and expose how soft priors reweight shortcuts while remaining less brittle than hard support under bounded corruption.

4.1. Soft Ontology Prior

  • Task-structural axioms guaranteed by the data-generating process remain hard, while benchmark and expert axioms may be soft or hardened in the HardPC⋆ baseline.Soft axioms downweight worlds through non-negative violation costs; β controls their strength, with β = 0 ignoring penalties and β → ∞ turning positive violations into exclusions.Source: §4.1 · pp. 4–5
  • The structured concept posterior combines task-consistent assignments, neural probabilities, and ontology scores, with training using log Zy/Zall rather than the improper surrogate log Zy alone.The all-label normaliser depends on θ, so it is required for the normalised likelihood.Source: §4.1 · p. 5

4.2. Tractable Compilation, Training, and Identifiability

  • Under compatible smooth, decomposable, deterministic compilation, injecting independent Bernoulli leaf evidence yields both partitions with WMC linear in the materialised circuit sizes.Circuits are compiled once per ontology/task pair; each training step injects neural evidence and evaluates two bottom-up logsumexp WMCs.Source: §4.2 · pp. 5–6
  • For two label-preserving shortcut maps, the ratio of ontology-weighted scores depends exponentially on β times their difference in expected violation energy.The shortcut energy is its expected weighted soft-axiom violation under the true distribution.Source: §4.2 · p. 6
  • At deterministic label-preserving point masses, Zy equals Zall; in the hard limit, any map with positive violation energy loses label-restricted support.Thus the soft reweighting factor does not distinguish such point masses at the exact deterministic limit.Source: §4.2 · p. 6
  • A bounded per-world energy perturbation ε limits soft-prior log-density change to 2βε, while one corrupted hard axiom can exclude positive-probability worlds and yield infinite KL.The hard collapse also produces a vanishing gradient on pθ.Source: §4.2 · p. 6

5. Experiments

Across the benchmarks, the soft-prior method improves concept recovery, most clearly when the prior supplies information hidden by task labels. Ablations and corruption tests show that recovery depends on informative prior content and objective normalization, while targeted shortcut injection can still bias the decoder.

5. Experiments

  • The evaluation covers an audited real-world BDD-OIA-6c view, controlled MNIST-Add, sparse-training MNIST-Half, and MNLogic-3bit-CNF, with ablations varying prior richness, strength, normalization, and axiom use.The primary concept metric is raw-head macro F1; results use ten random seeds.Source: §5 · p. 6, §5.1 · p. 7

5.2. Main benchmark results

  • On BDD-OIA-6c, Ours has the best non-Oracle raw concept F1, with a significant ten-seed gap (paired t p < 0.05; bootstrap 95% CI [0.009, 0.053]).All non-Oracle methods reach similar task accuracy.Source: §5.2 · p. 7, Table 1 · p. 8
  • MNLogic-3bit-CNF is less discriminative because several concept maps are nearly exchangeable, yielding high seed variance; results are consistent with using non-guaranteed benchmark CNF priors softly.Source: §5.2 · p. 8
  • On MNIST-Add, the level-3 ordering prior gives the largest concept-F1 improvement because it supplies left/right information hidden by the sum label.When that information is removed in L2/L1, gains shrink.Source: §5.2 · p. 8, Table 2 · p. 8
  • Ours reaches a near-oracle structured concept map in 9/10 MNIST-Add runs, while DPL‡, SPL†, HardPC⋆, and BEARS§ (k=1) reach the threshold in none.The threshold is structured concept F1 above 0.85; the raw perception head improves less than the structured decoder.Source: §5.2 · pp. 8–9, Table 3 · p. 9

5.3. Ablation study

  • Removing the MNIST-Add ordering prior causes the largest prior-richness drop, while removing parity leaves performance close to the empty-prior setting.On BDD-OIA-6c, removing the Zall denominator causes the largest concept-F1 collapse despite flat task accuracy; disabling soft penalties or hardening expert axioms causes smaller drops.Source: §5.3 · p. 9

5.4. Robustness to noisy ontologies

  • Random deletion of soft axioms gradually lowers structured concept F1, while targeted shortcut injection biases the decoder toward the shortcut despite violating worlds remaining in support.The targeted corruption lies outside the bounded random-deletion model and is identified as the main soft-compilation failure mode.Source: §5.4 · pp. 9–10, Figure 2 · p. 10

6. Conclusion

Soft ontology priors are incorporated into a label-normalised WMC objective, with hard support and soft penalties evaluated through compiled circuits. The conclusion reports improved concept recovery on BDD-OIA-6c and resolution of most MNIST-Add shortcut modes, while task accuracy remains insensitive.

6. Conclusion

  • Targeted ontology information improves concept recovery on BDD-OIA-6c and resolves most MNIST-Add shortcut modes, while task accuracy alone remains insensitive.The label-normalised WMC objective combines hard support and soft penalties for tractable posterior inference.Source: §6 · p. 10

A.1. Proof of Proposition 1

  • The joint circuit enumerates worlds satisfying hard ontology support and the task constraint, with fixed weights encoding the soft ontology score.Neural literal evidence is injected before evaluating the circuit root.Source: §A.1 · p. 14
  • One bottom-up pass evaluates the circuits in O(|C_y^{O,φ}| + |C_O|) time, and the log-space recurrences are differentiable.Decomposability, smoothness, and determinism support propagation without double-counting worlds.Source: §A.1 · p. 14

A.2. Proof of Proposition 2

For label-preserving deterministic maps, the proof compares label-restricted ontology scores and derives exponential down-weighting from their ratio. The claimed prior-induced bias applies before the posterior becomes an exact one-hot assignment.

A.2. Proof of Proposition 2

  • The ratio of label-restricted ontology scores for two label-preserving maps gives the stated exponential down-weighting.The proof takes population expectations using the fact that its indicator is one on the data support.Source: §A.2 · p. 14
  • The proposition describes prior-induced score and gradient bias during non-degenerate training and structured decoding, not conditional-likelihood odds at an exact one-hot vertex.At a label-preserving point mass, the same assignment is the only non-zero term in both partitions, so the soft scores cancel.Source: §A.2 · p. 14
  • In the hard limit β → ∞, any label-preserving map with positive ontology violation receives zero label-restricted support.Source: §A.2 · p. 14

A.3. Proof of Proposition 3

The proof bounds soft-prior log-density changes under bounded energy perturbations and contrasts this with hard-support collapse under a corrupted axiom. The compilation appendix describes circuit construction, training evaluation, and benchmark circuit sizes.

A.3. Proof of Proposition 3

  • A per-world energy perturbation bounded by ε yields a log-density perturbation bounded by 2βε, with ε = O(ρK∆) under the stated per-axiom noise model.The corresponding KL bound is also at most 2βε.Source: §A.3 · pp. 14–15
  • A single flipped hard axiom excluding a world with positive data probability can make its prior probability zero, diverge KL, and eliminate the gradient at that world.Source: §A.3 · p. 15

Appendix B. Compilation Details

  • The training pass injects neural literal log-likelihoods into the circuits and returns the normalized loss −log Z_y + log Z_all.Axioms are compiled over local scopes, shared-variable components are conjoined, and the task circuit is filtered through the ontology circuit.Source: Appendix B · p. 15
  • Across the reported benchmarks, the WMC portion requires a few thousand log-space operations per example and was not the memory bottleneck relative to neural passes.Task-specific enumeration keeps the materialized circuits small.Source: Appendix B · pp. 15–16

Appendix C. Datasets and Prior Specifications

The appendix specifies the concept vocabularies, ontologies, and data splits used across the benchmarks. It distinguishes task-induced and benchmark-provided priors from audited expert priors.

BDD-OIA-6c concept vocabulary and stop rule

  • BDD-OIA-6c uses six audited binary traffic concepts and a deterministic stop rule, with pre-training rule–label agreement of 0.971.The concept mapping requires candidate expert axioms to have below 5% violation on real concept labels and rule-derived labels to agree with RSBench test labels on at least 95%.Source: Appendix C · p. 16

MNIST-Add ontology richness levels

  • MNIST-Add’s ontology levels progress from no prior (L0) to one-hot constraints (L1), parity auxiliaries (L2), and a soft ordering prior (L3).The ordering prior uses non-negative weights for digit pairs where the first digit exceeds the second, with smaller digits assigned to position 1.Source: Appendix C · p. 16

MNIST-Half splits and ontology

  • MNIST-Half restricts the MNIST-Add ontology to five digits and trains on seven ordered pairs, with the 18 complementary pairs forming the OOD split.The in-distribution pairs are (0, 0), (0, 1), (1, 0), (2, 3), (3, 2), (2, 4), and (4, 2).Source: Appendix C · p. 16

MNLogic prior and splits

  • MNLogic treats its CNF as a soft distribution prior: 80% of samples come from the three in-distribution combinations and 20% violate the CNF.The OOD split draws from the seven remaining combinations; a hard-support encoding would be infeasible for the training data.Source: Appendix C · p. 16

Prior provenance

  • Prior provenance distinguishes task-induced digit schemas, benchmark-provided CNFs, and expert traffic rules used only in the audited six-concept view.The primary 21-concept BDD-OIA view has no expert prior because its test-label audit found no supported annotation invariant; per-split sample counts are listed for each benchmark.Source: Appendix C · pp. 16–17, Table 5 · p. 17

Appendix D. Implementation Details

Implementation uses benchmark-specific perception networks and training schedules, with Adam optimization and fixed batch sizes.

Networks

  • BDD-OIA-6c uses a two-layer MLP over 2048-dimensional features, while MNIST tasks use a shared two-convolution LeNet with 128-dimensional features.The LeNet is shared across the two digit positions for MNIST-Add and across three binary digit positions for MNLogic.Source: Appendix D · p. 17

Optimization

  • All benchmarks use Adam with learning rate 10^-3 and zero weight decay.The reported sweeps specify the selected β or prior-weight setting in their corresponding table.Source: Appendix D · p. 17

Training schedule

  • BDD-OIA-6c, MNIST-Add, and MNLogic train for 30 epochs; MNIST-Half trains for 60 epochs following the BEARS sparse-training protocol.Batch size is 256 for BDD-OIA-6c and 128 for the other benchmarks; each epoch processes the fixed BDD-OIA split or 30,000 generated or paired samples, depending on the benchmark.Source: Appendix D · p. 17

Appendix E. Additional Experimental Results

Additional experiments compare sparse concept supervision with ontology-only training and examine OOD diagnostics, prior strength, and structured decoding. The reported results distinguish direct perception-head concept scores from label-conditioned circuit decoding.

Additional diagnostics

BDD-OIA

  • BDD-OIA-6c uses an audited six-concept traffic view with rule–label accuracy of 0.971 before training; the 21-concept view remains a no-prior baseline.The audit rejected unsupported annotation invariants on real test labels, so no expert claims are encoded in the 21-concept view.Source: Appendix E · p. 18

Concept-supervision sweep

  • Even λsup = 1 far exceeds the best ontology-prior-only result, Ours-L3 with raw F1 0.498.The concept-supervision sweep is reported on MNIST-Add at L=1 and β=1.Source: Appendix E · p. 17, Table 8 · p. 19

Raw-θ vs. structured-posterior concept F1

  • On MNIST-Add, structured concept F1 is far higher than raw-θ F1 for every circuit-equipped method, especially Ours, whose ordering prior enables near-perfect circuit decoding.The predicted sum constrains the digit pair; the main tables use raw-θ F1 to avoid crediting concept recovery to label-conditioned decoding.Source: Appendix E · p. 18, Table 9 · p. 19

Related papers

Loading 2610.30391v1…