PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Faithfully Solve Ontology Engineering Tasks? A Mechanistic Interpretability Study

Bohui Zhang, Albert Meroño-Peñuela, Elena Simperl

arXiv:2610.38326v1cs.AI

TL;DR

Behavioral evaluations of LLMs’ ontological understanding do not explain the mechanisms behind their outputs, and prior mechanistic interpretability work has largely focused on instance-level facts. MIOE tests ontology-level predictions with interchange interventions across entity typing and subsumption prediction. The results indicate mechanistic accessibility but only partial disentanglement from class-specific information.

  • Problem

    Behavioral evaluations show whether ontology-related outputs are correct but do not reveal whether internal computations track the relevant relation.

    Source: Introduction · p. 1

  • Method

    MIOE uses interchange interventions on entity typing and subsumption prediction to test target-answer transfer alongside preservation of class-specific auxiliary information.

    Source: Introduction · p. 1, Task Formulation · p. 3

  • Results

    Ontology-level predictions are mechanistically accessible but only partially disentangled; multi-task objectives and contiguous layer-window interventions improve preservation and intervention quality.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    Correct ontological predictions do not by themselves imply a clean or modular internal mechanism, and current methods do not cleanly separate ontology variables from broader class-specific information.

    Source: Conclusion · p. 7

    The analysis calls for broader model families, richer ontology constraints, and tests across domains, prompts, and complex reasoning contexts.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Ontological knowledge underpins large language model (LLM) performance on knowledge-intensive tasks. However, existing evaluations of LLMs' ontological understanding largely rely on behavioral correctness, leaving the underlying mechanisms unexplained. Existing mechanistic interpretability work has largely focused on instance-level factual knowledge, leaving ontological knowledge under-explored. To bridge this gap, we introduce MIOE, a causal-abstraction benchmark that evaluates interchange interventions for two ontology engineering tasks: entity typing and subsumption prediction. MIOE tests whether learned internal intervention targets can change an ontology-level target answer while preserving class-specific auxiliary information. Across Qwen3-8B and Qwen3-4B, we evaluate SAE-, DBM-, and DAS-based methods, including multi-task and multi-layer variants. Distributed subspace methods give the strongest results and generalize across prompt settings, but the induced changes remain only partially separated from auxiliary class-specific information. Multi-task objectives and contiguous layer-window interventions improve this balance, while class-level analyses reveal substantial variation across ontology classes. These results suggest that ontological predictions are mechanistically accessible in LLMs, although current methods do not yet isolate a cleanly disentangled causal mechanism for ontological knowledge.

Introduction

Behavioral evaluations show whether LLMs produce correct ontology-related outputs, but not whether their internal computations track the relevant relation. MIOE tests ontology-level predictions through causal interventions while measuring preservation of class-specific information.

Introduction

  • Behavioral correctness evaluations do not reveal whether internal computations produce ontology-related answers by tracking the relevant relation.The paper frames this as a question of faithfulness: whether an output reflects the internal process that produced it.Source: Introduction · p. 1
  • MIOE evaluates interchange interventions for entity typing and subsumption prediction, pairing target relations with class-specific auxiliary attributes and prompt variants.Its setup measures both effects on the target relation and preservation of non-target subject information.Source: Introduction · pp. 1–2
  • The study reports that ontology-level predictions are causally accessible but only partially disentangled, with multi-task objectives and contiguous multi-layer interventions improving intervention quality.The research questions also examine how method, representation site, prompt type, and ontological class affect intervention effects and auxiliary preservation.Source: Introduction · p. 2

Related Work

Prior mechanistic interpretability research spans sparse-feature analysis and causal abstraction, but benchmark evaluations have largely left high-level ontological knowledge under-explored. LLM ontology-engineering studies cover varied tasks and report uneven capabilities.

Mechanistic Interpretability

  • Interpretability work uses sparse autoencoders to decompose activations and causal abstraction to test whether high-level causal models align with neural computation.Benchmarks assess distinct interpretability desiderata, while prior evaluations largely focus on attributes, factual recall, or synthetic circuits rather than ontological knowledge.Source: Mechanistic Interpretability · p. 2

LLMs for Ontology Engineering

  • LLM ontology-engineering research spans tasks including subsumption inference, ontology learning, alignment, and evaluation.Prior studies report both successful extraction or generation of ontological knowledge and limitations on specialized tasks.Source: LLMs for Ontology Engineering · p. 2
  • Reported ontology-related performance is uneven: LLMs may succeed on some tasks while failing on others, suggesting incomplete and unevenly accessible ontological knowledge.Prior work also reports memorization of some ontological knowledge alongside implicit knowledge used for reasoning.Source: LLMs for Ontology Engineering · p. 2

The MIOE Benchmark

MIOE frames ontology engineering as a causal-abstraction test: an intervention should transfer a target taxonomic prediction from a source subject while preserving the base subject’s auxiliary predictions. Its dataset and evaluation vary subjects, prompts, and intervention behaviors.

Task Formulation

  • MIOE tests whether an internal target can transfer a source subject’s taxonomic prediction while preserving the base subject’s class-specific auxiliary predictions.It covers entity typing and subsumption prediction, using subjects whose target classes differ.Source: Task Formulation · pp. 2–3

Dataset

  • The dataset organizes subjects by target classes and class-specific auxiliary attributes, with 54 classes for instance of and 39 for subclass of.It is sampled and refined from Wikidata, WordNet, and schema.org.Source: Dataset · p. 3
  • Ontology templates provide controlled target and auxiliary queries, while Wikipedia-derived prompts test interventions in contextualized language.The Entity setting holds out subjects while reusing training templates; the Context setting holds out templates while reusing training subjects.Source: Dataset · p. 3

Evaluation

  • Cause measures transfer to the source subject’s target class, whereas Iso measures preservation of the base subject’s auxiliary value.Interventions are evaluated on retained prompts where the unmodified model already gives an accepted answer, and scores are macro-averaged across groups to prevent larger groups dominating.Source: Evaluation · pp. 3–4

Experiments

The experiments instantiate interchange interventions at residual-stream sites using feature-based, coordinate-mask, and distributed-subspace methods. SAE learns sparse latents before selecting target-related features, while DBM learns a sparse selector over residual coordinates.

Mechanistic Interpretability Methods

  • The study evaluates SAE, DBM, and DAS, their multi-task variants MDBM and MDAS, and extends DAS and MDAS to multi-layer interventions.Most methods intervene at one fixed layer and subject-token position; multi-layer variants use separate subspaces across a small layer window.Source: Mechanistic Interpretability Methods · p. 4
  • SAE first learns sparse latent representations from activations, then selects latent features associated with the target ontological relation.A sparse linear classifier identifies the feature set, and the intervention substitutes source values for selected features while retaining base values for the others.Source: Mechanistic Interpretability Methods · p. 4
  • DBM learns a sparse selector over residual-stream coordinates, using a relaxed mask for coordinate-wise interpolation between base and source activations.In the hard-mask limit, selected coordinates come from the source and unselected coordinates remain from the base.Source: Mechanistic Interpretability Methods · p. 4

LDBM =

DAS models an intervention target as a low-dimensional linear subspace and exchanges only the representation component within that subspace. The intervention leaves the orthogonal complement unchanged.

  • DAS parameterizes the target feature as a low-dimensional linear subspace rather than learning a full d × d rotation.Source: LDBM = · p. 4
  • The intervention replaces the base representation’s subspace component with the source representation’s corresponding component.Source: LDBM = · pp. 4–5
  • The orthogonal complement remains unchanged, and DAS is trained on retained Cause examples.Source: LDBM = · p. 5

LDAS =

Multi-task variants train interventions to replace target answers while preserving auxiliary answers. Multi-layer DAS and MDAS extend this approach across adjacent residual-stream layers, and the experiments compare configurations on Qwen3 models.

Multi-task DAS & DBM

  • MDBM and MDAS add auxiliary-invariance examples because the original objectives train target-answer changes without explicitly preserving auxiliary answers.MDBM uses a DBM intervention with an L1 penalty; MDAS uses DAS with no penalty.Source: Multi-task DAS & DBM · p. 5

Multi-Layer DAS & MDAS

  • Multi-layer DAS jointly optimizes layer-specific subspaces across an ordered set of adjacent residual-stream layers.Lower-layer interventions can affect later base states during the same forward pass.Source: Multi-Layer DAS & MDAS · p. 5

LML-DAS(ΘS) =

Setup

  • Experiments cover Qwen3-4B and Qwen3-8B; multi-layer DAS and MDAS use contiguous two- and four-layer windows on Qwen3-8B.Single-layer experiments evaluate all 36 layers; multi-layer configurations include 256 × 4, 512 × 2, and 512 × 4 subspaces.Source: Setup · p. 5

Results

Distributed subspace methods, especially MDAS, achieve the strongest reported single-layer Disentangle results across both axiom types. Multi-task and contiguous layer-window interventions improve the balance between target changes and auxiliary preservation, but outcomes remain layer- and class-dependent.

Results

  • MDAS achieves the highest single-layer Disentangle scores across both axiom types: 0.592/0.587 for instance of and 0.513/0.542 for subclass of on Entity/Context.The same method ordering holds for Qwen3-4B.Source: Results · p. 5, Table 2 · p. 6
  • Across Entity and Context, the best multi-task configurations improve Iso over their single-task method families by 0.081 on average while reducing Cause by 0.052.Disentangle improves because the preservation gain outweighs the reduction in Cause; increasing intervention dimension can instead improve Cause while reducing Iso in some settings.Source: Results · p. 5
  • Cause is weak in early layers, rises through middle layers, and often peaks in middle or middle-to-late layers; later layers can have strong Iso but very low Cause.The layer profiles compare single-layer MDAS with rank 2048 against four-layer MDAS with 256 × 4 contiguous windows.Source: Results · pp. 5–6, Figure 1 · p. 6
  • For Qwen3-8B instance of, four-layer windows reach 0.667 Disentangle with DAS 512 × 4 and 0.666/0.663 with MDAS 256 × 4/512 × 4.At total dimension 1024, 256 × 4 outperforms 512 × 2 for both DAS and MDAS; the multi-layer results compare against single-layer settings of the same method and total dimension.Source: Results · p. 6, Figure 2 · p. 6, Table 3 · p. 7
  • Intervention success varies substantially across ontology classes: concrete, regular-profile classes often score higher, while broader or heterogeneous classes show lower scores.Role gaps also vary: protein and video game are stronger as sources, while national park and tourist attraction are stronger as bases; similar class rankings recur across Qwen3-4B and Qwen3-8B.Source: Results · pp. 6–7, Figure 3 · p. 7
  • The results suggest ontology-oriented predictions are causally accessible through learned intervention targets, but separation from auxiliary class-specific information remains partial and class-dependent.Source: Results · p. 7

Conclusion

MIOE tests ontological knowledge through interchange interventions that target entity-typing and subsumption predictions while preserving class-specific auxiliary information. Experiments find these predictions mechanistically accessible but only partially disentangled, with intervention success varying across classes.

Conclusion

  • MIOE pairs target ontology variables in entity typing and subsumption prediction with class-specific auxiliary attributes to test target-answer changes while preserving non-target information.Source: Conclusion · p. 7
  • Ontology-level predictions are mechanistically accessible but remain only partially disentangled from class-specific information.Multi-task objectives and contiguous layer-window interventions improve auxiliary-answer preservation; intervention success also varies substantially across ontology classes.Source: Conclusion · p. 7
  • Correct ontological predictions do not by themselves establish a clean or modular internal mechanism, and future work should test richer constraints, broader model families, and stability across domains, prompts, and complex reasoning contexts.Source: Conclusion · p. 7

Dataset Details

The dataset samples diverse ontology classes and subjects while avoiding cases likely to fail behavioral filtering or introduce time-sensitive ambiguity. It distinguishes instance-level from class-level auxiliary attributes and provides materials for reconstructing benchmark instances.

Dataset construction

  • Subject selection balances broad domain and popularity coverage against avoiding obscure cases likely to fail behavioral filtering, retaining at least ten candidate entities or subclasses per class before filtering.Subjects are selected from common but diverse ontology classes, with popularity and structured-information proxies guiding entity and subclass sampling.Source: Dataset construction · p. 10
  • The construction avoids highly time-sensitive subjects and attributes, queries ground truth from Wikidata, cross-validates available Wikipedia infobox values, and manually checks conflicts or missing aligned evidence.Source: Dataset construction · p. 10
  • Auxiliary attributes are instance-level for entities and class-level for subclasses, with the latter including abstract values such as an academic discipline’s general study object.Source: Dataset construction · p. 10
  • The anonymous code-and-data appendix includes class files, attribute dictionaries, prompt templates, context prompts, and construction scripts; release is planned under a research-permissive license upon publication.Source: Dataset construction · p. 10

Implementation Details

The implementation adapts released RAVEL methods to ontology-specific prompts, labels, and splits, and adds multi-layer DAS/MDAS. Methods select or learn intervention targets, while multi-layer variants operate across contiguous layer windows.

Implementation Details

  • The SAE, DBM/MDBM, and DAS/MDAS implementations adapt released RAVEL code to ontology-specific prompts, labels, and splits, with a multi-layer DAS/MDAS extension; experiments use one NVIDIA A100 GPU.The code appendix specifies the software environment, while Table 4 lists model settings, input lengths, batch sizes, seeds, and realized training-pair counts.Source: Implementation Details · p. 10

Method-specific Implementation

  • SAE trains a single-layer ReLU encoder and linear decoder on target-relation activations, then uses an ℓ1-regularized linear SVM to select latent features that are replaced during intervention.The evaluated latent expansion ratios are 2 and 4; the model is trained with reconstruction error plus a mean-absolute-activation penalty.Source: Method-specific Implementation · p. 10
  • DBM/MDBM learn residual-coordinate mask logits and exchange coordinates selected by thresholding the relaxed mask at 0.5; MDBM adds auxiliary examples to Cause training.DBM uses cause-only answer-token supervision, while MDBM mixes Cause and auxiliary examples.Source: Method-specific Implementation · p. 10
  • DAS/MDAS learn row-orthonormal maps while freezing the language model; DAS trains on Cause examples and MDAS mixes in auxiliary examples.Training uses autoregressive cross-entropy on answer tokens, and the learned subspace parameters are optimized without additional regularization.Source: Method-specific Implementation · p. 10

Multi-layer Realization

  • Multi-layer realization caches clean source activations, then runs the base input with interventions applied in ascending layer order so later base activations reflect earlier interventions.Layer-specific parameters are optimized jointly under the same autoregressive objective.Source: Multi-layer Realization · pp. 10–11
  • Experiments evaluate contiguous two- and four-layer windows with independent rank-k DAS/MDAS maps per layer, totaling 1,024 or 2,048 intervention dimensions.The two-layer sweep uses k = 512; the four-layer sweep uses k = 256 or 512. All reported multi-layer results intervene at block outputs at the subject position.Source: Multi-layer Realization · p. 11

Additional Results

Class-level analyses show that intervention performance varies systematically across ontology classes and depends on base–source direction; class rankings are stable across model sizes, but the separation of target and auxiliary information remains uneven.

Class-Level Intervention Analysis

  • For instance of, class-averaged Iso exceeds Cause, while for subclass of, Cause is comparable to or higher than Iso, especially in the base role.The analysis attributes the entity-typing bottleneck mainly to changing the target answer, while subsumption prediction is more entangled with surrounding semantic and attribute information.Source: Class-Level Intervention Analysis · p. 11
  • Class-level Disentangle rankings are stable across Qwen3-8B and Qwen3-4B, but base-role and source-role rankings are only weakly related.The role asymmetry means a class that is effective as an intervention base need not be effective as a source for rewriting other subjects.Source: Class-Level Intervention Analysis · p. 11
  • High-scoring classes tend to have compact, regular attribute profiles, whereas broader or more heterogeneous classes tend to score lower.Examples include airport, city, and currency for instance of, and flower or measuring instrument in the subclass-of base role.Source: Class-Level Intervention Analysis · p. 11

Model Role Cause Iso Disent. ∆

  • Reciprocal Disentangle gaps have medians of 0.132–0.179, and 29.8–41.8% of eligible class pairs differ by at least 0.2.Only 5.0–12.1% exceed 0.4; the model-size ordering varies by axiom type, and the distributions are descriptive because pairs sharing classes are dependent.Source: Model Role Cause Iso Disent. ∆ · p. 12

Qwen3-4B Single-Layer Results

Layer-Wise Intervention Profiles

Highest-Cause Single-Layer Configurations

  • The highest-Cause single-layer configurations are selected separately for each model, axiom type, and method.The corresponding table reports Entity-split results, while Table 10 selects configurations by Entity Disentangle score and evaluates those same configurations on Context.Source: Highest-Cause Single-Layer… · p. 13, Table 11 · p. 14, Table 10 · p. 14

Class # Entity Attributes

  • The dataset class tables list class counts and associated attributes for the instance-of and subclass-of datasets.The instance-of table lists entities and attributes; the subclass-of table lists subclasses and attributes.Source: Table 12 · p. 28, Table 13 · p. 29

Related papers

Loading 2610.38326v1…