How Language Models Organize and Structure Moral Knowledge

Orion Reblitz-Richardson

Source-linked AI summary

How Language Models Organize and Structure Moral Knowledge

Orion Reblitz-Richardson

arXiv:2608.27402v1Aug 27, 2026cs.CLcs.AIcs.LG

TL;DR

The paper asks whether language models organize moral knowledge beyond detecting moral content. It trains six foundation-specific probes and analyzes their directions across models, finding integrated foundation geometry and partially compositional dilemma representations, while MFT group structure remains unsupported by an underpowered test.

Problem
Binary moral detection does not establish whether models distinguish moral foundations or represent relationships among them.
Method
The study trains six MFT-specific linear probes, analyzes normalized probe-direction geometry, and tests it across architectures, scales, datasets, and fragility conditions.
Results
Foundation directions are distinct and share a positive moral-specific component, while dilemma directions overlap their component-foundation subspaces 2.7× more than a mismatched-pair baseline.
Takeaways & Limitations
Language models represent integrated moral foundations and moral dilemmas with both component-based and conflict-specific structure rather than only pre-resolved judgments.
Takeaways & Limitations
The test of MFT’s individualizing/binding partition is underpowered because only 20 unique partitions exist, so a small group effect cannot be excluded.

Abstract

from arXiv · show

How do large language models (LLMs) organize moral knowledge? Models detect moral content broadly, but detection is a low bar. We ask whether they go further, distinguishing moral foundations from one another and organizing the relationships between them geometrically. We train six independent linear probes on open-weight language models, one per Moral Foundations Theory (MFT) category (care/harm, fair/cheat, lib/oppress, loy/betray, auth/subv, sanc/degrade), and examine how the resulting directions relate to each other in representation space. We find the directions neither collapse into a single moral detector nor isolate from one another. Rather, they span a near-maximal number of independent dimensions while sharing a positive common component. The shared component is the signature of integration, and it is moral-specific relative to a matched non-moral concept battery built identically (mean pairwise cosine 0.26 vs. 0.013). The geometry is consistent across architectures and scale and reaches its integration regime early in pre-training, well before probe accuracy saturates. The structure the model discovers shows no evidence of the individualizing/binding distinction predicted by Moral Foundations Theory (an underpowered test: only 20 candidate partitions exist) but rather reflects corpus statistics. Extending to moral dilemmas, each dilemma direction partially composes from its component foundations, at 2.7x a mismatched-pair baseline, while the majority of its variance encodes conflict-specific structure. The model represents moral tension itself, not a pre-resolved judgment.

1 Introduction

The paper asks whether language models move beyond detecting moral content to represent distinct foundations and their relationships. Using foundation-specific probe geometry, it finds integrated, cross-architecture moral structure rather than collapse, isolation, or the predicted MFT grouping.

Motivation: Prior work treated morality as a binary feature, whereas this paper examines whether models distinguish foundations and encode relationships between them.The transition from detection to structured moral representation motivates the geometric analysis.
Approach: Six independent MFT probes define representation-space directions whose angular relationships reveal whether moral foundations are collapsed, isolated, or integrated.Integration means directions remain separated but non-orthogonal; collapse means convergence, while isolation means orthogonality without relational structure.
Scope: The analysis extends across OLMo-2 1B, OLMoE-1B-7B, and dense OLMo-2 7B to test architecture and scale consistency.The study compares dense and mixture-of-experts models and includes a larger dense model as a scale check.
Findings: Foundation directions are distinct yet positively related, yielding integration geometry rather than a single moral detector or unrelated moral slots.The reported mean pairwise cosine is approximately 0.22–0.27 across layers, and the six directions span five effective dimensions.
Findings: Moral-dilemma directions are partially compositional, combining their component foundations while retaining additional dilemma-specific structure.This extends the foundation-level geometry to scenarios in which two foundations conflict.
Findings: Output dilution degrades every foundation uniformly rather than selectively across architectures.The reported cross-architecture fragility gap is approximately 2.3×, with no reliably most or least robust foundation.
Contribution: The paper contributes probe-direction geometry as a bridge between binary moral probing and structured representations proposed by moral psychology.It characterizes foundation and dilemma geometry and compares per-foundation fragility across architectures.

2 Related Work

Prior probing work typically measures whether concepts are decodable, while moral-language-model research often evaluates outputs behaviorally. This paper instead studies geometric relationships among multiple decoded concepts, using MFT as a tractable structural test.

Probing literature: Linear probing has been used to decode syntactic, grammatical, relational, and world-knowledge properties from neural representations.The paper builds on the broader probing literature and the linear representation hypothesis.
Probing literature: Most probing studies report accuracy but discard probe parameters, while this work treats probe weight vectors as geometric information about concept relationships.The approach extends analysis from individual concept locations to relationships among multiple concept directions.
Geometric representations: Earlier work showed that high-level behaviors such as bias and refusal can be localized to specific activation-space directions.This paper extends that precedent from single directions to sets of related directions.
Geometric representations: The study applies representational similarity analysis to probe directions rather than directly to stimulus response patterns.Its cosine matrix compares the directions learned by probes for different concepts.
Moral foundations: MFT supplies six moral domains and a testable individualizing-versus-binding structural hypothesis, although alternative moral taxonomies exist.The paper uses MFT pragmatically because its categories provide a tractable direction set and an explicit grouping prediction.
Moral-language-model research: Behavioral benchmarks evaluate moral outputs, whereas this work asks whether models internally encode structured moral representations.The representational question differs from whether a model produces the correct moral judgment.
Contribution: This paper advances the series from binary moral encoding to foundation-specific directions and their inter-framework geometry.It reuses earlier probing and fragility protocols at the per-foundation level.

3 Methodology

The methodology trains foundation-specific linear probes across layers and models, extracts normalized probe directions, and analyzes their geometry, stability, and fragility. It also tests architecture, scale, dataset, and MFT grouping effects while recognizing limits from small training sets and underpowered comparisons.

Models and data: The study evaluates dense and mixture-of-experts models with matched layer counts, plus a larger dense model as a scale control.The comparison tests whether geometry depends on architecture or model scale.
Probe construction: Six binary probes are trained at each of 16 transformer layers for the six MFT foundations.The foundations cover care, fairness, loyalty, authority, sanctity, and liberty.
Probe construction: Mean-pooled 2048-dimensional activations from matched moral and neutral sentence pairs provide the probe inputs.The dataset contains 32 training pairs and 8 test pairs per foundation, with neutral counterparts controlling length, syntax, and topic.
Direction extraction: Each trained probe’s normalized weight vector is treated as the foundation’s direction, producing 6 × 16 = 96 directions per model.The vector is the normal to the classification hyperplane and represents the direction separating foundation-tagged moral content from neutral content.
Geometric analysis: Pairwise cosine matrices, effective dimensionality, and hierarchical clustering characterize collapse, isolation, integration, and possible foundation groupings.Effective dimensionality counts principal components explaining at least 90% of variance; Ward clustering uses cosine distance.
Geometric analysis: A permutation test compares within-group and between-group cosine similarity for MFT’s individualizing and binding partition.Group assignments are permuted 10,000 times to form the null distribution.
Reliability: Bootstrap resampling measures direction stability, with mean bootstrap cosine similarity above 0.8 indicating a stable direction.Each foundation-layer probe is retrained on 200 bootstrap samples.
Fragility analysis: Framework-specific fragility adds Gaussian noise to cached test activations and averages accuracy across noise seeds to estimate critical noise levels.The protocol uses multiple noise levels, censoring layers that never cross threshold at the grid maximum.

4 Results

Foundation probes are easily decodable from early layers, but their directions form a separated, positively aligned geometry rather than collapse. The shared component is much stronger for moral than matched non-moral concepts, while the geometry remains high-dimensional and does not recover MFT’s predicted groups.

Foundation-specific probe accuracy: All six foundation probes are linearly separable from the earliest layer, with every foundation reaching 100% peak accuracy.Authority/subversion reaches 100% at all layers; the minimum observed accuracy is 87.5% for care/harm at layer 1.
Framework geometry: Across stable layers, foundation directions remain separated, with mean pairwise cosine ranging from 0.232 to 0.274 rather than exceeding the collapse threshold.At layer 7, the representative mean cosine is 0.262; layer 0 is less stable despite having the strongest separation.
Framework geometry: Uniformly positive cosine similarities indicate a shared moral component, while effective dimensionality stays at the ceiling of 5 across all layers.The positive shared component distinguishes integration from isolation; the ceiling reflects six mean-centered directions.
Moral-specific integration: The moral shared component has mean pairwise cosine 0.26 versus 0.013 for matched non-moral probes, a paired difference of 0.223 with CI [0.202, 0.244].The matched non-moral probes also decode accurately, ruling out a dead-probe explanation for their near-zero cosine.
Moral-specific integration: A shared-neutral-pool construction reaches mean cosine 0.53, exceeding the moral value and showing that estimator and contrast construction affect cosine magnitude.The reported probe-weight estimate is 0.22 versus 0.41 for mean-difference directions, consistent with retained moral-versus-neutral contrast affecting magnitude.
MFT group structure: Hierarchical clustering and permutation tests do not recover the MFT individualizing/binding partition, although care–sanctity co-clustering appears at 10 of 16 layers.The partition test has minimum p = 0.32 and median p = 0.53; power is limited because only 20 unique 3–3 partitions exist.
Cross-layer and cross-architecture stability: Framework geometry is stable across layers and architectures: effective dimensionality remains 5, while OLMoE-1B-7B and OLMo-2 1B show similar cosine ranges.The dense model ranges from 0.216 to 0.274 and the MoE model from 0.219 to 0.287 across layers.

5 Discussion

The model develops integrated, multidimensional moral geometry early in training, but its organization reflects corpus statistics rather than MFT’s predicted grouping. Moral dilemmas combine component foundations with substantial conflict-specific structure.

Integration geometry: Mean pairwise cosine similarity of ≈0.22–0.27 shows a positive shared component, while the matched non-moral battery yields 0.013 versus moral 0.26.The paired difference is 0.223 with CI [0.202, 0.244], excluding zero.
Integration geometry: Effective dimensionality is 5 across layers, scales, architectures, and datasets, indicating distinct foundation directions rather than a single moral feature.The result is near-maximal for six directions and remains stable across the tested OLMo models and the independent vignettes dataset.
Corpus-shaped organization: The geometry does not recover MFT’s individualizing/binding partition, although the underpowered test cannot exclude a small effect.Only 20 partitions were enumerated, with an observed minimum p of 0.32; care–sanctity clustering instead persists across dataset choices.
Corpus-shaped organization: Unsupervised clustering recovers the labeled foundations weakly, reaching adjusted mutual information of 0.03 at best.The model’s dominant unsupervised structure is therefore not MFT-organized foundation structure.
Fragility: Every foundation is approximately 2.3× more fragile in MoE than dense models, with no reliably most or least robust foundation within architectures.Scale normalization removes the apparent per-foundation ordering and complexity gradient.
Training trajectory: Integration geometry emerges within the first few thousand steps, before probing accuracy saturates, then is gradually refined as mean cosine declines from 0.34 to 0.29.The trajectory supports an early layout-discovery phase followed by slower differentiation.
Moral dilemmas: Dilemma directions overlap their component-foundation subspace 2.7× more than mismatched pairs, yet approximately 88% lies outside that subspace.Matched membership is 0.118 versus 0.044, so dilemmas are partly compositional but retain conflict-specific structure.
Moral dilemmas: Near-balanced component loading (¯α = 0.486) indicates that dilemmas retain both competing foundations rather than encoding a pre-resolved judgment.The same conflict-as-tension profile is reproduced at 7B scale.

6 Conclusion

Across tested models, language models represent moral foundations as integrated, multidimensional directions and moral dilemmas as partly compositional but largely conflict-specific structures. The geometry is consistent across architectures and scale, and the method generalizes to other related concept taxonomies.

Foundation geometry: Effective dimensionality = 5 and uniformly positive mean pairwise cosine of ≈0.22–0.27 indicate distinct foundation directions with a shared component.The shared component is ∼20× larger than in the matched non-moral battery: 0.26 versus 0.013.
Fragility: Every foundation is ∼2.3× more fragile in MoE than dense models, without reliable within-architecture differences among foundations.The binding/individualizing group difference is also non-significant.
Moral dilemmas: Dilemma directions overlap their component-foundation subspace ∼2.7× more than mismatched pairs, while the remaining ∼88% encodes conflict-specific structure.Near-balanced loading and preservation across dense and MoE models and 1B to 7B support a tension representation beyond component foundations.
Broader method: Probe-direction geometry can analyze inter-concept structure for other value taxonomies, political orientation, and safety-training representations.The method applies wherever related concepts can be isolated by binary linear probes.

Appendices Supplementary material.

The appendices provide supplementary tables for per-foundation probe accuracy and bootstrap direction stability across models and layers.

Supplementary tables: Table 2 reports per-foundation probe accuracy across layers for OLMo-2 1B, with all foundations exceeding 0.6 at every layer.Each cell uses 16 test examples per foundation.
Supplementary tables: Table 3 reports per-foundation probe accuracy across layers for OLMoE-1B-7B.
Supplementary tables: Table 4 reports bootstrap direction stability using mean cosine with the full-data direction over 200 resamples.Values below the 0.8 stability threshold are marked with ∗.

B Bootstrap Direction Stability Tables

Bootstrap stability is weakest in the earliest layers and mostly stable in layers 6–15, where the headline integration result is confirmed.

Stability by layer: Layers 0–2 show widespread instability, with all six foundations below the stability threshold.
Stability by layer: Layers 6–15 are mostly stable, with authority at layer 8 the only noted exception.Authority at layer 8 has stability 0.792.
Stability by foundation: Sanctity/degradation is the most stable foundation at 13 of 16 layers, while care/harm is the least stable at 10 of 16.
Stable core: The stable core of layers 6–15 confirms the integration signature with effective dimensionality = 5.

C Cosine Similarity Matrices

The six foundation directions remain positively related across layers, with liberty, authority, and loyalty forming the strongest pairings. Their cross-boundary similarities indicate geometry organized by empirical distributional structure rather than MFT grouping.

The reported matrices cover the six foundation probe directions at selected OLMo-2 1B layers.
At layer 0, liberty–authority has the highest pairwise cosine similarity (0.333), followed by liberty–loyalty (0.273).
At layer 15, liberty–authority and loyalty–authority remain the strongest pairings, with cosine similarities of 0.393 and 0.415.
The strongest pairings cross the MFT individualizing/binding boundary, supporting organization along empirical distributional axes rather than theoretical MFT groups.

D Permutation Tests for MFT Group Structure

The exact permutation test found no significant evidence that foundation directions follow MFT’s individualizing/binding split. Observed within-group similarity was near zero or often below between-group similarity.

The test compares mean within-group and between-group cosine similarity for MFT’s two groups of three foundations.
20 distinct group assignments were enumerated exactly, making the p-value the fraction of partitions at least as extreme as observed.
No layer reached significance; the minimum p-value was 0.40.
Significance was reachable at p = 1/20 = 0.05, but the observed statistics were near zero and frequently negative.
Reproducibility: The experiments used OLMo-2 1B and OLMoE-1B-7B, with reported wall times ranging from ∼5 to ∼45 minutes.

E.3 Reproducibility notes

The reproducibility setup specifies probe optimization, deterministic probing data, activation pooling, and publicly available code.

Probe training: Linear probes use nn.Linear(2048, 1), BCE loss, Adam with lr = 10−2, and 50 training epochs without weight decay or scheduling.
Probing dataset: The 240-pair minimal-pair dataset contains 40 examples per MFT foundation and is version-controlled with automated validation.
Activation collection: Activations are mean-pooled across sequence positions, and OLMoE activations are collected after expert combination.
Code availability: Experiment scripts, the probing dataset, and figure-generation code are publicly available in the DeepSteer repository.

F Causal Validation (Preliminary)

Preliminary interventions suggest that foundation directions influence generation, but the uncontrolled analyses do not establish causal localization. Ablation and injection effects are direction-specific in reported tests, while lacking matched null controls.

Scope: Representation geometry shows where foundation information is decodable, whereas whether generation uses that information is a separate causal question.
Direction ablation: Ablating a foundation direction selectively degrades that foundation’s continuations, with specificity changing from −0.12 at layer 4 to −0.32 at layer 14.
Steering injection: Steering specificity rises with injection strength, from +0.08 at α = 1 to +3.59 at α = 20.
Steering injection: The same directions that decode foundations also steer generation toward them in proportion to dose.
Limitations: Without random-direction or channel-matched controls, the interventions cannot separate foundation-specific action from generic intervention effects.

G Dilemma subspace membership: matched vs. mismatched baseline

Dilemma directions show greater membership in the spans of their own component foundation pairs than in non-component foundation-pair spans. This matched-over-mismatched relationship holds across layers and replicates across model scales and architectures.

Matched vs. mismatched membership: Matched subspace membership exceeds the mismatched baseline at every layer.Matched spans contain each dilemma’s two component foundation directions; mismatched spans use foundation pairs sharing no component.
Matched vs. mismatched membership: 0.091 matched versus 0.039 mismatched cross-layer mean membership, a 0.052 gap with 95% CI [0.037, 0.069].This cross-layer-mean gap is the unbiased figure; both bootstraps resample the 15 dilemmas.
Matched vs. mismatched membership: 0.118 matched versus 0.044 mismatched per-pair-peak mean membership, or ∼2.7×, with a 0.074 gap and 95% CI [0.053, 0.100].The peak statistic is a max-over-layers extremum and is biased upward.
Cross-model replication: The matched-over-mismatched margin replicates on OLMo-2 7B and OLMoE-1B-7B.Reported peak values are 0.090 versus 0.032 for OLMo-2 7B and 0.118 versus 0.