Scaling Inherently Interpretable Language Models

Guide Labs Team; Andreas Madsen; Aya Abdelsalam Ismail; Giang Nguyen; Isaac Plant; Muawiz Chaudhary; Nathaniel Monson; Saqib Azim; Zhichen Guo; Julius Adebayo

Source-linked AI summary

Scaling Inherently Interpretable Language Models

Guide Labs Team, Andreas Madsen, Aya Abdelsalam Ismail, Giang Nguyen, Isaac Plant, Muawiz Chaudhary, Nathaniel Monson, Saqib Azim, Zhichen Guo, Julius Adebayo

arXiv:2608.07594v1Aug 6, 2026cs.CLcs.AI

TL;DR

Interpretability is often treated as a tradeoff against language-model capability, with explanations added after opaque training. This paper instead trains interpretability into the modeling process and finds that it improves with scale while Steerling-8B remains competitive with models trained on 2–16× more compute.

Problem
The paper tests whether interpretability constraints impose a growing capability penalty as language models scale.
Method
The authors build faithfulness conditions into the model’s data, architecture, objective, and losses, using shared concept variables for attribution and intervention.
Results
Across three orders of magnitude of compute, interpretability improves with scale, while Steerling-8B reaches within approximately 10% of open peers trained on 2–16× more compute.
Takeaways & Limitations
Interpretability can be specified, optimized, and measured during training rather than retrofitted afterward or treated as a capability tax.
Takeaways & Limitations
Steering causes substantial quality drops and fails to activate roughly one third of less-frequent concepts because concept injection is out of distribution during inference.

Abstract

from arXiv · show

Interpretability is often treated as a tax on capability: language models are trained as opaque systems, then explained after the fact, with methods whose reliability is difficult to establish. In this work, we challenge this premise. Rather than reverse-engineering a model, we make interpretability a constraint of the training pipeline, optimized alongside the language modeling objective. Across three orders of magnitude of compute, on both autoregressive and diffusion language models, interpretability scales with capability rather than against it. Surprisingly, model representations become more disentangled and aligned with human-understandable concepts with scale. We instantiate the training-time recipe with Steerling-8B, a diffusion language model with a causal attention mask. For any group of generated tokens, Steerling-8B attributes the output to relevant input tokens, human-understandable concepts, and training data. This enables closed-loop intervention: diagnose an output through its concept or feature attribution, retrieve similar training data, and correct the behavior through concept steering without retraining. Steerling-8B remains competitive with open peer models trained on substantially 2-16x more compute, suggesting a different scaling paradigm: interpretability can be designed into training, and it improves with scale.

1 Introduction

The paper makes interpretability a training constraint rather than a post-hoc explanation, and shows that interpretable models can remain competitive while scaling. Steerling-8B provides token, concept, and training-data attributions that also enable concept-based intervention.

Inherent interpretability: Inherent interpretability builds faithful explanatory conditions into the data, architecture, objective, and losses, making attribution a trained interface rather than an auxiliary visualization.This approach directly challenges the assumption that human-meaningful structure necessarily weakens model performance.
Three model understanding axes: Steerling-8B traces predictions to affecting input tokens, human-understandable concepts, and training data across three model-understanding axes.Input attribution uses a trained absence baseline, while concept attribution identifies representational concepts contributing to outputs.
From explanation to control: Concept directions support closed-loop control because amplifying or suppressing a concept directly edits the model’s forward-pass computation.The same interfaces used for explanation therefore enable intervention without retraining.
Atlas: Atlas canonicalizes hundreds of millions of tags into over 33,000 concepts and annotates over 1 trillion tokens across web text, code, mathematics, and academic prose.Its chunk-level annotator addresses the lack of a concept library suitable for modern pretraining corpora.
Scaling: Across three orders of magnitude of compute, concept modules impose a small fixed per-backbone scaling offset rather than a scale-growing interpretability cost.IsoFLOP sweeps compare autoregressive, causal-diffusion, and corresponding concept-equipped model families; interpretability metrics improve with scale.
Scaling and results: Steerling-8B performs within approximately 10% of open peer models’ average benchmarks despite peers using roughly 2–16× more compute.It is trained on 1.2 trillion tokens plus 150 billion midtraining tokens while carrying interpretability constraints throughout training.

2 Background

This section introduces Transformer and language-model notation, contrasts autoregressive and diffusion training objectives, and reviews concept bottleneck models for interpretable prediction and generation.

Transformer notation: Transformers map input tokens to contextual hidden states and project each state into vocabulary logits; the paper summarizes this notation in Table 1.Each hidden state summarizes its token together with contextual information.
Autoregressive language models: Autoregressive language models predict the next token sequentially, conditioning each prediction on all preceding tokens.The model uses pθ(x_i | x_<i), with negative log-likelihood averaged across sequence positions.
Diffusion language models: Diffusion language models (Austin et al., 2021a; Ou et al., 2024; Sahoo et al., 2024; Shi et al., 2024) reverse corruption to reconstruct original tokens, with masked variants training only on masked positions.Masked diffusion models replace tokens independently with [MASK] and optimize cross-entropy over masked positions across noise levels and sequences.
Concept bottleneck models: Concept Bottleneck Models (Koh et al., 2020) insert supervised human-interpretable concepts between inputs and outputs, training concept and prediction losses.The representation is c = ϕ(x), and the predictor produces y = ψ(c).
Concept bottleneck generative models: Concept Bottleneck Generative Models add an unsupervised channel for information beyond fixed supervised concepts and use orthogonality to separate the channels.The unknown channel u absorbs remaining information while an orthogonality loss encourages it to differ from known concept embeddings.

3 A recipe for building interpretable models

The recipe modifies every stage of the standard language-model training pipeline to enforce human-interpretability constraints. It targets input, concept, and training-data similarity attribution while addressing why post-hoc methods do not guarantee faithful explanations.

3.4 The interpretable recipe: The recipe modifies data curation, architecture and loss design, optimization, and evaluation to introduce human-interpretability constraints throughout training.Each modification is traced to a specific condition of the formal definition, with the full pipeline summarized in Figure 2.
3.1 Interpretability requirements: The required interface answers what input mattered, which human-understandable concepts drove the output, and which training examples resemble it.These correspond to input attribution, concept attribution, and training-data similarity attribution; similarity attribution does not claim causal training influence.
3.2 Why standard methods are insufficient: Post-hoc probes, sparse autoencoders, gradients, perturbations, and chain-of-thought can fail to establish that explanations reflect the model’s actual predictive computation.The failures include presence-versus-use gaps, reconstruction without causal role, infinitesimal sensitivity versus finite interventions, out-of-distribution baselines, and unfaithful language explanations.
3.2 Why standard methods are insufficient: Post-hoc explanations are also model-contingent: equally accurate models can produce contradictory attributions, motivating interpretability guarantees built into training rather than inferred afterward.The recipe is presented as addressing these validity and agreement failures, while its elements draw on earlier work rather than constituting an entirely new recipe.
3.3 Formalization: Inherent interpretability guarantees specified attribution queries, not total mechanistic transparency, and semantic faithfulness can still fail when a learned concept encodes hidden information beyond its human-readable label.Concept leakage occurs when the attribution tracks a feature’s causal effect but its semantic description incompletely characterizes the feature.

4 Data

Atlas constructs a large-scale concept-annotated corpus and concept library through a three-stage pipeline, then trains an annotator for arbitrary text. Human evaluations show that Atlas concepts are recognizable, preferred to distractors, and competitive with or better than independently generated human labels.

Atlas pipeline: Atlas uses three stages: high-recall span tagging, canonical concept clustering and naming, and training a concept annotator for arbitrary text.The pipeline moves from sampled documents to free-form tags, then to a canonical library and scalable annotation.
Pipeline validation: Pipeline validations found relevant tags, coherent clusters, and acceptable annotator predictions, with tag and concept ratings centered near 3.6 and 3.5 respectively.For the trained annotator, per-chunk averages center between 2.5 and 3.5, and 95.5% reach at least 2, though predictions are lower and more dispersed than original annotations.
Tag extraction: Stage 1 produced nearly 44 million annotated chunks and approximately 500 million tags, reduced to 14 million unique tags before clustering.The raw tag space captures recurring content and stylistic patterns but remains redundant and non-canonical before Stage 2.
Clustering and naming: Clustering coherence improves through k = 40,000–80,000, so Atlas initializes 80,000 clusters while limiting computational cost and semantic splitting.The resulting concept names are generated by an LLM from statistical regularities, making human interpretability an empirical question rather than an assumption.
Concept library: The final canonical library contains 33,732 concepts spanning science, technology, philosophy, medicine, law, and other domains.The library balances granularity and clarity despite unavoidable conceptual overlap and hierarchy.
Human interpretability: Atlas labels scored 3.98 versus 3.50 for human labels, received top-two ratings 79% versus 63%, and beat human labels with probability 0.62.This pilot comparison supports human endorsement of Atlas labels, while not certifying every library concept individually.

5 Inherently interpretable architecture

The architecture combines masked diffusion with block-causal attention to preserve joint, phrase-level processing while enabling autoregressive-style efficient inference. A concept bottleneck then decomposes hidden states into inspectable channels, making predictions exactly attributable to known, unknown, and residual components.

Causal Diffusion: Causal Diffusion combines masked diffusion with bidirectional attention within blocks and causal attention across blocks, omitting Block Diffusion’s clean training copy.The standard masked diffusion loss is applied to a single sequence with a block-causal mask.
Causal Diffusion: The resulting model preserves diffusion’s within-block parallelism and any-order flexibility while enabling autoregressive-style KV caching across previously generated blocks.Previously generated blocks’ keys and values are reused across denoising steps, then the current block is appended to the cache.
Concept module: The concept module decomposes each transformer hidden state into known-concept, unknown-concept, and residual components before the language-modeling head.Residual dropout discourages reliance on the residual channel, while low-rank factorization reduces the parameter cost of the unknown-concept embeddings.
Concept module: Because the language-modeling head is linear, every output logit is an exact additive function of concept activations and the residual, enabling faithful attribution and concept steering.Known and unknown concepts and the residual form separate, inspectable channels that provide a modular interface to predictions.
Training: A scheduled teacher-forcing substitution replaces predicted known-concept representations with ground-truth representations with probability αknown(s) during training.The schedule addresses unreliable early known-head predictions and prevents language-modeling loss from pushing labeled-concept activations toward unrelated information.

6 Interpretability capabilities

Steerling combines attribution and inference-time concept steering to explain why outputs occur and control model behavior. Its attribution views cover influential inputs, internal concepts, and retrieved training data, while steering amplifies or suppresses concepts without updating weights.

Overview: Steerling explains outputs through input-token, internal-concept, and training-data attribution, while concept steering controls behavior at inference time.Figure 18 presents the three attribution views for one output chunk; steering acts directly on learned concept representations.
6.1.1 Input attribution: Input attribution uses Integrated Gradients from the learned [MASK] embedding baseline to identify which prompt tokens most influenced an output.Because diffusion training repeatedly predicts masked tokens, [MASK] represents “no information” and defines a meaningful attribution path.
6.1.2 Concept attribution: Concept attribution decomposes output logits into known-concept, unknown-concept, and residual contributions, then ranks concepts by summed chunk-level effects.The residual captures the portion of the output not explained by the concept inventory.
6.1.3 Training data attribution: Training-data attribution retrieves semantically similar training chunks for provenance and auditing, rather than estimating causal effects through influence functions.The method bridges Steerling’s internal output representation space to an index built from embeddings produced by another model.
6.2 Steering: Concept steering injects normalized learned concept directions into hidden states without weight updates, enabling amplification toward or suppression away from known and unknown concepts.The steering strength is calibrated per concept so a shared target τ produces adaptive effects across concepts.
6.2.2 Suppression: Suppression combines negative hidden-state injection with a ReLU-gated logit mask, preventing negative steering from promoting tokens anti-aligned with the target concept.Amplification uses γ > 0, whereas suppression uses γ < 0.

7 Interpretability Metrics

Section 7 operationalizes interpretability as measurable training-time properties rather than post-hoc analysis. Its metrics test concept detection, concept-module attribution, disentanglement, and alignment with human-assigned meanings.

Metric design: The metric suite directly tests whether the architecture detects concepts, routes predictions through them, and keeps known and unknown concepts disentangled.These correspond respectively to the concept, reconstruction, and independence training objectives.
Concept Loss: Concept loss evaluates held-out concept detection using OR-aggregated binary cross-entropy between predicted concept presence and pipeline-generated chunk-level labels.A low value indicates that the module identifies concepts in unseen text as it learned during training.
Concept Independence Loss: Concept independence measures linear statistical dependence between known and unknown representations, with lower values indicating more independent information.The linear kernel matches the additive bottleneck and linear language-model head, where output logits decompose linearly.
Concept Contribution: Concept contribution measures the fraction of total logit magnitude attributable to known and unknown concept terms rather than the residual.Computed over held-out predictions, it evaluates attribution faithfulness at the logit level; higher values indicate predictions are mostly explained by the concept module.
Known Concept Alignment: Known concept alignment asks whether top vocabulary tokens projected from each concept embedding semantically match its human-assigned label and description.An LLM judge scores the alignment from 1 to 5, although this property is not directly supervised by the concept or language-modeling objectives.

8 Scaling laws

Across autoregressive and diffusion model families, interpretability-by-design preserves compute-optimal scaling with only a small fixed per-backbone offset, while all four interpretability metrics improve with compute. Small-scale fits also extrapolate to Steerling-8B’s validation loss within 0.11 nats and predict three of four interpretability metrics within error bounds.

Parameter overhead: The concept module’s relative parameter overhead falls from ∼89% at 10M parameters to ∼4% at Steerling and under 1% at frontier scales.The module scales as O(d), while backbone parameters scale as O(d^2L).
Scaling results: Interpretability-by-design imposes a small per-backbone offset on compute-optimal scaling, not a scaling tax, while all four interpretability metrics improve with compute across both backbones.Concept Loss and Concept Independence Loss decrease, whereas Concept Contribution and Known Concept Alignment increase.
Scaling-law extrapolation: 0.11 nats: small-scale fits predict Steerling-8B’s validation loss with this gap under the joint Chinchilla form.Steerling-8B achieves L*=2.72, while the joint fit predicts L*=2.61 at 7.6 × 10^22 FLOPs.
Interpretability scaling: Model size primarily drives concept identification and disentanglement, whereas training data primarily drives Known Concept Alignment.Concept Loss and Concept Independence Loss have near-vertical contours, while Known Concept Alignment has near-horizontal contours over the fitted range.
Interpretability extrapolation: Three of four interpretability metrics are predicted within error bounds from small-scale fits, while the fourth exceeds its prediction in the favorable direction.Some fitted asymptotes are constrained by natural bounds, so they should be interpreted as bounds rather than precise plateau estimates.

9 Steerling-8B: Pretraining

Steerling-8B is pretrained on a concept-annotated 1.2T-token corpus using an interpretable causal-diffusion architecture with several empirically selected design choices. During training, capability and interpretability mostly remain healthy, but late-stage degradation reveals failures tied to the masking curriculum, concept-head training, corpus composition, and missing steering operations.

9.1 Dataset: The 1.2T-token pretraining corpus combines quality-filtered web, academic, mathematical, and code data, with chunk-level concepts annotated through Atlas.The mixture is dominated by Nemotron-CC-HQ and includes peS2o, arXiv, OpenWebMath, Algebraic Stack, StarCoder, Wikipedia, and Wikibooks; per-source counts appear in Table 9.
9.2 Architecture and training choices: Steerling-8B uses block size b = 64, keeps unknown concepts at m = 3n, and factorizes their embedding matrix with rank R = 256, reducing its size roughly 15×.Block size 64 lowers validation loss without changing interpretability metrics, while increasing m from 3n to 5n provides no measurable gain.
9.3 Pretraining run: Capability metrics plateau or peak during training, while concept contribution rises from 0.62 to 0.85 and concept independence loss sharply worsens late.HellaSwag and PIQA remain near their peaks, ARC-Challenge dips slightly at the end, while MMLU and WinoGrande decline after midpoint; concept alignment peaks near 4 before settling.
9.4 Lessons learned: Late concept entanglement arises when hard masking coincides with exhausted teacher-forcing floors, while limited math and code and absent training-time steering explain additional failures.The corpus contains slightly over 100B math and code tokens, and the authors attribute the completed run’s issues to masking, concept-head entanglement, corpus imbalance, and no steering operations during training.

10 Steerling-8B: Mid-training

A 150B-token mid-training run targets pretraining’s masking, data, independence-loss, and teacher-forcing weaknesses while improving downstream capability, interpretability, and steering. The resulting model recovers lost knowledge and reasoning performance without sacrificing existing strengths.

Data mixture: The code-augmented mixture is adopted because it balances reasoning, math, and code and uniquely improves every benchmark over the pretrained model.The ablation compares math-heavy, balanced, code-augmented, and code-only 10B-token compositions.
Masking schedule: Uniform masking restores knowledge better than heavy masking: MMLU reaches 0.416 versus 0.280 under the 80% schedule and 0.298 for the pretrained model.The two schedules are close on most benchmarks, while the heavy schedule is slightly stronger on math tasks.
Training procedure: Teacher forcing is annealed away so every bottleneck concept is model-predicted, while interleaved steering training leaves LM Harness performance essentially unchanged.αknown decreases from 0.5 to 0 and αunknown remains 0; steering is evaluated as a 10B-token ablation.
Overall results: Mid-training improves every benchmark: MMLU rises 17 percentage points, math 30 percentage points, and code about 7 percentage points on average, with no capability trade-off.The benchmarks that were already healthy remain healthy.
Interpretability: Mid-training reduces concept independence loss by 19% and increases concept contribution, yielding cleaner separation and stronger interpretability.The two-term penalty introduced during mid-training mitigates the late-pretraining instability.
Steering: Steering improves after mid-training: mean concept rises from 1.072 to 1.247, mean quality from 0.972 to 1.064, and harmonic mean from 1.020 to 1.148.Mean sample harmonic also rises from 0.843 to 0.963.

11 Related work

Prior work shows that post-hoc explanations can vary across equally performant models, motivating architectures and training procedures that constrain interpretability inherently. Steerling combines this perspective with diffusion language modeling, scaling-law analysis, and attribution and steering methods.

The problem for post-hoc interpretability: Post-hoc explanations are unstable because equally accurate models can use different internal mechanisms and produce contradictory feature attributions or counterfactuals.The Rashomon set formalizes near-optimal models whose feature importance can vary substantially, while alternative decision boundaries can invalidate counterfactual recommendations.
How inherent interpretability addresses this?: Steerling constrains the Rashomon set during training by fixing the concept library, routing outputs through shared concept variables, and enforcing a common absence baseline.Different runs may still learn different parameters, but the concepts, attribution computation, and meaning of absence remain fixed.
Diffusion language models: Diffusion language models generate tokens through iterative denoising, and masked diffusion provides a learned [MASK] baseline, parallel generation, and arbitrary-order interventions that support interpretability.Steerling extends this substrate with Causal Diffusion, a block-causal formulation.
Scaling laws and interpretability: Interpretability-related scaling work finds that larger sparse autoencoders recover finer features and that neuron populations become more selective and monosemantic as base models grow.These studies examine whether post-hoc probes improve when the probe or underlying model is scaled, alongside established performance scaling laws for autoregressive and diffusion language models.
Attribution and steering: Steerling relates to architectural interpretability methods such as Concept Bottleneck Models and to post-hoc techniques for input, concept, and training-data attribution, while using concept steering for inference-time intervention.Its input attribution uses Integrated Gradients with [MASK] rather than an unlearned zero or padding baseline; existing steering methods typically inject learned directions into hidden activations (Zou et al., 2023; Turner et al., 2023).

12 Conclusion

Across three orders of magnitude of compute, interpretable training produced a small, fixed compute-optimal scaling offset rather than a compounding capability penalty. The work frames interpretability as a training-time contract while acknowledging that its current design choices are initial instantiations and that Steerling-8B is evidence of a viable research program, not a finished system.

Results: Across three orders of magnitude of compute, interpretable structure shifted compute-optimal scaling by a small, fixed offset rather than a penalty that compounds with scale.This result held in the tested settings for autoregressive and causal-diffusion language models.
Training recipe: Interpretability entered through conditions enforced in the data, architecture, objective, and losses, rather than through post-hoc explanation add-ons.The recipe was designed so that removing each component breaks a specific condition required for faithful explanations.
Limitations: The concept library, additive bottleneck, chunk-level concept supervision, and supervised-finetuning endpoint are first instantiations that remain open to substantial improvement.The concept library was fixed before training and focused mostly on topics describing training-document segments.
Future directions: Future directions include concept-based post-training rewards, inspectable and correctable agent decisions, and hierarchical, adaptive concept libraries.These prospects extend the framework beyond fixed concept libraries and the current training setup.
Implications: Steerling-8B is presented as evidence that frontier-grade interpretable models and agentic systems with decomposable, auditable, steerable actions are feasible, rather than as a finished system.The conclusion explicitly treats the model as evidence that this research program is viable.
Conclusion: Interpretability, steerability, and other reliability requirements can be specified as a contract, optimized during training, and measured as models scale instead of being retrofitted or treated as a capability tax.The conclusion proposes this as a different scaling paradigm for capable AI systems.

Authorship and Credit Attribution

The manuscript was a collaborative effort spanning data, modeling, infrastructure, post-training, product, and writing, with responsibilities described as primary rather than exclusive. Team members jointly developed, validated, trained, released, and deployed the Steerling system.

Collaboration and Writing: All team members contributed to writing, while the work itself received iterative feedback across data annotation, architecture, training infrastructure, post-training, and product.The statements of work identify primary responsibilities rather than exclusive ownership.
Data: Nathaniel Monson, Saqib Azim, and Julius Adebayo built the Atlas data annotation system and its supporting annotation, clustering, labeling, deduplication, and indexing components.Monson led LLM annotation infrastructure, the concept library, and human validation; Azim led Stage 2 and baseline embedding annotators.
Model Architecture, Training Recipe, Scaling Laws, and Training: Aya Abdelsalam Ismail led the model architecture, training recipe, and concept module, while Aya Ismail and Andreas Madsen devised the causal diffusion attention formulation and Giang Nguyen led mid-training steering.Nguyen also led steering validation and production implementation and contributed to attribution and training-recipe design.
Infrastructure: Andreas Madsen developed the dataloader and distributed multi-node training framework, Julius Adebayo wrote the pre-training package, and both managed the internal SLURM cluster.These infrastructure contributions supported research and production model training.
Model Post-Training and Open Source: Andreas Madsen led Steerling post-training, while Aya Abdelsalam Ismail and Giang Nguyen led the release of the base and instruct models with attribution and steering artifacts.Madsen adapted supervised fine-tuning to block-diffusion and interpretability training and fixed correctness issues across training and evaluation.
Post-Processing, Clarity Platform, and Supervision: The team also delivered concept naming, the Clarity product and serving infrastructure, and project supervision through contributions led by Nathaniel Monson, Zhichen Guo, Isaac Plant, Andreas Madsen, Muawiz Chaudhary, and Julius Adebayo.Clarity included frontend, API, microservice, gRPC, serving, attribution, and product backend systems.

Architecture … D.2 Phase 1 agreement and coherence

The appendix specifies the training-data attribution and concept-evaluation pipelines, then shows that lifted-word evidence is often semantically recoverable but imperfectly coherent and only moderately agreement-consistent. The attribution system indexes roughly 11 billion chunks and links response chunks to similar training sources through learned representation mapping and approximate search.

A Symbol reference / C Atlas: From documents to concepts / D Additional details on the human interpretability study: The appendix organizes concept-module notation by role and extends the Atlas from documents to concepts while positioning the study as additional methodological and statistical detail.The supplied materials identify notation tables, the text-assigned concepts evaluation prompt, and the appendix’s focus on agreement, robustness, and model-based analyses.
B.1 Training data attribution: The attribution pipeline indexes approximately 11 billion training chunks as 1024-dimensional embeddings using FT-Qwen3-Embedding-0.6B and searches them with a FAISS IVFPQ index.The inverted-file structure restricts comparisons to coarse clusters, making nearest-neighbor retrieval tractable at corpus scale.
B.1 Training data attribution: At inference, each response chunk is mean-pooled from per-token hidden states into a 4096-dimensional representation combining known-head, unknown-head, and residual components.This representation encodes the chunk’s meaning as represented by the language model.
B.1 Training data attribution: A roughly 15-million-parameter, two-hidden-layer MLP transducer maps each 4096-dimensional chunk representation into the corpus index’s 1024-dimensional embedding space.It is trained with a cosine-similarity objective to preserve semantic content relative to the target embedding.
B.1 Training data attribution: End to end, the system encodes and pools a response chunk, transduces it into corpus-embedding space, and retrieves the most similar training chunks by approximate nearest-neighbor search.The retrieved chunks serve as attributed training sources for the response.
D.1 Sampling and annotation protocol: The human interpretability study sampled 100 concepts across taxonomy branches, exposed annotators only to lifted-word lists, and collected 303 Phase 1 naming responses from 9 annotators.Phase 1 asked annotators to name concepts and rate recognizability on a 1–5 scale; Phase 2 compared pipeline labels with human labels and distractors under randomized, blinded presentation.
D.2 Phase 1 agreement and coherence: 55% of Phase 1 responses formed recognizable concepts, with mean coherence 3.52, while 17% were incoherent or noisy.Annotators rated 27% borderline, showing that lifted-word evidence is informative but not uniformly interpretable.
D.2 Phase 1 agreement and coherence: The Phase 1 coherence ratings showed moderate agreement: ICC(1) was 0.43, 53% of concepts were unanimous, and within-concept variability was 0.65 versus 1.05 overall.The results support meaningful but imperfect semantic recovery across short word lists and diverse technical domains.

D.3 Ordinal mixed-model analysis … F Symbol and notations

The study finds that pipeline-generated labels receive higher blind fit ratings than human labels, with results robust to modeling choices, linked to lifted-word coherence, and well-powered at the aggregate level. The section also documents study limitations, the LLM-judge protocol for known concept alignment, and notation used in scaling-law analysis.

D.3 Ordinal mixed-model analysis: 2.38 odds ratio: pipeline labels received higher fit ratings than human labels, with 79% versus 63% top-two ratings and posterior probability 0.99.The analysis uses a Bayesian cumulative-link mixed model for ordinal 1–5 ratings with crossed random effects, taking human labels as the reference category.
D.4 Assumption-light robustness checks: +0.47 points: simpler paired and Gaussian mixed-model checks likewise found pipeline labels rated higher than human labels.The paired comparison gave probability 0.62 that a pipeline label outscores a human label, with cluster-bootstrap 95% CI [0.58, 0.66] across 402 pairs.
D.5 Dependence on lifted-word coherence: +0.42 and +0.46 correlations: higher Phase 1 coherence predicted higher Phase 2 fit for human and pipeline labels, respectively.The pipeline advantage remained positive across coherence bins and was +0.22, +0.54, and +0.52 for low-, mid-, and high-coherence concepts.
D.6 Power analysis: Power exceeded 0.99 at roughly 20 fully rated concepts and was essentially 1.00 at the realized sample size for the aggregate label comparison.The simulation supports power for the stratified pilot’s aggregate comparison, not certification of every individual concept in the full library.
D.7 Limitations: The human study validates Atlas’s labeling operation rather than every concept individually, and lifted-word ambiguity, domain expertise, and limited evidence scope constrain interpretation.Phase 2 compares names relative to lifted-word evidence rather than full source-document evidence, making the result conservative in one respect and incomplete in another.
E Known concept alignment judge: Known Concept Alignment uses Mistral-Small-3.1-24B-Instruct at temperature 0 to score a concept label, description, and top-K embedding-scored tokens from 1 to 5.The complete judge prompt is supplied in Figure 33.
F Symbol and notations: Table 23 groups notation for scaling-law analysis into compute and resources, validation losses, power-law parameters, and interpretability scaling.The supplied passage identifies the table’s organizational scope but does not provide its individual symbols or values.

G Architectures, IsoFLOP slices, and hyperparameters · H ELBO estimation for validation loss · H.1 Estimators

The scaling sweep uses shared backbone and optimization configurations across four model families and IsoFLOP slices, while diffusion validation loss is estimated with four Monte Carlo ELBO schemes whose biases can affect scaling measurements. The estimators range from fixed-rate cross-entropy to blockwise ELBO estimation matched to causal block-diffusion training.

G Architectures, IsoFLOP slices, and hyperparameters: All four backbone families share one architecture, while +Concept variants add concept-module heads; sequences use length 4096 throughout.Backbone parameter counts exclude embeddings, whereas +Concept totals include concept-module heads but exclude factorized unknown-head concept embeddings.
G Architectures, IsoFLOP slices, and hyperparameters: The sweep uses four IsoFLOP target budgets per family, with four to six model sizes per slice within ±15% of each target.+Concept families begin at higher targets because concept-module per-token FLOPs are non-negligible for small backbones.
G Architectures, IsoFLOP slices, and hyperparameters: All scaling-law runs fix AdamW, learning-rate, batch-size, warmup, and architectural defaults across model sizes and families.The optimizer uses β1 = 0.9, β2 = 0.95, ε = 10−8, weight decay 0.1 excluding embeddings, gradient clipping 1.0, and peak learning rate 4×10−4.
G Architectures, IsoFLOP slices, and hyperparameters: Diffusion families use causal blocks of size 64 and uniformly sampled noise t ∼U(0.05, 0.95), while +Concept modules distinguish stable and anneal phases.Annealing adds top-k sparsification to both heads; αknown(t) and αunknown(t) control ground-truth component usage during training.
H ELBO estimation for validation loss: Diffusion per-token NLL requires ELBO estimation, and estimator-dependent absolute-loss bias can propagate into L∞ values and scaling exponents.This comparison evaluates four estimators on CDLM checkpoints, averaging each over approximately 100M validation tokens.
H.1 Estimators: The first two estimators are fixed-rate cross-entropy at τ = 0.5 and a nine-bin uniform-grid approximation over t ∈[0.05, 0.95].The fixed-rate measure is closest to the training objective but is not an ELBO bound; the grid uses tk = k/10 for k = 1, …, 9.
H.1 Estimators: The MDLM ELBO estimator samples one noise level per batch and is unbiased for the per-token NELBO, while differing from clipped CDLM training because it integrates over [0, 1].It is the estimator reported in section 8.2 and matches formulations used in Sahoo et al. (2024); Nie et al. (2024); Ni et al. (2025); von Rütte et al. (2025); Sahoo et al. (2026).
H.1 Estimators: The per-block ELBO matches causal block-diffusion training with independent noise per 64-token block and remains unbiased while reducing Monte Carlo variance.Each validation sequence contributes 64 independent noise levels, and the blockwise estimator covers a wider distribution of t values per batch than the MDLM estimator.

H.2 Results … Steerling-8B pretraining details

Across CDLM scaling analyses, compute-optimal parameter exponents are robust to estimator choice but loss values and asymptotes depend strongly on estimator and mask rate. Annealed CDLM+Concept evaluation shifts compute-optimal allocation and sharply improves Concept Independence Loss while leaving most interpretability metrics stable.

H.2 Results: The four estimators locate similar compute-optimal parameter counts but produce absolute-loss offsets of up to one nat across IsoFLOP slices.The per-slice parabolas have comparable minima despite substantially different vertical positions.
H.2 Results: αP fits overlap across ELBO estimators, clustering between 0.602 and 0.632, while αL varies modestly from −0.071 to −0.039.The compute-optimal parameter scaling is therefore robust to estimator choice, whereas loss curves are vertically offset.
H.3 Effect of mask rate on scaling exponents: At fixed mask rates, αL flattens from −0.120 at t = 0.1 to −0.013 at t = 0.9, while αP forms groups near 0.60 for t ≤0.7 and 0.62 for t ≥0.8.Higher mask rates approach the random-token floor, reducing the benefit of additional compute.
H.4 Discussion: Estimator-dependent L∞ spans roughly 1.9 to 3.4, so absolute irreducible losses are not directly comparable across diffusion methodologies using different ELBO estimators.The 1.5-nat spread exceeds the variation reported across diffusion scaling papers, although within-methodology comparisons remain meaningful.
I Annealing each IsoFLOP checkpoint: CDLM IsoFLOP analysis uses independently annealed checkpoints because final schedule decay materially lowers validation loss, despite higher evaluation cost than constant-correction estimates.For +Concept models, annealing also changes αknown, αunknown, and top-k sparsification schedules.
Steerling-8B pretraining details: The supplied passages do not report substantive Steerling-8B pretraining details.No evidence node for that subsection was provided, so no further claim is added.
I.1 Validation loss: Annealing shifts αP from 0.574 to 0.686 while leaving αL essentially unchanged and moving slice-wise L∗ values by at most 0.11 nats.The shift reflects asymmetric loss improvements across model sizes, yielding systematically larger compute-optimal models after annealing.
I.2 Interpretability: Three interpretability metrics remain nearly stable under annealing, whereas Concept Independence Loss decreases 3-10×.Known Concept Alignment changes from β = 0.447 pre-anneal to 0.437 annealed, while schedule changes affect the representations used by Concept Independence Loss.

J Pretraining recipe ablations … N.2 Final mid-training Steerling-8B configuration

The ablations establish a final recipe that preserves capability while improving or maintaining interpretability, and diagnostics link metric changes to masking and teacher-forcing schedules. Mid-training uses real Nemotron tokens because they best recover knowledge and mathematics, with the final configuration reported in Table 36.

J Pretraining recipe ablations; K Final pretraining Steerling-8B configuration; M Steering judge: The ablation program evaluates capability with validation loss, MMLU soft score, and HellaSwag, and interpretability with concept contribution and known concept alignment.The continuous capability metrics retain signal at the 1B-parameter scale, where hard accuracy is near chance.
J Pretraining recipe ablations; J.1 Diffusion block size; J.3 Unknown concept capacity; J.4 Unknown embedding factorization; J.6 Concept teacher forcing schedule; J.7 Unknown concept teacher forcing schedule: The final ablation choices retain interpretability while favoring b = 64, m = 3n, factorized unknown embeddings, and 0.5 teacher-forcing floors.Block size 64 improves validation loss and HellaSwag without changing interpretability; m = 5n provides no measurable gain; factorization is roughly 15× smaller with no capability cost; and the known floor of 0.5 preserves alignment.
J.2 Diffusion masking schedule: The moving Gaussian masking schedule is adopted because it has a slight validation-loss edge, although the two schedules are otherwise very close.MMLU soft score and HellaSwag largely overlap, while interpretability metrics track each other after an early transient.
J.4 Unknown embedding factorization: Factorizing the unknown embedding with rank R = 256 makes it roughly 15× smaller, preserves capability and known concept alignment, but lowers concept contribution from ∼0.90 to ∼0.80.The low-rank bottleneck reduces parameters and per-step compute while increasing reliance on the residual.
J.5 Use of the residual term: Removing the residual forces concept contribution to 1.0 by construction but causes a persistent capability penalty, so the residual is retained.HellaSwag ends near 0.31 without ε versus 0.36 with it; validation loss remains worse and MMLU soft score is weaker.
J.6 Concept teacher forcing schedule: Holding the known teacher-forcing floor at 0.5 preserves alignment near ∼3.7, whereas decay to 0.0 stalls it near 2.6 without affecting capability.The schedule begins with full teacher forcing, uses cosine decay, and then holds at its floor.
L Pretraining diagnostic: During pretraining, capability peaks or plateaus as the masking curriculum moves from center 0.2 toward 0.8, while interpretability degradation coincides with the hard masking regime and teacher-forcing transitions.MMLU and WinoGrande peak before the curriculum crosses 0.5, validation loss plateaus, and concept independence loss rises sharply in the final third.
N.1 Nemotron: real, synthetic, and mixed; N.2 Final mid-training Steerling-8B configuration: Real Nemotron tokens are selected for mid-training because real, synthetic, and mixed sources improve substantially over the base and remain close overall, while real tokens lead on MMLU and GSM8K.Each source is evaluated in a 10B-token run from the final pretraining checkpoint; the final mid-training configuration is listed in Table 36.