The Emergent Symbolic Structure of Artificial Neural Networks

Core approach

• DISCOVER replaces a neural network’s representation-generating process with a single interpretable TPR-based equation, while changing behavior minimally or not at all.TPRs encode structures as role-filler pairs translated into vectors.Source: §1 · p. 2

Scope of findings

• Emergent symbolic structure appears across multilayer perceptrons, recurrent neural networks, and Transformers trained on synthetic sequence-manipulation tasks.This indicates the finding generalizes across three neural-network classes.Source: §1 · p. 2
• All seven analyzed LLMs exhibit emergent symbolic structure, including systems trained on naturalistic data.The models are Gemma-3-27b, GPT-2-XL, GPT-OSS-20b, Pythia-12b, Qwen3-14b, OLMo-2-13B, and Llama-3.1-8b.Source: §1 · p. 2

Cross-domain analysis

• GPT-OSS representations are well approximated by TPRs across arithmetic, logic, coding, and language, supporting implicit symbolic structure during symbolic tasks.These four domains have traditionally been viewed as symbolic.Source: §1 · p. 2

Behavioral interventions

• Targeted representation edits produce expected behavioral changes, such as changing clever from a subject adjective to an object adjective and yielding the corresponding sentence interpretation.This links the identified representational structure to model behavior rather than merely describing it.Source: §1 · p. 2

Systematicity

• DISCOVER generalizes to unseen filler-role combinations, such as using scientist as a sentence subject after training without that combination, demonstrating systematic composition.Together, the findings support the hypothesis that standard neural networks learn implicit symbolic structure important for intelligent behavior.Source: §1 · p. 2

2.1 Understanding the representations of neural networks

• Interpretability research analyzes neural networks’ internal representation vectors, including how interpretable features are encoded and causally influence behavior (Elman, 1990; Bau et al., 2017; Adi et al., 2017; Rogers, Kovaleva, and Rumshisky, 2020; Ravfogel et al., 2021; Hernandez and Andreas, 2021; Elhage et al., 2022; Nanda, Lee, and Wattenberg, 2023; Engels et al., 2025; Todd et al., 2024; Bhalla et al., 2026; Templeton et al., 2024).The Linear Representation Hypothesis models representations as sums of concept vectors and unifies methods including additive analogies, linear probes, and sparse autoencoders (Park, Choe, and Veitch, 2024; Mikolov, Yih, and Zweig, 2013; Ettinger, Elgohary, and Resnik, 2016; Alain and Bengio, 2017; Hupkes, Veldhoen, and Zuidema, 2018; Belinkov, 2022; Cunningham et al., 2023).Source: §2.1 · p. 3

2.2 Structure and the binding problem

• Additive concept representations cannot encode order, creating a binding problem because symbolic domains depend on which elements occupy which positions.The distinction applies to language, mathematics, logic, and computer code, where reversing elements can change meaning (von der Malsburg, 1981, 1999; Treisman, 1999; Roskies, 1999; Greff, Van Steenkiste, and Schmidhuber, 2020).Source: §2.2 · p. 3

2.3 Tensor Product Representations

• Tensor Product Representations encode symbolic structures by binding filler vectors to role vectors with tensor products, summing the resulting matrices, and applying an affine transformation x 7→Wx + b.The resulting linearly-transformed TPR is W(Σ_i f_i ⊗ r_i)+b, preserving role-filler structure while resizing and reshaping the representation.Source: Figure 2.1 · p. 4, §2.3 · p. 5
• TPRs resolve binding through linear unbinding, and the paper hypothesizes that neural networks implicitly use this strategy when performing symbolic tasks.Unbinding is exact when role vectors are linearly independent and can remain highly accurate otherwise.Source: §2.3 · p. 5

3 The central technique: DISCOVER

• DISCOVER was introduced by McCoy et al. (2019) for recurrent networks and is extended here to a broader range of models, including large-scale LLMs.Its purpose is to test whether neural-network encodings can be represented by interpretable symbolic structures.Source: §3 · p. 5

3.1 Running example: A recurrent neural network reversing a list

• The running example uses encoder–decoder GRU networks trained to reverse letter lists, requiring the encoding to preserve both letters and their positions.Ten independently trained copies reached 100% test accuracy on unseen reversal items.Source: §3.1 · pp. 5–6

3.2 The DISCOVER procedure

• DISCOVER trains a linearly transformed TPR to minimize mean squared error against target encodings, then feeds the approximations into the original decoder and measures complete-sequence accuracy.Success means the TPR-based encoding is functionally equivalent to the target encoding for the decoder’s test outputs.Source: §3.2 · p. 6

3.3 Role schemes

• The analysis tests five role schemes—left-to-right, right-to-left, bidirectional, Wickelroles, and bag-of-words—while treating sequence letters as fillers.These hypotheses differ in how each letter’s positional or contextual role is assigned.Source: §3.3 · pp. 6–7

3.4 Results

• Bidirectional TPRs achieved at least 99.98% approximation accuracy across all 10 list-reversal networks, supporting bidirectional symbolic structure in their representations.Right-to-left roles performed nearly as well, whereas left-to-right and bag-of-words roles performed poorly, with the latter serving as a null hypothesis for lacking structure.Source: §3.4 · p. 7

3.5 Notes on the scope and interpretation of DISCOVER

• A successful DISCOVER approximation shows that the TPR captures information used by the target model, but not necessarily that it captures no additional information.The supervised method requires hypotheses about role schemes, and it characterizes representational structure rather than explaining how the network computes it; TPR success is not guaranteed.Source: §3.5 · pp. 7–8

4.1 Tasks

• The models performed copying, reversing, or interleaving on letter lists, tasks requiring representations of which letters appear and where they appear.Sequences had lengths 1–6, used 26 capital letters with repetition allowed, and comprised 57,000 unique input-output pairs per task.Source: §4.1 · p. 8

4.2 Neural network architectures

• The study compared MLPs, GRUs, standard Transformers, and bottleneck Transformers within a shared sequence-to-sequence encoder-decoder framework.The bottleneck Transformer forces the decoder to access only the first encoder vector, requiring that vector to encode the entire sequence.Source: §4.2 · p. 9

4.3 DISCOVER results

• DISCOVER tested five role schemes across 600 models, approximating single encoder vectors for MLPs, GRUs, and bottleneck Transformers and all token encodings for standard Transformers.Bidirectional roles generally outperformed unidirectional alternatives in ways aligned with task structure; bag-of-words performed poorly, whereas Wickelroles underperformed despite having 729 versus 21 roles.Source: §4.3 · pp. 9–10
• 0.973 was the lowest average bidirectional approximation accuracy across 12 architecture-task combinations, while every other setting exceeded 0.99 (Figure 4.1).Bidirectional roles provided strong approximations across all architecture-task combinations, including the reversing bottleneck Transformer.Source: §4.3 · p. 9, Figure 4.1 · p. 10

5.1 Conditions; 5.2 Models

• The study analyzes seven open-weight LLMs at five layers across lists, subject-verb-object sentences, and syntactically varied complex sentences.Lists contain three to five unique nouns; the sentence conditions include controlled occupation-term templates and varied constructions such as conjunctions, subordinate clauses, and passive voice.Source: §5.2 · p. 11, §5.1 · p. 11

5.3 Do the periods actually encode the preceding sentence or list?

• Period-unpacking models recover preceding contexts perfectly for subject-verb-object sentences, typically above 85% for lists and above 50% for complex sentences.Exact word-for-word output is required, so successful decoding depends on structural information that distinguishes alternative word orders.Source: §5.3 · p. 12, Figure 5.1 · p. 12

5.4 DISCOVER results

• Bidirectional roles approximate list encodings well, while predecessor roles perform substantially worse despite using 301 roles versus 12.Lists contain unique nouns, making predecessor role-filler pairs sufficient to reconstruct each list.Source: §5.1 · p. 11, §5.4 · p. 13
• For subject-verb-object sentences, DISCOVER achieves perfect approximation with distinct subject, verb, and object roles, which distinguish reversed subject-object assignments.The sentence dataset permits any noun as subject or object, making separate subject and object roles necessary.Source: §5.1 · p. 11, §5.4 · pp. 13–14
• Across 7 LLMs, DISCOVER closely approximates period encodings, capturing most preceding list or sentence structure and providing evidence for TPR structure in LLM representational spaces.Figure 5.2 evaluates approximations through period-unpacking accuracy; appropriate role schemes achieve high accuracy across lists, subject-verb-object sentences, and complex sentences.Source: §5.4 · pp. 13–14, Figure 5.2 · p. 13

6 Arithmetic, logic, coding, and language in an LLM

• The experiments address whether structure found in LLM encodings is used by the LLM itself by approximating all input representations across layers and tokens, rather than only period encodings.The analysis examines GPT-OSS on math, logic, computer code, and language to study how a vector-based LLM performs seemingly symbolic computations.Source: §6 · p. 14

6.1 Tasks

• The task suite spans arithmetic, syllogistic deduction, Python list-function execution, passivization, tense reinflection, and yes/no question formation across four symbolic domains.Prompts include correct-answer examples for in-context learning, and the language tasks use syntactically variable subject-verb-object sentences.Source: Table 1 · p. 15, §6.1 · p. 16

6.2 Model

• GPT-OSS performs all six tasks well: its lowest accuracy is 0.76 on syllogisms, followed by 0.94 on passivization, while every other task exceeds 0.96.The study analyzes one model because the experiments required a little over 3,000 GPU hours.Source: §6.2 · p. 16

6.3 DISCOVER results

• DISCOVER replaces GPT-OSS’s input representations across all layers and tokens with closed-form TPR approximations, while leaving uniform instructions unchanged and testing whether outputs remain correct.Five role schemes compare single-pair versus multi-pair representations and task-agnostic versus task-specific structure; task-specific (all) uses pair-based roles to capture token interactions.Source: §6.3 · pp. 16–18, Figure 6.1 · p. 17
• For all six tasks, task-specific (all) DISCOVER approximations outperform the other role schemes and come extremely close to GPT-OSS’s own accuracy, with the largest average gap 2.36% on arithmetic.The results support that each token encodes multiple role-filler pairs whose semantics correspond to task-specific structure.Source: §6.3 · p. 18

7 Causal interventions

• TPR constituent surgery edits representations by subtracting one role-filler pair and adding another, changing the represented structure without assuming information is localized to individual vector dimensions.Because a complete TPR is the sum of its role-filler pairs, the same closed-form edits can be applied to target-model representations predicted by DISCOVER.Source: §7 · p. 19

7.1 Letter sequence models

• Causal interventions on letter-sequence models achieved near-perfect accuracy across architectures and tasks, using bidirectional-role representations from DISCOVER.The analysis included MLPs, GRUs, and bottleneck Transformers with single-vector encodings, excluding standard Transformers because their multi-vector representations would make interventions overwrite whole vectors.Source: §7.1 · p. 20

7.2 Period encodings

• Interventions replacing words in lists or nouns in sentences achieved high accuracy, coming close in all cases to the target model’s accuracy.The list interventions replaced one word in length-five lists, while sentence interventions replaced nouns in templates of the form “The NOUN VERBED the NOUN.”Source: §7.2 · p. 20

7.3 Symbolic tasks in GPT-OSS

• Across 31 intervention types spanning six GPT-OSS tasks, edits achieved 0.903 average intervention accuracy, demonstrating sensitivity to the constituent structure of the approximating TPRs.The interventions edit one part of the symbolic structure and test whether GPT-OSS changes its behavior appropriately.Source: §7.3 · p. 21
• GPT-OSS interventions successfully changed fillers and structural roles, including arithmetic values, code elements, and modifier positions, with filler examples reaching 0.978 accuracy.Role-changing interventions modify an element’s structural position by replacing its current role-filler pair with a new one; GPT-OSS uses RoPE rather than explicit positional vectors, so the edited positional information is emergent.Source: §7.3 · p. 21, Figure 7.4 · p. 22
• Local filler edits remained nearly as effective as full edits, whereas local role edits performed substantially worse, indicating localized token identity but distributed structural representations.Interventions respecting sentence structure achieved high accuracy, while those preserving only linear order but not sentence structure achieved low accuracy.Source: §7.3 · p. 22

7.4 Causal intervention summary

• Across all DISCOVER-approximated model types, fine-grained edits to internal representations changed behavior, showing sensitivity to the TPRs’ compositional structure.This causal result supports the claim that the identified symbolic structure is behaviorally implicated rather than merely descriptive.Source: §7.4 · p. 22

Interpretation

• Generalization to novel pairings is diagnostic of systematic role-filler binding, because DISCOVER lacks a basis for representing unseen combinations when roles and fillers are not systematically composed.White-box validation supports this interpretation: DISCOVER generalizes for models known to use systematic binding but fails for models known not to use it.Source: §8 · p. 23

Methods

• DISCOVER is trained while withholding selected role-filler pairs, then evaluated on examples containing increasing numbers of those unfamiliar combinations; the target models are trained normally.This procedure tests systematic structure in the target representations rather than systematic generalization by the target models themselves.Source: §8 · p. 23
• The strong chance baseline is 1/n! for examples with n unseen role-filler pairs, because unmatched fillers can only be randomly assigned to the remaining slots without systematic binding.A weaker vocabulary-random baseline, with accuracy (1/|V_f|)^n, is omitted because it is usually near zero.Source: §8 · p. 23

Results

• DISCOVER robustly generalizes above baseline for almost all target models, including letter-sequence models, period encodings, and GPT-OSS on all tasks except arithmetic.Some letter-sequence cases retain near-ceiling accuracy, while period encodings generalize substantially above baseline across all models; GPT-OSS shows strong performance for larger numbers of unseen pairs.Source: Results · p. 24, Figure 8.1 · p. 24, Figure 8.2 · p. 24

9 Discussion

• Across many architectures and training tasks, neural networks exhibit linearly transformed TPR structure that supports targeted behavioral interventions and out-of-distribution generalization.These results suggest the structure is systematic, general, and relied upon during model behavior.Source: §9 · p. 25

9.1 The relationship between symbols and neural networks: A symbolic victory?

• The findings support limitivism: neural networks internally use symbols, but their symbolic structure is approximately rather than precisely realized.DISCOVER’s average causal-intervention accuracy on GPT-OSS was 0.903, while period-unpacking accuracy rose from 0.71 on GPT-OSS encodings to 0.96 on bidirectional DISCOVER approximations.Source: §9.1 · pp. 25–26

9.2 Other types of role-filler representations

• DISCOVER identifies a broad role-filler structure shared by major vector symbolic architectures, but cannot determine which specific architecture a target model uses.Future work should distinguish among these equivalent formalisms and investigate deviations that may integrate systematic and fuzzy computation.Source: §9.1 · p. 26, §9.2 · p. 27

9.3 Extensions of DISCOVER

• Future DISCOVER studies should explain how networks construct and process TPR structure, extend it to continuous or recursive fillers and roles, and scale analyses to broader stimuli.Broader applications may require unsupervised discovery rather than manually hypothesized fillers and roles.Source: §9.3 · p. 27

9.4 Implications for mechanistic interpretability

• The results suggest that mechanistic interpretability should analyze multiplicative filler-role combinations, not only neurons, heads, directions, or circuits.Such combinations may be causally relevant units underlying existing and future interpretability methods.Source: §9.4 · p. 27

9.5 Why do neural networks perform so poorly at compositional generalization?

• Compositional representations need not yield compositional generalization if networks learn only encountered role-filler combinations.This possibility could explain failures on novel positions and combinations despite internally compositional representations.Source: §9.5 · p. 28

9.6 Might the brain use TPRs?

• TPR structure might extend from artificial networks to the brain, but this possibility requires empirical tests on brain recordings with matched stimuli.DISCOVER’s fits to LLMs and LLMs’ fits to brain responses provide indirect motivation, not strong evidence for TPRs in biological cognition.Source: §9.6 · p. 28

Analyzing neural networks with TPRs

• DISCOVER extends earlier architecture-specific or limited applications to diverse neural architectures, including modern LLMs, while retaining the capacity for causal interventions.Earlier work analyzed simplified LSTMs, RNNs, sentence embeddings, SCAN models, and BERT; DISCOVER addresses architecture generality and complexity limitations.Source: §10 · p. 29
• DISCOVER identifies TPR structure across neural architectures and symbolic-domain LLMs, shows that this structure causally affects behavior, and demonstrates generalization to novel role-filler pairs.These contributions support the claim that systematic role-filler structure is broadly used for symbolic tasks.Source: §10 · p. 29

Symbolic mechanisms in neural networks

• Unlike mechanism- and behavior-focused studies of symbolic processing, this paper analyzes how models represent structured inputs, particularly role-filler representations and variable binding.Prior work studies internal processing mechanisms, behavioral binding, or variable-like subspaces; this paper focuses on representational structure.Source: §10 · p. 29

Analyzing structure in neural networks

• Unlike decoding methods that read structural information from vectors, DISCOVER uses encoding to reconstruct the complete representational space and analyze its global structure.The paper differs from other complete-space mapping approaches in its methods, analyzed models, and TPR-based representational structure.Source: §10 · pp. 29–30

Integrating TPRs into neural networks

• By revealing which structured representations naturally emerge in trained networks, DISCOVER may inform explicit TPR-based architectures and inductive biases that encourage robust or corrected representations.This proposed application includes identifying shortcomings to overcome or robust structures to encourage with less training.Source: §10 · p. 30

11 Conclusion

• Neural and symbolic computation need not conflict: neural networks can construct symbolic representations that closely approximate their learned representations through interpretable closed-form equations.The study analyzes fully systematic domains and identifies understanding representations in partially systematic settings as an important future direction.Source: §11 · p. 30

A Linearly-transformed TPRs vs. basic TPRs

• DISCOVER approximates neural-network representations with linearly transformed Tensor Product Representations rather than basic TPRs.The transformation supports target models whose representations are vectors rather than matrices and allows flexible representational dimensions.Source: A Linearly-transformed TPRs vs. basic… · p. 38
• Linear transformations preserve shared symbolic structure despite differences caused by coordinate reordering, rotation, stretching, or other non-one-to-one alignments.Without the transformation, approximation would require corresponding TPR and target-model elements to appear in the same order and coordinates.Source: A Linearly-transformed TPRs vs. basic… · pp. 38–39

B Technical details for letter sequence models

• Target models are evaluated against exact output sequences ending with an <EOS> token, with outputs truncated at the first <EOS> or padding token.All models use dropout proportion 0.1, and correctness requires producing the precise target sequence followed by <EOS>.Source: §B.1 · p. 39

B.1 Target models

• The MLP embeds and concatenates padded sequences, predicts each output position independently through softmaxes, and exposes its first-ReLU representation to DISCOVER.It uses embedding size 10, hidden size 64, maximum sequence length 6, and vocabulary size 29.Source: §B.1 · p. 39
• The GRU, standard Transformer, and bottleneck Transformer provide recurrent, tokenwise attention, and single-vector sequence-encoding targets for DISCOVER.DISCOVER approximates the GRU encoder’s final hidden state, each standard-Transformer encoder hidden state, and the bottleneck Transformer’s first-token representation.Source: §B.1 · p. 39
• Target-model accuracies on letter-sequence manipulation tasks are summarized as means across 10 reruns with error bars spanning two standard deviations.The figure reports the performance of the target models used in the sequence experiments.Source: Figure B.1 · p. 40

B.2 Training target models

• Target models used negative log likelihood loss, with teacher forcing during training for autoregressive decoders but not during evaluation.Source: §B.2 · p. 40
• Target-model training began with learning rate 0.001 and batch size 32, evaluating validation loss every 300 batches.Source: §B.2 · p. 40
• Models saved the best validation checkpoint, halved the learning rate after five unimproved evaluations, and stopped when a sixth decrease would apply.Source: §B.2 · p. 40

B.3 Training DISCOVER models

• DISCOVER used 20-dimensional filler and role embeddings, yielding 400-dimensional TPRs compressed to the target models’ 64-dimensional hidden size by a trainable biased linear layer.Source: §B.3 · p. 40
• DISCOVER minimized mean squared error between target and DISCOVER encodings while using the target models’ batch size, evaluation schedule, learning-rate decay, stopping rule, and dataset splits.Source: §B.3 · p. 40
• Approximation accuracy was the proportion of test inputs for which the target decoder produced the correct output sequence from DISCOVER encodings.Source: §B.3 · p. 40

C Analyzing models trained on alphabetizing a list

• Sorting models achieved high task accuracy, while DISCOVER found all were reasonably approximated by bag-of-words roles, with order-sensitive roles still often outperforming them.Average accuracy was 0.973 for MLPs and over 0.99 for the other architectures; the sorting task did not require representing input order.Source: C Analyzing models trained on… · pp. 40–41, Figure C.1 · p. 41
• Best role-scheme approximation accuracies were 0.979 for MLPs, 0.935 for GRUs, 0.871 for Transformers, and 0.979 for bottleneck Transformers.The remaining headroom may reflect rough rather than systematic TPR structure, or a need for role schemes combining linear, alphabetical, and contextual information.Source: C Analyzing models trained on… · p. 41

D Context-free grammar for complex sentences

• The complex-sentence CFG provides constrained variable templates rather than realistic syntactic analyses, and generated sentences were capitalized, comma-normalized, and period-terminated before LLM input.Relative clauses were restricted to otherwise simple sentences and could not be nested; the grammar contained 80 nouns, 16 V verbs, and 10 adjectives.Source: D Context-free grammar for complex… · p. 41, Figure D.1 · p. 42

E Rogue dimensions and z-scoring

• LLM representations were z-scored before DISCOVER approximation or decoder training to reduce the influence of behaviorally unimportant rogue dimensions.Z-scoring gives vector dimensions comparable prominence and is h−µ divided by σ, where µ and σ contain per-dimension means and standard deviations.Source: E Rogue dimensions and z-scoring · pp. 41–42
• Because z-scoring and its inverse are linear transformations, equivalent behavior can be obtained by incorporating them into model weights without changing expressive capacity.For GPT-OSS symbolic-task evaluation, inverse z-scoring translates DISCOVER approximations back into the original GPT-OSS representation space.Source: E Rogue dimensions and z-scoring · pp. 42–43

F.1 Information about the LLMs we studied

• LLM period encodings were analyzed at approximately 0%, 25%, 50%, 75%, and 100% of processing, beginning at layer 1 after the embedding layer.Table 3 records the selected layers and hidden-state sizes, with positions sometimes inexact because layer counts were not always divisible by four.Source: §F.1 · p. 43, Table 3 · p. 43

F.2 Dataset details

• The datasets contained 81,920/10,240/10,240 subject-verb-object train/validation/test sentences, 100,000/1,000/5,000 list examples, and 200,000/5,000/5,000 complex sentences.Subject-verb-object data were generated from all template-compatible sentences and split 80%/10%/10%.Source: §F.2 · p. 43

F.3 Training period-unpacking models

• Period-unpacking models are 6-layer, 16-head decoder-only Transformers with hidden size 1024, feedforward size 4096, and 1024-dimensional compressed LLM period inputs.A trained linear layer compresses each LLM period encoding to size 1024 before the Transformer processes it.Source: §F.3 · pp. 43–44
• For each condition, LLM, and layer, five period-unpacking models reconstruct lists or uncapitalized, separately punctuated sentences, ending with a special end-of-sequence token.List targets exclude preambles and punctuation, whereas sentence targets retain the complete sentence with punctuation tokenized separately.Source: §F.3 · p. 44

F.4 DISCOVER

• DISCOVER compares bidirectional, predecessor, subject-verb-object, syntactic, and bag-of-words role schemes, with scheme availability determined by the list or sentence condition.Syntactic roles are dependency-path labels, while bag-of-words assigns every filler the same role and therefore captures no structure.Source: §F.4 · pp. 44–45
• DISCOVER models approximate target representations with mean-squared error using condition-specific filler and role embeddings, matching the decoders’ data splits and checkpoint-based validation.Lists use 300-dimensional filler and 100-dimensional role embeddings; both sentence settings use 100-dimensional filler and role embeddings.Source: §F.4 · p. 45

G Period encodings with syntactic outputs

• The syntactic-output experiment changed only the period-unpacking decoder, reusing the original DISCOVER approximations to test whether syntactic targets improved syntactic-role performance.New decoders produced either linearized constituency parses or linearized dependency parses, while the original decoder reproduced sentences as linear strings.Source: G Period encodings with syntactic… · pp. 45–46
• Across sentence, constituency-parse, and dependency-parse decoders, bidirectional roles remained the best-performing DISCOVER scheme, indicating that period encodings prioritize sequential over syntactic information.The same DISCOVER approximations were evaluated with the original linear decoder and newly trained syntactic decoders; other token encodings might still exhibit stronger syntactic structure.Source: G Period encodings with syntactic… · p. 46, Figure G.1 · p. 46

H.1 Datasets and GPT-OSS’s performance

• GPT-OSS is evaluated on six tasks spanning arithmetic, syllogistic reasoning, code execution, passivization, tense reinflection, and question formation.The datasets use generated examples with task-specific constraints, including order-of-operations demonstrations, existence assumptions, variable binding, syntactic transformations, and tokenizer-based controls.Source: §H.1 · pp. 46–48

H.2 Role schemes

• The role schemes range from task-agnostic bag-of-words and bidirectional schemes to task-specific roles that encode structural positions within each symbolic task.Task-specific roles represent arithmetic operators and digit properties, syllogistic sentence positions, function and variable bindings, and syntactic positions.Source: §H.2 · p. 49

H.3 DISCOVER

• DISCOVER approximates only token representations that vary across prompts, including variable representations of later formatting tokens caused by preceding stimulus-dependent tokens.One DISCOVER model is trained for each of GPT-OSS’s 25 layers, sharing role–filler hypotheses across layers while allowing learned parameters to vary by layer.Source: §H.3 · pp. 49–50

I.1 Number of intervention examples

• GPT-OSS causal-intervention experiments use 100 interventions per condition, reflecting the greater computational cost of these experiments.The letter-sequence models and LLM period encodings instead use datasets of 1,000 interventions.Source: §I.1 · p. 50

I.2 LLM period encodings

• Causal interventions on lists and sentences use fixed templates, restricted decoders, and DISCOVER models to replace randomly selected words or subjects and objects.The list condition uses length-5 lists and bidirectional roles, while the sentence condition uses two-clause templates with six subject, verb, and object roles.Source: §I.2 · p. 50

I.3 GPT-OSS in symbolic domains

• In symbolic-domain interventions, replacing a role–filler pair requires editing every token whose representation is hypothesized to contain that filler from its occurrence onward.The task-specific (all) scheme is used, subtracting the original role–filler pair and adding the replacement pair at each affected token position.Source: §I.3 · p. 51, Table 4 · p. 51

I Technical details for causal interventions

• Additional causal-intervention procedures, examples, role-changing details, and accuracy metrics are provided in Supplement S2.These supplementary materials cover all intervention categories used for GPT-OSS.Source: §I.3 · p. 51

J Testing for structure-sensitivity of causal interventions in GPT-OSS

• DISCOVER tests paired representations and edit vectors that share linear movements but differ structurally, checking whether each edit works only with its structurally matching representation.The analysis evaluates all four pairings: two starting representations crossed with two edit vectors derived from subject/object phrase or clause shifts.Source: J Testing for structure-sensitivity of… · p. 52
• The structure-sensitivity analysis covers prepositional-phrase and relative-clause interventions, forward and backward adjective movements, subject- and object-modifying structures, and passive and tense tasks.These intervention types test whether the causal edits track abstract syntactic roles across varied sentence configurations.Source: J Testing for structure-sensitivity of… · p. 53
• Across all 8 intervention cases, matched edits outperformed mismatched edits in the predicted diagonal pattern, showing sensitivity to sentence structure rather than purely linear position.The clearest pattern occurred for tense,PP,front; mismatched interventions were substantially less accurate in weaker cases such as tense,RC,back.Source: Figure J.1 · p. 52, J Testing for structure-sensitivity of… · p. 53

K.1 General note: Restricted templates

• DISCOVER training generally varies sentence structure to distinguish competing role schemes, while restricted templates are used for withheld role-filler tests to prevent leakage across linear and syntactic hypotheses.Strict templates ensure that withholding a role-filler pair under one role scheme also withholds its corresponding pair under another plausible scheme.Source: §K.1 · pp. 53–54

K.2 Letter sequence models

• Letter-sequence experiments use 10 reruns per setting and bidirectional withholding, with four OOD test splits containing 1, 2, 3, or 4 withheld role-filler pairs.Pairs involving B and E select regularization through OOD validation; the other four withheld pairs provide OOD testing.Source: §K.2 · p. 54

K.3 Period encodings

• Period-encoding experiments test lists of length 5 and repeated-verb sentences, withholding nouns from positions and evaluating OOD splits with increasing numbers of withheld role-filler pairs.List evaluations use 5 splits with 1–5 withheld pairs; sentence evaluations use 4 splits with 1–4 withheld pairs and six syntactic roles.Source: §K.3 · pp. 54–55
• All considered LLM layers show robust period-encoding generalization substantially above chance, except that some first-layer results perform at or below baseline.Figure K.1 reports means across 10 reruns with error bars showing two standard deviations and dashed strong-baseline lines.Source: §K.3 · p. 55, Figure K.1 · p. 55

K.4 GPT-OSS performing symbolic tasks

• GPT-OSS symbolic-task experiments withheld role-filler pairs from consistent structural and linear positions, reserving disjoint pairs for OOD validation and testing across arithmetic, syllogisms, code, and language tasks.Constrained templates and fixed list structures ensured that withheld fillers corresponded to consistent positions; each DISCOVER model used a task-specific role scheme.Source: §K.4 · pp. 55–56

L.1 Letter sequence models

• Table 5 reports OOD validation accuracies for letter-sequence models with two withheld role-filler pairs, comparing unregularized λ = 0 with regularization using the best λ.Source: Table 5 · p. 56

L Regularization for OOD generalization

• L2,1 regularization addresses role-filler memorization by encouraging unused embedding dimensions while preserving a flexible embedding size; λ is selected on held-out OOD validation pairs and evaluated on disjoint OOD test pairs.The loss adds L2,1 penalties to the role and filler embedding matrices alongside the mean-squared approximation loss.Source: L Regularization for OOD generalization · pp. 56–57

L.2 LLM period encodings

• Regularization made a qualitative difference for LLM period encodings, with λ = 0.001 adopted broadly because it performed best or nearly best across tested models and layers.For Llama-3.1, λ = 0.001 achieved 0.896 versus 0.898 for the best λ.Source: Table 6 · p. 56, §L.2 · p. 57

L.3 GPT-OSS performing symbolic tasks

• For GPT-OSS symbolic tasks, the λ search was conducted separately for each of the six tasks using OOD validation results.The results are reported in Table 6 alongside the LLM period-encoding search.Source: Table 6 · p. 56, §L.3 · p. 57

M Generalization to novel role-filler pairs in white-box models

• The white-box validation experiment froze randomly initialized TPR and Atomic Pair encoders while training Transformer decoders, preventing training from creating systematic structure in the nonsystematic architecture.The TPR encoder had explicit systematic role-filler binding, whereas the Atomic Pair Encoder independently embedded each role-filler pair.Source: M Generalization to novel role-filler… · pp. 57–58
• DISCOVER generalized strongly to novel role-filler pairs in the white-box TPR model but failed completely in the Atomic Pair model, validating this test as evidence for systematic binding.Both models performed above 0.99 on copying, and both supported high in-distribution DISCOVER accuracy; only novel-pair generalization separated them.Source: M Generalization to novel role-filler… · p. 58, Figure M.1 · p. 58

Letter sequence results

• In letter-sequence experiments, bidirectional roles outperform Wickelroles in every condition despite using 21 rather than 729 roles, while left-to-right and right-to-left roles differ substantially despite each using 6 roles.The comparisons are sufficient to identify the relevant sequences, except that Wickelroles can conflate some sequences; this limitation does not explain all of their low performance.Source: N Is DISCOVER performance merely… · p. 59

Period encoding results

• In period-encoding lists, bidirectional roles consistently outperform predecessor roles despite using 12 rather than 301 roles, even though the unique elements make predecessor roles sufficient to identify each sequence.Source: N Is DISCOVER performance merely… · p. 59

GPT-OSS results

• In GPT-OSS, task-specific (all) outperforms bidirectional (all) in every condition, while task-specific (self) slightly outperforms bidirectional (all) in code execution with 99 versus 1,329 roles.Although task-specific (all) uses more roles in 5 of 6 conditions, it uses fewer in syllogisms; causal-intervention evidence also supports task-specific syntactic structure in two linguistic tasks.Source: N Is DISCOVER performance merely… · p. 59

• DISCOVER differences across role schemes are not primarily driven by role count, supporting the interpretation that approximation accuracy reflects how well a scheme captures the target representations’ causal structure.This is the section’s overall conclusion across the examined settings.Source: N Is DISCOVER performance merely… · p. 59