PaperlayerView exact revision on arXiv

Source-linked AI summary

The Emergent Symbolic Structure of Artificial Neural Networks

R. Thomas McCoy, Paul Soulos, Tal Linzen, Paul Smolensky

arXiv:2608.29530v1cs.CLcs.AI

TL;DR

Neural networks perform well in domains that appear to require symbolic structure despite using continuous vectors, raising questions about how their representations support this behavior. The paper uses DISCOVER and TPRs to approximate those representations, finding symbolic structure across neural networks and LLMs, with targeted interventions changing behavior.

  • Problem

    The paper asks how neural networks using continuous vectors can perform well in domains whose structure has traditionally been modeled symbolically.

  • Method

    DISCOVER approximates neural-network representations with linearly-transformed Tensor Product Representations that encode fillers together with their roles.

  • Results

    TPR approximations closely match representations across neural networks and LLMs, including GPT-OSS tasks in arithmetic, logic, coding, and language, while behavior changes minimally if at all.

  • Takeaways & Limitations

    The findings support the possibility that neural systems can be simultaneously neural and symbolic by constructing symbolic representations in vector space.

  • Takeaways & Limitations

    DISCOVER characterizes representational structure but does not explain how networks produce vectors with that structure.

Abstract

from arXiv · show

Modern systems in artificial intelligence (AI) somehow excel in domains for which they seem poorly suited. Intelligence has traditionally been modeled as operating over structured combinations of symbols, such as logical formulas. However, the strongest modern AI systems are based on neural networks, which instead represent information in continuous vectors. Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas. How do they do it? In this work, we propose a potential answer: Despite appearances, perhaps the internal representations of neural networks implicitly realize symbolic structure. In support of this hypothesis, we show that the vector representations of a variety of neural networks can be closely approximated with symbolic structures: we can replace the network's entire representation-generating process with a closed-form equation instantiating a symbolic structure, and the network's behavior remains largely unchanged. This finding holds for both small-scale neural networks trained to manipulate lists as well as large language models (LLMs) operating in four domains that are central in symbolic traditions: arithmetic, logic, computer code, and language. Further, our symbolic approximation allows us to modify an LLM's behavior in targeted ways via precise interventions on its internal representations, showing that the LLM's behavior is reliant on the symbolic structures we have identified. This work provides a potential way to reconcile longstanding symbolic conceptions of intelligence with the vector-based nature of modern AI.

1 Introduction

The paper asks how continuous-vector neural networks achieve strong performance in domains that appear to require symbolic structure. It tests whether their representations implicitly realize symbolic structure and finds this pattern across neural networks and LLMs.

  • Motivation: Neural networks use continuous vectors yet excel in traditionally symbolic domains such as language, mathematics, and coding.This contrast motivates the paper’s mechanistic-interpretability question.
  • Approach: DISCOVER approximates neural-network representations with vectors that explicitly encode symbolic structure using Tensor Product Representations.TPRs pair fillers with roles that denote their positions in a structure.
  • Core finding: A single interpretable TPR-based equation can replace a network’s entire representation-generating process while changing behavior minimally if at all.The approximation is used as a closed-form replacement for the original representation-generating process.
  • Core findings: Emergent symbolic structure appears across multilayer perceptrons, recurrent neural networks, and Transformers trained on synthetic sequence-manipulation tasks.The finding spans all three analyzed neural-network classes.
  • Core findings: Emergent symbolic structure also appears in seven LLMs trained on naturalistic data and in GPT-OSS tasks spanning arithmetic, logic, coding, and language.The four-domain analysis provides evidence that LLM representations use implicit symbolic structure for symbolic tasks.
  • Behavioral relevance: Intervening on internal representations can change model behavior in targeted ways, such as relocating an adjective from the subject to the object.Changing “clever” from a subject adjective to an object adjective makes the model behave as if it modified “lawyer.”
  • Behavioral relevance: DISCOVER generalizes to new filler-role combinations, including a word appearing in a role absent from its training examples.This supports systematic composition of fillers with roles.
  • Conclusion: Together, the findings support the hypothesis that standard neural networks can learn implicit symbolic structure important for intelligent behavior.The conclusion is framed as support for the hypothesis rather than as a claim that every neural network has this structure.

2 Background

The background frames a limitation of additive concept-vector representations: they do not encode how elements are arranged. It introduces TPRs as a vector-space formalism that binds fillers to roles, with DISCOVER testing whether neural networks use this strategy.

  • Interpretability: Mechanistic interpretability studies how neural networks operate internally, including the vectors that serve as their representations.The paper focuses on representational analysis within mechanistic interpretability.
  • Existing framework: The Linear Representation Hypothesis represents a neural vector as a sum of vectors for the concepts it contains.This framework unifies several representational-interpretability methods.
  • The binding problem: Additive concept vectors cannot distinguish ordered structures such as “cats chase dogs” and “dogs chase cats.”The same summed encoding would be assigned despite the sentences having different meanings.
  • The binding problem: The binding problem asks how a neural network can bind different pieces of information to the positions they occupy in a structure.Position information is needed to represent relations such as subject and object.
  • Tensor Product Representations: TPRs represent symbolic structures as fillers paired with roles, then translate those role-filler pairs into vectors.For example, the fillers in “cats chase dogs” can be paired with subject, verb, and object roles.
  • Tensor Product Representations: Linearly-transformed TPRs bind each filler and role with a tensor product, sum the products, and apply a linear transformation to produce the final encoding.The transformation allows superficial changes to the vector space without changing relevant TPR properties.
  • Tensor Product Representations: TPRs support recovering which filler occupies each role through linear unbinding, exactly when role vectors are linearly independent.Unbinding can still work with high accuracy when linear independence does not hold.
  • Paper hypothesis: The paper tests whether this TPR strategy is implicitly used by neural networks when they perform symbolic tasks.This is the core hypothesis motivating the subsequent analyses.

3 The central technique: DISCOVER

DISCOVER tests whether neural-network encodings implicitly realize symbolic structure by replacing them with interpretable Tensor Product Representations and checking whether the original decoder still behaves correctly. In the running example, role schemes determine how sequence positions are represented, with bidirectional roles closely matching reversal-task encodings, while the method's interpretation remains bounded.

  • Running example: The running example uses a sequence-to-sequence GRU network whose encoder maps a letter list to one vector and whose decoder produces the reversed list.Ten independently trained copies reached 100% test accuracy on unseen reversal items.
  • DISCOVER procedure: DISCOVER trains a linearly-transformed TPR to approximate a target network’s encodings, then feeds those approximations into the target decoder to test behavioral preservation.The TPR model minimizes mean squared error between its encodings and the target encodings.
  • Role schemes: DISCOVER compares alternative role schemes, including left-to-right, right-to-left, bidirectional, Wickelroles, and bag-of-words representations for sequence letters.These schemes differ in whether roles encode directional position, neighboring letters, or no positional distinctions.
  • Results: 99.98% was the lowest approximation accuracy across 10 reruns for the bidirectional TPR, which achieved perfect or near-perfect accuracy in every rerun.Right-to-left roles performed nearly as well, whereas left-to-right and bag-of-words performed poorly for reversal.
  • Interpretability: DISCOVER produces an interpretable closed-form equation for representations that makes black-box encodings substantially more human-understandable.The approximation replaces the network’s representation-generating process with a symbolic TPR-based expression.
  • Scope and interpretation: A successful approximation does not guarantee that the target network uses exactly the same information structure as the DISCOVER model, because the latter may be more expressive.The authors also distinguish identifying representational structure from explaining how the network computes that structure.

4 Letter sequence models

The study tests whether symbolic Tensor Product Representation structure emerges across neural architectures trained on structured letter-sequence tasks. DISCOVER finds strong approximations across tasks and architectures, with role schemes reflecting task-dependent processing.

  • 4 Letter sequence models: The models perform copying, reversing, or interleaving on letter lists, requiring representations of both letters and their positions.Sequences contain one to six capital letters, with 57,000 unique input-output pairs generated per task.
  • 4 Letter sequence models: Four architectures—MLPs, GRUs, Transformers, and bottleneck Transformers—were trained in sequence-to-sequence settings.The bottleneck Transformer restricts its decoder to the first encoder position, forcing that vector to combine information from the entire sequence.
  • 4 Letter sequence models: DISCOVER evaluates five role-scheme hypotheses by approximating the representations produced by each trained sequence-manipulation model.The analyses include 600 DISCOVER models across architectures, tasks, reruns, and role schemes.
  • 4.3 DISCOVER results: 0.973 is the lowest average bidirectional approximation accuracy across the 12 architecture-task combinations; all other combinations exceed 0.99.Bidirectional roles outperform the relevant unidirectional alternatives overall, while left-to-right tends to fit copying and right-to-left tends to fit reversing.
  • 4.3 DISCOVER results: Bag-of-words roles perform poorly for structure-dependent tasks, whereas they approximate models trained to sort lists alphabetically.Wickelroles also underperform bidirectional roles despite having 729 roles versus 21, indicating that parameter count alone does not determine approximation success.
  • 4.3 DISCOVER results: DISCOVER provides strong approximations across all tested tasks and architectures, although the role scheme instantiated varies with the task or architecture.The results support TPR structure as a broad approach for encoding structured information in these neural networks.

5 LLM representations

The study examines whether LLM period representations encode preceding lists and sentences, first testing whether those representations can be unpacked and then approximating them with DISCOVER. Across seven LLMs and multiple linguistic conditions, the approximations preserve substantial structural information, with role schemes differing by condition.

  • 5 LLM representations: The analysis hypothesizes that a period representation encodes the entire preceding sentence or list because it is the earliest position with access to the preceding context.The study tests this assumption before applying DISCOVER.
  • 5 LLM representations: The experiments analyze period encodings after noun lists, subject-verb-object sentences, and syntactically varied complex sentences.The study uses seven open-weights LLMs and examines period representations at five layers in each model.
  • 5.3 Do the periods actually encode the preceding sentence or list?: DISCOVER approximates LLM period encodings by training a TPR-based model and evaluating its outputs through independently trained period-unpacking models.The evaluation asks whether the approximation preserves the information used by the unpacking model.
  • 5.3 Do the periods actually encode the preceding sentence or list?: Period-unpacking models reconstruct all subject-verb-object inputs perfectly, with typical accuracy above 85% for lists and above 50% for complex sentences.Exact word-for-word reconstruction requires recovering structural information, not merely the words present.
  • 5.4 DISCOVER results: Subject-verb-object role schemes achieve perfect approximation accuracy, using distinct subject, verb, and object roles.Distinct subject and object roles distinguish sentences such as The photographer called the teacher from the reversed argument order.
  • 5.4 DISCOVER results: Bidirectional roles consistently outperform syntactic roles for complex sentences, suggesting that period encodings mainly capture linear order rather than syntactic structure.Across seven LLMs, DISCOVER closely approximates period encodings, providing evidence for TPR structure in their representational spaces.

6 Arithmetic, logic, coding, and language in an LLM

The authors analyze GPT-OSS on symbolic tasks spanning arithmetic, logic, coding, and language, replacing its input representations with DISCOVER TPR approximations. Task-specific role-filler structure closely reproduces the model’s performance across all six tasks.

  • Tasks: GPT-OSS is evaluated on six tasks across arithmetic, syllogistic logic, Python code execution, and three English language transformations.The language tasks are passivization, tense reinflection, and question formation.
  • Method: DISCOVER replaces GPT-OSS’s input-token representations at each layer with TPR approximations while leaving task instructions unchanged.A separate DISCOVER model is trained for each of GPT-OSS’s 25 layers.
  • Role schemes: Task-specific (all) outperforms the other role schemes on all six tasks.This scheme represents each token and preceding tokens using roles formed from task-specific positions for both tokens.
  • Results: DISCOVER approximation accuracy remains extremely close to GPT-OSS’s own task accuracy, with the largest gap being 2.36% on arithmetic.The result supports representations containing multiple role-filler pairs with task-specific structural semantics.

7 Causal interventions

The authors test whether TPR components are causally involved in model behavior by editing role-filler components of neural representations. These interventions produce appropriate behavioral changes, including in GPT-OSS symbolic tasks.

  • Editing TPRs: TPR constituent surgery edits a representation by subtracting one role-filler component and adding another.Because a complete TPR is the sum of its role-filler components, this operation changes a selected constituent while preserving the others.
  • Intervention procedure: The intervention procedure applies DISCOVER-predicted role-filler edits to representations obtained from the target model.The edits precisely change every coordinate of the representational vector by the specified amount.
  • GPT-OSS results: Across 31 causal intervention types on six GPT-OSS tasks, average intervention accuracy is 0.903.The result indicates that GPT-OSS behavior changes appropriately when components of the approximated TPR structure are edited.
  • Intervention types: Editing filler values generally achieves high accuracy, while editing structural roles tests whether elements can be moved within task representations.Examples include changing an arithmetic operand or moving an adjective to a different syntactic position.
  • Distributed structure: Local interventions remain nearly as effective for filler edits but perform substantially worse for role edits.This pattern suggests token identity is relatively localized, whereas structural information is distributed across tokens.

8 Testing generalization to novel role-filler pairs

The authors test whether DISCOVER can compose role-filler pairs it never encountered during training. It generally generalizes beyond chance, supporting systematic role-filler binding in neural representations.

  • Method: DISCOVER is trained with selected role-filler pairs withheld, then evaluated on examples containing those unseen combinations.The experiments cover letter-sequence models, period encodings, and GPT-OSS symbolic-task encodings.
  • Baselines: The strong chance baseline for n unseen role-filler pairs is 1/n!, whereas a weaker random-filler baseline is (1/|V_f|)^n.The strong baseline assumes arbitrary assignment of the unseen fillers to the roles.
  • Generalization results: DISCOVER generally retains above-baseline performance on withheld role-filler pairs across sequence models and period encodings.The interleaving Transformer is the main exception among the letter-sequence cases, and some runs show substantial variability.
  • Interpretation: The generalization findings support systematic binding between roles and fillers rather than representations containing only atomic concepts.The authors state that this binding may replace or supplement atomic-concept encoding.
  • Generalization results: For GPT-OSS, DISCOVER substantially outperforms the baseline on larger numbers of unseen role-filler pairs in every task except arithmetic.These results are reported for the symbolic-task encodings shown in Figure 8.3.

9 Discussion

The paper argues that neural networks are neither purely eliminative nor precisely symbolic: they approximately realize systematic symbolic structure, while their deviations may support additional capabilities. It also identifies important limits in the current evidence, including task specificity, incomplete understanding of mechanisms, and the inability to generalize conclusions directly to brains.

  • The relationship between symbols and neural networks: Across model types and training tasks, neural networks exhibit implicit, emergent symbolic structure that approximates Tensor Product Representations.The structure appears across architectures and tasks rather than as a quirk of one model type.
  • The relationship between symbols and neural networks: Targeted interventions and out-of-distribution generalization indicate that the identified symbolic structure is systematic and contributes to model behavior.The approximations support controlled behavioral modifications and generalize to vocabulary elements in novel positions.
  • The relationship between symbols and neural networks: The results challenge eliminativism by showing that neural networks without explicit symbols nonetheless use implicit symbolic structure in internal processing.The paper therefore treats symbols as relevant both to training and to the systems’ internal computations.
  • The relationship between symbols and neural networks: The findings support limitivism: models approach symbol systems but realize them approximately rather than precisely.DISCOVER achieved high but imperfect accuracy, with GPT-OSS causal interventions averaging 0.903.
  • The relationship between symbols and neural networks: Deviations from precise symbol structures may reflect useful integration of systematic compositional computation with fuzzy, statistical computation.The authors identify understanding these deviations as an important direction for future work.
  • Extensions of DISCOVER: DISCOVER currently analyzes specific tasks and domains, and future work must extend it to broader stimuli, potentially through unsupervised discovery of fillers and roles.The paper also calls for studying how networks produce, process, and acquire the identified structure.
  • Why do neural networks perform so poorly at compositional generalization?: Compositional representations may fail to yield compositional generalization when networks learn only role-filler combinations encountered during training.The paper uses word2vec as an in-principle illustration of this distinction.
  • Might the brain use TPRs?: The findings do not establish that biological brains use TPRs, because artificial and biological neural networks differ and brain-recording studies are still required.The authors present extension to the brain as a possibility for empirical investigation, not a demonstrated result.

10 Related work

The related-work discussion situates the paper among studies analyzing or incorporating symbolic structure in neural networks. Its central distinction is a complete, encoding-based reconstruction of representations, combined with broad architectural coverage, symbolic-domain analyses, causal tests, and generalization to novel role-filler pairs.

  • Analyzing neural networks with TPRs: TPRs were proposed as a formalism for representing symbolic structures through fillers paired with roles, and this paper analyzes whether such structure emerges in standard neural networks.The discussion connects the paper’s approach to Smolensky’s earlier proposal that networks might learn TPRs.
  • Analyzing neural networks with TPRs: DISCOVER extends earlier analyses of learned role-filler representations from restricted or architecture-specific settings to broader models and representational structures.Prior work applied DISCOVER to RNNs, SCAN, sentence embeddings, and BERT, whereas the current paper expands its scope.
  • Core contributions: The paper’s core contributions are identifying TPR structure across architectures and symbolic domains, showing causal involvement in LLM behavior, and demonstrating generalization to novel role-filler pairs.These findings support the claim that systematic role-filler structure is broadly used for symbolic tasks.
  • Symbolic mechanisms in neural networks: The paper studies representations rather than processing mechanisms, while related interpretability work identifies neural mechanisms with aspects of symbolic processing.Variable binding is highlighted as a relevant connection between these lines of work.
  • Analyzing structure in neural networks: Decoding methods show that particular information is present in vectors, whereas DISCOVER aims to reconstruct the complete representational structure.This distinction motivates the paper’s emphasis on analyzing the structure of vectors rather than merely reading out selected properties.
  • Analyzing neural networks with TPRs: The paper differs from related mapping approaches in its methods, analyzed models, and reliance on TPR-based representational structure.The high-level strategy of mapping representational spaces is shared, but the concrete analyses are distinct.
  • Integrating TPRs into neural networks: Unlike work that explicitly incorporates TPRs into networks, this paper tests whether implicit TPR structure emerges naturally in standard neural networks.The related literature often uses explicit TPR-like structures to improve compositional abilities.

11 Conclusion

The paper argues that neural and symbolic computation need not conflict: neural networks can construct symbolic representations approximated by interpretable, closed-form TPR equations. It also identifies partially systematic domains as an important direction beyond the fully systematic tasks studied here.

  • 11 Conclusion: The resulting TPR equation can replace a network’s representation-generating process while preserving behavior with minimal change.The paper frames this as evidence that a single system can be simultaneously neural and symbolic.
  • 11 Conclusion: The analyzed tasks are fully systematic, whereas many naturalistic-language domains are only partially systematic and require fuzzy statistical inferences alongside systematic structure.The conclusion presents understanding representations in partially systematic domains as an important future direction.
  • 11 Conclusion: DISCOVER approximates neural representations with linearly transformed TPRs, accommodating vector size and superficial reordering, rotation, or stretching.The linear transformation maps matrix-based TPRs to the vector representations used by target models and captures representational degrees of freedom that preserve conceptual structure.
  • 11 Conclusion: Target models generate intended output sequences followed by an <EOS> token, and evaluation ends at the first generated <EOS>.This output convention defines how sequence completion is interpreted for the target models.

B.2 Training target models

Target models are trained with negative log likelihood, while DISCOVER models approximate their representations with mean squared error using matching data splits and evaluation procedures. Approximation accuracy measures whether the original decoder still produces the correct output from DISCOVER representations.

  • B.2 Training target models: Target models use negative log likelihood loss, with teacher forcing during training for autoregressive decoders but not during evaluation.Training uses an initial learning rate of 0.001, batch size 32, and validation checks every 300 batches.
  • B.2 Training target models: The DISCOVER TPR uses 20-dimensional filler and role embeddings, producing 400 dimensions before compression to the target models’ 64-dimensional hidden vectors.A trainable linear layer with bias performs the 400-to-64-dimensional compression.
  • B.2 Training target models: DISCOVER models minimize mean squared error between target and approximated encodings while using the target models’ dataset splits.They otherwise follow the same batch size, evaluation frequency, learning-rate decay, and stopping procedure.
  • B.2 Training target models: DISCOVER representations are evaluated by feeding test-set approximations into the target decoder and measuring the proportion of correct output sequences.This metric directly tests behavioral preservation under representation replacement.

C Analyzing models trained on alphabetizing a list

Alphabetizing does not require preserving input order, and the trained models are generally approximated well by bag-of-words structures. However, residual approximation gaps remain for some architectures, leaving open whether richer role schemes or rougher structure are needed.

  • C Analyzing models trained on alphabetizing a list: The alphabetical role scheme assigns each letter a role based on its position in the alphabet.For example, in Q M Z C, the roles correspond to fourth, second, fifth, and third alphabetic positions, respectively.
  • C Analyzing models trained on alphabetizing a list: Sorting models are reasonably approximated by the bag-of-words role scheme, unlike models trained on order-sensitive sequence tasks.The result suggests that networks do not deeply encode structure when the task does not require it.
  • C Analyzing models trained on alphabetizing a list: Best-role-scheme approximation accuracies are 0.979 for the MLP, 0.935 for the GRU, 0.871 for the Transformer, and 0.979 for the bottleneck Transformer.The reported values are averaged across reruns and use alphabetical roles for the MLP and Transformer, and Wickelroles for the GRU and bottleneck Transformer.
  • C Analyzing models trained on alphabetizing a list: The remaining headroom may reflect rough rather than fully systematic TPR structure, or a role scheme combining linear, alphabetical, and contextual information.These are presented as possible explanations rather than established causes.
  • C Analyzing models trained on alphabetizing a list: The complex-sentence stimulus grammar is a template generator rather than a claim about reasonable English syntactic analysis.It limits sentence complexity while expressing variable templates compactly.

F.3 Training period-unpacking models

The period-unpacking models decode LLM period representations into preceding lists or sentences, enabling DISCOVER approximations to be evaluated through reconstruction. The analysis compares five role schemes and finds that bidirectional roles outperform syntactic roles for complex sentences.

  • F.3 Training period-unpacking models: Period-unpacking models are decoder-only Transformers that receive an LLM period encoding and generate the preceding list or sentence.The encoding is first compressed to 1024 dimensions before entering a six-layer, 16-head Transformer.
  • F.3 Training period-unpacking models: For lists, the target output excludes the preamble and punctuation; for sentences, the model generates the complete sentence with standardized capitalization and punctuation.These output conventions define the reconstruction targets for the three experimental conditions.
  • F.3 Training period-unpacking models: The five role schemes are bidirectional, predecessor, subject-verb-object, syntactic, and bag-of-words.Each scheme is restricted to the conditions for which its role definition applies, while bag-of-words assigns every filler the same role.
  • F.3 Training period-unpacking models: Bidirectional roles substantially outperform syntactic roles when DISCOVER approximations are decoded for complex sentences.The comparison is surprising given prior evidence that LLMs encode substantial syntax.
  • F.3 Training period-unpacking models: Bidirectional roles remain best-performing across linear-string, constituency-parse, and dependency-parse decoders.The same DISCOVER approximations are reused because only the period-unpacking model changes.

H.1 Datasets and GPT-OSS’s performance

This section describes six symbolic-task evaluations for GPT-OSS and the prompt, dataset, and role-scheme procedures used to analyze its representations.

  • Tasks: The six tasks cover arithmetic, syllogistic reasoning, code execution, passivization, tense reinflection, and question formation.The syntax tasks use context-free grammars, while code execution involves Python list manipulations.
  • Datasets: Arithmetic examples test order of operations in expressions containing multiplication and addition.The expressions use integers from -9 to 9.
  • Datasets: Syllogism prompts provide two premises and four candidate conclusions, with the model returning the first conclusion entailed under an existence assumption.The premises and conclusions use quantified statements about occupation nouns, optionally modified by adjectives.
  • Datasets: The code-execution task asks GPT-OSS to predict outputs of Python functions that manipulate lists through operations such as reversal, sorting, and repetition.The functions are named change and alter and vary across queries.
  • Datasets: Passivization, tense reinflection, and question formation require transformations of syntactically varied English sentences generated from context-free grammars.Tense reinflection requires identifying each verb’s subject to select singular or plural present-tense morphology.
  • Analysis procedure: DISCOVER analyzes GPT-OSS representations using candidate role schemes, including bidirectional and task-specific roles, while training separate models for each of 25 layers.The same hypothesized role-filler pairs are used across layers, but learned DISCOVER parameters vary by layer.

J Testing for structure-sensitivity of causal interventions in GPT-OSS

The analysis tests whether GPT-OSS causal interventions respond to sentence structure rather than merely to tokens’ linear positions. Across eight intervention cases, the results support structure-sensitive behavior.

  • Intervention design: Eight intervention cases compare edits that share linear-position changes but differ in their sentence-structural changes.The cases vary prepositional phrases, relative clauses, subject or object modification, and adjective movement direction.
  • Intervention design: Structure-sensitive interventions should show matched edit vectors working better than mismatched vectors, whereas linear-only interventions should show similar effectiveness across pairings.The analysis evaluates all four combinations of two starting representations and two edit vectors.
  • Results: In all 8 cases, mismatched interventions had lower accuracy than matched interventions to some degree, producing the predicted diagonal pattern.The diagonal pattern was especially clear for tense,PP,front and weaker but present for tense,RC,back.
  • Results: The authors conclude that these interventions are sensitive to sentence structure rather than being driven purely by linear position.This addresses the concern that strong linear-role approximations might explain the intervention effects.
  • Methodological controls: DISCOVER is trained on structurally varied stimuli in the main experiments so linear and syntactic role schemes are not equivalent by construction.Variation deconfounds linear positions from structural positions, enabling comparison of their fit to the target model.

K.4 GPT-OSS performing symbolic tasks

This section describes out-of-distribution generalization and regularization analyses for GPT-OSS’s symbolic-task representations. The procedure withholds role-filler pairs and uses regularized DISCOVER models to test systematic structure.

  • OOD design: The OOD experiments withhold selected role-filler pairs across arithmetic, syllogisms, code execution, passivization, tense reinflection, and question formation.The withheld pairs are partitioned into validation and test sets, with task-specific constraints on positions and templates.
  • Training: DISCOVER models use task-specific (all) role schemes and altered stopping schedules for these constrained-template experiments.Training stops after the third learning-rate decrease following non-improving evaluations.
  • Generalization challenge: Large embedding dimensions can let DISCOVER memorize individual role-filler pairs instead of learning consistency across shared roles or fillers.The method addresses this risk because overly small embeddings could also fail to capture the target structure.
  • Regularization: L2,1 regularization augments mean squared error with penalties on the role and filler embedding matrices, encouraging unused embedding dimensions.The regularization weight λ is selected using separate OOD validation and test splits.
  • Regularization results: 0.54 to 0.97: regularization raised OOD accuracy in the interleaving bottleneck Transformer letter-sequence condition.Other letter-sequence settings had unregularized accuracy at ceiling, showing that the effect varied by condition.
  • Regularization results: Regularization made a qualitative difference for LLM period encodings, and λ = 0.001 was adopted broadly after searches across three LLMs.For Llama-3.1, λ = 0.001 achieved 0.896 versus 0.898 for the best λ.
  • Validation: Generalization to novel role-filler pairs succeeded for the white-box TPR model but completely failed for the Atomic Pair model lacking systematic binding.In-distribution DISCOVER accuracy was high for both models, so OOD generalization distinguishes them.

N Is DISCOVER performance merely driven by the number of roles?

This section tests whether DISCOVER’s role-scheme differences are explained by the number of roles rather than structural fit. Comparisons across sequence, list, and GPT-OSS experiments argue against role count as the primary explanation.

  • Confound: A central confound is that role schemes with more roles may perform better simply because they provide more parameters.The concern is whether DISCOVER accuracy reflects structural fit or role-count differences.
  • Letter sequences: 21 vs. 729: bidirectional roles outperform Wickelroles in every letter-sequence condition despite using far fewer roles.Left-to-right and right-to-left roles also use the same number of roles, yet often differ in performance.
  • Lists: 12 vs. 301: bidirectional roles consistently outperform predecessor roles in the lists condition despite using fewer roles.The lists contain unique elements, so predecessor roles are sufficient in principle to identify each sequence.
  • GPT-OSS: In GPT-OSS, task-specific (all) outperforms bidirectional (all) in every condition, including cases where it uses fewer roles.In code execution, task-specific (self) reaches 99 roles versus 1,329 for bidirectional (all) while slightly outperforming it.
  • Conclusion: The authors conclude that role-count differences do not primarily drive DISCOVER approximation accuracy.They interpret role-scheme accuracy as evidence about how well a scheme captures the causal structure of representations.
Loading 2608.29530v1…