Source-linked AI summary
Encoded Early, Used Late: Where Transformers Begin to Act on an Inferred Partner's Expertise
Mika Okamoto, Gabriele Sarti
TL;DR
The paper asks whether partner expertise inferred across a conversation is readable before it causally affects model output. Using controlled multi-turn dialogues, probing, and counterfactual patching, it finds early decodability but substantially later causal influence, with scope limited to one model and synthetic corpus.
Problem
The paper asks whether the separation between decodability and causal use also holds for partner expertise inferred across a dialogue rather than stated in the input.
Method
The study constructs EXPERTCOLLAB dialogues, extracts participant-perspective activations, probes partner expertise across layers, and uses counterfactual patching with controls to test causal propagation.
Results
The attribute peaks at layer 8 with 0.79 probe accuracy, falls near chance by layer 40, and patch effects rise from 0.01 at peak decoding to 0.86 at layer 36 and above 0.90 from layer 40.
Takeaways & Limitations
Decodability does not establish function, so probing should be paired with intervention when locating or steering inferred partner-conditioned behavior.
Takeaways & Limitations
All results use one model and one synthetic, same-model corpus of 28 base conversations, so layer boundaries may depend on architecture.
Abstract
from arXiv · showhide
A transformer can make an attribute linearly decodable in its residual stream at a depth where that attribute does not yet influence the output. This gap between where information is readable and where it is used has been shown for attributes stated directly in the input. We ask whether it also holds for an attribute the model must infer gradually over a conversation, namely how expert its dialogue partner is. Using ExpertCollab, a corpus of multi-turn research-planning dialogues between model-played personas at four expertise levels, we find that partner expertise is most decodable in the early layers and falls to near chance before the midpoint of the network. Counterfactual patching shows that injecting the expertise difference at the layer of peak decodability barely changes a fixed late-layer readout, whereas the same difference injected past the midpoint propagates almost completely, a separation of more than an order of magnitude. A content-matched random control and a probe-free diagnostic place the transition at the same early layer, and a statically specified control attribute stays decodable throughout. An inferred relational attribute is therefore represented well before it becomes causally active, which bounds where any attempt to read out or steer partner-conditioned behavior must intervene. We use one model on a synthetic corpus as an initial demonstration.
1 Introduction
The paper asks whether partner expertise inferred across dialogue shows the same gap between decodability and causal use found for stated attributes. It introduces a controlled framework and argues that inferred expertise is readable early but becomes causally active only later.
- Motivation: Prior work shows that an attribute can be decodable before the layer where it shapes model output.Skip connections can preserve an early-written feature downstream without later computation consuming it.
- Research question: Partner expertise is studied as an inferred relational attribute signaled through a conversational partner’s writing rather than stated in the input.The paper focuses on whether this attribute can be separated into readable and causally used representations.
- Approach: EXPERTCOLLAB contains multi-turn research-planning dialogues between model-played personas at four expertise levels, enabling analysis of where partner expertise is decoded and causally affects output.The study extracts activations immediately before the model generates each next turn.
- Contributions: The paper contributes a controlled paradigm that combines probing with counterfactual patching, content-matched controls, calibration controls, and a probe-free diagnostic.These components are designed to distinguish readable representations from representations that propagate causally to output.
- Contributions: EXPERTCOLLAB withholds explicit credentials, so expertise must be inferred from how each persona engages.The corpus is intended to support study of an attribute that is never directly stated.
- Contributions: The central structural finding is that inferred expertise is decodable early but causally inert there, becoming causal only past the network midpoint unlike the stated control attribute.The paper also proposes a falsifiable hypothesis linking the decode-to-causal gap to how much an attribute must be inferred.
2 Related Work
The paper extends readable-versus-used analyses from attributes stated in the input to a relational attribute accumulated across dialogue. It connects the question to transformer depth, behavioral conditioning, and intervention-based causal analysis.
- Prior work: Prior studies found that decodability and causal influence can diverge for input-stated attributes in vision, sparse-autoencoder, and language-model settings.This paper extends that question to an attribute accumulated across dialogue turns.
- Transformer depth: Transformer depth is treated as a sequence of functional stages, with a reported midpoint transition where representations begin aligning with vocabulary space.The paper relates its causal-onset boundary to this organizational transition.
- Behavioral relevance: Locating when an inferred partner attribute becomes causal matters for behaviors conditioned on users, including sycophancy.The paper frames causal localization as a prerequisite for acting on such representations.
3 Methodology
The methodology generates controlled dialogues, extracts participant-perspective activations, probes partner expertise across depth, and uses counterfactual patching to test causal propagation. It includes stated-attribute, random-vector, calibration, and probe-free controls.
- Pipeline: The pipeline generates dialogues, extracts participant-perspective activations, probes partner expertise, and patches activations to test causal influence.These four stages correspond to the study’s figure-1 workflow.
- Corpus: EXPERTCOLLAB contains 28 synthetic 12-turn research-planning dialogues between two model-played agents drawn from four expertise levels across four domains.Personas specify communication styles and blind spots without stating credentials; all transcripts pass a leakage screen.
- Probe analysis: Figure 2 compares pooled cross-validated probe accuracy by layer with accuracy across layers and turns for partner expertise.The figure also contrasts inferred partner expertise with the statically specified own-level control.
- Activation extraction and probing: Activations are sampled every fourth layer from 0 to 76 of Llama-3.3-70B Instruct at the position immediately before the model generates its next turn.Both speakers are pooled into 56 instances, and L2-regularized logistic probes are fit by layer and turn for partner expertise and own-level control.
- Counterfactual patching: Counterfactual patching adds a source-minus-target activation difference at a chosen layer and runs the forward pass to a fixed last-layer readout.The analysis uses dialogue pairs differing only in one participant’s expertise and reports a normalized propagation effect.
- Controls: A norm-matched random vector tests content specificity, while a probe-free cosine-dissimilarity check tests whether patched and unpatched final activations differ.These controls complement the fixed-readout probe measurement.
4 Results
Partner expertise is most readable early but becomes causally influential only later in the network. Probe-based and probe-free diagnostics converge on this transition, while random controls do not reproduce it.
- Partner-expertise accuracy peaks at 0.79 at layer 8 and decays to near chance by layer 40.The speaker’s own level remains above 0.60 at every layer, whereas the surface-feature baseline reaches 0.32 against 0.25 chance.
- The normalized causal effect is 0.01 at the peak-decoding layer, reaches 0.86 at layer 36, and exceeds 0.90 from layer 40.The same-layer calibration averages 1.01, and the transition separates early null effects from late propagation by more than an order of magnitude.
- The probe-free cosine dissimilarity is flat initially, rises sharply from layer 8, and saturates near 0.40 by layer 32.This matches the causal-effect transition without using a trained classifier.
- A norm-matched random control shows no comparable rise, indicating that the causal transition is specific to the expertise direction.
- Late-layer patches change about one in three top-five next tokens, whereas peak-decoding-layer patches leave the output almost unchanged.
5 Conclusion
The study finds that inferred partner expertise is written into the residual stream before it becomes causally active. This supports treating decodability and functional use as distinct questions, while limiting the result to an initial demonstration.
- Partner expertise is most decodable at layer 8 but causally inert there, while the same difference injected near layer 36 recovers most of the source signal.The causal onset aligns with a midpoint transition, unlike the stated control attribute.
- Decodability does not establish function because residual-stream information can remain readable without being consumed by later computation.
- The paper hypothesizes that the gap between peak decodability and causal onset grows as an attribute becomes more inferred rather than stated.The proposed test varies how explicitly an attribute is provided and examines whether causal onset shifts monotonically.
- The evidence is limited to one model and one synthetic, same-model corpus of 28 base conversations, with architecture-dependent layer boundaries and possible self-generation-pattern recognition.The authors describe the work as an initial demonstration and identify cross-model replication as a next step.
A Reproducibility note
The reproducibility materials specify the released prompts, generation setup, corpus composition, persona design, and model-serving distinction used for activation analysis.
- The appendix releases the dataset and analysis code, with verbatim prompts and hyperparameters as the authoritative specification.
- Dialogues use meta-llama/Llama-3.3-70B-Instruct-Turbo through Together AI, with 12 turns per conversation and a 400-token cap per turn.
- The corpus contains 28 base conversations from seven persona pairings and four research scenarios across four expertise levels.The pairings include same-level and cross-level combinations of high schooler, junior researcher, researcher, and professor personas.
- Expertise levels are coded HS 0, JR 1, R 2, and PROF 3, and all 28 conversations pass the leakage control unchanged.
- The generation endpoint is a separately served model variant from the Llama-3.3-70B-Instruct model used for probing and patching.
- Persona prompts specify engagement style, blind spots, and interaction tendencies while forbidding explicit credentials, so expertise is signaled behaviorally.
B.3 Shared task framing and conversation style
The shared task template frames every exchange as collaborative research planning and keeps expertise implicit through a common conversation style. The pipeline then screens transcripts for leakage and extracts each speaker’s generation-boundary activations from its own perspective.
- System-prompt assembly template: Each system prompt combines a persona prompt, a shared task block, and a shared conversation-style block.
- Shared task framing and conversation style: The shared task asks collaborators to design a research study, including measurements, comparisons, and execution.
- Shared task framing and conversation style: The shared conversation-style block requires natural in-character interaction, adaptation to the partner’s perceived experience, and short messages.This shared block is intended to keep expertise implicit across personas.
- Shared task framing and conversation style: The four scenarios cover three machine-learning topics and one soil-science topic.
- Shared task framing and conversation style: A leakage detector excludes turns stating titles, credentials, or years of experience, while tolerating implicit signaling through behavior; all 28 conversations pass.
- Participant-perspective construction: For each target turn, the pipeline rebuilds the chat from the speaker’s perspective and extracts the final-token activation at the generation boundary.The model’s own prior turns appear as assistant messages and the partner’s prior turns as user messages.
- Participant-perspective construction: Activations are sampled every fourth layer from layers 0 through 76, and both speakers are pooled into 56 instances.
C.3 Linear probe training
The analysis trains regularized probes at each layer and turn, using counterfactual pairs and calibrated controls to distinguish partner-level encoding from causal propagation.
- Probe training: Probes use standardized 8192-feature activations with L2-regularized multinomial logistic regression and fixed inverse regularization strength 0.01.The saga solver runs for up to 5000 iterations; cross-validation uses stratified k-folds with k = max(2, min(5, cmin)).
- Counterfactual construction: The primary ablation varies one participant’s expertise while holding the other participant’s persona and scenario fixed.Patching the fixed participant’s residual stream tests how that participant encodes the partner’s changing expertise.
- Causal measurement: The normalized causal effect compares patched confidence with source and target baselines at a fixed last-layer readout.The readout-layer baseline is computed separately for each turn, and cells with baseline differences below 0.05 are masked.
- Controls: Calibration gives an average trivial-recovery effect of 1.01, while same-level null pairs produce zero effect.These controls test whether the measurement recovers a directly applied patch and remains zero when the source-target difference vanishes.
D.4 Behavioral-linkage metrics
The paper links activation patching to output-level behavior with token-overlap and final-representation-shift metrics. Both measures place the transition from weak early effects to stronger late effects at similar depths.
- Behavioral linkage: Top-5 token overlap declines from 0.97 at the first layers to about 0.67 by layer 32.The gradual decline indicates a small, graded output footprint for shallow patches rather than an abrupt early behavioral switch.
- Probe-free diagnostic: Final-layer cosine dissimilarity stays flat through early layers and rises sharply from layer 8.This probe-free representation shift locates the same transition as the probe-based effect.
- Behavioral linkage: 0.653 and 0.660 normalized recovery effects are obtained for patches toward higher- and lower-expertise sources, respectively.The corresponding 95% intervals are [0.597, 0.713] and [0.614, 0.704], indicating direction-symmetric recovery.
- Turn consistency: Recovery is near zero at early layers and near complete at late layers across all six speaker turns.The transition is consistent across conversational position, supporting a dependence on network depth rather than turn index.
E.1 Surface-feature baseline
The surface-feature baseline tests whether early partner-expertise decoding can be explained by simple lexical statistics. It performs above chance but substantially below the neural probe, while causal recovery remains uniform across turns.
- Surface-feature baseline: 0.32 accuracy is achieved by the five-feature lexical classifier, versus 0.786 for the neural probe and 0.25 chance.The lexical features are hedging rate, mean sentence length, question rate, type-token ratio, and vocabulary complexity.
- Surface-feature baseline: The 0.45 accuracy gap between the lexical baseline and neural probe indicates that early encoding captures structure beyond these surface statistics.The comparison uses the same pooled partner-level target and cross-validation protocol.
- Causal-effect profile: Early-layer null and late-layer recovery are uniform across all six turns in the normalized causal-effect analysis.Figure 5 includes 48 pairs, with 20.1% of cells masked.
- Probe behavior: Peak-cell probe errors concentrate between neighboring expertise levels rather than distant levels.The confusion pattern is consistent with a graded expertise signal.
F Toward a general principle
The paper frames the decodability–causality separation as a possible organizing principle for transformer residual streams and proposes testing whether inference demands shift causal onset later.
- General principle: The readable-versus-used separation appears across several directly specified attributes and here for an attribute inferred over dialogue.The authors present this cross-attribute pattern as a candidate organizing principle rather than a settled conclusion.
- Evidence base: The partner-level confusion matrix is shown for the pooled probe at layer 8, turn 5, with n=56.The matrix is fit on all data for visualization.
- Hypothesis: The proposed hypothesis is that the depth gap between peak decodability and causal onset grows with how much an attribute must be inferred.The contrast between partner level and stated own level is presented as one data point supporting this falsifiable hypothesis.
- Testable prediction: Varying an attribute from explicitly stated to indirectly inferable is predicted to move causal-onset depth monotonically later.Controlled prompt manipulations could turn the qualitative observation into a quantitative relationship.