PaperlayerView exact revision on arXiv

Source-linked AI summary

Signatures of semantic search in the activations of large language models

Luke Leckie, Peter M. Todd, Jacob G. Foster

arXiv:2609.35599v2cs.AI

TL;DR

The paper asks whether LLMs represent switching and clustering as distinct search behaviors. Using mechanistic interpretability and activation steering, it finds distinct representational signatures and that steering generic activation directions biases switching.

  • Problem

    The study asks whether LLM switching is coupled to generic activation-space directions, which would indicate distinct representations of switching and clustering.

    Source: §4 · pp. 7–8

  • Method

    The authors use the J-lens to examine concept activations and steer residual-stream activations during semantic fluency task generation.

    Source: §1 · p. 3, §3.4 · p. 7

  • Results

    LLMs show distinct representational signatures during exploration and exploitation, and steering generic activation directions biases switching rates.

    Source: §5 · pp. 9–10, §4.2 · p. 9

  • Takeaways & Limitations

    The findings extend the semantic foraging framework to AI and suggest avenues for interpreting and modulating LLM search tasks.

    Source: §5 · p. 10

    The study uses the semantic fluency task, and it remains unclear whether its reported dynamics extend to naturalistic speech or other search tasks.

    Source: §5 · p. 10

Abstract

from arXiv · show

When recalling lists of concepts (e.g., animals) during the semantic fluency task (SFT), both humans and large language models (LLMs) organise their output into clusters of related items (e.g., sea animals) that are punctuated by strategic switches between clusters. In humans, this pattern can be explained by a semantic foraging process, whereby distinct neural and behavioural signatures accompany within-cluster production ("exploit") and between-cluster switching ("explore"). Whether LLMs likewise represent these two search regimes within their internal states is unknown. Here, we apply a range of mechanistic interpretability techniques to provide evidence for this. In Study 1, we use the Jacobian lens (J-lens), which maps intermediate-layer residual-stream representations to token-level activations, to show that concept-level activations predict switching. First, we find that switching coincides with low next-token activations. Moreover, the probability of switching rises as the set of strongest J-lens activations (the J-space) becomes depleted of items from the category currently being produced, analogous to explore-exploit decision-making during patch foraging. We then show that middle-layer J-lens activations of abstract category-related labels (e.g., "water") increase in anticipation of switching into that category. We confirm these representations to causally influence switching by deriving steering vectors that target category switching. In Study 2, we identify generic residual stream directions that are activated during and in anticipation of switching. By steering activations along these directions, we bias increased or decreased rates of switching. Our study extends the semantic foraging framework to artificial intelligences and provides evidence that LLMs maintain distinct representational signatures for exploration and exploitation as they verbalise conceptual information.

1 INTRODUCTION AND RELATED WORK

Semantic fluency has been framed as a foraging process that balances exploiting related concepts against exploring new clusters. This paper tests whether LLMs show internal signatures of these regimes and finds evidence of concept- and category-level representations linked to switching.

1 INTRODUCTION AND RELATED WORK

  • Patch foraging balances exploitation of known, depleting resources against exploration for novel, uncertain resources.Animals typically leave a depleting patch based on the value of continued exploitation relative to expected opportunities elsewhere.Source: §1 · pp. 1–2
  • Human semantic fluency output is organized into clusters of related items separated by strategic switches, with distinct behavioral and neural signatures accompanying those regimes.Switching is associated with longer response latencies and heightened posterior cerebellum and hippocampus activation; activity in those regions ramps during clustering.Source: §1 · p. 2
  • Although LLMs produce clustered and switched concept lists, it was unknown whether these patterns reflected a foraging-like process or only resembled human output.Prior interpretability work linked language changes and other behaviors to internal representations, but had not studied or manipulated activations during natural semantic foraging.Source: §1 · p. 2
  • The study finds that LLMs show distinct representational signatures for exploration and exploitation during semantic search.Its analyses link concept activations and generic activation-space directions to switching behavior.Source: §1 · p. 3

2 EXPERIMENTAL SET-UP

The experiments test five open-weight instruction-tuned LLMs on animal-list semantic fluency. They analyze or steer activations across layers while generating sampled responses under a shared setup.

2 EXPERIMENTAL SET-UP

  • Experiments used five open-weight instruction-tuned LLMs ranging from 2-billion to 9-billion parameters, with activation analyses or steering at regularly spaced residual-stream layers.Experiments were iterated over 100 random seeds per model-layer combination.Source: §2 · p. 3
  • Each model was prompted to name as many different animals as possible in a comma-separated list.Outputs were capped at 200 tokens; generations used temperature 0.9 and top-p 0.95.Source: §2 · p. 3
  • A switch was identified when a named concept shared no category labels with the preceding concept.Animal categories followed an extended version of Troyer’s designations, allowing an animal to belong to multiple categories.Source: §2 · p. 3

3 STUDY 1: CONCEPT ACTIVATIONS INDICATE AND ANTICIPATE SWITCHING

During the Semantic Fluency Task, LLM concept activations signal clustering and switching, while category-level activations anticipate and can steer switches. J-space depletion also predicts increased switching.

3.1 ITEM ACTIVATIONS DIFFER BETWEEN CLUSTERING AND SWITCHING

  • Switch-participating items have lower next-token probabilities and lower pre-emission J-lens activations than cluster-participating items.The activation difference appears from early-middle layers in most models and strengthens toward late layers.Source: §3.1 · pp. 4–5, Figure 2 · p. 5

3.2 J-SPACE CONCEPT DEPLETION PREDICTS SWITCHING

  • Switching probability rises as the number of unproduced, current-category items in the J-space falls, with a stronger effect than depletion of the overall category pool except for Qwen.The effect is generally most salient in middle layers and robust across tested J-space sizes.Source: §3.2 · p. 5

3.3 ACTIVATION OF ABSTRACT CATEGORY-RELATED LABELS ANTICIPATES SWITCHING

  • Category-related labels activate more during clustering within their target category, and target-label activation rises before a switch, peaks at the switch, then declines.Unrelated labels also tend to rise before switching but drop sharply at the switch, consistent with competition among categories.Source: §3.3 · p. 6, Figure 3 · p. 6

3.4 CONCEPT ACTIVATIONS HAVE CAUSAL INFLUENCE ON SWITCHING

  • Both item-level and category-related steering induce switches into target categories across models, with different layer-depth profiles.For Gemma-9B, target-switch probability reached 0.68 with item-level steering at layer 40 and 0.45 with category-related steering at layer 32.Source: §3.4 · p. 7, Figure 4 · p. 7

4 STUDY 2: GENERIC SIGNATURES OF SWITCHING IN ACTIVATION SPACE

Study 2 tests whether switching and clustering have generic activation-space signatures beyond representations of particular categories. Residual-stream directions predict switching and can be steered to change switching rates, with effects varying by model and layer.

4 STUDY 2: GENERIC SIGNATURES OF SWITCHING IN ACTIVATION SPACE

  • Study 2 asks whether residual-stream directions represent switching as a generic behavior, rather than only switching into particular categories.The authors frame this as a test of whether LLMs represent exploration and exploitation as generic behaviors.Source: §4 · pp. 7–8

4.1 COMPARING RESIDUAL STREAM ACTIVATIONS BETWEEN CLUSTERING AND SWITCHING

  • The analysis compares residual-stream states immediately before a switch with states at the switch, using cross-validated logistic regression to predict switching versus clustering.PCA reduces layer activations to 30 dimensions; folds exclude categories from both training and test sets.Source: §4.1 · p. 8
  • Switching was predictable above chance from both anticipatory and event-position activations; Gemma-9B reached AUC 0.72 at layer 26 and 0.87 at layer 16, respectively.Anticipatory prediction was relatively consistent across layers, whereas event-position prediction was strongest in middle layers.Source: §4.1 · p. 8
  • As clusters grew, activations increasingly loaded onto the anticipatory direction, paralleling ramping activation during human clustering.The reported increase was significant across all models (χ2 ≥20.44, p < 0.0001).Source: §4.1 · p. 8

4.2 STEERING SEMANTIC EXPLORATION-EXPLOITATION

  • The steering direction is reconstructed by projecting logistic-regression coefficients through PCA loadings, then injected into selected residual-stream layers with signed strength.Matched-norm random directions orthogonalized against the steering direction serve as perturbation controls.Source: §4.2 · p. 8
  • Steering the generic direction changed switching rates: at Gemma-9B layer 26, positive steering raised the rate from 0.29 to 0.70 and negative steering lowered it to 0.07.Steering effects were model- and layer-dependent, and positive steering was more consistent than negative steering.Source: Figure 5 · p. 9, §4.2 · p. 9

5 CONCLUSIONS

The conclusions report distinct exploration- and exploitation-related signatures in LLM semantic fluency and parallels with human findings. Their generality beyond the controlled task and relatively small models remains uncertain.

5 CONCLUSIONS

  • LLMs showed switching uncertainty, category-label activation, and ramping switch-associated activation during clustering, paralleling reported human SFT patterns.The cited human patterns include elevated switching latency, category-semantic representations, and ramping neural activation.Source: §5 · p. 9
  • The authors suggest that, if the findings generalize, related activation signatures could modulate semantic pace, coherence, and coverage in routine conversations or exploration in applied search tasks.These applications are presented conditionally, not as demonstrated outcomes.Source: §5 · p. 10
  • Generality to naturalistic speech and other search tasks is unclear, and whether the reported dynamics arise in frontier-scale models remains an open question.The study used models described as relatively small compared with present frontier models.Source: §5 · p. 10

6 AI-USE STATEMENT

The authors report using generative AI for code generation, not for writing or idea generation, and state that a human checked the code and accepts responsibility for the final work.

6 AI-USE STATEMENT

  • Generative AI tools were used to generate code, including Claude code and Opus models.Source: §6 · p. 10
  • The authors state that generative AI was not used for writing or idea generation.Source: §6 · p. 10
  • LL checked the code manually for correctness and takes responsibility for the final content, including AI-assisted text, claims, or artifacts.Source: §6 · p. 10

A.1 SWITCH/CLUSTER DESIGNATION

Switches are designated when consecutive concepts share no category labels, after out-of-category items are omitted. Token-level analyses use first-token matching for multi-word concepts, while steering vectors use only single-token concepts.

A.1 SWITCH/CLUSTER DESIGNATION

  • Out-of-category verbalized concepts were omitted from switch/cluster designation; they comprised 2.4% of pooled named items, ranging from 0.0% to 5.5% across models.Source: §A.1 · p. 12
  • For multi-word concepts in token-level activation analyses, concepts were matched to tokens using their first token.Source: §A.1 · p. 13
  • Only single-token concepts were used to construct steering vectors.Source: §A.1 · p. 13

A.2 STATISTICS

The section describes seed-level statistical tests for activation differences, depletion effects, anticipatory ramping, and steering. Multiple comparisons across layers or lags are corrected where applicable.

A.2 STATISTICS

  • Tests used each seed as one replication, with per-model comparisons across layers or lags and Benjamini-Hochberg correction for multiple tests.Linear regression with Wald’s test assessed paired activation differences; one-sided p-values were doubled where reported.Source: §A.2 · p. 13
  • J-space depletion was compared with raw category-pool depletion using point-biserial correlations across five layer-depth bins per model.The analysis tested whether the J-space depletion correlation was more negative than the raw-depletion correlation.Source: §A.2 · p. 13
  • Anticipatory loading was tested against remaining cluster items, while steering effects were compared with magnitude-matched noise using Fisher’s exact test.The test direction followed the steering scale’s sign.Source: §A.2 · p. 13

A.3 ROBUSTNESS ANALYSIS

The robustness analysis varied J-space size in Gemma models and found the depletion effect robust across the tested sizes for Gemma-9B, and down to size 5 for Gemma-2B.

A.3 ROBUSTNESS ANALYSIS

  • J-space sizes of 5, 15, and 25 tokens were tested in Gemma models; sizes above 25 were excluded because most Troyer categories contain fewer than 25 animals.Source: §A.3 · p. 13
  • Gemma-9B results were robust across all tested J-space sizes, whereas Gemma-2B results were robust down to the smallest tested size, 5.Source: §A.3 · p. 13
  • The size-robustness comparisons are reported in Table S5.Source: §A.3 · p. 13

A.4 NOISE INJECTION

The supplementary material describes norm-matched orthogonal noise controls, plots J-lens activations around emissions, and lists statistical comparisons for activation, depletion, and steering analyses.

A.4 NOISE INJECTION

  • Random control steering vectors are orthogonal to the target steering direction and receive the same perturbation norm as the corresponding constructed vectors.The norm is |s|rL, using the same layer-specific residual norm and steering coefficient.Source: §A.4 · p. 13

A.5 SUPPLEMENTARY FIGURES

  • Figure S1 plots z-scored J-lens activations around item emissions and the switching-minus-clustering activation difference; stars mark significant comparisons.Lag 0 is emission, with negative and positive lags denoting before and after emission.Source: Figure S1 · p. 14

A.6 SUPPLEMENTARY TABLES

Related papers

Loading 2609.35599v2…