PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Frequency Tagging Probe (HFTP): A Unified Approach to Investigate Syntactic Structure Representations in Large Language Models and the Human Brain

Jingmin An, Yilong Song, Ruolin Yang, Nai Ding, Lingxi Lu, Yuxuan Wang, Wei Wang, Chu Zhuang, Qian Wang, Fang Fang

arXiv:2510.13255v1cs.CLcs.NE

TL;DR

It remains unclear whether LLMs represent hierarchical syntax through mechanisms comparable to those in the human brain. HFTP probes frequency-domain representations in model neurons and brain recordings, finding left-hemisphere alignment alongside divergent trends across upgraded models.

  • Problem

    Whether LLMs represent human-like hierarchical sentence structure remains unclear despite their syntactic capabilities.

    Source: §1 · p. 2

  • Method

    HFTP uses frequency-domain analysis to probe syntactic representations in LLM neurons and human-brain recordings and compare them with representational similarity analysis.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    LLM representations align more strongly with the brain’s left hemisphere, while Gemma 2 shows improved alignment and Llama 3.1 weaker alignment than their predecessors.

    Source: §4.3 · p. 9, §5 · p. 10

  • Takeaways & Limitations

    HFTP provides a framework for studying syntactic representations and model–brain alignment across language models and the human brain.

    Source: §1 · p. 2, §4.3 · p. 10

    The sEEG data primarily involve Chinese stimuli, and only a small set of model architectures and parameter scales was evaluated.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) demonstrate human-level or even superior language abilities, effectively modeling syntactic structures, yet the specific computational modules responsible remain unclear. A key question is whether LLM behavioral capabilities stem from mechanisms akin to those in the human brain. To address these questions, we introduce the Hierarchical Frequency Tagging Probe (HFTP), a tool that utilizes frequency-domain analysis to identify neuron-wise components of LLMs (e.g., individual Multilayer Perceptron (MLP) neurons) and cortical regions (via intracranial recordings) encoding syntactic structures. Our results show that models such as GPT-2, Gemma, Gemma 2, Llama 2, Llama 3.1, and GLM-4 process syntax in analogous layers, while the human brain relies on distinct cortical regions for different syntactic levels. Representational similarity analysis reveals a stronger alignment between LLM representations and the left hemisphere of the brain (dominant in language processing). Notably, upgraded models exhibit divergent trends: Gemma 2 shows greater brain similarity than Gemma, while Llama 3.1 shows less alignment with the brain compared to Llama 2. These findings offer new insights into the interpretability of LLM behavioral improvements, raising questions about whether these advancements are driven by human-like or non-human-like mechanisms, and establish HFTP as a valuable tool bridging computational linguistics and cognitive neuroscience. This project is available at https://github.com/LilTiger/HFTP.

1 Introduction

The introduction motivates a unified way to compare hierarchical syntax representations in language models and the human brain. It presents HFTP as a frequency-domain probe for locating these representations and studying their alignment.

1 Introduction

  • The paper reports that syntactic processing recruits distinct mechanisms in the brain, particularly in left inferior frontal and posterior temporal regions.The introduction links increasing syntactic complexity to graded activation in these areas.Source: §1 · p. 1
  • LLMs can produce syntactically regular language, but whether they represent human-like hierarchical sentence structure remains disputed.Prior findings differ on whether their success reflects structural representations or shallow statistical heuristics.Source: §1 · p. 2
  • HFTP builds on hierarchical frequency tagging to separate sentence- and phrase-level structure from lexical and prosodic regularities for model–brain alignment.The introduction frames frequency-domain markers as a bridge between biological and artificial systems.Source: §1 · p. 2
  • The paper develops HFTP to characterize syntactic representations across LLM computational modules and compare them with human-brain representations.The approach applies neuron-wise probing to models and population-level analysis to the brain.Source: §1 · p. 2

2 Related work

Prior work finds syntactic processing across distributed brain regions and syntactic information in language models, but existing methods have made systematic comparison across models and neural populations difficult.

2 Related work

  • Human syntax engages a distributed, interconnected cortical network, including left inferior frontal and posterior temporal regions [Friederici and Brauer, 2009, Hagoort, 2013, Blank et al., 2016].Hierarchically structured strings also engage left inferior frontal gyrus and posterior superior temporal regions.Source: §2 · pp. 2–3
  • Language models encode syntax-sensitive dependencies and hierarchical structure, but methods used to study them are difficult to apply to human brain activity [Linzen et al., 2016, Kuncoro et al., 2018, Hewitt and Manning, 2019].Prior work also describes syntactic knowledge distributed across transformer layers [Clark, 2019, Tenney et al., 2019, Manning et al., 2020].Source: §2 · p. 3
  • Brain–language-model alignment studies report contributions from multiple linguistic properties, while methodological inconsistencies impede systematic comparison of syntactic encoding across models and neural populations [Caucheteux et al., 2021, Caucheteux and King, 2022].Intervention studies report cross-layer effects from syntactic properties such as tree depth and top constituents.Source: §2 · p. 3

3 Methods

The methods apply frequency-domain probing to model activations and intracranial recordings, then compare syntactic representations across the two systems. They use structured corpora and controls to isolate sentence- and phrase-level signals before testing model–brain correspondence.

3 Methods

  • The workflow covers syntactic corpora and LLM architectures, model and brain probing, and cross-system comparison of frequency-domain representations.The section is organized into corpus/model, LLM probe, sEEG probe, and alignment components.Source: §3 · p. 3

3.1 Data and LLMs

  • Chinese four-syllable and English four-word sequences form two-phrase sentences; a word-order-randomized control preserves lexical items while disrupting fixed positional patterns.Naturalistic text tests generalizability, and the model–brain experiments use Chinese corpora adapted from prior work.Source: §3.1 · p. 3
  • HFTP is applied to six models, and sub-token activations are averaged at the syllable or word level before frequency analysis.The models are GPT-2, Gemma, Gemma 2, Llama 2, Llama 3.1, and GLM-4; the passage defines an MLP neuron as a unit in a Transformer's intermediate MLP hidden layer.Source: §3.1 · p. 4

3.2 Syntactic structure probe in LLMs

  • For LLMs, 1-Hz and 2-Hz amplitudes are tested against 95% confidence intervals from 1,000 permutations to identify significant syntactic neurons.Sentence- and phrase-specific neurons are then selected using experimental-versus-control z-score deviations greater than two standard deviations at 1 Hz and 2 Hz, respectively.Source: §3.2 · p. 5

3.3 Syntactic structure probe in the human brain

  • The human-brain probe analyzes sEEG from 26 native Chinese speakers listening to Chinese sentence and phrase corpora, using ITPC across trials and anatomically grouped ROIs.Channels are assessed against a permutation-derived 95% confidence interval; 1 Hz denotes sentence channels and 2 Hz phrase channels.Source: §3.3 · pp. 5–6

3.4 Alignment of syntactic structure representations of LLMs with the human brain

  • Alignment is measured by constructing SRDMs from cosine similarities between syntactic-condition frequency representations, then correlating model and brain matrices with RSA.The method also defines global model–brain, model–region, and regional contribution measures.Source: Figure 2 · p. 4, §3.4 · p. 6

4 Experiments

HFTP uses frequency-domain representations to examine syntactic processing in LLM neurons and human sEEG channels. The experiments find sentence and phrase representations in both, but with distinct patterns across model layers and brain regions, alongside stronger model–brain alignment in the left hemisphere.

4.1 MLP neurons represent sentences and phrases in LLMs

  • All six models had sentence-, phrase-, and shared-selective neurons; maximum syntactic-neuron proportions fell from 11% in Llama 2 to 4.5% in Llama 3.1 and from 27% in Gemma to 22% in Gemma 2.The paper suggests this may reflect a shift in computational resource allocation, but does not establish that explanation.Source: §4.1 · pp. 6–7
  • HFTP identified selective sentence, phrase, shared, and non-sensitive frequency patterns in MLP neurons, with sentence and phrase tags at 1 Hz and 2 Hz, respectively.Frequencies above 2 Hz were set to zero for smoothness in the plotted representation.Source: §4.1 · pp. 6–7, Figure 3 · p. 7
  • Sentence and phrase neuron proportions covaried across layers in all six models, with reported correlations including Gemma (r = 0.994), Gemma 2 (r = 0.994), and GLM (r = 0.993).The authors interpret the correlations as evidence of similar underlying sentence and phrase processing mechanisms in LLMs.Source: §4.1 · pp. 7–8

4.2 Sentences and phrases representations in the human brain

  • In the brain, phrase-channel proportions decreased from lower to higher ROIs while sentence-channel proportions increased; sentence and phrase channel counts were not significantly correlated across ROIs.The reported correlations were left: r = -0.169, p = 0.870; right: r = -0.197, p = 0.539.Source: §4.2 · p. 8, Figure 6 · p. 8

4.3 Alignment of syntactic structure representations between LLMs and the human brain

  • Gemma 2 exceeded Gemma (sentence S = 0.644 vs 0.582; phrase S = 0.628 vs 0.575), whereas Llama 3.1 aligned below Llama 2 at both levels.GPT-2 had the highest average correlation across both sentence and phrase levels (S = 0.654, L).Source: §4.3 · p. 9
  • Model–brain similarity was higher for the left than right hemisphere across sentence and phrase corpora, with region-wise effects concentrated in core language cortex.The highlighted regions include STG, MTG, and IFG.Source: Table 1 · p. 9, §4.3 · p. 9, Table 2 · p. 10

5 Conclusion

The conclusion presents HFTP as a unified way to examine syntactic representations in LLMs and the human brain. It reports hierarchical processing and left-hemisphere alignment in LLMs, while noting that model–brain correspondence varies across newer model versions.

  • HFTP examines neuron-wise sentence and phrase representations in LLMs, population-level brain patterns, and generalization to naturalistic text.Source: §5 · p. 10
  • LLMs show hierarchical syntactic processing and alignment with left-hemisphere brain activity, but their representational mechanisms diverge from those in human cortical regions.Source: §5 · p. 10
  • Gemma 2 shows improved brain alignment while Llama 3.1 shows weaker human–model correlations despite enhanced task performance; the conclusion also notes possible benefits and risks of interpretability.The stated risks include optimizing persuasive manipulation and privacy harms if interpretability is linked with personal neural data.Source: §5 · p. 10

6 Limitation

The evaluation has cross-linguistic and model-coverage limits: sEEG data are primarily from Chinese stimuli, and only a small set of model architectures and scales was tested.

  • Although LLM analyses used Chinese and English corpora, the sEEG data were collected in China and primarily involved Chinese stimuli.Source: §6 · p. 10
  • English sEEG evidence is preliminary, based on one Chinese native participant; deeper cross-linguistic analyses and validation with native English speakers remain future work.Source: §6 · p. 10
  • Only a small set of model architectures and parameter scales was evaluated, leaving universal mechanisms of model–brain syntactic alignment unclear.Source: §6 · p. 10

A Alignment pipeline for syntactic processing between LLMs and the human brain

The alignment pipeline transforms model and brain responses into frequency-domain representational dissimilarity matrices, then compares their structure across layers and brain channels. It aggregates these comparisons into whole-brain and region-specific alignment metrics.

A Alignment pipeline for syntactic processing between LLMs and the human brain

  • For consistency, the pipeline uses the sentence and phrase Chinese corpora for both systems, with word-order-randomized controls and the final 32 syllables of each sequence.Each corpus has 40 trials of 36 syllables; SRDM calculation divides them into six conditions of 20 trials.Source: A Alignment pipeline for syntactic… · p. 15
  • Model activations are transformed with FFT and pairwise condition cosine similarities form a neuron-level SRDM; brain-channel ITPC yields a channel-level SRDM.The LLM and human measures both capture frequency-domain representations.Source: A Alignment pipeline for syntactic… · p. 15
  • The pipeline compares LLM-layer and brain-channel SRDMs using Spearman correlation, selecting each layer’s top 100 channels and testing sentence/phrase-channel overlap with a chi-square test.A significant overlap indicates alignment in structure processing between LLMs and brain ROIs.Source: A Alignment pipeline for syntactic… · p. 15
  • Model-brain similarity averages each layer’s Spearman correlations with its top 100 channels, while model-region similarity averages correlations for top channels within a specified ROI.The equations define the layer and channel indices and the channel set belonging to each region.Source: A Alignment pipeline for syntactic… · p. 15

B Predictive encoding control analysis

A predictive encoding analysis provides an orthogonal check on model–brain alignment by predicting sEEG frequency responses from LLM layer features. Its results reproduce key RSA patterns, including greater left-hemisphere alignment and model-upgrade differences.

B Predictive encoding control analysis

  • For each model layer, significant-neuron time courses are averaged by block, transformed with FFT, and represented by real and imaginary coefficients from 0.5–2 Hz.The predictor matrix stacks block-frequency samples; channel targets concatenate ITPC amplitudes across blocks and frequencies after aligning model and sEEG frequency axes.Source: B Predictive encoding control analysis · p. 16
  • Ridge regression predicts channel responses from layer features using five random 70/30 train/test splits, with the penalty selected by inner cross-validation on training data.Predictive accuracy is the average split-wise Spearman correlation on held-out targets; model-side feature selection avoids using sEEG data for selection.Source: B Predictive encoding control analysis · p. 16
  • Predictive scores replace SRDM correlations in the RSA aggregation, yielding comparable model–brain and model–region metrics from top-100 channel rankings.Sentence-corpus scores average across sentence, phrase, and combined neuron types; phrase-corpus scores use phrase-selective neurons only.Source: B Predictive encoding control analysis · p. 16
  • Between-model differences in brain-level alignment are significant for sentence and phrase corpora, and alignment is consistently higher in the left hemisphere.Sentence: F = 43.37, p < 0.001, η2 = 0.008; phrase: F = 13.80, p < 0.001, η2 = 0.008. FDR-controlled region-level effects occur in left-dominant language cortices, including STG, MTG, and ITG.Source: B Predictive encoding control analysis · pp. 16–17
  • Gemma 2 exceeds Gemma in sentence-corpus left-hemisphere model–brain similarity (0.469 vs. 0.467), while Llama 3.1 falls below Llama 2 (0.463 vs. 0.472).For the phrase corpus, Gemma 2 and Gemma tie at 0.465, while Llama 3.1 is slightly above Llama 2 (0.451 vs. 0.446); phrase gaps compress, likely because phrase analyses use fewer, noisier frequency samples.Source: B Predictive encoding control analysis · p. 17

C Bilingual sentence- and phrase-level representations in LLMs

Across multilingual models, language-specific and bilingual syntactic neurons show different layer distributions. Llama 2, Llama 3.1, and GLM-4 show increasing bilingual-neuron proportions in deeper layers, while Gemma-family and GPT-2 patterns differ.

C Bilingual sentence- and phrase-level representations in LLMs

  • Language-specific syntactic neurons tend toward final layers in Llama 2, Llama 3.1, and GLM-4, while bilingual-neuron proportions increase in deeper layers.Figure 7 depicts cross-language neural representations from Gemma, Gemma 2, Llama 2, Llama 3.1, and GLM-4.Source: C Bilingual sentence- and phrase-level… · pp. 18–19, Figure 7 · p. 18
  • Gemma has language-specific and bilingual neurons in both early and deep layers, whereas Gemma 2 shows this pattern only in early layers.The passage contrasts the models’ observed layer distributions.Source: C Bilingual sentence- and phrase-level… · p. 19
  • GPT-2 has roughly balanced language-specific and bilingual neuron distributions across layers, with English-specific neurons more prominent early and Chinese-specific neurons more concentrated late.The GPT-2 used here was pretrained on Chinese text with UER, unlike the original GPT-2, which cannot handle Chinese.Source: C Bilingual sentence- and phrase-level… · pp. 18–19
  • Llama 3.1 has fewer Chinese-specific neurons in its final layer than English-specific neurons, and fewer bilingual neurons than the other five LLMs.Although pretrained on 176 languages [Dubey et al., 2024], it appears less specialized in Chinese, which may explain these patterns.Source: C Bilingual sentence- and phrase-level… · p. 19

D Preliminary bilingual HFTP test on native Chinese speakers

A preliminary test in one native Chinese speaker found sentence- and phrase-rate peaks in English as well as Chinese. The appendix also describes how contribution ratios quantify regional representation among channels aligned with LLM layers and reports which regions contributed most across models.

D Preliminary bilingual HFTP test on native Chinese speakers

  • English listening showed robust 1 Hz sentence- and 2 Hz phrase-rate peaks, mirroring Chinese in one native Chinese speaker.The English materials were manually translated four-word sequences; the authors hypothesize that English speakers who understand Chinese would show similar peaks.Source: D Preliminary bilingual HFTP test on… · p. 19, Figure 8 · p. 19

E Contribution Ratios of LLMs

  • Contribution ratio measures each brain ROI’s share of the top 100 channels for a model layer, normalized by that ROI’s overall representation.The appendix presents these results for six LLMs and both hemispheres.Source: E Contribution Ratios of LLMs · pp. 19–20
  • Across LLMs, left-hemisphere A1 and STG and right-hemisphere Insula, Temporal Pole, and Amygdala contributed more to syntactic alignment.The authors suggest the pattern may involve language-specific and auxiliary regions.Source: E Contribution Ratios of LLMs · p. 20

F HFTP on naturalistic corpus

HFTP produced hierarchical frequency patterns in naturalistic Chinese and English text, while Wikipedia-derived corpora showed weaker and more variable tracking. The appendix also lists the model and corpus materials used.

F HFTP on naturalistic corpus

  • Naturalistic Chinese 8- and 9-syllable and matched English 8- and 9-word corpora showed four pronounced peaks, with analogous patterns across languages and sequence lengths.For eight-unit sequences, peaks were 0.5, 1.0, 1.5, and 2.0 Hz; for nine-unit sequences, they shifted to approximately 0.44, 0.89, 1.33, and 1.78 Hz.Source: F HFTP on naturalistic corpus · p. 20, Figure 9 · p. 21, Figure 10 · p. 21
  • Wikipedia-derived Chinese and English eight-unit corpora retained a 0.5 Hz sentence peak and phrase-rate peaks at 1.0, 1.5, and 2.0 Hz, but tracking was weaker and more variable.The passage attributes attenuation to irregular text features that reduce cross-sentence periodicity and harmonic reinforcement.Source: F HFTP on naturalistic corpus · p. 20, Figure 13 · p. 22
  • The authors speculate that shared tokenization statistics and prosodic templates produce two- and three-beat groupings, while clause phrasing yields a half-sentence rhythm.They associate the nine-unit 0.89 Hz peak with a prosodic break near the clause midpoint.Source: F HFTP on naturalistic corpus · p. 20

G Model details

  • The study used six LLMs, with model size, layer count, attention heads, and MLP-neuron counts summarized in Table 5.The supplied passage does not report the parameter values.Source: G Model details · p. 21

H Syntactic corpora

  • The corpus set included controlled Chinese four-syllable and English four-word syntactic materials, naturalistic Chinese and English corpora, and Wikipedia-derived out-of-domain validation corpora.The naturalistic materials included eight- and nine-unit sequences in each language; Wikipedia materials used eight units.Source: H Syntactic corpora · pp. 21–22

I Brain ROIs

The appendix identifies mapped brain ROIs and provides figure captions for model-specific contribution ratios across hemispheres, alongside a hierarchical-frequency figure for an English Wikipedia corpus.

I Brain ROIs

Related papers

Loading 2510.13255v1…