PaperlayerView exact revision on arXiv

Source-linked AI summary

What Shapes Brain Alignment of Large Language Models during Dialogue? Insights from Language Production and Comprehension

Yuko Nakagi, Yuya Chiba, Yasue Kishino

arXiv:2610.22961v1cs.CLcs.LGq-bio.NC

TL;DR

Dialogue brain alignment has been less understood than alignment during passive comprehension, particularly across processing modes, regions, and LLM layers. Using dialogue fMRI and LLM representations, this study analyzes alignment and representational properties; it reports mode-dependent patterns, associations with intrinsic dimension, and varying feature contributions.

  • Problem

    Prior work mainly examined passive comprehension, leaving dialogue alignment across production, comprehension, brain regions, and LLM layers—and its representational contributors—insufficiently understood.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The study analyzes dialogue fMRI alongside LLM representations across layers and brain regions, examining intrinsic dimension and removing interpretable dialogue-related features.

    Source: §1 · p. 2, §4.1 · p. 3

  • Results

    Dialogue brain alignment varies across processing modes, regions, and layers; it is strongly associated with intrinsic dimension, while linguistic and communicative feature contributions also vary across these dimensions.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The findings suggest that LLMs capture aspects of human language and dialogue processing during both production and comprehension, supporting evaluation across multiple processing modes.

    Source: §7 · p. 8

    The findings rely on one natural-dialogue fMRI dataset, primarily casual Japanese conversations about everyday topics, so their generalizability remains unestablished.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Brain alignment of large language models (LLMs) offers a way to investigate representational similarities between LLMs and the human brain during language processing. While previous studies have mainly focused on passive language comprehension, little is known about such alignment during dialogue, in which speakers engage in both language production and comprehension. In this study, using fMRI data collected during dialogue, we examine brain alignment during language production and comprehension across LLM layers and brain regions. We assess its association with the intrinsic dimension (ID) of LLM representations and the contributions of several interpretable linguistic and communicative features. Our results reveal distinct patterns of brain alignment between production and comprehension, as well as strong associations with ID in brain regions associated with each processing mode. Furthermore, we show that the contributions of linguistic and communicative features also vary across processing modes, LLM layers, and brain regions.

1 Introduction

Prior brain-alignment research has largely examined passive comprehension, leaving dialogue alignment across processing modes, brain regions, and LLM layers insufficiently understood. This study analyzes dialogue fMRI and LLM representations to examine alignment and its representational properties.

1 Introduction

  • Prior studies linked LLM representations to brain activity, with alignment strongest in intermediate layers, while mainly studying passive comprehension.The introduction frames dialogue as interactive language production and comprehension, with utterances conveying linguistic information and communicative intentions.Source: §1 · p. 1
  • Using dialogue fMRI and representations from multiple LLMs, the study examines production and comprehension alignment across brain regions and layers, its association with intrinsic dimension, and four dialogue-derived features.The reported patterns vary across processing modes, brain regions, and layers; alignment is strongly associated with intrinsic dimension, especially in regions closely associated with each mode.Source: §1 · pp. 1–2
  • The authors describe this as the first decomposition of dialogue brain alignment by representational features at multiple levels, extending analysis beyond the language network to higher-order cognitive regions involved in dialogue.Source: §1 · p. 2

2 Related Work

Prior work established LLM-to-brain prediction during comprehension and identified representational factors associated with alignment. Dialogue research has begun examining production and comprehension, but fine-grained alignment patterns and their relation to model representations remain open questions.

2 Related Work

  • During comprehension, intermediate LLM layers predict brain activity particularly effectively, and proposed alignment factors include surprisal, linguistic information, and representational complexity.Prior studies used brain encoding models to predict brain activity from LLM representations.Source: §2 · p. 2
  • Dialogue studies have examined shared and unique production–comprehension representations, context lengths, and speech-model representations, but fine-grained whole-brain alignment remains limited.Source: §2 · p. 2
  • Whether dialogue brain alignment varies with LLM representational properties had remained unexplored.Source: §2 · p. 2

3 Dataset and Models

The study pairs dialogue fMRI from seven native Japanese speakers with Japanese-tuned LLMs. Its dataset analysis uses temporally aligned dialogue transcripts and held-out sessions, while the model analysis centers on Llama-3.1.

3.1 fMRI Dataset

  • The Natural Dialogue fMRI Dataset contains recordings from seven native Japanese speakers conversing with an interlocutor outside the scanner on everyday topics.Runs lasted 7 minutes 10 seconds, and both speakers’ transcripts were aligned to one-second fMRI repetition-time segments.Source: §3.1 · p. 2
  • After excluding short sessions and two runs with missing transcripts, each participant contributed three sessions of 7–11 runs for three-fold leave-one-session-out evaluation.Each fold trains on two sessions and tests on the held-out session.Source: §3.1 · pp. 2–3

3.2 Large Language Models

  • The study uses three Japanese-tuned, conversational post-trained causal decoder-only Transformers with 8–21B parameters, presenting Llama-3.1 as the main analysis model.Gemma-2 and GPT-OSS results are presented in the Appendix.Source: §3.2 · p. 3

4 Methods

The methods extract context-sensitive LLM states, use voxel-wise encoding models to measure production and comprehension alignment, correlate alignment with representational intrinsic dimension, and remove interpretable dialogue features to assess their contributions.

4.1 Extraction of Latent Representations

  • At each time point, the input combines the target speaker’s current utterance with preceding dialogue from both speakers, and the final-token representation is extracted at context lengths 1, 4, 16, and 32.The procedure extracts participant and interlocutor representations separately.Source: §4.1 · p. 3

4.2 Brain Alignment of LLMs

  • Separate production and comprehension voxel-wise encoding models map LLM representations to fMRI activity, accounting for BOLD delays of 2–9 seconds and evaluating prediction with test-set Pearson correlation.Weights are estimated by ℓ2-regularized linear regression, with regularization selected per voxel using training data.Source: §4.2 · p. 3

4.3 Comparison between Brain Alignment and Representational Complexity

  • The study estimates layerwise intrinsic dimension with GRIDE and uses Spearman correlations to compare its association with alignment across processing modes and regions.Intrinsic dimension is described as estimating the number of dimensions underlying the representation space’s nonlinear structure.Source: §4.3 · p. 4

4.4 Removal of Interpretable Features Related to Dialogue

  • LEACE concept scrubbing sequentially removes linearly decodable information about bunsetsu count, lexicon, part of speech, or dialogue acts, then compares alignment and probing before and after removal.Feature labels are represented as one-hot or multi-hot vectors; LEACE is fit on training data.Source: §4.4 · p. 4

5 Results

The results show that brain alignment differs across production and comprehension, and that its relationships with representational complexity and dialogue features vary across brain regions. They also identify distinct regional patterns in the contributions of linguistic and communicative features.

5.1 Brain Alignment of LLMs during Dialogue

  • Production and comprehension show distinct spatial patterns: production alignment is strong in auditory, sensorimotor, and parietal regions, while comprehension alignment is strong in auditory, sensorimotor, and frontal regions.Whole-cortex alignment rises from early layers and peaks at intermediate layers for both modes, and increases with longer dialogue context; ROI trajectories differ, including a comprehension increase toward output layers in A1.Source: §5.1 · pp. 4–5, Figure 1 · p. 5

5.2 Representational Properties and Brain Alignment during Dialogue

  • Layerwise ID correlates strongly with whole-cortex alignment for both modes when context length is at least 4, while ROI associations are stronger for comprehension in auditory regions and production in SM.Correlations are largely comparable between modes in AG, MFG, and IFG.Source: §5.2 · pp. 5–6, Figure 2 · p. 6

5.3 Effects of Interpretable Features Related to Dialogue

  • Across processing modes, feature contributions vary by layer and region: L contributes most broadly, BC dominates around SM and auditory regions, and DA tends to dominate higher-order frontal regions.L’s largest contribution is particularly pronounced in production; BC makes a significantly greater unique contribution in A1 during comprehension for five participants, while L does so in AG for all participants (P < 0.001 for both).Source: §5.3 · pp. 6–8, Figure 3 · p. 7

6 Discussion

Production and comprehension show similar layerwise alignment trajectories, peaking at intermediate layers, but differ across brain regions and in how representational properties relate to alignment. ID and dialogue-derived features reveal distinct patterns across processing modes, layers, and regions.

6 Discussion

  • Production and comprehension alignment follow similar layerwise trajectories, both peaking at intermediate LLM layers.This pattern is consistent with prior findings on passive comprehension and evidence of neural activity shared across production and comprehension.Source: §6 · p. 8
  • Alignment is consistently high across layers in SM for production and in A1 and STG for comprehension.The analysis uses whole-brain voxel-level fMRI during dialogue.Source: §6 · p. 8
  • LLM representation ID strongly correlates with whole-cortex alignment in both modes, with stronger associations in regions primarily engaged by each mode.In language-related regions, the association is comparable between production and comprehension.Source: §6 · p. 8
  • LLM lexical features make the largest contribution to alignment, particularly for production; BC and L are prominent earlier, while POS and DA become more prominent in higher-order regions from intermediate layers.The authors suggest production alignment may be more influenced by concrete linguistic information, while comprehension may additionally involve abstract, context-dependent information; DA-related information tends to be dominant in frontoparietal and temporoparietal regions.Source: §6 · p. 8

7 Conclusions

The study finds that LLM feature contributions to brain alignment during dialogue vary across processing modes, layers, and brain regions. The findings suggest that LLMs capture aspects of human language and dialogue processing in both production and comprehension.

7 Conclusions

  • Feature contributions to dialogue brain alignment vary across processing modes, LLM layers, and brain regions.The authors conclude that LLMs capture aspects of human language and dialogue processing during both production and comprehension, and highlight evaluating alignment across multiple processing modes.Source: §7 · p. 8

Limitations

The findings are bounded by the models, interpretable features, and dialogue data examined. The study also uses text transcripts rather than multimodal signals.

Limitations

  • The analysis used three open-source LLMs, and whether the findings hold for a wider variety of models and larger LLMs remains unexamined.A broader, more diverse model set is needed to distinguish general representational properties from characteristics of the models examined.Source: Limitations · p. 9
  • The four interpretable features do not exhaustively characterize dialogue information, and other factors may contribute to alignment.The authors identify extending the feature set as important for a more complete understanding.Source: Limitations · p. 9
  • Findings rely on one fMRI dataset of natural dialogue, primarily casual Japanese conversations about everyday topics, so generalizability across datasets and dialogue contexts is unestablished.The authors call for study across broader dialogue settings and communicative contexts.Source: Limitations · p. 9
  • The publicly available dialogue data are limited to text transcripts, so the analyses omit audio, facial expressions, gaze, and gestures.The authors note that incorporating multimodal signals may provide a more comprehensive account of alignment between multimodal LLM representations and brain activity.Source: Limitations · p. 9

Ethical Considerations

The study reanalyzes publicly available fMRI and dialogue data collected with ethics approval, using anonymized data without personally identifiable information.

Ethical Considerations

  • The analyses used anonymized, publicly available data collected with ethics committee approval and accessed no personally identifiable information.The authors characterize the study's ethical risks as minimal.Source: Ethical Considerations · p. 9

A Additional Methods

The additional methods specify the selected brain regions and models, correct fMRI activity for head motion, estimate intrinsic dimension, and construct interpretable dialogue features.

A.1 ROI Selection

  • The analysis uses eight dataset-provided ROIs, including regions selected for language processing, production, and comprehension.The selected regions include STG, STS, MTG, AG, MFG, IFG, SM, and A1.Source: §A.1 · p. 13

A.2 Large Language Models

  • Three Japanese-language Swallow models are analyzed: Llama-3.1-Swallow, Gemma-2-Swallow, and GPT-OSS-Swallow.The first two were instruction-tuned; GPT-OSS-Swallow underwent continued pretraining and supervised fine-tuning.Source: §A.2 · p. 13

A.3 Head-Motion Correction

  • Voxel-wise encoding models predict activity from a head-motion feature, and their predictions are subtracted before brain-alignment analyses.The regression and cross-validation procedures match those described in Section 4.2.Source: §A.3 · p. 13

A.4 Estimation of Intrinsic Dimensions

  • GRIDE estimates intrinsic dimension from nearest-neighbor ratios, then selects the candidate scale with the lowest median relative change across layers.The estimation assumes locally uniform density and independent ratios across data points; candidate scales exclude the smallest and largest evaluated scales and restrict s to ≤8.Source: §A.4 · pp. 13–14

A.5 Interpretable Features Related to Dialogue

  • Four dialogue-related features encode bunsetsu count, everyday-conversation lexicon, part of speech, and dialogue acts at the TR-segment level.Dialogue-act tags are assigned manually by two annotators, with a tag retained only when both agree.Source: §A.5 · p. 14

B Additional Results

Across three LLMs, dialogue brain alignment peaks at intermediate layers and shows consistent processing-mode-specific regional patterns. Feature removal substantially reduces decodable information about each targeted feature.

B.1 Brain Alignment of LLMs during Dialogue

  • Across all three LLMs, brain alignment peaks at intermediate layers, with high production alignment in SM and AG and high comprehension alignment in A1 and STG.Comprehension alignment in A1 rises again near the output layers; layerwise patterns are similar across participants.Source: §B.1 · pp. 14–15

B.2 Probing Performance of Interpretable Features

  • For production and comprehension, ridge-regression correlations decrease substantially after feature removal across all tested features.The authors interpret this as evidence that removal substantially reduces linearly decodable information about the targeted features.Source: §B.2 · p. 15

B.3 Effects of Interpretable Features Related to Dialogue

Across Gemma-2 and GPT-OSS, supplementary results match the Llama-3.1 pattern: lexicon contributes most to alignment, especially during production. Spatial and ROI-level feature patterns are broadly consistent across participants and models.

B.3 Effects of Interpretable Features Related to Dialogue

  • Lexicon makes the largest contribution to brain alignment across layers in Gemma-2 and GPT-OSS for both processing modes, especially production.The supplementary tables report percentage decreases after feature removal for whole-cortex-averaged alignment.Source: §B.3 · p. 15, Table 4 · p. 16, Table 5 · p. 16
  • Across participants and models, spatial patterns of dominant features are similar, particularly for bunsetsu count and lexicon.During comprehension, POS and dialogue acts also dominate in temporoparietal, parietal, and frontal voxels from intermediate layers onward.Source: §B.3 · p. 15
  • Across participants and models, the relative magnitudes of each feature’s unique contribution across ROIs are broadly consistent.The supplementary boxplots show these ROI-level contributions for production and comprehension in Llama-3.1, Gemma-2, and GPT-OSS.Source: §B.3 · p. 15
Loading 2610.22961v1…