PaperlayerView exact revision on arXiv

Source-linked AI summary

SCRIBE: Scalable Contextual Representations of Interpretable Brain-Grounded Semantics

Camila Kolling, Omer Moussa, Emin Çelik, Mariya Toneva

arXiv:2610.21518v1cs.AIcs.CL

TL;DR

Because meaning varies with context, studying natural-language semantics in the brain is challenging. SCRIBE uses LLMs to generate interpretable, context-aware features, aligns with human judgments, and predicts neural responses beyond measured syntax.

  • Problem

    Meaning varies with context, making real-world linguistic semantics difficult to study in the brain.

    Source: §1 · p. 1

  • Method

    SCRIBE uses an ensemble of open-source LLMs to generate context-dependent semantic features within a neuroscience-informed space.

    Source: §1 · p. 2

  • Results

    SCRIBE reaches 75–81% of the human noise ceiling on semantic judgments, improves neural prediction with context, and retains unique brain-predictive variance beyond measured syntax.

    Source: §5 · p. 9

  • Takeaways & Limitations

    SCRIBE offers a scalable framework for studying contextual semantics in the brain using explicit, human-interpretable dimensions.

    Source: §5 · p. 9

    SCRIBE scores are systematically lower than human ratings on average, so applications relying on absolute scores may require calibration.

    Source: §5 · p. 9

Abstract

from arXiv · show

Understanding how the human brain constructs meaning from real-world language remains a fundamental challenge, as semantics depend on context that varies in natural settings. To enable the study of real-world linguistic meaning in the brain, we introduce SCRIBE, a framework grounded in a neuroscience-informed semantic space that leverages an ensemble of open-source large language models to generate interpretable, word-level semantic representations of arbitrary text. In a behavioral study with 921 participants, we show that human semantic feature judgments vary systematically with context, and that SCRIBE captures these shifts, reaching 75-81% of the estimated human noise ceiling. Using brain activity recorded while participants listen to naturalistic stories, we further demonstrate that SCRIBE predicts neural responses in language-selective brain regions, significantly outperforming non-contextual semantic representations. Variance partitioning reveals that SCRIBE explains unique brain-predictive information beyond syntactic features, suggesting that its representations capture aspects of linguistic meaning not accounted for by the measured syntactic features. Together, these findings establish SCRIBE as a scalable and interpretable foundation for studying natural language semantics in the brain.

1 INTRODUCTION

Meaning varies with linguistic context, making real-world semantics difficult to study using fixed annotations. SCRIBE applies LLMs within an interpretable semantic feature space, and the paper reports alignment with human judgments and improved neural prediction.

Motivation

  • Meaning varies with context, while the effectively unbounded range of natural contexts complicates experimental design and collection of human annotations.For example, “knife” can evoke harm or a neutral functional meaning depending on the sentence.Source: §1 · p. 1

Prior limitations

  • Traditional feature-based representations are interpretable but assume fixed meanings across contexts and often rely on manual annotation and fixed vocabularies.These constraints make them difficult to scale to arbitrary natural-language inputs.Source: §1 · p. 1

SCRIBE

  • SCRIBE uses an ensemble of open-source LLMs to extend a neuroscience-informed semantic feature space to arbitrary words in context.This approach is intended to combine contextuality and scalability with interpretable semantic dimensions.Source: §1 · p. 2

Main findings

  • SCRIBE captures context-dependent human semantic judgments at 75–81% of the estimated human noise ceiling and significantly outperforms out-of-context representations in neural prediction.The authors also report unique brain-predictive information beyond syntactic features.Source: §1 · p. 2

2 RELATED WORK

Prior approaches offer either interpretable but often static semantic features or contextual LLM representations that are difficult to interpret. SCRIBE uses LLMs to annotate context-sensitive concepts within a cognitively grounded, explicit semantic space.

Feature-based interpretable semantic representations

  • Feature-based approaches use interpretable, neurobiologically grounded dimensions but typically rely on out-of-context representations and manually annotated, limited vocabularies.SCRIBE instead uses LLMs to generate context-aware features in a semantically grounded space.Source: §2 · p. 2

LLM-based contextual semantic representations

  • Contextual LLM embeddings predict brain activity but are difficult to interpret and relate explicitly to semantic processing.Prior work also attributes part of pretrained LLM brain alignment to syntax.Source: §2 · p. 2

SCRIBE

  • SCRIBE combines a cognitively grounded semantic space, a target concept, and its linguistic context to generate property scores that are aggregated across models for brain prediction.LLMs are prompted independently for each semantic property.Source: Figure 1 · p. 3

Bridging interpretability and contextual semantic representations

  • SCRIBE and QAEmb both use LLMs to produce explicit, cognitively motivated word-level features, but SCRIBE targets conceptual semantics in Binder et al.’s (2016) neurocognitive space.QAEmb provides a broader framework spanning neuroscience-motivated linguistic properties.Source: §2 · p. 3

3 METHODS

SCRIBE maps concepts, optionally in context, to interpretable semantic feature scores. The methods describe its feature space and annotation, human-alignment and brain-encoding evaluations, and choices about words and context.

3.1 SCRIBE: INTERPRETABLE AND CONTEXT-AWARE SEMANTIC REPRESENTATIONS

  • SCRIBE uses 65 neurocognitively grounded semantic properties and averages four open-source LLMs’ independently prompted scores to represent concepts in context.The same interpretable feature space supports context-dependent profiles and automated annotation of naturalistic stimuli.Source: §3.1 · pp. 3–4

3.3 MODELING SEMANTIC REPRESENTATIONS IN THE BRAIN

  • Brain encoding uses word-level features to predict fMRI responses from seven participants listening to stories, with models trained on nine stories and tested on one held-out story.Features are aligned to the signal, augmented with time-lagged copies, and fit using ridge regression; evaluation normalizes correlations by voxel-wise noise ceilings.Source: §3.3 · p. 4

3.3.1 ANNOTATION DESIGN: WORD TYPES AND CONTEXT SIZE

  • SCRIBE annotates content words and uses the preceding 10 tokens for context, with broader word sets outperforming more restrictive ones and gains from additional context saturating.These choices were selected through ablation studies.Source: §3.3.1 · pp. 4–5

3.2 EVALUATING ALIGNMENT WITH HUMAN SEMANTIC JUDGMENTS

  • For isolated concepts, SCRIBE aligns significantly with human semantic ratings, with the ensemble achieving the best performance.The human-alignment evaluation reports Pearson correlation and MSE, and normalizes contextual-condition correlations by the estimated human noise ceiling.Source: §3.2 · p. 4, Table 1 · p. 5

4 RESULTS

SCRIBE represents semantic features in context and is evaluated against human judgments, neural responses, and syntactic controls. The results show context-sensitive human judgments, improved brain prediction from contextual scores, and predictive information beyond measured syntax.

4.1 SCRIBE GENERATES SEMANTIC FEATURES THAT ALIGN WITH HUMAN JUDGMENTS

  • For isolated concepts, the ensemble reached r = 0.77 and MSE = 1.07, with correlations above 0.6 across evaluated models and MSEs below permutation baselines.The ensemble showed the strongest alignment, supporting its use as a scalable approximation of human semantic ratings for isolated concepts.Source: §4.1 · p. 5

4.2 SCRIBE CAPTURES CONTEXTUAL SHIFTS IN HUMAN SEMANTIC JUDGMENTS

  • In a 921-participant study of 323 concept–property pairs, judgments followed HT > O > LT (all directional p < .001) across low- and high-target contexts.The contexts were generated to decrease or increase specific concept–property associations; the resulting norms are described as the first human-annotated context-dependent semantic feature norms and will be publicly released.Source: §4.2 · pp. 5–6, Table 2 · p. 6
  • Mean-of-eight reliability was ICC(1, 8) = 0.84 (O), 0.74 (LT), and 0.72 (HT); out-of-context judgments were most consistent.Noise ceilings were 0.92 (O), 0.86 (LT), and 0.85 (HT); all mean-of-eight reliability estimates were above chance (p ≤ .001).Source: §4.2 · p. 6
  • SCRIBE reproduced HT > O > LT (all directional p < .001) and reached 80% (O), 81% (LT), and 75% (HT) of human noise ceilings.Correlations with mean human ratings were r = 0.73 (O), 0.69 (LT), and 0.64 (HT).Source: §4.2 · p. 6

4.3 CONTEXT-AWARE SCRIBE SCORES ARE GOOD PREDICTORS OF BRAIN ACTIVITY

  • SCRIBEC significantly outperformed SCRIBEO across the whole brain and examined language-related ROIs, with contextual gains concentrated in language-related regions and more pronounced in the right hemisphere.The variants use the same semantic feature space and annotation procedure; SCRIBEC achieved performance comparable to QAEmb.Source: §4.3 · p. 7, Figure 2 · p. 7
  • SCRIBEC explained more unique variance than QAEmb on average in all four examined ROIs, with significant differences in LTC and AG.The methods had comparable whole-brain alignment, and further analyses found complementary representations and robustness after dimensionality reduction.Source: §4.3 · pp. 7–8, Figure 3 · p. 8

4.4 SCRIBE CAPTURES BRAIN-PREDICTIVE INFORMATION BEYOND SYNTACTIC FEATURES

  • After controlling for measured syntax, SCRIBEC retained more unique brain-predictive variance than QAEmb across language-related ROIs, significant overall in late-language regions and specifically in LTC and AG.SCRIBEC shared significantly less variance with syntax than contextual Llama2-7b in IFG/MFG, while contextual Llama2-7b embeddings had higher predictivity but encoded a broader mixture of linguistic information.Source: §4.4 · pp. 8–9

5 DISCUSSION

SCRIBE predicts neural responses better when it incorporates context, and its contextual semantic judgments align with human ratings. It offers an interpretable, scalable framework, with findings bounded by the study’s stimulus design, semantic space, and evaluation conditions.

5 DISCUSSION

  • SCRIBE uses LLM annotation in a cognitively grounded feature space to extend interpretable semantic representations from isolated concepts to arbitrary language contexts.The authors argue that interpretability and contextuality need not be opposing goals.Source: §5 · p. 9
  • Across 921 participants, contextual feature judgments shifted reliably, and SCRIBE reached 75–81% of the human noise ceiling on semantic ratings.The agreement was comparable to that between two independently collected sets of human ratings.Source: §5 · p. 9
  • Contextual SCRIBE significantly improves fMRI response prediction over its out-of-context representation, particularly in language-related cortical regions.It retains more unique brain-predictive variance than QAEmb after accounting for independently defined syntactic features.Source: §5 · p. 9
  • Scores depend on the underlying LLMs and prompts, and the findings concern averaged judgments and controlled, LLM-generated contexts.Scores are systematically lower than human ratings, so applications using absolute scores may require calibration; the fixed semantic space may also miss aspects of meaning.Source: §5 · p. 9
  • The syntactic control space is intentionally minimal, leaving richer structural and low-level or lexical-statistical features untested.Future work could extend SCRIBE to more diverse datasets and domains, connect semantic features to model internals, and expand the semantic space.Source: §5 · p. 9

ETHICS STATEMENT

The behavioral study received institutional ethics approval, obtained informed consent, and protected participant privacy.

ETHICS STATEMENT

  • Participants gave informed consent, and the study was approved by the authors’ institutional Ethical Review Board.Source: ETHICS STATEMENT · p. 10
  • Participants received USD 12 per hour, or USD 2 for the approximately 10-minute study.Source: ETHICS STATEMENT · p. 10
  • No personally identifiable information was collected or released; contextual ratings will be released in anonymized form with random participant codes.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

SCRIBE extends an interpretable, neuroscience-informed semantic feature space to context-dependent annotation with LLMs. Its appendix describes the feature space, annotation prompts, prompt robustness, and contextual scoring.

AI USE STATEMENT

  • The authors used generative AI to create controlled linguistic contexts and assist with manuscript editing, and reviewed and verified all AI-assisted outputs.The authors take responsibility for the methods, analyses, claims, and final content.Source: AI USE STATEMENT · p. 10

A.1 COGNITIVELY-GROUNDED, INTERPRETABLE SEMANTIC FEATURE SPACE

  • SCRIBE preserves the interpretability of Binder et al.’s semantic feature space while extending it to contextual settings through LLM-based annotation.The space has 65 properties spanning sensory, motor, spatial, temporal, and abstract dimensions; Table 3 lists 68 queried properties, recombining three split bipolar properties to yield 65.Source: §A.1 · p. 14, Table 3 · p. 15

A.2 LLM-BASED SEMANTIC ANNOTATIONS

  • SCRIBE prompts LLMs to rate how strongly a target concept is associated with each semantic property, producing scores with greedy decoding at temperature 0.The resulting first-number scores are deterministic under this decoding setup.Source: §A.2 · p. 17

A.2.1 OUT-OF-CONTEXT SEMANTIC ANNOTATION

  • Out of context, each concept receives 65 independently generated property ratings on a 0–6 scale, aggregated into a vector comparable to Binder et al.’s human ratings.The ratings are computed for the same concepts used in the human dataset.Source: §A.2.1 · p. 17

A.2.2 ABLATION OF PROMPT COMPONENTS

  • Prompt phrasing and formatting changes altered ensemble correlation with human ratings by at most 0.015, while the original prompt performed best.Removing property definitions caused the largest drops; removing the expert role caused a smaller drop. Table 5 reports the ablation results.Source: §A.2.2 · p. 18, Table 5 · p. 19

A.2.3 CONTEXTUAL SEMANTIC ANNOTATION

  • Contextual annotation conditions each concept–property rating on its sentence, allowing scores to reflect meaning modulated by context.Context windows include the k tokens preceding and including the target word.Source: §A.2.3 · p. 20

A.3 SYNTHETIC DATASET FOR CONTROLLED CONTEXTUAL EVALUATION

This appendix describes controlled contextual evaluation and the fMRI evaluation setup. It also reports that LLM annotations align with human semantic ratings beyond a shuffled-assignment baseline.

A.3 SYNTHETIC DATASET FOR CONTROLLED CONTEXTUAL EVALUATION

  • The controlled evaluation selects concept–property pairs across baseline strengths, then generates contexts intended to decrease or increase each association.The design aims to test whether semantic features respond predictably to targeted contextual manipulations while controlling for baseline constraints on change.Source: §A.3 · p. 22, §A.3.1 · p. 22

A.3.1 CONCEPT AND PROPERTY SELECTION

  • The selection procedure uses low, mid, and high bins of [0, 2), [2, 4), and [4, 6], retaining 57 properties with at least eight concepts per bin.It selects eight concepts per bin for each retained property, yielding 24 concepts per property; the fixed thresholds align with the original rating scale and support interpretable assignments.Source: §A.3.1 · p. 22, Figure 5 · p. 23

A.3.2 CONTEXTUAL SENTENCE GENERATION

  • For each selected pair, llama3.1-8b-instruct generates one sentence intended to decrease and another intended to increase the target property score.The generated sentences are designed to manipulate the target concept’s semantic interpretation in a controlled manner.Source: §A.3.2 · p. 23

A.4.1 FMRI DATASET

  • The fMRI subset contains stories listened to by seven participants, with encoding models trained on nine stories and evaluated on one held-out story.The subset excludes one participant with incomplete data and one story unavailable in the QAEmb feature set.Source: §A.4.1 · p. 26

A.4.2 ROIS DETAILS

  • Language regions are defined using a subset of the Glasser atlas’s 180 regions of interest per hemisphere.The selected ROI labels are listed in Table 8.Source: §A.4.2 · p. 26, Table 8 · p. 26

A.4.3 ESTIMATED NOISE CEILING

  • Voxel-wise noise ceilings use within-participant repetitions to estimate explainable variance; filtering below 0.4 leaves 30,000–50,000 voxels per participant.The noise ceiling estimates the maximum explainable variance for each voxel.Source: §A.4.3 · p. 26

B.1.1 PERMUTATION TEST

  • Across models, true annotations yielded lower MSE and higher Pearson correlation than shuffled concept–property assignments, indicating systematic semantic relationships.The permutation test repeated the shuffling procedure 1,000 times; Figures 6 and 7 show the score distributions for true and permuted assignments.Source: §B.1.1 · p. 27, Figure 6 · p. 28, Figure 7 · p. 29

B.2.1 LARGE-SCALE HUMAN BEHAVIORAL STUDY

Human semantic judgments shift systematically with linguistic context, and averaging eight raters yields reliable measurements. The study also shows that contextual judgments are less reliable than out-of-context judgments, informing the noise ceiling used to assess model alignment.

B.2.1 LARGE-SCALE HUMAN BEHAVIORAL STUDY

  • The study collected 11,983 ratings from 921 participants across 323 concept–property pairs and 969 annotation items, averaging 12.4 ratings per item.Pairs were evaluated in high-target, low-target, and out-of-context conditions.Source: §B.2.1 · p. 30
  • Human ratings followed HT (M = 4.23) > O (M = 3.23) > LT (M = 2.67; all comparisons p < .001), with no further LT decrease for low-baseline pairs.The low-baseline result is consistent with a floor effect.Source: §B.2.1 · pp. 30–31, Table 10 · p. 32

B.2.2 RELIABILITY OF HUMAN JUDGMENTS AND NOISE CEILING

  • With eight ratings per item, reliability ranged from 0.72 to 0.84; out-of-context judgments were more reliable than contextual judgments, which did not differ significantly.Reaching 0.90 reliability was estimated to require about 14 raters per item out of context and 25–29 in context.Source: §B.2.2 · p. 33, Figure 8 · p. 34

B.2.3 ALIGNMENT OF SCRIBE WITH HUMAN JUDGMENTS RELATIVE TO THE NOISE CEILING

SCRIBE aligns with human semantic ratings across conditions, with performance expressed relative to estimated human reliability. Its errors also include a systematic score offset, while its alignment remains similar when compared with an established out-of-context norm.

B.2.3 ALIGNMENT OF SCRIBE WITH HUMAN JUDGMENTS RELATIVE TO THE NOISE CEILING

  • The ensemble achieved the highest correlation with the mean of eight human ratings in all three conditions; its MSE was best in both contextual conditions, while qwen2.5-7b had slightly lower MSE out of context.Source: §B.2.3 · p. 35
  • SCRIBE rated properties about one point below humans in O and LT; this offset accounted for 42–45% of MSE in those conditions and 20% in HT.The MSE decomposition separates systematic offset, scale difference, and imperfect correlation.Source: §B.2.3 · p. 35, Table 15 · p. 36
  • Binder norms correlated with out-of-context human ratings at r = 0.77, or 84% of the ceiling, close to SCRIBE’s 80%.Using all ratings rather than balanced subsamples raised raw correlations but left ceiling-normalized alignment at 0.80, 0.80, and 0.75 for O, LT, and HT.Source: §B.2.3 · p. 35

B.2.4 RESULTS ON THE FULL SYNTHETIC DATASET

Across the full synthetic dataset, model-generated semantic features show the intended contextual shifts, and brain-prediction performance generally benefits from contextualized annotations. The voxel-wise maps localize contextualization-related gains mainly to language-related cortex, with stronger effects in the right hemisphere.

B.2.4 RESULTS ON THE FULL SYNTHETIC DATASET

  • All tested models and the ensemble showed the intended HT > O > LT score ordering across all pairs and within mid- and high-baseline bins; the ensemble showed significant effects in every bin.For low-baseline pairs, two individual models did not lower scores further in LT, mirroring the human floor effect.Source: Table 16 · p. 37

B.3.1 ANNOTATION DESIGN: WORD TYPES AND CONTEXT SIZE

  • Across most models, contextual information significantly improved brain-prediction performance, with k = 10 yielding the best results and All Semantic POS producing the highest overall correlations.The reported context-size pattern aligns with prior reports of gains that eventually saturate (Toneva & Wehbe, 2019).Source: §B.3.1 · p. 38, Table 17 · p. 38, Table 18 · p. 39

B.3.2 VOXEL-WISE DIFFERENCE BRAIN MAPS OF OUT-OF-CONTEXT AND CONTEXTUAL SCRIBE

  • Contextualization-related voxel-wise gains were concentrated in language-related cortical regions and were stronger in the right hemisphere.The maps contrast contextual SCRIBEC with out-of-context SCRIBEO across subjects.Source: Figure 9 · p. 42, §B.3.2 · p. 42

B.3.3 COMPLEMENTARITY BETWEEN QAEMB AND SCRIBE

SCRIBE and QAEmb contribute complementary brain-predictive information: their combined features improve held-out prediction over either representation alone. SCRIBE’s performance also remains similar after reducing its feature space from 65 to 33 dimensions.

B.3.3 COMPLEMENTARITY BETWEEN QAEMB AND SCRIBE

  • Both SCRIBEC and QAEmb explained unique neural-response variance, with SCRIBEC explaining more in LTC (p = 0.039) and AG (p = 0.023); voxel maps predominantly favored SCRIBE, especially in language-related areas.The voxel-wise pattern was less pronounced in Subject 8.Source: §B.3.3 · p. 43, Figure 10 · p. 44, Figure 11 · p. 45
  • The combined QAEmb + SCRIBEC representation achieved 0.2022 ± 0.0146 whole-brain noise-ceiling-normalized correlation, significantly exceeding SCRIBEC (0.1837 ± 0.0144) and QAEmb (0.1778 ± 0.0125).The combined representation also significantly outperformed both individual representations in the late-language network, LTC, IFG/MFG, and AG.Source: §B.3.3 · p. 43, Table 21 · p. 44

B.3.4 CONTROLLING FOR FEATURE-SPACE DIMENSIONALITY

  • Reducing SCRIBEC from 65 to 33 dimensions preserved whole-brain performance: 0.1837 ± 0.0144 versus 0.1859 ± 0.0126 after selection (p = 0.5781).Performance was also highly similar across the examined language and semantic ROIs.Source: Table 22 · p. 46, §B.3.4 · p. 46

B.3.5 VARIANCE PARTITIONING WITH SYNTACTIC FEATURES

Variance partitioning shows that SCRIBE retains more brain-predictive information beyond measured syntactic features than QAEmb, while sharing less variance with those features.

B.3.5 VARIANCE PARTITIONING WITH SYNTACTIC FEATURES

  • SCRIBE shares significantly less variance with syntactic features than QAEmb in LTC and AG, with the same numerical direction in Late Language and IFG/MFG.Its greater brain-prediction accuracy relative to QAEmb is not accompanied by greater overlap with measured syntactic information.Source: §B.3.5 · p. 47, Figure 12 · p. 47
  • SCRIBE retains significantly more unique brain-predictive variance than QAEmb in Late Language, LTC, and AG, with no significant difference in IFG/MFG.This variance remains after accounting for the syntactic feature space.Source: §B.3.5 · p. 47, Figure 12 · p. 47
  • SCRIBE’s greater unique variance and lower syntactic-feature overlap relative to QAEmb support a more semantically selective interpretation of its neural predictivity.The syntactic feature space includes part-of-speech tags, dependency length, and constituency tree depth.Source: §B.3.5 · p. 47

B.3.6 COMPARISON WITH LLM EMBEDDINGS

SCRIBE predicts brain responses better than non-contextual Llama2-7B embeddings, while contextual Llama2-7B performs better overall. Variance partitioning nevertheless finds unique SCRIBE-predictive information beyond syntactic features.

B.3.6 COMPARISON WITH LLM EMBEDDINGS

  • 0.1837 ± 0.0144 vs. 0.1185 ± 0.0106: SCRIBE’s whole-brain prediction accuracy exceeds non-contextual Llama2-7B embeddings by 55%, while contextual embeddings score 0.2387 ± 0.0215.SCRIBE also significantly outperforms non-contextual embeddings in Late Language, AG, LTC, and IFG/MFG.Source: §B.3.6 · p. 49, Figure 14 · p. 49
  • SCRIBE shares significantly less variance with syntactic features than contextual Llama2-7B in IFG/MFG, with the same numerical direction in Late Language and LTC.The comparison does not establish which linguistic information drives contextual Llama2-7B’s higher overall prediction performance.Source: §B.3.6 · p. 49, Figure 15 · p. 50
  • Permutation tests found significant unique SCRIBE-explained variance beyond syntactic features in all four language-related ROIs (all p ≤.002).Despite contextual Llama2-7B’s higher overall brain-prediction accuracy, SCRIBE retains unique predictive information.Source: §B.3.6 · p. 49, Figure 16 · p. 50

Related papers

Loading 2610.21518v1…