PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying Meaning with SCAPE-T: Semantic Complexity from Encoder-Only Transformer Attention

Maxim Khovansky, Lev Mukhanov

arXiv:2610.33441v1cs.CLcs.AI

TL;DR

Semantic complexity lacks a generally adopted measure that directly targets meaning rather than surface difficulty. This paper defines it as semantic code length and introduces SCAPE-T, which aggregates encoder-attention signals. Under word-length control, SCAPE-T outperforms tested baselines on held-out domains and leads on an abstractness set.

  • Problem

    Existing complexity measures often target readability, lexical difficulty, or surface features, while semantic complexity remains underexplored.

    Source: §I · p. 1

  • Method

    SCAPE-T estimates semantic code length using encoder attention signals for inverse-weighted self-focus, CLS focus, and inter-head redundancy.

    Source: §I · p. 2, B. Applying the definition to… · p. 4

  • Results

    Under word-length control, SCAPE-T reaches 78.5% pairwise accuracy, 12.7 percentage points above the strongest baseline, and it also leads on the abstractness set.

    Source: A. Primary Results · p. 7, §VI · p. 8

  • Takeaways & Limitations

    The results support SCAPE-T’s ability to isolate semantic integration under length control and transfer across genres.

    Source: §VI · p. 8, §VIII · p. 8

    Hyperparameters are tuned once on a single development contrast, leaving overfitting risk and stability across alternative splits insufficiently characterised.

    Source: §VII · p. 8

Abstract

from arXiv · show

Automated semantic complexity estimation remains underexplored in NLP, despite important applications in linguistics and interpretability. This paper defines semantic complexity as the code length required to uniquely specify a text's semantic meaning, and operationalises it via SCAPE-T, an attention-derived metric based on encoder-only transformers. Existing measures either model reader effort (readability, lexical difficulty), which correlates with surface features (word length, syntax), or quantify surface features directly (type-token ratio, word frequency). SCAPE-T targets semantic integration directly by combining three submetrics: (i) inverse-weighted self-focus (token-to-self), (ii) CLS focus (token-to-CLS), and (iii) inter-head redundancy (cosine similarity across heads), using layer-weighted aggregation; data-driven learning is confined to hyperparameter tuning for calibration. Development uses philosophical prose vs. MCTest stories; evaluation is on children's stories vs. technical Wikipedia articles, with pairwise accuracy on contrasts as the primary criterion. Baselines include readability, syntactic/lexical, and LLM-derived measures. SCAPE-T substantially outperforms all tested baselines under word-length control, with a pairwise accuracy of 78.5% (95% bootstrap CI: 76.1-80.4, N=10k pairwise comparisons; +12.7 percentage points over the strongest baseline); without that control, readability heuristics overtake, consistent with their reliance on surface features. On a short-terms abstractness dataset, SCAPE-T also leads. Ablations show each submetric contributes materially. Overall, the results show SCAPE-T isolates conceptual structure from lexical/syntactic confounds better than existing methods, and generalises across two held-out domains.

I. INTRODUCTION

The paper defines semantic complexity as the code length needed to specify meaning and introduces SCAPE-T, an encoder-attention metric intended to measure it directly. Its evaluation tests whether the metric distinguishes semantic complexity beyond surface features.

I. INTRODUCTION

  • Semantic complexity is defined as the minimum per-token code length needed to uniquely specify a text’s meaning relative to a fixed situation model and empirical world state.The paper frames semantic complexity as distinct from readability or lexical sophistication.Source: §I · p. 1
  • SCAPE-T combines inverse-weighted self-focus, CLS focus, and inter-head redundancy through layer-weighted aggregation.Calibration tunes hyperparameters on token-length-matched development contrasts without additional task supervision or model-weight training.Source: §I · p. 2
  • Because semantic-complexity ground truth is unavailable, evaluation uses proxy contrasts and pairwise accuracy, with held-out domains testing generalisation.The development contrast pairs Hegel and Kant prose with MCTest stories; testing contrasts technical Wikipedia with children’s stories.Source: §I · p. 2
  • Under word-length control, SCAPE-T outperforms tested baselines on held-out domains by 12.7 percentage points and also leads on the abstractness set.Without length control, readability heuristics overtake SCAPE-T.Source: §I · p. 2
  • The paper’s contributions include formalising semantic complexity and proposing an evaluation protocol that controls word length and adds a style- and domain-neutral abstractness set.Ablations are reported to confirm that each attention-derived submetric contributes.Source: §I · p. 2

A. Readability and Psycholinguistic Predictors

Prior complexity measures largely target readability, predictability, or manually represented meanings rather than semantic integration. The review motivates a scalable, label-free metric based on model-internal evidence.

A. Readability and Psycholinguistic Predictors

  • Readability and psycholinguistic predictors capture surface form and vocabulary difficulty but do not directly isolate semantic integration.These include readability formulas, lexical statistics, and human-anchored features such as age of acquisition and familiarity.Source: A. Readability and Psycholinguistic… · p. 3

B. Information-Theoretic, Formal, and Discourse-Based Proxies

  • Predictability and compression measures track token-level predictability, while formal semantic frameworks require hand-crafted representations that limit automation and scalability.Discourse models address semantic organisation more directly but rely on parsers and inventories and can be sensitive to genre and parser error.Source: B. Information-Theoretic, Formal, and… · p. 3

C. Attention- and Representation-Based Methods

  • An attention-based autoencoder is trained and evaluated on lexical-complexity annotations, while SCAPE-T uses encoder attention signals to target semantic integration.The review cautions that attention is a diagnostic measure here, not a causal explanation of model predictions.Source: C. Attention- and Representation-Based… · p. 3

D. Summary

  • The identified gap is a scalable, label-free metric that targets semantic integration using model-internal evidence.The reviewed alternatives include surface predictors, predictability proxies, non-automated semantic frameworks, discourse models, and neural methods trained on lexical labels.Source: D. Summary · p. 4

A. Why encoder self-attention?

The paper uses frozen encoder self-attention to estimate input-referential semantic code length, then motivates attention-derived measures through assumptions about sentence-level meaning, token relations, and nonredundant channels.

A. Why encoder self-attention?

  • Encoder-only self-attention is chosen because it is conditioned on input tokens alone, unlike decoder and encoder–decoder attention that incorporate output-side information.The frozen encoder serves as the fixed situation model and its empirical parametric knowledge as a proxy for the empirical world state.Source: A. Why encoder self-attention? · p. 4, B. Applying the definition to… · p. 4

B. Applying the definition to encoder-only transformers

  • SCAPE-T treats sentence-level emergence, subsentential emergence, and nonredundant channel utilisation as monotonic correlates of higher semantic complexity.The submetrics are theoretically motivated correlates, not exhaustive estimators, and can be extended with additional justified correlates.Source: B. Applying the definition to… · p. 4

1) Estimator 1. CLS focus (from A1):

  • CLS focus compares attention directed to [CLS] with attention to other columns, estimating sentence-level content that emerges beyond constituent meanings.In BERT-style encoders, [CLS] aggregates sentence-level information.Source: 1) Estimator 1. CLS focus (from A1) · pp. 4–5
  • Inverse-weighted self-focus estimates token interdependence, while head redundancy discounts duplicated relational channels beyond self- and CLS-directed links.Lower self-focus corresponds to greater subsentential relational constraints; larger redundancy coefficients indicate more independent relational channels.Source: 1) Estimator 1. CLS focus (from A1) · p. 5
  • The redundancy comparison considers heads in previous layers and earlier-indexed heads in the same layer, making the estimator invariant to permutations of later heads.A threshold discounts baseline cosine similarity attributed to shared inputs and chance overlap.Source: 1) Estimator 1. CLS focus (from A1) · p. 5

4) Layer aggregation:

Layer aggregation gives greater weight to intermediate-to-late encoder layers, while retaining a uniform component to calibrate that emphasis.

4) Layer aggregation:

  • The aggregation uses a normalised mixture of Gaussian and uniform layer weights, with the Gaussian centred at μ and variance σ^2.The uniform component allows the strength of the layer emphasis to be calibrated.Source: 4) Layer aggregation · p. 5
  • The bell-shaped weighting profile emphasises intermediate-to-late layers because prior evidence places semantic encoding there more strongly than in early or very top layers.Early layers are described as emphasising lexical and syntactic processing, while very top layers are more task-specific.Source: 4) Layer aggregation · p. 5
  • The Gaussian component’s mean and variance set the location and spread of the layer emphasis.The uniform mixture allows the emphasis to be adjusted.Source: 4) Layer aggregation · p. 5

5) Composite estimator:

The composite estimator combines its specified submetrics into one score, while SCAPE-T treats them as correlates of minimum description length rather than an exhaustive estimate. Its design allows additional correlates when they meet stated constraints.

5) Composite estimator:

  • The composite estimator combines the specified submetrics into one score, with Base length-normalized to compensate for CLS-focus sequence-length dependence.The length-normalization exponent is retained for tuning more complex dependencies in attention distributions.Source: 5) Composite estimator · p. 5

C. Scope, correlates, and extensibility

  • The three base submetrics are theoretically motivated correlates, not an exhaustive estimate of minimum code length; added correlates must have a justified monotonic relationship and remain input-referential.Source: C. Scope, correlates, and extensibility · pp. 5–6

5) Composite estimator:

  • ISF applies a post-aggregation transform, and separating it from Base improved performance empirically on the development set.Multi-sentence texts receive the arithmetic mean of their per-sentence scores.Source: 5) Composite estimator · p. 5, Figure 1 · p. 6

IV. DATA & EVALUATION

The evaluation uses simple-versus-complex sentence contrasts, a separate short-term abstractness lexicon, and pairwise accuracy to compare scores across groups.

A. Sentence datasets.

  • The development contrast contains 200 token-length-matched pairs; the test WLC set contains 100 pairs matched on average word length and the 90th-percentile maximum word length.The source collections include philosophical prose and MCTest stories for development, and children’s stories and technical Wikipedia pages for testing; all sets require BERT token length of at least 10.Source: A. Sentence datasets · p. 6

B. Abstractness lexicon (short terms).

  • The abstractness lexicon contains 236 human-checked terms of at most two words, organized into four classes from physical references to highly abstract concepts.Each term is scored as a standalone input.Source: B. Abstractness lexicon (short terms) · p. 6

C. Evaluation.

  • Pairwise accuracy compares scores between complex and simple groups; for abstractness, it compares Category 4 with Categories 1 and 2, excluding Category 3 for clearer contrasts.Ties count as one-half.Source: C. Evaluation · p. 6

D. Hyperparameter tuning.

SCAPE-T is tuned once on a token-length-controlled development contrast, then evaluated with frozen settings. It leads the tested baselines on the word-length-controlled test and short-term abstractness set, while readability and lexical heuristics score highest without word-length control.

D. Hyperparameter tuning.

  • A compact (1 + λ) evolutionary strategy tunes hyperparameters once by maximizing pairwise accuracy on the token-length-controlled development set, then freezes settings for test sets and the abstractness lexicon.No test data is used for tuning.Source: D. Hyperparameter tuning · p. 6

A. Primary Results

  • On the uncontrolled test set, lexicon scores 94.3%, Flesch–Kincaid 93.1%, and Zipf 85.4%, while SCAPE-T reaches 85.1%.Source: A. Primary Results · p. 7
  • On the token-length-controlled development set, SCAPE-T scores 87.1% (bootstrap 95% CI: 0.81–0.91, N=40,000 pairwise comparisons).Source: A. Primary Results · p. 7
  • On the short-terms abstractness dataset, SCAPE-T leads with 82.3%, ahead of Flesch–Kincaid (78.9%) and the lexicon baseline (74.4%).In the linguistic-bias test, French children’s sentences scored higher in 78% of pairs against the development complex group; Scottish English dialectal terms showed a similar tendency.Source: A. Primary Results · p. 7
  • 78.5% pairwise accuracy on WLC Test (95% bootstrap CI: 76.1–80.4) beats DHH lexicon complexity (65.8%) by +12.7pp; Table II reports the comparison.Source: Table II · p. 7, A. Primary Results · p. 7

B. Ablations

SCAPE-T remains strongest under word-length control and on short abstractness items, while ablations identify CLS focus and layer weighting as especially informative. Its transfer and layer-specific patterns also expose scope and interpretation questions.

B. Ablations

  • CLS focus and layer weighting produce substantial ablation drops on both sets, while self-focus drops are smaller but significant and head redundancy matters on abstractness only.The self-focus drop has a 95% bootstrap CI of -5.67 to -3.42.Source: B. Ablations · p. 7, Table III · p. 7, Table IV · p. 7
  • Removing layers 2 and 10 causes substantial drops, layers 1 and 3 yield minor gains, and other layers show no notable changes.The discussion suggests early syntactic scaffolding and late semantic integration as a possible explanation for layers 2 and 10’s importance.Source: B. Ablations · p. 7, Figure 2 · p. 7, §VI · p. 8

VI. DISCUSSION

  • Under word-length control, SCAPE-T leads the tested baselines, and it also leads on the abstractness set; readability and lexical baselines score highest without control.The discussion attributes their uncontrolled advantage, at least in part, to sensitivity to predictive surface cues.Source: §VI · p. 8
  • Hyperparameters tuned on the length-controlled Hegel/Kant–MCTest contrast transfer to technical Wikipedia prose versus children’s stories.Source: §VI · p. 8
  • Because SCAPE-T reads English BERT Base attention, non-standard or non-English inputs may receive inflated scores; multilingual encoders are advised for such data.Source: §VI · p. 8
  • The discussion characterizes CLS focus and layer weighting as performance drivers, self-focus as adding local-context sensitivity, and redundancy as more relevant for short inputs.Early-plus-late reliance for interdependence remains an open limitation to probe.Source: §VI · p. 8

VII. LIMITATIONS

The conclusion reports that SCAPE-T outperforms tested baselines under word-length control and that its submetrics and aggregation scheme contribute materially. The limitations qualify its model portability, knowledge source, evaluation scope, and current extensions.

VII. LIMITATIONS

  • A single frozen BERT Base situation model, English tokeniser, and CLS-style aggregator constrain portability; comparability across models is not established.The metric is a diagnostic proxy for semantic integration, not a causal account of model reasoning.Source: §VII · p. 8
  • Pretraining-corpus knowledge brings coverage gaps and biases, though their effect on estimates is considered unlikely to dominate and requires investigation.SCAPE-T uses parametric knowledge as a proxy for true world state.Source: §VII · p. 8
  • Ablations concentrate layer importance at layers 2 and 10, especially layer 2, whose near-exclusive importance requires explanation.Emphasising mid-to-late layers appreciably improves performance.Source: §VII · p. 8
  • Proxy contrasts are not gold standards; length control can distort natural length distributions, and sentence-level scoring only indirectly reflects discourse phenomena.Preliminary observations also indicate inflated scores for non-English or dialectal inputs processed with an English encoder.Source: §VII · p. 8

VIII. CONCLUSION

  • Under word-length control, SCAPE-T outperforms readability, lexical, syntactic, language-model, and compressor baselines on held-out domains and leads on a style- and domain-neutral abstractness set.The conclusion describes this as a proof of concept for using encoder attention patterns to gauge semantic complexity.Source: §VIII · pp. 8–9
  • Each submetric and the aggregation scheme contribute materially, while complementary internal signals remain untested.The conclusion identifies model portability, discourse-aware variants, semantic-content challenge sets, and cross-model calibration as future directions.Source: §VII · p. 8, §VIII · p. 9

APPENDIX A ABBREVIATIONS, ACRONYMS, AND TERMINOLOGY

The appendix identifies abbreviations, technical acronyms, and shorthand, and outlines preprocessing and text-normalisation steps used before scoring.

APPENDIX A ABBREVIATIONS, ACRONYMS, AND TERMINOLOGY

  • Table V covers abbreviations, technical acronyms, and shorthand.Source: Table V · p. 11

A. Overview

  • The pipeline normalises text, tokenises and groups sentences, cleans sentences, projects content words, then scores sentences and averages them to document level.Source: A. Overview · p. 11

B. Text-level normalisation

  • Normalisation replaces ellipses according to the next non-space character and expands common abbreviations and citation shorthands once, from left to right.Repeated punctuation is also collapsed.Source: B. Text-level normalisation · p. 11
  • Normalisation examples include expanding titles and Latinisms, standardising bibliographic shorthands and “etc.”, and removing periods after Roman numerals.Source: B. Text-level normalisation · p. 11

C. Segmentation and sentence grouping

The appendix defines how text is segmented and grouped into sentences, then describes sentence-level rejection, repair, filtering, and final cleaning.

C. Segmentation and sentence grouping

  • Segmentation treats maximal non-space spans excluding specified punctuation as words and individual punctuation marks as punctuation; delimiters end sentences, while trailing fragments are kept.Source: C. Segmentation and sentence grouping · p. 11

D. Per-sentence cleaning

  • Each raw sentence undergoes hard rejection, repairs, a repeated fixed-point filter, and final sanitation; rejection covers direct speech, formatting artefacts, and number-led markers.Repairs and filtering remove or normalise stray punctuation, prefixes, and references; final cleaning appends a delimiter when needed and discards sentences beginning with lowercase letters.Source: D. Per-sentence cleaning · pp. 11–12

E. Content-word projection (before scoring)

Before scoring, the pipeline removes punctuation and selected stop words, then averages sentence scores to form a deterministic document-level result.

E. Content-word projection (before scoring)

F. Aggregation to document level

G. Determinism

  • Preprocessing is deterministic given the input because all steps are functional and the frozen encoder runs in evaluation mode with attention outputs enabled.Source: G. Determinism · p. 12

APPENDIX C HYPERPARAMETER BOUNDS

Loading 2610.33441v1…