PaperlayerView exact revision on arXiv

Source-linked AI summary

Articulation Upper Bounds for Language Models

Alexander Spangher, Daniel E. Ho, Diyi Yang, Noah Goodman, Sanmi Koyejo

arXiv:2610.19721v1cs.AIcs.CLcs.LG

TL;DR

The paper asks how much meaning can be conveyed between LLMs when language captures only part of what speakers intend. It optimizes messages, measures receiver recovery, and studies transmission as model capability scales. Across 1,270 concepts, most were transmissible now or may be with future scaled models, while 65 were bounded.

  • Problem

    Language captures only a subset of perceptions, intuitions, and intents, motivating measurement of how well concepts can be articulated between LLMs.

    Source: §1 · p. 1

  • Method

    The paper optimizes prompts with ϵ-estimation, measures receiver recovery with a reconstruction bottleneck, and uses observational scaling laws across model capability.

    Source: §1 · p. 2, §2 · p. 3, §5 · p. 11

  • Results

    Across 1,270 concepts, 321 were transmissible now, 884 may be transmissible with future scaled models, and 65 were bounded.

    Source: §4 · p. 10

  • Takeaways & Limitations

    Humans had moderate-to-high agreement on 76% of LLM-bounded concepts, so the observed articulation limits are executor-relative.

    Source: §1 · p. 2

    The unsupervised reconstruction measure can reward a receiver for consistently restating a rule that is wrong.

    Source: §5 · p. 12

Abstract

from arXiv · show

Language captures only a subset of the perceptions, intuitions and intents that form meaning between speakers. What, then, are the limits of language? We take steps towards measuring this in LLMs as follows: for a set of concepts, we measure how well each concept can be transmitted between two LLMs as their capabilities scale. This requires novel measurement tools: (1) optimal messages: we introduce a prompt optimization algorithm that beats existing algorithms by up to 14% on benchmarks and accurately estimates prompt upper bounds. (2) A transmission measure: the sender is asked to label text; the receiver then tries to recover the sender's prompt, and recovery is measured; we find this correlates strongly with human label error. With these measures, we use observational scaling laws to study how transmissibility of a concept, conveyed with optimal messages, increases as transmitter and receiver capacity grows: does it asymptote below or above a threshold? Measuring 1.2k concepts in eight domains, we find concepts that (a) can be fully articulated; these concepts usually depend on local patterns, e.g. "joke ends with surprise"; (b) might be articulated by future, stronger models; these rely on whole-text structural patterns, e.g. "story's worldbuilding coherence"; (c) concepts that face articulability limits; these concepts' meaning, we find, is tied to identity and social-group awareness, e.g. "comic voice". Interesting, these same concepts are not all challenging for humans: humans have moderate-to-high levels of agreement in 76% of concepts that LLMs struggle with; articulation limits, we find, are executor relative. Finally, we characterize the nature of improvement in each case: while recoverable concepts can be articulated through definitions and explanations, irrecoverable concepts cannot be transmitted, even with examples, and might require novel learning mechanisms.

1 INTRODUCTION

Language may not capture every concept speakers intend to convey, yet articulation matters when LLMs act through rubrics or constitutions. The paper introduces tools to measure message quality and transmission, then uses them to study which concepts remain articulable as models scale.

1 INTRODUCTION

  • ϵ-estimation beats existing prompt optimization algorithms (Agrawal et al., 2025; Opsahl-Ong et al., 2024) by up to 14% at equal budgets across six benchmarks.It builds a search space of meaning units, combines them into prompts, and estimates missing search steps; its mid-run improvement estimates have ≈5% error.Source: §1 · p. 2
  • The reconstruction bottleneck measures transmission by having a receiver reconstruct the sender’s prompt from its annotations, and it reliably predicts supervised performance where ground-truth labels exist.It provides an unsupervised transmission measure.Source: §1 · p. 2, Figure 1 · p. 2
  • Across 1.2k concepts in eight domains, local-pattern concepts were already articulable, structural-pattern concepts may become articulable with scale, and identity/social-group concepts plateaued.The examples include joke mechanics, worldbuilding coherence, and comic voice, respectively.Source: §1 · p. 2
  • Humans had moderate-to-high agreement on 76% of LLM-bounded concepts, indicating that the observed articulation limits are executor-relative.Here, the executor is the model running the prompt, such as the transmitter.Source: §1 · p. 2
  • For concepts where transmission is possible, explicit explanations, mechanics, and descriptions increase transmission most in optimal messages; examples offer only limited evidence for improving transmission in less feasible cases.The authors cannot conclude that examples or demonstrations increase transmission for those less feasible concepts.Source: §1 · p. 2

2 THE Transmitter-Message-Receiver PIPELINE

The paper defines articulability as concept recovery between a transmitter and receiver, then develops prompt optimization and reconstruction methods to study that transmission. Its results show gains for ϵ-estimation and support reconstruction as an unsupervised measure and optimization objective.

2 THE Transmitter-Message-Receiver PIPELINE

  • Articulability is the limit of concept transmissibility over optimal messages and transmitters and receivers of infinite capacity.More articulable concepts can be transmitted at lower capacity and with worse messages.Source: §2 · p. 3

2.1 THE Message: FINDING OPTIMAL MESSAGES WITH ϵ-ESTIMATION

  • ϵ-estimation builds prompts from behavior-changing semantic units, searches combinations through addition and pruning, then estimates unseen units’ value to project an upper bound.The estimate is conditional on the declared samplers and their search space.Source: §2.1 · pp. 4–5

2.2 THE Transmitter-Receiver: RECONSTRUCTION BOTTLENECK

  • The reconstruction bottleneck asks a receiver to infer the transmitter’s prompt from labeled examples, then measures recovery on held-out items against controls.Recovery assumes both models can express the generating rule; lexical traces can resemble understanding without successful application to new items.Source: §2.2 · pp. 5–6

2.3 PERFORMANCE RESULTS

  • The evaluation uses six benchmarks and 1.2k annotation tasks, each representing a criterion that a rubric might contain.The annotation tasks come from nine domains; six of those datasets have human labels.Source: §2.3 · p. 6, §2.3.2 · p. 6

2.3.1 ϵ-ESTIMATION RESULTS

  • ϵ-estimation outperforms existing methods by up to 14% at equal call budgets on HotpotQA, HoVer, PUPA, and AIME, and is not significantly worse on any benchmark.Its upper-bound estimates have median prediction error ≈.02 eight units ahead.Source: §2.3.1 · p. 6

2.3.2 RECONSTRUCTION BOTTLENECK RESULTS

  • Reconstruction optimization beats an LLM-critic objective by +.033 AUC (99 wins to 54 losses), while reconstruction MI correlates .37–.68 with AUC across criteria.Reconstruction matches, but does not beat, description-based labels; those labels have an advantage because annotations partly used description text.Source: §2.3.2 · pp. 6–7

3 ARTICULATION UPPER BOUNDS: OBSERVATIONAL SCALING LAWS FOR TRANSMITTER-RECEIVERS

The study fits transmission against model capability and identifies rising, reaches, and bounded concepts. The regimes differ in the patterns they capture and in how explicit messages or examples help.

3 ARTICULATION UPPER BOUNDS: OBSERVATIONAL SCALING LAWS FOR TRANSMITTER-RECEIVERS

  • With one executor withheld, fitted curves predicted its recovery with median error .062, or 9% of held-out value.Median error was .062 versus .075 for a straight line, .051 for a logistic fitted regardless of AICc, and .149 for the mean of remaining executors.Source: §3 · p. 7
  • Rising concepts skewed toward whole-text structure, reaches toward local patterns, and bounded concepts toward identity, social-group awareness, and beyond-text features.Whole-text structure and verifiable features each accounted for 28% and 29% of rising metrics; local patterns accounted for 34% of reaching metrics, while identity/social-group awareness and beyond-text features accounted for 23% and 15% of bounded tasks.Source: §3 · pp. 7–8, Figure 5 · p. 9
  • 884 concepts were still rising, 321 reached the estimated ceiling within observed capability, and 65 remained bounded below the reference threshold.The fits use recovery against capability index z; bounded concepts remain partly inarticulable relative to the instrument.Source: Figure 3 · p. 8
  • Figure 4 shows regime composition by domain, with each bar summing to 100% of that domain’s fitted tasks and domains ordered by bounded share; bounded mass was confined to humor, peer review, news homepages, and marginally creative writing.Source: Figure 4 · p. 8
  • Definitions and explanations were effective at mid scale, while the full packet performed best for 66% of concepts at 70B.At scales of 3B parameters or less, no articulation strategy emerged; explanations performed best for 47% of concepts at 7B.Source: §3 · p. 8
  • Examples barely improved bounded-concept recovery: gains averaged +.006 bits for Qwen2.5-14B and Llama-70B, and were slightly negative for three smaller transmitters.Only four criteria exceeded the reconstruction-noise band for both larger models; example effects varied by model.Source: §3 · p. 9
  • Name-term frequency and concept diffuseness did not explain acquisition, while post-training improved reaches and rising recovery more than bounded recovery.Post-training added +.036 on reaches, +.034 on rising, and +.008 on bounded constructs; bounded criteria showed no acquisition in any tested model family.Source: §3 · p. 10

4 DISCUSSION

The discussion argues that bounded transmission clusters around social and identity-related concepts, and persists despite larger models, more labels, and examples. Human agreement and model-family differences mean these limits should be interpreted as executor-relative.

4 DISCUSSION

  • The 65 bounded concepts primarily concern the identities and social groups of text creators and audiences, and remain below the planted reference even with 90 labels or GPT-5.6.The authors also report that bounded concepts are labeled reliably, while the largest models agree least on them.Source: §4 · p. 10
  • Pretraining frequency and concept form did not predict acquisition, and post-training improved reaches and rising transmission but not bounded constructs.Base and post-trained models were bounded on the same constructs.Source: §4 · p. 11
  • Among 37 bounded concepts with human studies, humans agreed moderately or highly on 76%, so the limits are executor-family relative rather than universal.The paper also reports imperfect overlap between human agreement and LLM transmission for rising and reaches concepts.Source: §4 · p. 11
  • The authors call for research into learning mechanisms for identity- and social-group-related knowledge that may not be articulable by the tested model families or strategies.They note that such concepts may matter for frontier tasks.Source: §4 · p. 11

5 CONCLUSION

The conclusion presents tools for estimating message quality and transmission, then reports that most measured concepts are transmissible now or with scale, while a small socially anchored set remains bounded. The appendices describe the unit-sampling, screening, and subset-selection procedures behind the estimates.

5 CONCLUSION

  • Across 1,270 concepts from eight domains, most concepts can be articulated between language models now or with scale, while a small identity- and social-group-related set remains bounded.Examples and larger models largely do not change performance for that bounded set.Source: §5 · pp. 11–12
  • Among 37 bounded concepts with prior human-measurement studies, 76% showed moderate-to-high human agreement.The other 28 of the 65 bounded concepts had no study the authors could find.Source: §5 · p. 11
  • The authors identify unsupervised reconstruction and regime assignment as limitations, and restrict their study to eight creative and qualitative domains.They report that 163 concepts had less than a .6 chance of landing in their printed regime under resampling, and say factual or procedural concepts may behave differently.Source: §5 · p. 12

APPENDIX CONTENTS

  • The appendices organize documentation of the instruments and capability index, scaling analyses, bounded-regime explanations, protocols, and data.Appendices A–E cover instruments and the capability index; F–I cover scaling; J covers bounded-regime explanations and tests; K–L cover protocols and data.Source: APPENDIX CONTENTS · p. 24

A UNIT SAMPLING AND THE CAUSAL SCREEN IN DETAIL

  • The unit pool draws candidates from web retrieval, direct LLM proposals, and optimizer trajectories, but the recorded pools used proposals and trajectories, not web-retrieved units.Every ceiling is relative to this declared generating process; candidates are split into fragments and screened by a single-pass marginal against the starting prompt.Source: A UNIT SAMPLING AND THE CAUSAL SCREEN… · p. 24

B THE SUBSET-SELECTION ALGORITHM BEHIND MΩ: APPROXIMATION GUARANTEES, THEIR ASSUMPTIONS AND THE MEASURED SUBMODULARITY CHECK

  • The subset-selection appendix defines choosing at most k units to maximize measured recovery and states which approximation guarantees depend on assumptions.It also describes what the procedure does when an assumption fails and assembles the sampling-to-ceiling pipeline.Source: B THE SUBSET-SELECTION ALGORITHM… · p. 24

B. Inside the audit

The audit distinguishes the benchmark selector from analyzed alternatives and tests the conditions behind its per-instance bounds. Selection-log diagnostics show little measured within-pool headroom on some benchmarks, but the bounds remain conditional on assumptions and noisy scores.

B. Inside the audit

  • Pools of at most 15 units are solved by exhaustive enumeration, while larger pools use greedy selectors; benchmark experiments use only the first large-pool selector.The exact branch certifies the evaluated empirical objective on the fixed probe set, not the population optimum; the large-pool selector is a heuristic when its hypotheses are unverified.Source: B. Inside the audit · p. 25
  • Benchmark runs use a prefix sweep of marginal-ranked units, followed by one delete pass, one add-back pass, and separate-panel confirmation; multi-pass scoring comes afterward.The production selector does not run enumeration or double greedy; LiveBench used the earlier 32-unit cap and shipped GEPA’s prompt.Source: Figure 7 · p. 25, B. Inside the audit · p. 26
  • After greedy selection, sorted leftover marginals and the submodularity ratio γ yield an instance-specific upper bound on how much a feasible subset could improve the selected set.At γ = 1, the bound reduces to Minoux’s classical online bound; the multiplicative worst-case counterpart is reported only to characterize the objective in advance.Source: B. Inside the audit · p. 27
  • Across evaluated prefixes, summed single-unit gains exceeded joint gains by factors of 2–61, while measured add-back gaps were small where available.At the best selection-batch prefix, U2 was .710 versus .700 on HotpotQA at γ = 1 and .892 versus .892 on IFBench; one-pass scoring and seed-relative marginals make the additivity test a necessary condition, not proof of submodularity.Source: B. Inside the audit · p. 27
  • Because raw R is non-monotone and submodularity is unestablished, U2 bounds the free-disposal idealization R↑ rather than raw R, and sampled γ can make the plug-in estimate too small.When the sampled ratio clamps to zero, U2 is withheld and the selected prompt is reported as measured, not certified.Source: B. Inside the audit · pp. 27–28
  • Finite-item scoring makes marginal ranks and accepted decisions noisy, so exact-oracle approximation ratios do not transfer and no noise-adjusted ratio is claimed.Paired marginal evaluation, confirmation re-evaluations, and multi-pass test scoring reduce noise’s effect but do not establish a noise-adjusted guarantee.Source: B. Inside the audit · p. 28

C.1 SUPERVISED BENCHMARKS: COMPARISON AGAINST PROMPT OPTIMIZERS, CALL BUDGETS AND RUN-TO-RUN SPREAD

The section compares ϵ-estimated prompts with prompt-optimization baselines under matched task-call budgets and reports paired score contrasts. Mω beats all three baselines, although the comparisons are session-specific and some benchmark gains vary across selection runs.

C.1 SUPERVISED BENCHMARKS: COMPARISON AGAINST PROMPT OPTIMIZERS, CALL BUDGETS AND RUN-TO-RUN SPREAD

  • Mω beats all three baselines in the paired comparison, with Table 2 reporting mean item-level differences and 95% bootstrap intervals.The HotpotQA row is the equal-call session, with GEPA run at 16,700 calls.Source: §C.1 · p. 29, Table 2 · p. 29
  • Table 3 comparisons use five independent test passes and paired item-level bootstrap intervals, while absolute scores are session-specific and only within-session differences are comparable.LiveBench shipped GEPA’s 600-call prompt unchanged, so that row contains no unit-selection effect.Source: Table 3 · p. 29, §C.1 · p. 29
  • GEPA received more calls than Mω on LiveBench and AIME, while GEPA+Merge and MIPROv2 ran at their own default budgets.Mω’s logged spend was 4,616 versus 18,708 calls on LiveBench and 2,490 versus 6,870 on AIME; HotpotQA and IFBench matched the stated GEPA call budgets.Source: §C.1 · p. 29
  • At equal calls, Mω led GEPA by +.077 on HotpotQA; on HoVer, three selection runs yielded gains over GEPA ranging from +.035 to +.068.The HoVer runs scored .572, .539, and .541 against GEPA’s .507 in the reported comparisons; the uniform Table 3 session gave .567 for the row-of-record Mω prompt.Source: §C.1 · p. 30

C.2 ESTIMATING THE MISSING VALUE: MISSING-MASS CURVES, BACKTEST OF THE VALUE CURVE, ASSUMPTIONS AND CALIBRATION OF THE ESTIMATE

Missing-mass and value-curve estimates bound the gains available within the sampled prompt-unit space, but their ceilings depend on how units are individuated and on calibration assumptions. The reconstruction checks show that recovery is operational rather than a matter of wording or message length.

C.2 ESTIMATING THE MISSING VALUE: MISSING-MASS CURVES, BACKTEST OF THE VALUE CURVE, ASSUMPTIONS AND CALIBRATION OF THE ESTIMATE

  • Fine clustering leaves novelty near 1, while coarse clustering yields bounded estimates, showing that apparent saturation depends on unit granularity.At coarse granularity, HoVer has the smallest, fastest-closing space and IFBench the most open; syntactic clustering conservatively inflates novelty.Source: Figure 9 · p. 31
  • The pooled median projection error is .018 at 1–3 units ahead, .020 at 4–8, and .029 at 9+ for the power-law backtest.Across all six benchmark trajectories, the backtest supports median error rather than a worst-case guarantee; error distributions are heavy-tailed for small fit windows.Source: §C.2 · pp. 31–32
  • With constraints and fit windows of at least 8 points, later prefixes exceed the fitted ceiling 2.4% of the time at horizons 1–8, but held-out benchmark misses remain.Held out, IFBench exceeds its ceiling 8.9% of the time at horizons 1–8 and HotpotQA 8.4% at 9+; calibration uses six trajectories, not proof.Source: §C.2 · p. 32
  • The ceiling is conditional on continued draws from the declared sampling process, not a bound over all possible prompts.Exchangeability sensitivity is substantial: HotpotQA’s pooled Chao1 estimate is 337, versus a leave-one-replicate-out range of 159–498 and a block-bootstrap lower limit of 23.Source: §C.2 · pp. 32–33
  • On five of six benchmarks, the constrained ceiling equals the best observed sweep score; only LiveBench projects above its maximum, by .010.The ceiling is not the best score plus the estimated unseen-unit value: unseen units are priced at zero on five benchmarks by construction.Source: §C.2 · p. 33

D.1 CHOICE OF MEASURE, AND CHECKS ON THE RELABELING CHANNEL

  • Behavioral relabeling scores operationalized recovery, while prompt similarity can reward lexical echoes and identification can select plausible rules without applying them.The prompt-similarity judge was wording-sensitive: changing a concept specification flipped 46% of arbiter verdicts.Source: §D.1 · pp. 33–34
  • Reversed reconstructions comprise 8% of rows but only 1.3% of mutual information, and MI correlates with balanced accuracy at Spearman .83 across metrics.The typical reversal pairs a transmitter that rarely says YES with a receiver that says YES to most items.Source: §D.1 · p. 34

D.2 MESSAGE LENGTH: DEFINITIONS, THE RECEIVER’S RULES AND THE SEARCH UNITS, AGAINST REGIME AND RECOVERY

  • Receiver-rule length carries no information about recovery in any regime: correlations range from −.14 to +.18, none with p < .08.Definition lengths and unit lengths likewise show no meaningful association with regime or marginal gain.Source: §D.2 · p. 35

D.3 RECONSTRUCTION-VALIDITY CORRELATIONS, THE ARBITER CHECK, AND WHAT IS VALIDATED AT WHICH LEVEL

The validation evidence varies by comparison level: direct MI-versus-error evidence is limited, and the study distinguishes judge validity from whether reconstruction MI predicts correctness. The reported correlations also rely on filtered, specification-relative labels and a stratified sample.

D.3 RECONSTRUCTION-VALIDITY CORRELATIONS, THE ARBITER CHECK, AND WHAT IS VALIDATED AT WHICH LEVEL

  • Validation is uneven across levels: executor capability is checked, but regime membership has no validation evidence, and criterion-level comparisons rely on planted controls and the arbiter check.The direct MI-versus-error correlation is only within a criterion across prompt forms; across-criteria mention-label correlations are within noise.Source: §D.3 · p. 36
  • The arbiter is a stronger judge with more context, not ground truth; the evaluated subset is stratified and restricted to its confident verdicts.The independent frontier arbiter used full-document context, was blind to silver labels and judge scores, and was certified on sealed anchor items.Source: §D.3 · p. 36
  • The table reports balanced-sample AUCs, not corpus-weighted estimates, and arbiter-filtered labels change with the written concept specification.Re-arbitrating items under a changed specification flipped 46% of verdicts, so prompts sharing the specification text are structurally advantaged.Source: §D.3 · p. 37
  • Within-criterion, across-prompt-form MI-versus-AUC correlations vary by task, and the pooled correlation is +.25 (permutation p = .40), a null, underpowered test.Across metrics, pooled MI association with arbiter-filtered-label AUC is +.25; within metric across prompt forms it ranges from −.08 to +.13, all p > .12.Source: §D.3 · p. 37

E EXTERNAL VALIDATION OF THE CAPABILITY SCALAR z

The appendix checks an executor-capability index against an external leaderboard and documents a model-specific incompatibility. Its observational scaling analysis separates regimes by fitted recovery behavior and finds bounded criteria more associated with truth conditions beyond the text.

E EXTERNAL VALIDATION OF THE CAPABILITY SCALAR z

  • Within leaderboard editions, battery z correlates with OpenCompass average at r ≈.64–.67; excluding Gemma-2-27B raises agreement to r = +.87 and ρ = +.90 (n = 5).Gemma-2-27B is the source of the discrepancy: it performs at chance on composite battery items despite stronger OpenCompass rankings than Gemma-2-9B.Source: E EXTERNAL VALIDATION OF THE… · p. 37
  • The external check covers only six exact-checkpoint matches, uses within-edition comparisons, and covers the local ladder rather than frontier points.OpenCompass scores are not comparable across editions, and its frontier entries are predominantly reasoning-mode variants while these probes disable reasoning.Source: E EXTERNAL VALIDATION OF THE… · p. 38

F OBSERVATIONAL SCALING BY REGIME: DEFINITIONS, FITS, TYPE

  • Regimes are fitted from reliability-corrected agreement with a leave-one-executor-out consensus, then classified by saturation, planted-reference ceiling, or slope.Noisy or undetermined fits are excluded, and the supervised Qwen3 ladder is contextual rather than refit in this analysis.Source: F OBSERVATIONAL SCALING BY REGIME:… · p. 38
  • At the top of the executor range, reaches recover .75, rising .68, and bounded .52; rising ceilings remain unidentified for 99% of fits.Bounded ceilings are attained within range with median CI width .105, while rising ceilings are extrapolations with median CI width .56.Source: F OBSERVATIONAL SCALING BY REGIME:… · p. 38
  • Bounded criteria have 32% beyond-text truth conditions and mean externality index .78, unlike rising and reaches, which are nearly identical on this axis.The bounded pattern is driven by identity and persona, community transformation, and reception outcomes, and persists within humor, news, and peer review.Source: F OBSERVATIONAL SCALING BY REGIME:… · p. 39

F.1 ROBUSTNESS OF THE REGIME FINDINGS: CURVE FORM, EXECUTORS, EXCLUSIONS, DOMAINS, CONTRASTS AND CAPABILITY AXIS

The regime patterns are broadly robust to measurement noise and several modeling checks, but individual classifications—especially bounded verdicts—depend on executor choice and capability-axis construction. Type signatures persist across codebook granularity, while the bounded regime is particularly sensitive to the axis used.

F.1 ROBUSTNESS OF THE REGIME FINDINGS: CURVE FORM, EXECUTORS, EXCLUSIONS, DOMAINS, CONTRASTS AND CAPABILITY AXIS

  • Regime curves use 14 pooled instruction-tuned open models; 77 of 1,347 curves are excluded as noisy, and 40 planted rules set the reference rather than being fitted.Saturating fits use a logistic when it wins AICc; non-saturating curves are rising at Spearman slope ≥ .35 and noisy otherwise.Source: §F.1 · pp. 39–41
  • The fitted curve form has a median held-out prediction error of 9% of the strongest executor’s value and over-predicts on 72% of metrics.Saturation takes precedence when the logistic beats the line by AICc; rising is the residual class when the line is preferred and Spearman slope is at least .35.Source: §F.1 · pp. 39–40, Table 9 · p. 45
  • Dropping the highest-z executor changes 12% of verdicts, with 106 of 321 reaches curves reclassified as rising.Recovery-noise draws preserve printed verdicts as the modal verdict for 94% of metrics, but bounded labels persist in only 65% of draws on average.Source: §F.1 · pp. 40–41, Table 10 · p. 45
  • Bounded criteria range from 15% of humor tasks to zero in math, notice-and-comment, patents, and press releases, while rising criteria comprise 65–86% in every domain.The domain pattern is broadly retained under recovery noise and removal of the strongest executor, but it changes under differently constructed capability axes.Source: §F.1 · p. 41, Table 11 · p. 45
  • Three of five regime-by-type contrasts remain significant under recovery noise, capability-index noise, and nuisance-regression checks, but only reaches/local patterns survives removal of the strongest executor.The rising/verifiable contrast is significant when pooled across domains (q = .013) but not within domain (q = .31), indicating that contrast reflects domain composition.Source: §F.1 · pp. 41–42, Table 12 · p. 46
  • The first principal component explains 80% of executor-by-metric variance, but the regime contrasts and bounded counts depend on the capability axis.On a differently constructed axis, 30% of verdicts change and the bounded group falls to 12; equally spaced ranks instead make 273 curves bounded.Source: §F.1 · p. 42, Table 13 · p. 46
  • The early saturation of local-pattern criteria is the only regularity that holds under every check except the circular recovery-based axis.Without each curve’s most capable executor, a third of reaches curves become rising and only the reaches/local-pattern contrast remains significant; a differently constructed axis also changes bounded counts and signatures.Source: §F.1 · p. 42

G. THE TWO CRITERION CODEBOOKS: DEFINITIONS, EXAMPLES AND HOW

  • The six-category Figure 5 codebook deterministically merges the 12-category labels, while a separate nine-type codebook defines truth conditions along a text-to-beyond-text axis.The six-category merges include verifiable features, identity and social-group awareness, and beyond-text features; the nine-type axis is applied with regimes held out of labeling.Source: G THE TWO CRITERION CODEBOOKS:… · p. 47, §G.1 · p. 47, Figure 14 · p. 47

G.3 BOUNDARY CASES, RELIABILITY, AND HOW THE TWO CODEBOOKS RELATE

The codebook’s least clear boundary is between craft and reception, where independent labeling passes disagree most. The six-category merge and nine-type truth-condition axis are distinct schemes, and their social categories do not align one-to-one.

G.3 BOUNDARY CASES, RELIABILITY, AND HOW THE TWO CODEBOOKS RELATE

  • The craft–reception boundary is least clear for criteria such as catharsis, which can describe either textual setup or an achieved audience response.Among 65 reception-labeled tasks, 32 were labeled reception in both nine-type passes and 10 craft in both.Source: §G.3 · pp. 48–49
  • The bounded externality shift replicates in the second labeling pass despite disagreements on reception labels.The nine-type labels have κ = .75 agreement, while the truth-condition axis has κ = .73.Source: §G.3 · p. 49
  • The six-category social-awareness grouping deliberately spans the axis: community transformation is at the interface, while identity and persona are beyond the text.Externality indices use the nine-type axis, not the six-category merge.Source: §G.3 · p. 49

H.1 EVERY VERSION OF THE EXAMPLES TEST: WHO PICKS THE EXAMPLES, WHICH LABEL THEY CARRY, AND WHICH MODELS RUN THEM

This section examines how choices about examples, labels, and models shape recovery across example tests, and how decompression and optimization contribute different kinds of content.

H.1 EVERY VERSION OF THE EXAMPLES TEST: WHO PICKS THE EXAMPLES, WHICH LABEL THEY CARRY, AND WHICH MODELS RUN THEM

  • Recovery averaged .046 bits with definitions and .048 with agreement-selected examples, while no task’s mean change separated from zero.Three criteria exceeded the arbitrary-example noise bound at both judges, against about 0.6 expected; overall, the small average change did not establish a general gain.Source: §H.1 · pp. 49–50, Table 14 · p. 50
  • At Llama-70B, judge-selected examples raised recovery +.028 bits over definitions for bounded criteria and controls, but gains over null examples were uncertain.The selected-six gains over null six were +.014 [−.016, +.048] for bounded criteria and +.026 [−.007, +.059] for controls.Source: §H.1 · p. 51
  • Examples did not meet the preregistered authorship threshold: the pooled gain over the two largest receivers of each family was +.022, versus a required gain above .05.The placebo stayed near chance for every receiver, with balanced accuracy from .475 to .508.Source: §H.1 · p. 51
  • +.005 bits at Qwen2.5-14B and +.007 at Llama-70B: real labeled items raised recovery, while transmitter-written examples never beat them by more than .001 bits.At Llama-70B, real items also reduced the number of criteria labeled all NO from 121 to 92 and raised label information from .421 to .473 bits.Source: §H.1 · pp. 51–52, Table 15 · p. 52
  • Combining the transmitter’s rule and four examples changed no ordering in the Qwen2.5-14B test, and the receiver’s rule plus four items still underperformed its rule alone.Conditions rose by .002 to .005 bits; over all 449 criteria, the rule recovered .040 bits and the rule-plus-four-items condition .034.Source: §H.1 · p. 54
  • Showing four chosen items recovered less than rule-based telling at both models, and receiver-selected items added nothing over random draws.At Qwen2.5-14B, four items recovered .022 bits versus .036 for the rule; at Llama-70B, they recovered .030 versus .043. Passing all 30 items matched the rule only on bounded criteria at Qwen2.5-14B and remained below it at Llama-70B.Source: §H.1 · p. 53, Table 17 · p. 54
  • For absurdism, the recovery-searched two-YES/two-NO set scored .187 bits on held-out jokes, below all three random sets (.217, .196, and .211).The selected set of any four recovered .288 bits, while random sets scored .211, .329, and .242; the chosen examples were ordinary Reddit jokes rather than recognizably absurdist ones.Source: §H.1 · p. 54

H.2 WHAT EACH PROCESS WRITES INTO THE PROMPT: CONCEPT CONTENT AGAINST MEASUREMENT CONTENT

  • Across 200 decompression-ladder steps, 203 GEPA rounds, and 360 Mω units, blinded tagging collapsed eight content codes to two with κ = .93, versus κ = .68 for eight-way coding.The decompression corpus spans ten annotation domains, rather than the eight used in the scaling analyses.Source: §H.2 · pp. 54–55
  • Decompression adds only concept content, while optimization writes measurement content unless the judge lacks an esoteric concept.Figure 15 reports decompression steps as 100% concept-content; the exception is optimization for esoteric concepts whose judge may not hold the concept.Source: Figure 15 · p. 54, §H.2 · p. 55
  • GEPA rounds are dominated by score anchors and format (75–77%), while Mω additions most often provide executor checking steps (51%).GEPA's frequent secondary additions are edge-case rules and input hygiene; low-fidelity rubrics receive more instrument discipline rather than concept re-teaching.Source: §H.2 · p. 55

I ACQUISITION AND PRETRAINING EXPOSURE: THE CORPUS-FREQUENCY

The paper tests whether a criterion’s corpus exposure predicts when models acquire it, using name-indexing and bank-scale acquisition measures. Neither test supports the exposure hypothesis, and the authors caution that name-term frequency is a coarse proxy for concept exposure.

Test scope

  • The test asks whether concept-vocabulary frequency predicts name-indexing order or acquisition onset, but the paper’s name-only crossing sample contains 37 metrics.The 37 crossings are described as a selected class, not a special one; the appendix uses them for the name-indexing test while bank-scale tests use all 1,270 fitted metrics.Source: I ACQUISITION AND PRETRAINING… · pp. 55–56

Exposure measure and limitation

  • Exposure measures mean log10 corpus frequency of criterion-name terms in Dolma, a proxy because the evaluated models’ pretraining corpora cannot be queried.The authors call name-term frequency coarse because it conflates common English words with concept exposure; a co-occurrence estimator is left for future work.Source: I ACQUISITION AND PRETRAINING… · p. 56

Grain 1: name-indexing order

  • Across 37 concepts with fitted name-only crossings, no tested exposure correlation was significantly negative, so the preregistered support criterion was not met.These 37 metrics are the cleanest name-indexing cases; the authors report that measured properties of this selected class do not differ from the full bank of 1,270 fitted metrics.Source: I ACQUISITION AND PRETRAINING… · p. 56

Grain 2: acquisition onset

  • For 1,270 fitted metrics, exposure and acquisition onset were nearly uncorrelated (ρ = +.033, CI [−.022, +.088]), and exposure separated none of the three scaling regimes.The only domain intervals excluding zero were creative writing (+.12) and humor (+.13), both opposite the exposure hypothesis, and press releases (−.18).Source: I ACQUISITION AND PRETRAINING… · p. 56

Residuals and concept form

  • Compact-checkable and diffuse concept types had indistinguishable exposure-adjusted onset residuals (+.006 vs −.013; difference +.02, p = .75), with no compact-type enrichment among earliest acquisitions.Exploratory type-level departures put identity/persona and external-bundle criteria early, while mechanical checks and audience fit fall late relative to exposure.Source: I ACQUISITION AND PRETRAINING… · p. 56

J SIMPLER EXPLANATIONS FOR THE BOUNDED REGIME THAN TACITNESS,

The appendix evaluates simpler explanations for the bounded regime, finding that the instrument’s capability axis and planted reference affect which criteria are classified as bounded. Several proposed explanations are not supported by the reported tests, while corpus disambiguation and unstated standpoint remain plausible but incompletely tested.

The instrument makes it

  • The instrument explanation is not ruled out: the bounded count ranges from 12 to 273 across capability axes, and criterion-level membership is sensitive to executor spacing and the planted reference.Regime sizes and domain composition survive measurement noise and executor perturbations, but the identity of particular bounded criteria does not.Source: J SIMPLER EXPLANATIONS FOR THE BOUNDED… · p. 57

The transmitter’s labels are one-sided

  • One-sided labels are not supported as the explanation: bounded criteria are no more one-sided than matched controls and recover about as well at a single receiver.Panel probes find bounded criteria rarer-positive, which changes which criteria are classified as bounded but not their number.Source: J SIMPLER EXPLANATIONS FOR THE BOUNDED… · p. 57

Thirty labeled items are too few

  • Thirty labels are not too few to explain bounded recovery up to 90 items, although a sample-complexity explanation at hundreds of items remains open.Recovery does not rise for bounded criteria from 30 to 90 items (−.014 [−.023, −.006] bits), while controls fall by the same amount.Source: J SIMPLER EXPLANATIONS FOR THE BOUNDED… · p. 57

The corpus never disambiguates the concept

  • Corpus ambiguity is supported for much of the bounded humor set: only 17 of 41 criteria have defined values on Reddit jokes, and contrastive items have not been directly tested.On published humor pieces, bounded criteria gain +.073 bits over controls; the authors propose contrastive items that separate each criterion from its nearest neighbor as a direct test.Source: J SIMPLER EXPLANATIONS FOR THE BOUNDED… · p. 57

The label depends on a standpoint the message never states

  • Identity and social-group criteria are overrepresented among bounded cases and may depend on an unstated label standpoint, but a message explicitly stating that standpoint has not been tested.The authors note that high human-agreement values come from trained coders using a closed codebook.Source: J SIMPLER EXPLANATIONS FOR THE BOUNDED… · p. 57

Labels are unreliable, or consensus is the ceiling

  • Unreliable labels or a consensus ceiling are not supported, and post-training adds recovery only where recovery already moves.Form reliability is the same across regimes, while the strongest executors agree least with one another on bounded criteria.Source: J SIMPLER EXPLANATIONS FOR THE BOUNDED… · p. 57

The panel stops too early

  • A beyond-panel executor leaves 0 of 65 bounded criteria above the planted reference, but its agreement with the panel does not establish correctness.GPT-5.6 is at z = 4.8 versus the panel’s best at 2.6; past the panel, the measured agreement reflects the panel’s reading, not correctness.Source: J SIMPLER EXPLANATIONS FOR THE BOUNDED… · p. 57

Tacitness

  • Tacitness is not directly tested: the longest message contains 90 labeled items, and the strongest executor is still evaluated through panel consensus.The authors identify contrastive items as an unrun experiment relevant to the alternative explanation.Source: J SIMPLER EXPLANATIONS FOR THE BOUNDED… · pp. 57–58

J.1 LABEL RELIABILITY, THE CONSENSUS, SKEW AND PROBE HALVES: FOUR CHECKS ON THE SHIPPED PANEL LABELS

The checks examine whether bounded-regime findings reflect label reliability, consensus, skew, probe selection, limited inputs, or model capability. They find several important qualifications: criterion-level verdicts are not stable across probe halves, and frontier-model and artifact tests complicate what a plateau means.

J.1 LABEL RELIABILITY, THE CONSENSUS, SKEW AND PROBE HALVES: FOUR CHECKS ON THE SHIPPED PANEL LABELS

  • The bounded plateau is not explained by unreliable labels or a consensus ceiling: bounded-label split-half reliability is .95, while top-three pairwise agreement is lowest on bounded criteria at .22.The top-three agreement is .35 for reaches and .36 for rising; the checks do not support either explanation.Source: §J.1 · p. 58, Table 18 · p. 58
  • Bounded verdicts are not stable for individual criteria across probe halves: only 15 of 65 shipped bounded criteria are bounded in both, while group size and domain composition are more robust.The even and odd halves classify 92 and 41 criteria as bounded and agree on 66% of verdicts overall.Source: §J.1 · p. 58, Table 18 · p. 58

J.2 ONE-SIDED LABELS DO NOT SEPARATE BOUNDED CRITERIA FROM THEIR CONTROLS

  • Bounded criteria are no more one-sided than matched controls, and their rule recovery is comparable when both label classes are present.Their plateau is not visible as low recovery at one receiver; it concerns how panel agreement changes with capability.Source: §J.2 · p. 59, Table 19 · p. 59

J.3 NINETY LABELED ITEMS IN PLACE OF THIRTY: RECOVERY DOES NOT RISE

  • 30, 60, or 90 labeled items do not lift bounded-criteria recovery: it plateaus by 30 items, with no criterion gaining more than .104 bits from extra items.The bounded change from 30 to 90 items is −.014 [−.023, −.006]; the interaction with controls is −.004 [−.015, +.008].Source: §J.3 · p. 59, Table 20 · p. 59

J.4 PAST THE PANEL: A FRONTIER EXECUTOR ON THE SAME INSTRUMENT

  • GPT-5.6 passes the function check well beyond the panel but does not lift bounded criteria: 0 of 65 exceed the planted reference, and median y is .31 for bounded criteria and controls.Its planted-rule AUC is .93 versus .86 for the panel’s strongest executor, but agreement y is relative to the panel consensus.Source: §J.4 · p. 60, Table 21 · p. 60

J.5 THE INPUT TEST: BOUNDED HUMOR CRITERIA SCORED ON ARTIFACTS THAT CONTAIN WHAT THEY NAME

  • Richer humor artifacts raise recovery for many bounded criteria, suggesting they were starved of their subject matter in the short-joke corpus; criteria needing an audience or field remain flat.At Qwen2.5-14B, the bounded gain on McSweeney’s pieces is +.066 [+.040, +.093], while the contest prompt adds nothing to either group.Source: §J.5 · pp. 60–61

J.6 BASE MODELS AGAINST THEIR POST-TRAINED SIBLINGS ON THE SAME INSTRUMENT

  • Post-training raises recovery on reaches and rising criteria but barely changes bounded criteria: pooled differences are +.036, +.034, and +.008 [−.005, +.021], respectively.The bounded gap is smaller than the other two by .026 [.013, .040]; the base-model readouts are descriptive because only some pairs pass the function and ladder gates.Source: §J.6 · p. 62, Table 22 · p. 62, Table 23 · p. 63

J.7 PUBLISHED HUMAN AGREEMENT ON THE BOUNDED, REACHES AND RISING CRITERIA

The section compares published human agreement with the criteria classified as bounded, reaching, or rising, while noting limits in how those studies match the criteria. Human agreement varies across these groups, and the accompanying protocol registry records preregistered studies and outcomes.

J.7 PUBLISHED HUMAN AGREEMENT ON THE BOUNDED, REACHES AND RISING CRITERIA

  • The bounded-criteria agreement estimates have limits: high values mostly come from trained coders using closed codebooks, and some reported coefficients pool ratings across many raters.The agenda-setting measure is an aggregate correlation, not inter-rater agreement.Source: §J.7 · p. 64
  • Among reaches and rising criteria with measurements, 4% show high agreement, 43% moderate, and 53% low; one study accounts for 131 of 357 low verdicts, and many construct matches are loose.Studies were found for 185 reaches criteria and 494 rising criteria.Source: §J.7 · p. 64, Table 26 · p. 67, Table 27 · p. 68
  • For bounded criteria, the closest published studies report 8 high, 20 moderate, 9 low, and 28 with no study found; the evidence search examined about 80 papers and retained 53.Agreement uses the single-rater coefficient when reported, with high at .80 or above and moderate from .40 to .79.Source: §J.7 · p. 64, Table 24 · p. 65, Table 25 · p. 66
  • No human agreement study was found for 36 bounded-criterion families and 60 rising-criterion families, spanning domains including humor, creative writing, mathematics, and news.The nearest studies for several additional families report no agreement statistic.Source: §J.7 · pp. 65, 66, 67, 69

K PREREGISTERED PROTOCOLS AND SCORING-SESSION FINGERPRINTS

  • The protocol registry links preregistered experiments to dated records and records scoring-session fingerprints for Table 3 cells.The appendix lists protocols and outcomes, including tests of examples, input richness, post-training, more labels, and frontier executors.Source: K PREREGISTERED PROTOCOLS AND… · p. 70

L REPRODUCIBILITY AND DATA STATEMENT

The paper identifies its code, benchmarks, and criterion bank as resources for reproducing and interpreting its analyses.

L REPRODUCIBILITY AND DATA STATEMENT

  • The supervised harness, reconstruction channel, and analysis scripts are in the project repository; executor runs used vLLM 0.25.1, and reconstruction experiments identify their models.The paper uses the public GEPA evaluation suite and the Spangher et al. (2026) criterion bank unchanged, adds no human annotation, and recruited no human subjects.Source: L REPRODUCIBILITY AND DATA STATEMENT · p. 70

Related papers

Loading 2610.19721v1…