PaperlayerView exact revision on arXiv

Source-linked AI summary

Verifiable, Articulable, and Tacit Components of Preference

Alexander Spangher, Sheldon Huang, Andreas Haupt, Noah D. Goodman, Diyi Yang, Daniel E. Ho, Sanmi Koyejo

arXiv:2610.03025v1cs.AIcs.CLcs.CYcs.LG

TL;DR

Preference judgments can rely on constructs that people share but struggle to articulate, while AI reward models commonly use explicit criteria. The paper measures these gaps across a large preference dataset and finds them across domains, with consequences for model alignment to human choices and generations.

  • Problem

    Tacit components of preference are understudied, although AI reward models commonly rely on verifiers, constitutions, and rubrics.

    Source: §1 · p. 2

  • Method

    The paper introduces CreativePreferences and compares program-based verifiers, rubric-based criteria, and dense models trained on text and preferences.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Articulability and verifiability gaps occur across domains, and the full VAT model agrees more with human judgments while emphasizing articulated metrics shifts generations away from inarticulated preferences.

    Source: §1 · p. 2, §8 · p. 15

  • Takeaways & Limitations

    The authors recommend studying where tacit knowledge emerges, considering human-in-the-loop approaches for less articulable tasks, and bounding expectations in highly intersubjective domains.

    Source: §1 · p. 2, §6.1 · p. 12

    Articulated scores depend on LLM judges, while observed labels can be noisy proxies for preference.

    Source: §9 · p. 16

Abstract

from arXiv · show

What makes a short story gripping; a news article newsworthy; or a math proof elegant? These constructs resist articulation or verification; their meaning is at least partially tacit. However, modern AI models are improved primarily via articulated constitutions, rubrics and verifiers (i.e. in RLAIF and RLVR); tacit components of preferences are typically understudied. We introduce a large, labeled preference dataset CreativePreferences, containing 2.8M texts labeled by 317M human preference judgments across 7 creative domains, with 42 benchmark tasks. We model these labels with executable programs, rubric banks and densely trained models (V, A and VAT, respectively). We observe robust articulability gaps, VAT-VA; and verifiability gaps, VAT-V; we estimate upper and lower bounds for each gap with a novel measurement approach that discovers articulable and verifiable metrics, identifies spurious variables and estimates the value of undiscovered metrics using capture-recapture. These gaps occur across all domains, even in domains traditionally treated as fully verifiable: correctness-centered domains (i.e. mathematics and software engineering) and claim- and novelty-centric domains (i.e. news, patents, peer review). The size of the gap varies based on domain (e.g. peer review and creative writing have the largest articulability gaps) and widens as more people take part in the judgment, consistent with Collins' collective tacit knowledge. We show two consequences: (1) on human generations, the full model more closely matches human preferences, often in disagreement with articulated criteria, and (2) in an analogy to Goodhart's law, articulating preference shifts it away from the tacit dimension. Articulability and verifiability gaps are consequential; we give recommendations on when tasks can be prompted; how learning mechanisms might improve; and when to leave judgments with humans.

1 INTRODUCTION

The paper studies preference components that people may share but struggle to articulate, a challenge for AI systems trained with explicit rubrics and verifiers. Across seven domains, it finds gaps between what models can learn from preferences and what criteria can articulate or verify.

1 INTRODUCTION

  • The paper addresses limited prior evaluation of tacit knowledge in preference modeling, despite reward modeling being well studied and used to develop AI agents.Prior work has covered a limited set of domains and has not focused on measuring tacitness.Source: §1 · p. 2
  • CreativePreferences contains 2.8 million texts and 317 million preference judgments across seven domains and 42 benchmark tasks.Tasks predict community votes, gatekeeper verdicts, and expert curations.Source: §1 · p. 2
  • The study models verifiable criteria with programs, articulable criteria with rubrics, and preference signal beyond those criteria with a dense VAT model.It defines articulability gaps as VAT − VA and verifiability gaps as VAT − V, and tests their robustness against alternative explanations.Source: §1 · p. 2
  • Articulability and verifiability gaps appear across every studied field, including math and software engineering, where articulability gaps reach 26% and 28%, respectively.The paper reports at least 85% articulability in code, math, journalism, and law, compared with 61–85% in humor, creative writing, and peer review.Source: §1 · p. 2
  • Gaps are associated with novelty, clear writing, and surprise, and grow with the number of judges (ρ = .46), consistent with collective tacit knowledge.Community voting has the least articulable shared preference, while curated preferences have the most.Source: §1 · p. 2
  • The full VAT model agrees more with human judgments, while emphasizing articulated metrics shifts generations away from inarticulated preferences; the reverse shift does not occur.The paper also recommends human-in-the-loop approaches for less articulable tasks and bounding expectations that AI may surpass humans in highly tacit domains such as creative writing.Source: §1 · p. 2

2 THE CREATIVEPREFERENCES DATASET

CreativePreferences brings preference signals from seven creative domains together to support cross-field study. It uses naturally occurring judgments, including votes, decisions, and curated selections.

2 THE CREATIVEPREFERENCES DATASET

  • The dataset treats a domain as creative when its field or experts decide the value of its products rather than predetermined criteria.The seven domains include traditionally studied reward-modeling fields such as math and coding, as well as creative writing and humor.Source: §2 · p. 3
  • CreativePreferences contains 317 million preference judgments across 2.8 million texts, 42 tasks, and seven domains, enabling large cross-field comparisons.The paper contrasts this scope with existing datasets that tend to cover one creative field, correctness-based tasks, chat or safety, or elicited labels.Source: §2 · p. 3
  • Its labels are found judgments rather than elicited pairwise comparisons, transmitted through expert curation, gatekeeper verdicts, or community votes.These signals respectively select winners from a pool, accept or reject submitted items, or aggregate members’ votes.Source: §2 · p. 3

3 METHODS

The methods compare learners given raw training data, text-based criteria, or formal checks to measure preference signal not captured by articulation or verification. They discover and test metrics, estimate the contribution of undiscovered metrics, and control for spurious features.

3 METHODS

  • The study measures average-reward differences among learners given the training data, a text compression, or a formula compression of the same data.The method expects these representations to be nested in signal capacity: formulas cannot carry more than text, and text cannot carry more than the data.Source: §3 · p. 4

3.1 MEASURING ARTICULABILITY AND VERIFIABILITY GAPS

  • V uses executable-program metrics, VA combines programs and prompted rubrics, and VAT trains a dense model on text, preferences, and metrics.The comparison represents formally checkable constructs, constructs articulable to an LLM, and constructs learned directly from data.Source: §3.1 · p. 4
  • Four metric-discovery methods propose criteria from preference outcomes, and the study implements discovered metrics as programs and rubrics optimized with a joint fidelity objective.The discovery methods include AutoMetrics, ExperiGen, HypoGeniC, and HypotheSAEs.Source: §3.1 · p. 4
  • Capture-recapture estimates the value of undiscovered metrics using discovery gains and the estimated missing mass of new proposals.The estimate assumes metrics are generally discovered in salience order and includes a sensitivity budget for rare, high-value criteria.Source: §3.1 · p. 5
  • Channel tests, path tests, and LLM adjudication identify spurious features, which are handled by stratifying data, stripping surface features, or residualizing scores.The same stratified and stripped datasets are used to train V, VA, and VAT; residualization is applied to each model’s score.Source: §3.1 · p. 6
  • The study checks robustness with bootstrap samples, data-group influence and modeling sensitivities, while reserving the test split for final scoring.Metric discovery uses training and evaluation splits; the upper- and lower-bound estimates use proposal counts rather than test labels.Source: §3.1 · p. 6

4 RESULTS

The results show articulability gaps across the studied tasks, including math and coding, and report that these gaps persist under tests of additional metrics, judge quality, model architecture, and alternative explanations.

4.1 ARTICULABILITY GAPS OCCUR ACROSS NEARLY ALL TASKS AND DATASETS

  • Math correctness checks have held-out AUC ≤.55 for answer acceptance, versus .65–.66 for articulated criteria.Table 2 compares programs’ checks with criteria articulated to a judge; its feature highlights encode univariate held-out AUC against the stated label.Source: Table 2 · p. 7
  • Math preference has articulability gaps up to 26% and coding up to 28%; verifiable metrics explain 13%–80% of shared preference across domains.The reported largest gaps include 34–39% of shared preference in creative writing and academic peer review.Source: §4.1 · pp. 6–7
  • Humor has relatively small articulability gaps but large verifiability gaps: only 34% of its articulable signal is verifiable, while articulated criteria carry about 85% of recoverable preference on r/Jokes upvotes.The strongest reported criteria include virality/shareability (.613) and misdirection/reveal (.607).Source: §4.1 · p. 7

4.2 ARTICULABILITY GAPS ARE ROBUST

  • On r/WritingPrompts upvotes, VA AUC rises .607→.665 as k increases 8→144, approaching an asymptote of .668.The study reports that adding more genuine and spurious metrics has minimal effects overall and does not close the gaps.Source: §4.2 · p. 7
  • Optimizing prompts adds at most .05, and estimating an optimal judge moves the articulated model by at most .031 across tasks; neither closes the gap to VAT.The reported asymptote for V and VA remains below VAT in all cases.Source: §4.2 · p. 8
  • Alternative metric-aggregation architectures do not change held-out AUC, and the component of VAT orthogonal to any metric explains 85% of the articulability gap.The VAT score is only partly reconstructible from articulated criteria (ρ = .56); tested corpus- and population-side alternatives also do not explain the gaps.Source: §4.2 · p. 8

5 ANALYSIS

The analysis finds that preference gaps vary with textual and social context, and that the full VAT model more often selects human-preferred items than VA. On generated text, the models favor different qualities, while their cross-rankings remain relative rather than ground truth.

5.1 WHY DO ARTICULABILITY GAPS OCCUR?

  • High-gap texts often feature clear writing or a twist, while corpus-relative indicators include novelty and the overuse of idioms.Annotators found within-text indicators in 62% of examples and corpus-relative indicators in 22%; jargon, reading grade, and judges’ education did not predict gaps.Source: §5.1 · p. 9
  • Expert-curated preferences are most articulable and community voting least; Figure 3 plots recoverable preference by channel, with one point per task.Peer-review articulability is 64% for paper prizes, 61% for paper acceptance, and 58% for citation percentile.Source: Figure 3 · p. 9, §5.1 · p. 9
  • Relative gaps do not correlate with predictability (ρ=.09), and remain varied after controls for 43–57 spurious variables per label.Creative-writing best-of lists show no articulability gap, while only 57% of shared preference in crowd-rated stories is articulable.Source: §5.1 · p. 9

5.2 CONSEQUENCES OF ARTICULABILITY GAPS

  • When VAT and VA rank human-text candidates differently, VAT’s choice is more often the human-preferred item, and regret is positive in 13 of 16 tasks.Regret also tracks the induced articulability gap (ρ = +.65), but a small gap does not guarantee low regret because decision margins differ.Source: Table 3 · p. 10, §5.2 · p. 10

5.1 WHY DO ARTICULABILITY GAPS OCCUR?

  • More evaluators per decision correlate with larger gaps (ρ=.46; n=28, p=.016), consistent with collective tacit knowledge.The deconfounded gap is flat in training-set size.Source: §5.1 · p. 10

5.2 CONSEQUENCES OF ARTICULABILITY GAPS

  • Gap size alone does not calibrate selection regret: Math.SE accepted answers have a smaller gap than votes but greater regret (+.068 versus +.014).Regret depends on within-group decision margins, so the same AUC gap can carry different costs across tasks.Source: §5.2 · p. 10
  • On generated text, VAT’s top pick averages at VA’s .7 percentile, while VA’s top pick averages at VAT’s .6 percentile.These cross-model rankings are relative and may both be wrong; the authors leave a reinforcement-learning run for future work.Source: §5.2 · p. 11

6 DISCUSSION

The discussion treats articulation and verification gaps as consequential but not proven measurements of human tacit knowledge. It recommends measuring gaps, exploring learning approaches, and deferring to humans where tacit knowledge cannot be recovered.

6 DISCUSSION

  • Across seven domains and 42 tasks, gaps persist despite stronger criteria and judges; VAT better reflects human choices, while optimizing articulated criteria shifts generations away from inarticulated preference.The authors caution that these gaps are not established as measurements of tacit knowledge.Source: §6 · p. 11
  • The authors do not establish that measured gaps equal human tacit knowledge, and whether current learning mechanisms can acquire all forms remains unclear.They distinguish somatic tacit knowledge rooted in bodily interaction from collective knowledge acquired through social systems.Source: §6 · p. 11
  • Across 7,236 generated stories, VAT emphasizes rhythm, subtext, ambiguity, and voice, whereas VA emphasizes escalation, payoff, and dialogue.The example models select different stories despite each receiving high scores on different criteria.Source: §6 · p. 11

6.1 RECOMMENDATIONS AND FUTURE WORK

  • Measure gaps where possible: small-gap tasks may suit prompts and rubrics, while larger gaps may signal underspecification risk.The authors call for further measurement approaches, including in-situ user designs, to examine different forms of tacit knowledge.Source: §6.1 · p. 12
  • Study how people acquire tacit knowledge to inform learning mechanisms, including emulation learning, proactive social feedback, and apprenticeship.The authors note that preference-pair elicitation is less scalable and tends to return articulated preferences under observation.Source: §6.1 · pp. 12–13
  • When learning methods cannot recover needed tacit knowledge, consider human-in-the-loop approaches or bound expectations for machine performance.The authors describe some social tasks as polymorphic: their changing context resists fixed rules and requires continued engagement with the group.Source: §6.1 · p. 13

7 RELATED WORK

The related-work discussion connects tacit knowledge to philosophy, rule-following, and reward modeling, while positioning this study as a broad computational analysis of preferences across creative fields.

Tacit knowledge, tacit rules and learning methods

  • Prior work traces tacit knowledge to professional judgment, community interpretation of rules, and somatic or collective origins.The discussion also describes learning through participation, imitation, shared intention, rehearsal, and feedback.Source: §7 · p. 13

Reward modeling: Rubrics, judges and reward signals

  • Reward models, rubrics, and verifiers are scrutinized because fixed criteria can underrepresent the construct they represent and may cease to be valid when optimized against.CreativePreferences uses found preferences from natural settings rather than elicited comparisons.Source: §7 · p. 14

Creative domains, expertise and agreement without definition

  • The study distinguishes itself by systematically combining many fields and comparing them using the same methods.The literature review describes evidence that evaluators can agree without articulating shared criteria, including in mathematics, software, and scientific review.Source: §7 · pp. 14–15

Discovering natural language criteria and program synthesis

  • The measurement approach combines criteria discovery with capture-recapture estimates of undiscovered criteria and an audit of spurious features.The authors position these contributions as a procedure built around existing discovery methods.Source: §7 · p. 14

8 CONCLUSION

The conclusion presents preference gaps as measurable across complex fields, including fields often treated as verifiable. It links those gaps to human-model agreement and recommends measurement, learning research, and human deferral where needed.

8 CONCLUSION

  • CreativePreferences covers 2.8M texts, 317M human judgments, 42 tasks, and seven creative domains, with gaps found in every domain.The conclusion says the gaps persist with more criteria, better prompts, stronger judges, and better aggregation.Source: §8 · p. 15
  • The full model agrees more with people’s actual choices, while optimizing generations toward articulated criteria shifts them away from tacit preference.The conclusion states that the gaps have not been shown to equate directly to human tacit knowledge.Source: §8 · p. 15
  • The paper recommends measuring gaps, studying human tacit-knowledge acquisition to inform learning, and delegating to humans when learning cannot recover it.It also recommends bounding expectations for machine performance in such tasks.Source: §8 · p. 15

9 LIMITATIONS

The authors treat the residual as an estimator-defined gap, not demonstrated tacit knowledge, and qualify its interpretation through measurement, labeling, and evaluation constraints.

9 LIMITATIONS

  • Single-seed AUCs are point estimates; sampling error is several AUC points on 528-item caption-finalist and 223-item peer-revealed splits, lowering significance there.The paper reports significant gaps on all other tasks.Source: §9 · p. 15
  • The residual survives tests of alternative explanations and grows with data and judge count, but the authors do not establish that it is tacit knowledge.They also do not distinguish shared unstated preferences from variation between individual raters; rater-level data are available only for peer review.Source: §9 · pp. 15–16
  • Articulation and verification upper bounds cover mined and automatically constructed metrics; a hand-written expert rubric might capture criteria those methods missed.A hand-written rubric would need to exceed the estimated upper bound to change the interpretation.Source: §9 · p. 16
  • Every articulated score depends on an LLM judge, while noisy proxy labels and model-relative residuals limit what the paper calls taste.The authors test multiple judges but note that different judges may measure differently.Source: §9 · p. 16
  • .612 versus .524 AUC across agency-stratified and frozen GroupKFold splits exposes agreement-label sensitivity; the full model scores .566 evaluation versus .639 test.The outcome label is stable across the two split designs, unlike the agency-agreement label.Source: §9 · p. 16

A.1 FULL DATASET INVENTORY

A THE CREATIVEPREFERENCES DATASET: INVENTORY AND AUDIT

  • Dataset construction uses institutional labels, matched rendering, anchored judging, stable content-hash splits, and declared confounds for deconfounding analyses.The appendix also documents filters, thresholds, matching, deduplication, and leak fixes; the district-court outcome collection is excluded because litigation rulings are not preferences about creative products.Source: §A.1 · p. 32

Appendix

  • The appendix describes moderation tasks as matched removed-versus-kept posts and production verdicts as institutional decisions partly based on information outside the text.The moderation family includes r/Jokes and nine subreddit macro-norm tasks; production-verdict tasks are small and costly.Source: §A.1 · p. 32

A.2 AUDIT TABLE: WHAT EACH TASK’S LABEL IS, AND WHAT SUPPORTS EACH COMPARISON

The audit table links each benchmark task to its decision context and label provenance, while the accompanying construction notes state shared corpus rules and define software-code preference channels.

A.2 AUDIT TABLE: WHAT EACH TASK’S LABEL IS, AND WHAT SUPPORTS EACH COMPARISON

  • Table 7 audits each Table 23 task by decision-maker, candidate set, decision-time information, label source, and scored-population size.The gap interval is based on paired grouped bootstrap draws, with the number of draws depending on which gap is reported.Source: §A.2 · p. 35, Table 7 · p. 35

B DATASET CONSTRUCTION AND BALANCING

  • Corpus construction uses enacted institutional or community labels, balances or groups producer-identifying variables, keeps splits stable by group-key hashes, and normalizes presentation across label classes.The appendix records filters, thresholds, matching variables, deduplication, and leak fixes for each corpus.Source: B DATASET CONSTRUCTION AND BALANCING · p. 37

B.1 SOFTWARE CODE

  • In software code, verdicts are maintainer merges or answer acceptance, curation is a deliberate pick among items, and community preference is anonymous answer voting.The covered texts include pull requests, Stack Overflow answers, and programming-contest submissions.Source: §B.1 · p. 37

B.1.1 GITHUB PULL REQUESTS (EXPERT VERDICT)

These appendix sections audit how preference labels, outcomes, and verdicts are constructed and evaluated across code, mathematics, law, academic review, journalism, and creative and curation tasks, with attention to balancing, leakage, confounds, pipeline artifacts, and executable checks. They assess what text-based signals, articulated criteria, and models can establish, documenting limits and how controls and task definitions affect measured predictability.

B.1.1 GITHUB PULL REQUESTS (EXPERT VERDICT)

  • After label-pure and flaky-batch exclusions, the GitHub ladder scored V .576, A .615, and V +A .632, versus repository base rate .706; unseen-repository V and A scored .549 and .596.The definitive ladder used 44,751 pull requests from 594 repositories; an earlier ladder was retracted after in-sample fitting, duplicate/conflicting joins, repository pooling, and train-test text leakage. Audits also found title flags, repository-level imbalance, and mislabeled Bors merges. The dense model is withheld because its non-standard backbone and 512-token limit make it incomparable.Source: §B.1.1 · pp. 37–39
  • The repo-stratified regression estimate was OR = 1.214, 95% CI [1.094, 1.347], p = 7 × 10−5, across 49,656 observations and 466 strata; it is not a strict pass-to-fail effect.Because pass-to-fail occurs in only 1.8% of sound mixed pull requests, the reporting rule uses gates, precision, and lift rather than pooled AUC for rare verifiable signals.Source: §B.1.1 · p. 40
  • For GitHub test-execution evidence, the repo-stratified estimate concerns a broader regression verdict, while no row is fully replay-certified under the project’s six-part contract.The passages distinguish structurally valid, repeat-supported, and replay-certified rows and caution against describing them simply as verified.Source: §B.1.1 · p. 40

B.1.2 STACK OVERFLOW ANSWERS (VERDICT, CURATION, COMMUNITY)

  • Stack Overflow supplies asker acceptance, within-question bounty awards, and community votes as three distinct judgments on programming answers, with question-grouped splits.The tasks contain 16,001 accepted-answer examples, 23,498 bounty examples, and 12,202 vote examples; the answer-age confound also applies and is handled as for math.StackExchange.Source: §B.1.2 · p. 40

B.1.3 COMPETITION PROGRAMMING EDITORIALS (EXPERT CURATION)

  • The final editorial-task scores were AtCoder .731, CodeChef .729, CodeForces .686, and LeetCode .568; the bank was at least as good as the dense model on every platform.LeetCode’s earlier .686 was a near-copy artifact, and CodeForces rose from .627 after negative enrichment. The passages also report a LeetCode source confound: source signature contributed beyond length and editorial style.Source: §B.1.3 · pp. 41–42

B.2 MATHEMATICS

  • The mathematics section covers maintainer merge decisions and asker acceptance as verdicts, bounty awards or editorial-approach judgments as curation, and answer votes as community preference.Its three corpora are mathlib pull requests, math.StackExchange answers with a bounty companion, and AoPS forum solutions.Source: §B.2 · p. 42

B.2.1 MATHLIB4 PULL REQUESTS (EXPERT VERDICT)

  • The topic-residualized mathlib ladder scored V .635, V′ .680, C .736, and A ≈.48; the earlier zero-review-thread friction label was retired as circular.Tactic profiles were promoted into V′, while topic was residualized because it did not add to V; author and copyright text were stripped from the diff.Source: §B.2.1 · pp. 42–43

B.2.2 MATH.STACKEXCHANGE ANSWERS (COMMUNITY; CURATION VIA BOUNTIES)

  • On math.StackExchange, earlier answers win 67.3% of within-question pairs, rising to 76.4% when the score difference is at least five; length alone predicts wins only 52.9%.The canonical v3.3 build balances by cross-fitted question-propensity decile and three-year bin, retaining 99,722 rows at 50/50 with a floor of .461.Source: §B.2.2 · p. 44
  • After balancing, the answer-word AUC is .6212 against a question-only floor of .4690, yielding an answer-word margin of +.152.The decile-by-class balance is exactly .5000 in all ten deciles, but post-publication edit markers remain a textual leak; the floor guarantee is specific to the stated TF-IDF/logistic-regression model family.Source: §B.2.2 · p. 44

B.2.3 AOPS FORUM SOLUTIONS (EXPERT CURATION)

  • TF-IDF predictability on verified-correct AoPS solutions falls from .726 in the full pool to .594, while V falls from .706 to .560.The paper recommends reporting the full-pool .73 with this two-regime nuance: full-pool predictability is substantially a solution-completeness gradient, while deviation from the editorial approach among correct solvers is rare (12%) and weakly predictable. A thanks-based residual removes measured confounds, with out-of-fold R2 = .522.Source: §B.2.3 · p. 45

B.3 LAW AND POLICY

  • The law-and-policy datasets distinguish institutional verdicts, agency actions on comments, and public co-signing, while legal outcomes are documented separately from preference benchmarks.The corpora cover patent prosecution, notice-and-comment rulemaking, and a twelve-domain legal-outcomes collection.Source: §B.3 · p. 46

B.3.1 PATENT PROSECUTION (EXPERT VERDICT)

  • 0.024 bits (2.5% of H(Y)) is recovered for patent outcomes; semantic-delta AUC is .605, while three attempts to improve outcome prediction were null.The earlier “1% verifiable vs 70% dense” comparison is retracted as confounded and cross-cohort; examiner leave-one-out leniency reaches AUC .681 but is only a leakage diagnostic. Alternative outcome variables changed marginals by −.0009, +.0007, and +.0065.Source: §B.3.1 · p. 47
  • Claim-level prior-art retrieval reaches only ∼20.6% examiner-cite recall, the pipeline’s binding limit, and the disclosure metric adds +.0002 marginal contribution to outcome prediction.The pipeline decomposes claims, retrieves references, localizes and verifies disclosure, then aggregates element verdicts; even when the judge lands on the examiner’s exact cited paragraph, it returns not-disclosed 27.3% of the time. The label and candidate-set design also limit interpretation: negatives are inferred from absent examiner action, and gold-reference presence nearly determines the label (AUC .9983).Source: §B.3.1 · pp. 47–48

B.3.2 NOTICE-AND-COMMENT RULEMAKING (VERDICT, CURATION, COMMUNITY)

  • V, A, V + A, and dense models score .595, .592, .593, and .588 on the pooled notice-and-comment outcome label, respectively, against .595 for character length alone.Outcomes are largely docket-level, and within-docket out-of-fold concordance is .498; the agree-label estimate is design-conditional (.612 versus .524).Source: §B.3.2 · pp. 49–50
  • Lifetime docket throughput correlates with deep verifier signal at ρ = +.635 (p = .003, n = 19), while headcount predicts only shallow verifiable signal (ρ = +.496, p = .022).After partialling out sampled-docket diversity, the deep-verifier correlation remains +.576 (p = .010), while the dense correlation falls to −.062.Source: §B.3.2 · p. 50

B.3.3 LEGAL OUTCOMES ACROSS TWELVE DOMAINS (APPENDIX-ONLY)

  • The twelve-domain legal-outcomes corpus is appendix-only and excluded from preference mapping because court rulings apply law to disputes rather than express preferences about creative products.The rebuilt corpus uses facts-only extraction and staged filtering to remove post-disposition material that leaked outcomes into an earlier pipeline.Source: §B.3.3 · pp. 50–51
  • Removing 2,639 extraction-leak rows (5.0%) moves lexical .740 →.719, articulated .641 →.630, and dense .784 →.782, widening the tacit gap .044 →.063.The defect arose when fact-free consent-remand forms received hallucinated fact blocks; the canonical pool also drops pro se and enforcer-agency confounds.Source: §B.3.3 · p. 51

B.4.1 ICLR PEER REVIEW (VERDICT, CURATION, COMMUNITY)

  • After venue control, the ICLR articulated lift is +.057 and the grouped value is .751; repairing a citation-join artifact lowered one cell’s AUC from .989 to .842.The earlier ∼.77 dense ceiling was pooled and venue-confounded and should not be compared with ICLR-only numbers.Source: §B.4.1 · pp. 52–53

B.4.2 BEST-PAPER AWARDS (EXPERT CURATION)

  • Citation-decile matching yields a .603 deconfounded text floor, versus .728 raw text-floor AUC and .635 citation-only AUC; within-venue-year AUC is .571 mean / .500 median.The final corpus contains 11,613 rows and 960 usable awards across 605 venue-years; text remains predictive beyond citations, but the signal is modest.Source: §B.4.2 · p. 54

B.5.1 PRESS RELEASES AND EDITORIAL PICKUP (EXPERT VERDICT)

  • Raising the press-release pickup threshold from k ≥1 to k ≥3 raises dense grouped AUC from .584 to .705; the adopted threshold avoids 88% single-outlet noise.The residual within-topic, cross-publisher craft signal is ≈+.046 over chance; publisher memorization contributes +.056 and topic selection +.038.Source: §B.5.1 · pp. 54–55
  • On the rebuilt press-release frame, company-grouped AUC is V .653, V A .681 (.744 with mined criteria), T .774, and V AT .771.The class imbalance at k ≥3 (1:27) is handled with stratified grouped cross-validation and threshold-free AUC.Source: §B.5.1 · p. 55

B.5.2 HOMEPAGE NEWSWORTHINESS (EXPERT CURATION)

  • Human annotators articulate ∼69–90% of homepage placements with classic news values, while within-snapshot pairwise AUC is ∼.65 against global .552.Across eight cleaning stages, the verifiable model moves .5535 →.572 and the full model .6208 →.630, so cleaning recovered little predictive signal.Source: §B.5.2 · p. 56

B.5.3 BBC MOST-READ, TWEET ENGAGEMENT, AND LINK-AGGREGATOR VOTES (COMMUNITY)

  • For BBC most-read, the earliest slice’s headline-length asymmetry makes the trustworthy AUC the length-decile-matched .60–.64, not the raw value.The task compares top-ten most-read headlines with same-day headlines from the same publisher; later-era slices draw both classes from the same homepage capture.Source: §B.5.3 · p. 57

B.6.1 R/WRITINGPROMPTS STORY UPVOTES (COMMUNITY)

  • The rebuilt r/WritingPrompts corpus contains 96,080 stories balanced at 48,040 per class across 70,453 unique prompts.Length balancing reduced the high-scoring versus low-scoring mean-length ratio from 1.30× to 1.001× after the clean rebuild.Source: §B.6.1 · p. 57

B.6.2 ROYALROAD: MARKET PICKUP, MAGAZINE CURATION, AND FOLLOWER COUNTS

  • RoyalRoad’s stub-status task is at chance under a certified instrument, consistent with market-pickup determinants lying mostly outside the opening chapters.The final pool contains 1,274 works after topic and era stratification; the text leaves narrative-register signals, while the popularity-matched subset was abandoned as too small.Source: §B.6.2 · pp. 58–59
  • The RoyalRoad magazine task is a pilot with 26 winners, while the follower-count task’s dense-over-bank margin measures an instrument limit rather than tacitness.The magazine bank reaches .624 [.550, .686] with a strong .787 anchor battery; the follower task’s anchor battery is weak and V +A coincides with a word-n-gram baseline. Kindle Scout is also pilot-scale (n=726), with fused value .792 [.627, .935], and its excerpt omits the publisher’s full manuscript and nomination statistics.Source: §B.6.2 · p. 59, §B.6.3 · p. 59

B.6.4 WIGLEAF TOP 50 FLASH FICTION (EXPERT CURATION)

  • .541 TF-IDF on Wigleaf’s full set matches an independent within-magazine .537, after the archive-only fix reduced provenance AUC from .90 to .500.The fix lowered Top50 recovery from ∼66% to ∼45%; cross-magazine TF-IDF .721 mostly reflected venue prestige, so the within-magazine result is reported.Source: §B.6.4 · p. 60

B.7.2 MODERATION REMOVALS: R/JOKES AND THE NINE MACRO-NORM COMMUNITIES (VERDICT; HUMOR AND LONG-FORM WRITING)

  • .807–.811 and .883–.884 are the rebuilt full-model score ranges for humor and long-form moderation tasks, .02–.03 below pre-rebuild values after removing an era component.Within a single subreddit, the articulated instrument recovers more of the dense model’s signal than pooled fitting, which can mistake cross-community norm differences for tacitness; r/Jokes articulates to .904 against a fused .923.Source: §B.7.2 · pp. 61–62

B.7.4 SNL CUT-FOR-TIME (PRODUCTION VERDICT)

  • V .55, A .44, and V +A .49 are all at chance on the SNL cut-for-time task after shared ASR removed a transcriber-authorship signature.The null is consistent with the cut decision depending on performance, timing, and the dress-rehearsal room rather than the script.Source: §B.7.4 · p. 62

B.7.5 NEW YORKER CAPTION CONTEST (EXPERT CURATION AND CROWD; EXCLUDED FROM CROSS-FIELD READOUTS)

  • .546 word and .545 character scores on the New Yorker hard-negative pool leave a −.001 gap after aggressive normalization removed typography and terminal-punctuation leakage.The initial embedding probe scored .996 AUC and fell to .730 after a first normalization round; the corrected transformation strips terminal punctuation and preserves raw text unchanged.Source: §B.7.5 · pp. 62–63

B.8 CROSS-CUTTING RULES

  • Stable-hash splits on natural units, separate confound and leakage cleaning, and identical normalization across pipelines are cross-cutting rules for these corpora.Near-perfect probe AUC can signal leakage: the caption embedding probe was .996 before repair, while Wigleaf provenance AUC fell from .90 to .500 after both classes used the same archive path.Source: §B.8 · pp. 63–64

C.1 EXECUTABLE VERIFIABILITY: WHAT A PROCEDURE CAN CERTIFY

  • 17 of the 940 V columns behind Table 23 are executable checks, all in GitHub pull requests; 918 are surface statistics and 5 are metadata or learned scores.Executable features certify item properties such as compilation, tests, reference matches, arithmetic identities, or item-linked lookups; Table 8 reports their applicability and AUC alongside existing V features and their combination.Source: §C.1 · p. 64

C.2 JUDGE PROMPTS, RUBRIC TEXTS, AND SCORING CODE FOR TABLE 2

  • Articulated criteria were scored with one zero-temperature, single-token prompt per item and criterion, while verifiable checks used execution fleets, deterministic programs, or caption features.The plot-arc proxy computes length adequacy, paragraph structure, and verb-density patterns, illustrating that replayable checks measure only what their programs compute—not the full named construct.Source: §C.2 · pp. 64, 66

C.1 EXECUTABLE VERIFIABILITY: WHAT A PROCEDURE CAN CERTIFY

  • Executable checks offer limited independent signal: deeper law and journalism checks cover below a quarter of items, AoPS final-answer AUC is .649 but redundant, and patents’ .702 is confounded by claim count (.721; residual about +.03).The authors interpret the verifiable share as a surface share everywhere except code and do not claim executable checks change paper-wide ordering; published-article tasks have little variance because editorial verification has already occurred.Source: §C.1 · p. 64, Table 8 · p. 65

C.3 MODELS USED AT EVERY STEP

  • After editorial-pickup rescoring, VA changed from .681 to .658 and VAT from .771 to .750, while T stayed .774; the T–VA gap widened.The refit used the judge-of-record protocol; four of 40 criteria became constant, the not-applicable rate fell to 21%, and the scrambled-text anchor AUC was .96.Source: §C.3 · p. 67
  • Competition editorial-approach rescoring changed VA from .754 to .734 [.687, .784] and VAT from .754 to .737, while V stayed .743 and T stayed .724.The judged criteria added nothing beyond the programs; the bank ensemble score anti-correlated with code length (r = −.59 with log characters).Source: §C.3 · p. 67
  • The full model used text-only Llama-3.1-8B LoRA training, while VAT candidates fused text and criterion scores or combined T and VA predictions.The candidate was selected on the evaluation split and reported on the test split, so the reported VAT score could be lower than T when the selected candidate lost on test.Source: §C.3 · p. 67

C.4 WHAT EACH MODEL WAS ALLOWED TO READ

  • In every audited task, the articulated judge read at least as much of each item as the full model, and it read more in both long-text tasks.For head-and-tail judge windows versus right-truncated full-model prefixes, the models saw different portions of the same item, and the effect of that difference was not measured.Source: §C.4 · p. 69, Table 10 · p. 69

C.5 ROBUSTNESS OF THE GAP ESTIMATES ACROSS MODEL FAMILIES

This section examines the stability of gap estimates across model families and bounds the value of undiscovered criteria, including empirical tests whose conclusions remain conditional on mining and rare-criteria assumptions. Four sealed campaigns showed positive monitor gains, while the ICLR campaign could not proceed under the frozen stopping rule; these tasks lacked an evaluation frame for head-to-head comparison and were excluded from Table 19.

C.5 ROBUSTNESS OF THE GAP ESTIMATES ACROSS MODEL FAMILIES

  • On story upvotes, V A ranges from .619 with Llama-3.1-8B to .660 with Gemma-4-31B, while V alone is .612 for every judge.All five judges order the anchors correctly; on joke upvotes alternative judges fall .05–.07 below the record, whereas notice-and-comment judges remain within .015 of one another and the record.Source: §C.5 · p. 70, Table 12 · p. 70, Table 13 · p. 71
  • Soft criterion scores change AUC by only +.003 for A and +.004 for V A, while judge-selected scoring templates change V A by at most .003.The soft readout’s median per-criterion Spearman ρ is .81 across 31 criteria, and neither judge’s preferred template raises V A.Source: §C.5 · p. 71
  • V A varies by only .023 AUC on math and .031 on Stack Overflow across judge families, with every alternative judge below the judge of record.Criterion-level rank correlations are much lower than bank-level agreement and rise with judge capability, indicating that judges differ in which criteria carry the signal even when recovery is similar.Source: Table 11 · p. 70, §C.5 · p. 71
  • The seven-judge ensemble improves articulated AUC by only +.004 on math and +.005 on Stack Overflow, while switching dense-model bases changes T by at most .007.The per-criterion mean across judges is slightly worse than the record judge; the Qwen dense model is also slightly weaker on both tasks, making the Llama model of record the conservative choice.Source: §C.5 · p. 72

D.1 DERIVATION AND EXTENSIONS

  • The missing-value bound estimates undiscovered value from Good–Turing missing mass and assumes salient criteria are found earlier; decorrelated proposers use overlap to estimate what their shared blind spots may miss.Because mining rounds adapt to prior discoveries, the missing mass describes the final-round proposal process rather than a stationary criteria population; truncation backtests provide an empirical check on the salience assumption.Source: §D.1 · pp. 72–73
  • A single undiscovered criterion would need to be worth at least 1.0 times the best discovery round on joke upvotes, 5.4 on notice-and-comment, 6.4 on citations, or 22 on story upvotes to overturn the surviving gaps.This rare-event analysis assumes at most one below-resolution criterion; the bound also identifies four campaigns whose bank plus whisker already reaches or exceeds the full model, so they are not read as tacit.Source: §D.1 · p. 73

D.2 BACKTESTING THE MISSING-VALUE BOUND

  • The backtest bound holds in 18 of 30 truncations, and in 13 of 16 at k ≥3; the bounded articulated ceiling remains below the full model on story upvotes (.674 < .792) and citation percentile (.782 < .884).Failures concentrate at early truncations, often when cumulative gain is zero or negative; the late-truncation test is partly expected because campaigns stop when gains flatten. Three under-mined tasks are treated as unresolved rather than tacit.Source: §D.2 · pp. 75–77, Table 16 · p. 76
  • Monitor gains were +.0030 for RoyalRoad followers, +.0036 for Stack Overflow accepted, +.0237 for Stack Overflow bounty, and +.0371 for Math.SE bounty.The respective campaign rounds, estimated means, and whiskers were: RoyalRoad, 2 rounds, ˆµ = .356, +.0017; Stack Overflow accepted, 3, ˆµ = .685, +.0078; Stack Overflow bounty, 4, ˆµ = .511, +.0248; Math.SE bounty, 5, ˆµ = .663, +.0730.Source: §D.2 · p. 78
  • The Math.SE bounty gap to the full model moved from .085 to .048, with degraded-audit flags in rounds 2 and 4.Both auditors missed one planted probe in those rounds, and the readmission rule was applied.Source: §D.2 · p. 78
  • The ICLR accept/reject campaign remained at round 0 because the paired refit-seed spread of VA on the monitor split was .012, above ε = .005.Its earlier pilot used a single self-proposer and had no valid Good–Turing estimate.Source: §D.2 · p. 78
  • The four further tasks had no evaluation frame for the head-to-head and therefore did not enter Table 19.Source: §D.2 · p. 78

D.2.1 METRIC DISCOVERY HEAD-TO-HEAD, AND THE ARTICULATED CEILING

The head-to-head comparison finds that articulated banks generally remain below the full model, while external metric banks improve the committed articulated ceiling. The spurious-feature audit combines statistical tests with an explicit construct judgment, and its validation recovers planted artifacts and historical leaks.

D.2.1 METRIC DISCOVERY HEAD-TO-HEAD, AND THE ARTICULATED CEILING

  • The head-to-head applies three published discovery methods to shared training inputs and evaluation frames, then compares their banks and unions with the campaign whisker and committed ceiling.ExperiGen is strongest among external methods, best on ten tasks and ahead of the authors’ bank on six; package-faithful HypotheSAEs is last on every task.Source: §D.2.1 · p. 78, Table 19 · p. 79
  • On joke upvotes, the eight-criterion RL-XAR rubric reaches VA .678 and does not close the gap, despite being the most efficient method per criterion.Adding it raises the external-method union from .707 to .717 and the union stacked on the authors’ bank from .716 to .721.Source: §D.2.1 · p. 79
  • The articulability gap survives on 12 of 13 bounded tasks; the whisker covers it on BBC most-read (−.005), while N&C rule change is at parity (+.001).No articulated model reaches the full model on any task; mean tacit share is .38 for the authors’ bank and .21 against the committed ceiling.Source: Table 19 · p. 79, §D.2.1 · p. 79
  • Ours plus whisker beats the best single external bank on 5 of 13 bounded tasks and loses on 8; against ours∪ext, it wins 4 and loses 9.The committed ceiling therefore takes the maximum of ours and ours∪ext before adding the whisker.Source: §D.2.1 · p. 79

E DECONFOUNDING THE FULL MODEL

  • The deconfounding appendix details the spurious-feature audit, the procedures used to stratify, strip, and residualize features, and how full-model performance changes as declared channels are removed.The main text summarizes this procedure in Section 3.1.Source: E DECONFOUNDING THE FULL MODEL · p. 80

E.1 THE SPURIOUS-FEATURE AUDIT

  • The audit treats spuriousness as a construct-validity judgment, using local tests to probe suspected paths and an evidence table that records each result and its assumption.Its channel tests probe what a criterion reads; path tests ask whether its label association survives controls such as within-stratum comparisons.Source: §E.1 · p. 80, Figure 5 · p. 81
  • The final construct audit assigns a feature to work quality or to capture, selection, or context; free-text rewriting is excluded because preserving unnamed tacit features cannot be verified.The audit runs different components depending on task evidence, with several further tests specified as the next increment.Source: §E.1 · p. 81
  • The validation battery routes planted artifacts and historical leaks, including a disguised quality criterion, while leaving the genuine controls unflagged.A blind adjudicator routed all nine test criteria correctly and all 156 planted probe pairs across 39 mining rounds correctly.Source: §E.1 · p. 82, Table 20 · p. 82

E.2 DECONFOUNDING PROCEDURE: WHAT WE STRATIFY, STRIP AND RESIDUALIZE

The appendix tests and removes label-predictive nuisance features before interpreting full-model gaps, then measures performance remaining after progressive partial-out. Residual dense-model performance is usually above chance but seldom above articulated banks, and conservative residualization limits what can be concluded about gaps.

E.2 DECONFOUNDING PROCEDURE: WHAT WE STRATIFY, STRIP AND RESIDUALIZE

  • Three tests classify candidate metrics as spurious: scrambled-text and n-gram channel checks, within-stratum prediction checks, and LLM adjudication of whether a feature measures context rather than work quality.A second model arbitrates LLM disagreements; declared metrics enter a nuisance ledger and are excluded as features.Source: §E.2 · pp. 82–83
  • The test battery routed both planted joke-label artifacts, including one disguised as craft, to the nuisance ledger while retaining all 112 genuine criteria.It also recovered bot-merge notices, [WIP] markers, and transcriber signatures identified by humans.Source: §E.2 · p. 83
  • Features present in over 60% of items are stratified; removable surface cues are stripped; other nuisance variables are cross-fitted out and checked with nonlinear probes.Linear partial-out can also remove genuine quality variance correlated with nuisance features, so residualized full-model values are lower bounds.Source: §E.2 · pp. 83–84

F FULL RESULTS

  • Held-out evaluations use task-specific grouping units with no group split across folds; the appendix reports complete results and robustness checks behind the main-text figures.Examples of declared nuisance variables include posting time for jokes, answer position for Stack Exchange, and datelines for press releases.Source: §E.2 · p. 83, F FULL RESULTS · p. 85

E.2 DECONFOUNDING PROCEDURE: WHAT WE STRATIFY, STRIP AND RESIDUALIZE

  • Deconfounding changes the target to choices conditioned on attributes such as length, topic, or genre, while residualization can remove shared quality variance.The authors accept this target change to avoid reporting gaps explainable by shortcuts.Source: §E.2 · p. 83

E.3 SPURIOUS-FEATURE DECAY CURVES FOR THE FULL MODEL

  • The decay analysis tracks held-out residual AUC as declared nuisance channels are added, reporting an asymptote only when the curve plateaus and otherwise using the full-channel residual as a bound.The table also reports placebo drops from row-permuted nuisance blocks to estimate the portion attributable to first-stage estimation noise.Source: §E.3 · p. 84, Table 22 · p. 85
  • Nonlinear probes recovered the joint nuisance block above null in seven of thirteen tasks, explaining 2%–11% of residual variance; nonlinear correction lowered residual AUC by at most .021 net of placebo.The linear probe was at null in every task; nonlinear recovery was much smaller than the block’s prior 20%–57% explanation of dense-score variance.Source: §E.3 · pp. 84–85
  • In 12 of 13 tasks the deconfounded dense model stayed above chance, but only three residuals exceeded their articulated banks; conservative partial-out makes the other ten inconclusive.The caption-contest finalist floor was .518, essentially chance; ICLR accept/reject residual performance (.654) fell below its bank (.668), and nuisance channels alone reached .690 nonlinearly.Source: §E.3 · p. 85

F.1 ALL RESULTS

This section presents benchmark comparisons of verifiable checks, articulated criteria, and the full model, alongside analyses of robustness, scaling, information, cross-task patterns, and field properties associated with tacit preference. Across nineteen tasks, it examines how language and participant characteristics relate to preference while noting that some findings are exploratory.

F.1 ALL RESULTS

  • Across nearly every task, V < V A ≤ max(T, V AT); the full model extends articulated criteria, while fusion helps most when its parent models are close.The reported full model is the better of T and V AT, and the fused candidate is selected on evaluation data and scored on a disjoint test split.Source: §F.1 · pp. 86–87, Table 23 · p. 87

F.2 CROSS-FIELD CHARTS: FULL INSTRUMENT LADDERS PER CHANNEL

  • Normalized articulability share is uncorrelated with the ceiling (ρ = −.09, p=.65, n=28), although raw gaps co-move with it (r=.49, p=.008).Matched-ceiling tasks still span normalized gaps of 0–44%; curation has the lowest ceilings and smallest articulability bands, verdicts are intermediate, and community tasks have the widest gaps (15–44%). Peer review has the widest verdict band, while the annotated creative-writing ladder defines verification and articulation gaps as adjacent instrument steps.Source: §F.2 · pp. 88–90

F.3 STANDARD ROBUSTNESS CHECKS: BOOTSTRAP, INFLUENCE, SUBSAMPLING AND FITTING CHOICES

  • On r/Jokes, leaving out one temporal stratum moves articulated AUC by .187; on Wigleaf, its gain over V spans zero and changes sign when any of 112 stories is omitted.Elsewhere, leave-one-group-out changes AUC by at most .003 and halving groups changes it by less than .01; tested fitting choices do not move gaps beyond reported intervals.Source: §F.3 · pp. 90–91

F.4 STANDARDIZED UNCERTAINTY FOR THE INSTRUMENT LADDERS

  • Paired articulated increments exclude zero on tasks with plentiful groups, except pooled long-form moderation, whose interval resamples only four communities.On peer review, half the reviewers’ recommendations predict decisions at AUC .881 [.879,.884], while the articulated side’s measured upward sensitivity is roughly .01–.02 AUC, rising to .05 for peer review under judge-optimized prompts.Source: §F.4 · p. 92, Table 25 · p. 92

F.5 FULL-MODEL DATA SCALING

  • At full data, deconfounded residuals reach +.013 [.007,.018] on math.SE bounty and +.006 [.002,.011] on Stack Overflow bounty, with no plateau in range.The articulated bank absorbs roughly 90% of what added data gives the dense model; historical sweeps show creative writing still climbing while math Q&A, legal outcomes, and press releases saturate by 30–60%.Source: §F.5 · p. 93, Figure 11 · p. 93, Figure 12 · p. 94

F.6 MEASURING ARTICULABILITY GAPS IN TERMS OF INFORMATION

  • The information view agrees with the AUC view in 40 of 47 populations, and verifiable-share rankings across views correlate at ρ = .93 (p < 10−4, n=43).The analysis calibrates out-of-fold predictions and uses a within-group permutation null; estimates are lower bounds, and small strata make absolute levels less reliable than orderings.Source: §F.6 · pp. 94–95

F.7 VERIFIABILITY TRACKS ARTICULABILITY ACROSS TASKS

  • Across 28 tasks, articulable and verifiable shares correlate at ρ = .63 (p < .001), but the association weakens to ρ = .23 (p = .24) after adjusting for nesting.On the earlier 23 tasks, the observed correlation .84 exceeds a nested-null correlation of .67 (p = .012); humor articulates 85% of shared preference but verifies only 30%, while peer review occupies the low-articulability corner.Source: §F.7 · p. 95

G WHY SOME FIELDS CARRY MORE TACIT PREFERENCE

  • The field-level tacit shares cluster: peer review and creative writing are highest, but the nineteen-task screen is exploratory and cannot isolate field properties reliably.Each averages .39, versus .11–.15 for the other five fields; roughly forty covariates were tried, and no individual result survives multiple-comparison correction.Source: G WHY SOME FIELDS CARRY MORE TACIT… · p. 95

G.1 JARGON DOES NOT PREDICT TACIT PREFERENCE

  • Jargon does not predict tacit share across tasks, fields, or within-task item splits, despite contrasting jargon density among fields with large and small gaps.Across nineteen tasks, rank correlation is −.13 (p=.60); field-level correlations are nonsignificant, and item jargon share yields AUC .40–.56.Source: §G.1 · p. 96

G.2 THE REST OF THE LANGUAGE OF THE TEXTS

  • Longer texts have larger tacit shares within the community channel, while novelty does not predict labels within any task.Length is the only surface property that survives controlling for composition versus answer; novelty AUC ranges from .37 to .56.Source: §G.2 · p. 97

G.3 HOW MANY PEOPLE DECIDE, AND FROM HOW MANY INSTITUTIONS

  • Participant counts cannot establish an independent effect: counts are label-derived or restate the judgment channel, while institutional coverage omits the largest-gap fields.Four fields with measured institutional structure have tacit shares within .03 despite ranging from two bodies to thousands; affiliations are unavailable for peer review, Reddit, and RoyalRoad.Source: §G.3 · pp. 97–98, Table 29 · p. 98
  • Hand-coded field characteristics show no producer-training relationship, a U-shaped decider-training pattern, and associations of smaller gaps with high-validity environments and formal correctness criteria.The criterion association is not necessary: humor, press releases, and notice-and-comment lack formal criteria but have small gaps; decider type reproduces the channel ordering.Source: §G.3 · p. 98

G.4 WHAT THE JUDGMENT IS ABOUT, AND A DIRECT TEST

This section tests how prompting, alternative explanations, and item-level descriptions relate to observed articulability gaps. It assesses the effects and limitations of these approaches, including mixed effects when descriptions are implemented.

G.4 WHAT THE JUDGMENT IS ABOUT, AND A DIRECT TEST

  • Compositions have a higher tacit share than answers (.30 vs .13); corpus memory closes 46% of the citation gap but nothing on joke upvotes, so it is not a general explanation.On citation percentile, corpus memory alone reaches .739 versus .627 for the criteria bank; the authors treat term-frequency neighbors as a crude memory, making the closure a lower bound on what retrieval could achieve.Source: §G.4 · pp. 98–99, Table 30 · p. 99

G.5 THE SOCIAL STRUCTURE OF ARTICULABILITY: SUBFIELDS AND ORGANIZATIONS

  • Criteria are shared across related decisions but community-specific weighting matters: within-community fitting improves articulated recovery, especially for long-form moderation, showing pooled gaps partly reflect heterogeneity.Per-criterion rank agreement is ρ = .76 between regulatory outcome and agreement labels versus ρ = .47 between academic curation and citations; long-form within-community AUC exceeds the pooled articulated score (.606) by .09–.23, with r/books reaching .837.Source: §G.5 · pp. 100–102, Table 31 · p. 102
  • In federal agencies, headcount predicts shallow verifier signal (ρ = .50, p = .022), while docket throughput predicts deep signal (ρ = .64, p = .003); with n = 19, this is one observed dissociation, not a general law.Merging layers reduces the throughput association to ρ = .30 (not significant), whereas the full V +Vdeep+A stack retains it (ρ = .48, p = .036).Source: §G.5 · p. 101, Figure 15 · p. 101

G.6 STRESS-TESTING THE FRAMEWORK: WHO DECIDES, AND CAN THE INSTRUMENT SAY “NOTHING”?

  • Verdict-channel articulability ranges from .92 for rule-governed moderation to .74–.79 for expert adjudication; on one text community, curated selection is most articulable and community engagement least.Across boundary tasks, fusion does not meaningfully underperform its better parent and helps most when the parent models are close; RoyalRoad reproduces the channel ordering using texts from one community.Source: Table 32 · p. 102, §G.6 · p. 103
  • SNL and pilot RoyalRoad market verdicts are not recoverable from text at their tested scales, while conditioning text models on criterion scores generally adds no signal beyond raw text.For SNL, V, A, and VA score .55, .44, and .49, with every CI straddling .5; the RoyalRoad pilot scores .486 [.390, .581], though the larger population shows weak significant signal. Conditioning has one peer-review exception (+.045 [.022, .067]) and no grid-wide relationship with text-only strength.Source: Table 32 · p. 102, §G.6 · p. 103

H.1 PRETRAINING DENSITY

  • Pretraining density does not predict tacit share (ρ = .12, permutation p = .73), contrary to the prediction that greater corpus exposure would reduce the gap.The authors caution that the density measure is coarse: each field’s count is dominated by one generic phrase.Source: §H.1 · p. 104, Table 33 · p. 104

H.2 SUBFIELD MIXTURE

  • Subfield conditioning and per-subfield criteria do not explain the gaps: within-subfield gaps stay as large or larger, and criteria mined for agencies do not improve within-agency AUC.Separate per-subfield models lose to the pooled bank inside every subfield; on notice-and-comment responses, own-agency criteria change within-agency AUC by −.002 on average.Source: §H.2 · pp. 104–105

H.3 POST-TRAINING

  • Base-versus-instruct comparisons do not identify post-training as the source of the gap: 8B scorers are degenerate, while 12B base judges fail the anchor in all three tasks.The authors also optimize judge-specific prompts using a label-free objective, but the supplied passage describes the procedure rather than its outcome.Source: §H.3 · pp. 105–106, Table 36 · p. 106
  • The best articulated model remains 0.08 AUC below the full model on peer review and 0.14 below it on creative writing, despite judge-specific prompt optimization.Across four judge families, optimized prompts move the articulated model by at most 0.05, only on peer review; the accepted rewrites converge on numeric anchors and explicit scoping.Source: Table 37 · p. 107, §H.3 · p. 107

H.4 RIVAL ACCOUNTS OF THE GAP

  • The gap is similar for seen and unseen deciders, while timing substantially confounds story-upvote results but has little effect on most other tested tasks.Identity gains are small or uncertain on bounty tasks; within posting-order quintiles, the story-upvote gap remains above the half-gap kill threshold.Source: §H.4 · pp. 107–109
  • Readout granularity, judge noise, and holistic judgments bound the measurement instrument, but none establishes that the residual gap is human-articulable.More detailed criterion readouts change the gap by at most a few hundredths; the peer-review banks are least reliable, and a direct yes/no judgment is explicitly treated as an instrument bound.Source: §H.4 · p. 108
  • At scale, label-blind story descriptions retain most of the full model’s advantage over the articulated model, but raw text performs better on the largest dataset.At 76,867 rows, descriptions score .771 versus .802 for raw text and retain 77% of the full model’s advantage over VA; their pilot advantage reflects data efficiency.Source: §H.4 · pp. 108–109
  • Across tasks, the gap grows with the number of deciders and social-outcome labels, while tests do not support general accounts based on label noise, relative labels, or norm drift.Exploratory label-shape results include a larger citation-percentile gap for older papers and a community-channel association with item length; temporal holdouts find staleness for press pickup and joke upvotes, but not peer review, spotlight selection, or news curation.Source: §H.4 · pp. 109–110
  • Item-level tests found no common property that identifies tacit items across both classes, and criteria proposed from gap contrasts did not close the gap beyond placebo criteria.The gap contrast can identify class-specific patterns, but exposure channels explain little except for press pickup, whose gap falls from .107 to .065 within company.Source: §H.4 · p. 111

H.5 WHAT THE TEXTS WITH THE LARGEST GAPS HAVE IN COMMON

  • Readers found varied text properties among high-gap items, but properties often mark preferred and not-preferred errors in opposite directions, so they weakly predict labels.Among the verified properties, 28 are judged from text alone, 3 relative to the question, 2 to the proposed rule, and 6 require outside knowledge; none is corpus-relative.Source: §H.5 · pp. 113–114
  • Corpus-relative descriptions generally failed as item-level criteria, while several novelty and curiosity properties survived, including oddity or question headlines, story payoff, title substitution, and “novel”.The bank over-credited the word “novel”: it appeared in 17% of accepted high-gap abstracts versus 25% of matched ones, and 39% versus 28% among rejected ones.Source: §H.5 · p. 115
  • Reading within existing ledger strata lowered the raw HashtagWars gap from .175 to .079; added channels shifted gaps by at most .012 except for press pickup, homepage placement, and N&C responses.For those exceptions, press pickup changed .107 to .087 (.065 read within company), homepage placement .084 to .073, and N&C responses .021 to .038.Source: §H.5 · p. 115
  • On six of 15 tasks, bank descriptions shrank the gap by at most .025 AUC (citations .194 to .169; Math.SE accepted answers .073 to .056; spotlight picks .100 to .088), while HashtagWars rose .175 to .218.VAT moved by at most .018. The table reports results on all held-out items and defines the gap as max(T, VAT) − VA in AUC on the raw frame.Source: Table 42 · p. 115, §H.5 · p. 115
  • Description checks were limited by gap-sorted item review, prior exposure to 150–400 task items, and text truncation at 1,500 characters.Math.SE titles omitted the asker’s attempt, making properties relative to it noisy; two prominent joke descriptions instead tracked missing data or subreddit culture.Source: §H.5 · p. 115

H.6 NEGATIVE RESULTS

The negative results test whether stronger or more targeted articulated instruments can close the gap, and identify where apparent progress instead reflects noisy estimation, confounded inputs, or limited evaluation settings. Across these tests, the reported alternatives do not displace the flat criteria bank or explain away the full model’s residual signal.

H.6 NEGATIVE RESULTS

  • −.058 AUC on creative-writing upvotes and −.027 on peer-review verdicts: metric trees lose to the flat bank on identical folds.Leaves have 5–25× less data, and hard routing adds estimation variance without more representational power than conditional criteria weighting; the result does not rule out criteria absent from the shared bank.Source: §H.6 · p. 118
  • The grouped-OOF stack improves frozen boosting by only +.003–.004, within seed spread; tested heads and nonlinearities provide no clear aggregation-capacity explanation for the gap.On creative writing and peer review, MLPs score .651–.659 vs .665 and .626–.640 vs .668, respectively; higher-order nonlinearity adds little or hurts across the reported tasks.Source: §H.6 · pp. 118–119
  • The criteria bank reconstructs full-model rankings with median ρ = .56 across twelve tasks, while adding residual signal recovers 85% of the story-upvote gap and 59% of the ICLR gap.The residual raises story-upvote AUC from .665 to .773 against the full model’s .792, and ICLR AUC from .668 to .732.Source: §H.6 · p. 119

I CONSEQUENCES OF THE GAP

  • Single-generator pools show no detectable divergence between rubric- and preference-model selection; their homogeneous quality makes instruments rank noise.A third of bank criteria collapse to constants on these pools, whereas diverse pools are needed for the predicted asymmetry; the consequences section also examines selection regret across tasks.Source: §H.6 · p. 119, I CONSEQUENCES OF THE GAP · p. 120

H.6 NEGATIVE RESULTS

  • Criteria conditioning is null where text-only models are strong, and its gain does not track text-only-model strength (Spearman −.14, p=.76).The nine within-community contrasts have intervals covering zero; peer-review spotlight is a single-task exception where the text-only model is weakest.Source: §H.6 · pp. 119–120
  • Audits rejected circular or confounded labels and a weak pseudo-label judge; patents were withdrawn because positive rows exposed gold-cited references to every model.The proposed valid patent check would require a pre-priority-date prior-art index queried identically for every row, which the authors had not built.Source: §H.6 · p. 120
  • Feature verbalizations poorly track full-model activations, while per-item criteria and rationales do not yet close the gap or yield broadly reliable gap properties.Verbalization fidelity is .00–.11 for eleven features and .24 for one; STaR label-recovery accuracy rises from 48% to 56.2% on 500 items, and item-level patterns are confounded or task-specific.Source: §H.6 · pp. 119–120

I.1 SELECTION REGRET ON EVERY TASK

This section examines selection judgments across tasks, combining evidence on selection regret and ranking behavior with research and examples documenting tacit, context-sensitive, and sometimes inconsistent decision-making. It also considers how formal criteria, cue training, and simple models relate to these judgments.

I.1 SELECTION REGRET ON EVERY TASK

  • Regret was positive in 13 of 16 tasks, negative in one and zero in two; the sign test over 14 non-zero tasks gave p=.0009.The negative result was caption crowd (−.250); GitHub PR merged and editorial pickup had zero regret.Source: Table 45 · p. 121, §I.1 · p. 121
  • Across eight grouped tasks, induced-gap/regret Spearman ρ had median +.65 (95% CI [.08, .77]); at matched AUC, removal identity did not change regret beyond refit noise.Seven of eight grouped tasks were positive and the one-sided Wilcoxon test gave p=.012; peer-review tasks were excluded because regret equals pairwise-AUC difference there, and low-group-count tasks could not resolve regret reliably.Source: §I.1 · p. 122, Table 46 · p. 122, Table 47 · p. 122

I.2 GENERATED POOLS ON THREE TASKS

  • Across 852 pools, the full model’s top pick ranked at the .71 percentile of the articulated model’s ranking and the articulated model’s pick at .69, a difference of .023 [.004, .042].The asymmetry may reflect top-tail interactions or the coarseness of the four-level articulated score; the study does not distinguish these explanations.Source: Table 48 · p. 123, §I.2 · p. 123

J.1 JOURNALISM

  • The 120 quotations are not cumulative evidence for one tacit component: 62 mark inability to explain reliable judgments, 27 expert disagreement, 8 institutional bias, and 26 failures of explicit prediction rules.Fifteen carry two marks; 12 are background.Source: J ADDITIONAL COLLECTED QUOTES · p. 123

J.2 MATHEMATICS

  • Mathematical choice is described as guided by delicate, felt rules that are difficult to state precisely, while mathematical beauty is disputed and context-dependent.Thurston also emphasizes that mathematical practice relies on social standards and communication beyond formal documents.Source: §J.2 · pp. 124–125

J.3 CODING

  • Programming theory can depend on similarities that cannot be expressed as rules, and code readability is intentionally left formally undefined to capture intuitive judgments.Buse and Weimer report significant, but not overwhelming, human agreement about readable code.Source: §J.3 · p. 125

J.4 ACADEMIC ARTICLES AND PEER REVIEW

  • Peer review combines spontaneous impressions, post hoc defensible judgments, and context-sensitive criteria; revised uniform styles characterize most, but not all, referees’ judgments.Hirschauer describes three phases of judgment, while Hug notes that a revised style fails to describe some referees’ judgments.Source: §J.4 · pp. 126–127

J.5 GRANT PANELS

  • Grant-panel standards are often taken for granted rather than formally spelled out, and reviewers may calibrate by developing an embodied impression of the proposal set.Panelists also describe using intuition and connoisseurship to reach judgments about excellence.Source: §J.5 · p. 127
  • Scientific novelty is described as fundamentally uncertain, with bias and chance embedded in outcomes.Roumbanis, “Peer Review or Lottery?” (2019) [iv] makes this claim about ex ante evaluation of scientific novelty.Source: §J.5 · p. 128

J.6 LAW, POLICY AND REGULATION

  • Legal judgment can rely on intuition that judges may be unable to articulate, while outstanding lawyers’ tacit knowing is evident in competent practice.Hutcheson (1929) [i] describes decision by feeling and hunch; Neumann (2000) [i] quotes Schön on tacit knowing-in-action.Source: §J.6 · p. 128

J.7 PATENTS

  • Halving patent-review time raised grant rates by 9 to 19 percentage points, or roughly 13 to 28 percent, even with the same examiner and written standard.Frakes & Wasserman (2014) report this result, illustrating how patent outcomes can vary with review conditions. [iv]Source: §J.7 · p. 129

J.8 CREATIVE WRITING AND PUBLISHING

  • Creative rubrics fix their dimensions at design time, while high-quality work may demonstrate value on unanticipated dimensions.Zou & Xu (2026) [iv] contrast rubric evaluation with expert-judgment transfer and argue for holistic judgment under full-dimensional constraints.Source: §J.8 · p. 130

J.9 HUMOR

  • Humor models lagged human matching accuracy by 30 points, and human-authored explanations were preferred over the best machine-authored ones in more than 2/3 of cases.The explanation comparison used ground-truth visual scene descriptors and few-shot GPT-4. [–]Source: §J.9 · p. 130

J.10 ART AND CONNOISSEURSHIP

  • Connoisseurship is described as private appreciation and a way of seeing, while scientific investigations have not conclusively settled authorship of many paintings attributed to Rembrandt.Eisner (1998) [–] characterizes connoisseurship; Gordenker et al. (2013) [iv] report that scientific research has had an impact without producing conclusive results.Source: §J.10 · p. 131

J.11 MEDICINE

  • Expert global ratings discriminated among clinician experience levels better than checklists, which may fail to capture increasing clinical competence.Hodges et al. (1999) [iv] found experienced clinicians scored better on global scales but worse on checklists; Martin et al. (1997) [iv] likewise found global ratings more discriminating.Source: §J.11 · p. 131

J.12 DESIGN STUDIOS AND TEACHING OBSERVATION

  • A single classroom observation by one observer had reliability between .27 and .45, and rater impressions could linger across sequential videos.Ho & Kane (2013) also report that a .1-point score difference could move a teacher 10 percentile-rank points. [ii]Source: §J.12 · p. 133
  • FFT factor analysis had R2 = .80, PLATO R2 = .575, and holistic MQI R2 = .84–.90; FFT exact agreement was 47.3–65.8% (kappa .05–.28).Kelly et al. [ii, iv] report single-factor structures; principals were better at identifying teachers at the extremes than distinguishing those in the middle [–].Source: §J.12 · pp. 133–134

J.13 JUDGED COMPETITIONS

  • In violin competitions, video-only accuracy was 50.5% versus 33.3% chance, while audio-only and audiovisual accuracy were non-significant; wine awards also showed weak cross-competition agreement.Tsay [i] found musicians’ sound-only accuracy was 25.7%, below the 33% chance rate, versus 47.0% for silent video; Hodgson [ii] found 84% of wines that won gold somewhere received no award elsewhere.Source: §J.13 · p. 134
  • Non-criterion variables jointly explained 44.1% of dressage-score variance, while figure-skating technical and program-component scores showed linear relationships that implied difficulty bias.Wolframm [iii, iv] analyzed 510 scores across seven events; gymnastics judging also showed frequent systematic bias despite high reliability, and dressage scores varied by judge and competition level [ii].Source: §J.13 · p. 134

J.14 HIGH-STAKES SELECTION

  • Nearly half of venture capitalists often made gut investment decisions, while unstructured interviews could produce confident impressions from random responses and worsen predictions.Gompers et al. [i] report 22% used NPV and 9% used no financial metrics; Dana et al. [iv] report the interview made predictions worse, and Highhouse [i, ii] notes low interrater reliability.Source: §J.14 · p. 135
  • “Fit” lacked a formal definition despite being reported as the most important interview-stage criterion by more than half of interviewees; admissions officers used three incompatible definitions of holistic review.Rivera [iii] describes a technically strong candidate rejected for perceived fit; Bastedo et al. [ii] identify whole file, whole person, and whole context as operational definitions.Source: §J.14 · p. 135

J.15 RULE EXTRACTION

  • After roughly one minute of instruction on one contour cue, novices’ accuracy in sexing hard chick photographs rose from 60.5% to 84%, and predictable expert-judgment variance rose from 4% to 67%.Biederman and Shiffrar [i] describe hands-on expert checking rather than simple diagrams; birdwatchers’ “jizz” and aircraft-spotting skills were also described as difficult for experts to articulate [i].Source: §J.15 · p. 135
  • A linear model built from each judge’s cue inputs beat that judge’s accuracy in all five studies, while experts’ articulated rules could remain wrong with unclear feedback.Camerer and Johnson note that statistical models need not describe experts’ actual processes and that simple linear combinations can outperform complex configural rules.Source: §J.15 · p. 135

J.16 CROSS-CUTTING

  • People can act according to rules they cannot state, and in one decision task only 26% detected when their chosen option was covertly swapped.Argyris, Putnam, and Smith [i] describe tacit “theories-in-use”; Johansson et al. [i] found participants could confabulate explanations for selecting the swapped option.Source: §J.16 · p. 136

Related papers

Loading 2610.03025v1…