Source-linked AI summary
Decoupling Is Not Identification: Supervised Evidential Learning in Next-Token Prediction
Ge Wang
TL;DR
The paper asks what makes a Dirichlet concentration measure training support, since decoupling prediction from concentration does not identify its meaning. It tests explicit exact-count supervision on character-level Moby-Dick with count labels withheld for some context types. Supervision generalizes to held-out types as a lexical-support ranking, but does not establish calibrated counts or useful error deferral.
Problem
Method
Results
Held-out-label ENTOP–S achieved partial correlation 0.201 ± 0.010 and matched-pair win 0.822 ± 0.021, while lexical-support ranking did not guarantee calibrated counts or useful error deferral.
Source: §4 · pp. 3–4, §5 · p. 5Takeaways & Limitations
The paper proposes confidence control, matched pairs, held-out evidence labels, a constant baseline, and a decision test as a minimum protocol for epistemic-concentration claims.
Source: §5 · p. 5The scope is one novel, one character tokenizer, five seeds, and a filtered label-consistent subset.
Source: §5 · p. 5
Abstract
from arXiv · showhide
A next-token probability says what a model predicts, not how much training support lies behind it. A Dirichlet head can represent this distinction by separating mean $m$ from concentration $S$, but decoupling does not identify what $S$ means. Here we propose an Evidential Next-Token Prediction (ENTOP) framework to audit this gap on character-level Moby-Dick, using exact 8-gram count as a reproducible lexical-support label and withholding count regression from 20% of context types. Standard implicit evidential training carries essentially no count signal beyond confidence on held-out-label types (partial Spearman $ρ= 0.001 \pm 0.014$), whereas explicit supervision generalizes ($ρ= 0.201 \pm 0.010$; matched-pair win $= 0.822 \pm 0.021$). CE predictive entropy is at chance for unseen 8-grams (AUROC $= 0.490 \pm 0.004$), while supervised vacuity reaches $0.772 \pm 0.003$, comparable with an indexed CE-representation baseline ($0.769$) but below the tautological corpus oracle ($1.000$). Neither longest-suffix nor representation-distance strata explain where amortization succeeds. Increasing count weight under the digamma objective improves support fit only by sacrificing prediction. A constant predictor wins natural log-RMSE, and vacuity does not improve error deferral. These results motivate a minimum evidence protocol---confidence control, matched pairs, held-out labels, a constant baseline, and a decision test---and show that concentration can pass identification while failing calibration and utility.
1 Introduction
The paper asks what label, if any, makes Dirichlet concentration measure training support, since emitting a second-order output alone does not make it epistemically meaningful. It audits this question by withholding count supervision from some context types while retaining their language-model training.
1 Introduction
- A second-order output is not epistemically meaningful merely because a network emits it; loss minimization can determine concentration through parameterization and regularization.Prior Networks train distributional uncertainty, while Posterior Networks use latent density and pseudo-counts [3] [5].Source: §1 · p. 1
- Withheld types still receive language-model training but no count regression, so success on them cannot come from directly fitting their count labels.The split withholds count loss from 20% of context types.Source: §1 · p. 1
- The audit combines prediction-loss comparisons, confidence-matched pairs, held-out labels, support variants, constant predictors, and risk–coverage tests.Its claim is limited to explicit supervision making concentration rank lexical support across context types, not calibrating counts or task risk.Source: §1 · p. 1
2 Evidential Next-Token Prediction Framework
ENTOP decouples next-token prediction direction from concentration, then tests whether explicit count labels give concentration a support-related meaning. The framework distinguishes that supervised target from an implicit digamma objective and includes a six-condition comparison.
2 Evidential Next-Token Prediction Framework
- The decoupled head makes E[π] = m exactly, so concentration S cannot algebraically change prediction.The minimum concentration is Kα0, with α0 = 0.1; decoupling gives representational freedom but not epistemic semantics.Source: §2 · pp. 1–2
- For αk ≥ 1, the tight bound is p̄max ≤ 1 − (K − 1)/S, so requiring p̄max ≥ 1 − ϵ requires S ≥ (K − 1)/ϵ.Equality occurs when each nonmaximal αj = 1.Source: §2 · p. 2
- ENTOP uses exact context count n(x) to define S*(x) = Kα0 + n(x) and trains concentration toward it with a squared log-count loss.Exact count is an auditable lexical-exposure target, not semantic evidence.Source: §2 · p. 2
- ENTOP–D uses the decoupled head with digamma expected cross-entropy and annealed KL regularization, without explicit supervision of S.The “D” denotes the digamma objective; “S” denotes explicit support supervision.Source: §2 · p. 2
- The six-condition grid separates count supervision from prediction objective, and Figure 1 depicts ENTOP–S’s separate targets for prediction direction and evidence strength.The held-out 20% still receive language-model training but no count regression.Source: §2 · p. 2, Figure 1 · p. 2
3 Experimental Design
The experiments use character-level Moby-Dick with exact context counts and a shared split that withholds count labels for 20% of context types. The design compares support measures under confidence controls and probes unseen contexts, interpolation, scale, and error deferral.
3 Experimental Design
- The corpus uses 1,006,806 training, 49,335 validation, and 130,653 test characters, with K = 48, L = 8, and 511,526 unique training context types.The chapter partitions are 1–110, 111–120, and 121–135.Source: §3 · p. 2
- A shared stratified split assigns 80% of context types to possible count supervision and withholds it from 20%, while language-model training samples all types.Primary support analysis filters for n ≥ 2 and q(x) ≥ 0.9, yielding 54,122 supervised and 13,616 held-out-label types.Source: §3 · p. 2
- Support is measured with partial Spearman correlation residualized against H(m) and mmax, plus fixed matched-pair win rates for confidence-matched contexts.The pair sets contain 884 supervised and 204 held-out pairs; a constant predictor supplies a scale baseline for three log-RMSE measures.Source: §3 · p. 3
- The boundary test defines unseen 8-grams by ntrain(x) = 0 and compares entropy and vacuity with indexed references, while suffix and CE-space distance strata probe interpolation.The longest-suffix oracle is an upper bound requiring a corpus index; error deferral sorts test errors by uncertainty, with joint-score weight selected on validation.Source: §3 · p. 3
4 Results
Explicit count supervision generalizes support ranking to held-out context types, while count-scale, prediction, and decision results limit what that signal establishes. The results also show that boundary detection and support ranking do not make vacuity useful for error deferral.
Results
- The constant beats ENTOP–S on natural held-out log-RMSE (0.304 ± 0.000 versus 0.575 ± 0.031), but ENTOP–S improves balanced (1.673 ± 0.000 to 1.030 ± 0.048) and n ≥20 RMSE (1.611 ± 0.000 to 0.963 ± 0.043).Natural weighting emphasizes low-count types, whereas balanced and high-count evaluations favor support separation.Source: §4 · pp. 3–4, Table 2 · p. 4
- ENTOP–S reaches held-out partial ρ=0.201 ± 0.010 and pair win=0.822 ± 0.021; ENTOP–D remains null (0.001 ± 0.014).Coupled residual correlations are −0.047±0.002/−0.036±0.007 for supervised/held-out types.Source: §4 · p. 3, Figure 2 · p. 4
- ENTOP–S vacuity reaches 0.772 ± 0.003 unseen-boundary AUROC, near the fixed CE-hidden kNN index (0.769) but below the exact corpus-query oracle (1.000); entropy is at chance (0.490 ± 0.004).The kNN comparator uses a 50,000-type bank; the oracle is exact by construction.Source: Figure 2 · p. 4, §4 · p. 4
- At β=10, D+count reaches held-out ρ=0.133 and pair win=0.809, but accuracy is 0.336 versus ENTOP–S’s 0.483 and boundary AUROC is 0.711 versus 0.772.The sweep is a seed-0 diagnostic trade-off, not a five-seed model comparison.Source: §4 · p. 4, Table 3 · p. 5
- Figure 3’s claim boundaries show natural RMSE favoring a constant, balanced and high-count evaluations favoring ENTOP–S, vacuity failing selective error prediction, and neither suffix length nor CE-h proximity yielding a monotonic support gradient.The suffix and proximity comparisons do not identify the generalization mechanism.Source: Figure 3 · p. 5
5 Discussion and Conclusion
Explicit labels make concentration rank lexical support across context types, but that identification does not establish calibrated counts or useful prediction risk. The authors therefore frame evidence claims as requiring controls and targets chosen for a decision.
5 Discussion and Conclusion
- Neither lexical-overlap strata nor the tested CE-h proximity proxy produces a monotonic gradient, leaving the amortization mechanism unresolved.Source: §5 · p. 5
- An earlier zero-structure pilot’s raw correlation collapsed after shared linguistic structure and confidence control; the authors propose five controls as a minimum protocol.These are confidence-adjusted correlation, matched pairs, held-out evidence labels, a constant baseline, and a decision test.Source: §5 · p. 5
- The evidence is limited to one novel, one character tokenizer, five seeds, and a filtered label-consistent subset; future targets should be decision-relevant and evaluated by that decision.Exact count is described as convenient rather than uniquely decision-relevant.Source: §5 · p. 5
- Explicit labels amortize lexical support across context types, approaching an indexed baseline without retaining its bank; support ranking is neither calibrated count nor prediction risk.Source: §5 · p. 5
- Decoupling lets ENTOP represent sharp predictions with little evidence, but does not by itself identify concentration’s meaning.Source: §5 · p. 5