Source-linked AI summary
Easy to anticipate, hard to compute: boundary dependence finds the computed outputs that entropy patching misses
Nicolás Vera Zúñiga
TL;DR
Byte-level patching uses next-byte entropy to allocate global computation, but predictable byte types can mask values that require computation. The paper tests boundary dependence, a label-free signal for where patch starts matter, and finds it outperforms entropy-based placement on computed results across several settings. The results suggest that patch allocation should reflect the global model’s needs, not only next-byte surprise.
Problem
Entropy-based patching allocates global computation according to next-byte uncertainty, leaving open whether that signal identifies positions whose values require computation despite predictable byte types.
Source: §1 · p. 1Method
Results
At tight budgets, boundary dependence improves computed-result accuracy over entropy across trained and adapted BLT models, with its advantage growing with model size in scratch-trained models.
Source: §1 · p. 2Takeaways & Limitations
For byte models, patch allocation should track what the decoder needs from the global model rather than only how surprising the next byte is.
Source: §13 · p. 12Under teacher forcing, adapted BLT-1B gets about 2% of final answers under every layout, so the reported layout gaps could not be measured end to end.
Source: §12 · p. 11
Abstract
from arXiv · showhide
Byte-level language models such as the Byte Latent Transformer (BLT) group bytes into patches and run their large global model once per patch. BLT starts a patch where a small model's next-byte entropy is high, so global compute goes where the next byte is hard to predict. We show that this rule has a systematic blind spot: positions whose type is predictable but whose value must be computed, such as the number after "=" in a worked math solution. Under tight patch budgets, entropy-triggered layouts skip these positions, and accuracy on them collapses. In Meta's BLT-1B with patch starts on 10% of bytes, the entropy rule puts a patch start at 16% of the computed results in GSM8K solutions and gets 19.0% of them exactly right; a boundary after each "=" at the same patch count gets 51.8%, and entropy combined with a label-free boundary-dependence signal gets 67.1% (default layout at 26% of bytes: 76.8%). The gap survives adapting BLT-1B to the budget with low-rank fine-tuning (32.9% vs 72.7%, three runs per rule, paired p < 1e-200) and grows with model size in byte models trained from scratch at a 10% budget: at 1M, 12M and 50M parameters, boundary dependence beats entropy on final answers by -1.6, +10.1 and +19.8 points, and at 50M it gets 35.9% of computed results against 13.9% (3 seeds each). BLT's entropy-jump rule helps neither target at 50M. The entropy trigger of Scratchpad Patching is likewise indistinguishable from random scratchpads on final answers (5.6% vs 5.9%, 5 seeds), while answer-start scratchpads give 38.1%. The effect is specific to computed values: copies and lookups gain little, and values the model cannot compute gain nothing. Boundary dependence, the rise in the model's own loss when a patch start is removed, measured per two-byte context, finds these positions without labels: combined with entropy it beats the hand-written rule on computed results.
1. Introduction
The paper identifies a blind spot in entropy-based patching: predictable byte types can precede values that require computation, so tight budgets may skip those results. It proposes boundary dependence as a label-free signal and situates the question among other patching and compute-selection methods.
1. Introduction
- At tight budgets, entropy patching can skip computed results because their byte type is predictable even when their value requires context the local model cannot see.The paper reports that the blind spot persists across trained and adapted models and grows with scale in models trained from scratch.Source: §1 · pp. 1–2
- Boundary dependence measures the model’s loss increase when a patch start is removed, averaged per preceding two-byte context, to find computed results without hand-written rules.The paper also reports that the effect is specific to computed values: copies and lookups gain little, while values the model cannot compute gain nothing.Source: §1 · p. 2
2. Background and related work
- BLT uses next-byte entropy to place patches, while other methods use word boundaries, learned boundaries, or inserted scratchpads triggered by entropy.Prior work evaluated Scratchpad Patching’s entropy trigger on bits per byte and other downstream tasks, but not math answer accuracy.Source: §2 · p. 2
- Unlike static-tokenizer studies of number segmentation, this paper asks which online-decided positions receive a fresh global step.It also distinguishes its patch-level selection question from token selection by excess loss and learned per-token routing.Source: §2 · p. 2
3. Setup
The evaluation tests BLT-1B under changed patch layouts and compares entropy-based and result-aware placement on GSM8K. At equal patch counts, placing boundaries after equals signs substantially improves exact accuracy on computed results over entropy placement.
3. Setup
- BLT-1B accepts explicit patch lengths, enabling inference-time layout changes; the evaluation compares entropy, boundaries after = with remaining starts filled by entropy, dependence, and their standardized sum.The corrected model restores the 512-byte sliding window; tight-budget layouts use thresholds fitted on training problems and applied position by position at test time.Source: §3 · p. 3
- The test set contains 294 GSM8K problems with 796 in-line computed results, scored by exact byte prediction given the true preceding text.Final answers are also reported; paired differences use exact McNemar tests and intervals resample whole problems.Source: §3 · p. 3
4. The blind spot in BLT-1B
- Under a tight patch budget, entropy drops computed results faster than other positions, covering 48% of computed results at 15% and only 16% at 10%.Source: §4 · p. 3
- +32.8 points at 10% and +19.1 points at 15%: boundaries after each = beat entropy on computed-result accuracy at equal patch counts.All layouts ran slightly above budget on test problems; Figure 1 shows entropy missing all six computed results in one solution while entropy plus dependence starts a patch before each.Source: Table 1 · p. 3, §4 · p. 4
5. Training at tight budgets
Training models with tight patch budgets tests whether entropy’s blind spot persists when models learn under those layouts. Across trained models, result-aware boundaries outperform entropy on computed results, and the advantage grows with model size.
Training at tight budgets
- At D = 128 and 1.1M parameters, every entropy seed is below every seed of the other two rules on both accuracies; hand-written and dependence seeds overlap on computed results at 20%.The table reports three seeds for computed results, final answers, and bits per byte.Source: Table 2 · p. 4
- At 10%, BLT’s jump rule gets 64.1% of final answers vs entropy’s 10.4%, but computed-result accuracy is 7.0% vs 7.1%.Computed-result accuracy does not improve with budget for any rule in these models, while MATH answers remain near 3%.Source: §5 · p. 4
- In the Figure 1 example, entropy misses all six computed results, while entropy plus dependence starts a patch before each at equal patch counts.The figure depicts one GSM8K solution at a 10% budget.Source: Figure 1 · p. 5
- The hand-written rule has the lowest bits per byte at every budget, while dependence costs 1.5–7% in bits per byte against entropy.Source: §5 · p. 6
- At entropy’s usual 25% budget, with D = 128, word starts plus math syntax (18.5% of bytes) gets 30.4% of computed results versus entropy’s 11.0%; word starts alone gets 15.5%, and 6-byte stride plus math syntax gets 16.6%.Source: §5 · p. 6
Scaling to 50M parameters
- The scaling study trains 1.1M, 11.7M, and 53.7M parameter models from scratch at a 10% budget on a 1.90 GB corpus.The corpus combines four OpenWebMath shards with GSM8K and MATH training solutions repeated to 10% of its bytes.Source: §5 · p. 6, Table 3 · p. 6
- At 53.7M parameters, dependence gets 78.9% of final answers vs entropy’s 59.0%, and 35.9% of computed results vs 13.9%; its advantage grows with size.The dependence advantage on final answers is −1.6 points at 1.1M, +10.1 at 11.7M, and +19.8 at 53.7M.Source: §5 · p. 6, Figure 2 · p. 7
- At 20%, entropy reaches 80.2% on final answers, matching dependence at 10%, but remains below dependence on computed results, 23.1% vs 35.9%.This comparison indicates that extra patches reach predictable answer starts, while computed-result placement remains a constraint at this scale.Source: §5 · p. 7
6. A label-free trigger: boundary dependence
Boundary dependence uses the model’s loss change around a patch boundary to find computation-sensitive positions without labels. Combined with entropy, it improves computed-result accuracy over both entropy and the hand-written rule, while final-answer gains depend on budget.
6. A label-free trigger: boundary dependence
- Boundary dependence measures the mean loss rise over the next four bytes, averaged by preceding two-byte context; inference uses a causal lookup table.After =, the mean rise is 0.88 nats, versus 0.13 on average.Source: §6 · p. 7
- For small trained models, the original recipe fails because entropy-trained references rarely place boundaries at results and therefore assign little loss to removing one.Using a reference trained with random patch starts and measuring the loss drop from adding a start fixes this failure.Source: §6 · p. 7
- +48.1 points at 10% and +29.5 points at 15%: entropy plus dependence beats entropy on computed results, and exceeds the hand-written rule by +10.4 points at 15%.The combined signal also places boundaries at operands and operators, not only after =.Source: §6 · p. 7
- +11.6 points over entropy on final answers at 10% (p = 3 × 10−5), but +2.4 points at 15% is not significant (p = 0.26).At 15%, entropy alone already reaches most final answers; dependence does not differ significantly from the hand-written rule.Source: §6 · p. 8
7. Adapting BLT-1B to the budget
Low-rank adaptation improves every layout but does not close the computed-result accuracy gap. Fine-tuned final answers saturate, so they no longer distinguish layouts.
7. Adapting BLT-1B to the budget
- The adaptation test trains rank-16 low-rank adapters for 1,500 steps on GSM8K solutions, with three runs per 10% layout.Adapters cover the global transformer and decoder cross-attention.Source: §7 · p. 8
- +39.8 points for entdep over entropy on computed results after adaptation; entdep also exceeds the hand-written rule by +10.0 points.The entdep–entropy difference is pooled over runs, with a +36.3 to +43.2 interval and p = 6 × 10−217.Source: Table 4 · p. 8, §7 · p. 8
- After fine-tuning, final answers saturate near 97% and no longer separate layouts; this is attributed to answers repeating the last computed result and fine-tuning teaching the copy.Source: §7 · p. 8
8. Which positions are affected
The placement advantage is concentrated on values computed from context beyond the local model’s reach. Copies and lookups gain little, while uncomputable targets gain nothing.
8. Which positions are affected
- The test compares four other target types under BLT-1B at 10%, using train-fitted thresholds.Source: §8 · p. 8
- Copies and lookups gain 1–3 points from a patch start, compared with 32.8 points for computed results.For computed results, the compared rule forces a boundary at every target and fills remaining starts with entropy.Source: §8 · p. 8, Table 5 · p. 8
- When BLT-1B cannot compute program-trace values or answer logic questions, placement has nothing to act on; logic answers remain at chance under every layout.Program-trace arithmetic reaches only 20.5% even with default patches.Source: §8 · p. 8
- With a 32-byte local-decoder window, the hand-written rule matches entropy on traces (21.8% vs 20.6%); at 8 bytes it leads by 4 points, but seeds overlap.The authors’ interpretation is that the blind spot requires a computable value whose operands lie beyond the local model’s reach.Source: §8 · p. 9
9. Scratchpad Patching
On final-answer accuracy, entropy-triggered scratchpads perform like random placement, whereas answer-start scratchpads perform much better. This task-level comparison differs from the prior trigger ablation’s bits-per-byte result.
9. Scratchpad Patching
- The comparison uses 16-byte fixed patches, with scratchpads on 6% of bytes and equal scratchpad counts across triggers.The small-model harness uses cross-attention over the partial patch with its mean as query.Source: §9 · p. 9
- The entropy trigger is indistinguishable from random scratchpads on final answers, while answer-start scratchpads give about six times the average accuracy.One of five answer-start seeds failed to train, reaching 8.3%.Source: §9 · p. 9, Table 6 · p. 9
- The final-answer result does not contradict the prior ablation favoring entropy on bits per byte, which can hide failures at a few answer-deciding positions.Source: §9 · p. 9
10. Compute
The section weighs compute savings against accuracy and reports several patch-selection alternatives that failed to improve on hand-placed boundaries.
10. Compute
- 77.3% computed-result accuracy at 59% of default compute matches the default layout’s 76.8%, though dependence alone fails on final answers.At 10% and 15% patch budgets, the dependence-table rule uses 48% and 59% of default compute, respectively.Source: §10 · p. 9, Figure 3 · p. 10
11. Negative results
- Rho-1-style excess-loss selection and simplified learned chunking lose to hand-placed boundaries in every tested setting.Source: §11 · p. 9
- At five seeds, per-digit and operand-aligned arithmetic boundaries no longer outperform matched random controls.They had looked promising at two seeds.Source: §11 · p. 9
- The learned patcher covers 83% of computed-result starts but, when trained into a model at 10%, scores 14.3% / 57.4% versus the table’s 17.1% / 67.4%.On held-out correlation, it matches each domain’s own table.Source: §11 · pp. 9, 11
- On GSM8K, computed results are easiest for the tested token models, while draft-target agreement is lower on computed-value tokens than text.The draft agrees on 53% of computed-value tokens and 86% of text, but is rarely confident on computed values (6.5%).Source: §11 · p. 11
- With cross-attention pooling and a decoder that sees across patch boundaries, most of the earlier synthetic split-computation penalty disappears.The earlier benchmark used sum pooling and a patch-local decoder; all reported results here use the cross-attention model.Source: §11 · p. 11
12. Limitations
The reported conclusions have boundaries in scale, evaluation setting, domain, trigger transfer, checkpoint conversion, and threshold selection.
Scale
- The scale trend comes from three model sizes on one corpus, while BLT-1B uses inference-time layout changes and low-rank adaptation rather than full tight-budget training.At 50M, the hand-written rule has two seeds, and the jump and double-budget results have one each.Source: §12 · p. 11
Teacher forcing
- Accuracies use teacher forcing; BLT-1B gets about 2% of final answers end to end under every layout, and scratch-trained models solve at most 1 of 100 per run.Source: §12 · p. 11
Domain
- The effect is established for arithmetic in worked solutions; code shows a small significant effect, while logic and program traces are inconclusive.For logic and traces, the models lacked the skill or the local model already saw the operands.Source: §12 · p. 11
Label-free trigger
- The dependence table is fitted and tested within the same domain and does not transfer across formats, covering 0% of program-trace values when fitted on GSM8K.On BLT-1B, entropy plus dependence is the best variant, whereas dependence alone is best in trained models.Source: §12 · p. 11
Checkpoint conversion
- BLT-1B results use an upstream fix restoring the missing 512-byte sliding window, which is not yet merged; the released conversion gives the same conclusions with different magnitudes.With the released conversion, results minus entropy at 15% and 10% are +25.0 and +26.6 points.Source: §12 · p. 11
Thresholds
- Original BLT-1B layouts selected the top R% of positions within each test problem, so selection could depend on later bytes; train-fitted thresholds reproduce the conclusions.With the released conversion, train-fitted thresholds give a +25.0-point result-minus-entropy gap at 15%, versus +21.5 points originally.Source: §12 · p. 11
13. Conclusion
The conclusion identifies a mismatch between byte surprise and the global computation needed for predictable-type outputs, and argues that boundary dependence can locate such positions without labels. The declarations provide code and data information and report no funding, competing interests, or human subjects.
13. Conclusion
- Entropy identifies surprising next bytes, but computed outputs can have predictable types and values that require computation; tight-budget entropy patching skips these positions.The conclusion says boundary dependence measures the loss increase when a patch start is removed and finds the positions without labels.Source: §13 · p. 12
- The authors conclude that byte-model compute allocation should follow what the decoder needs from the global model, rather than next-byte surprise.Source: §13 · p. 12
Declarations
- Code, raw logs, and a results registry are available in the project repository, with an archive listed in the declarations.Source: Declarations · p. 12
- The declarations report no funding or competing interests and state that experiments used public datasets and generated text without human subjects.They also say an AI coding agent worked under the author’s direction in the research loop.Source: Declarations · p. 12
Appendix A. Experimental standards
The experimental standards arose after early conclusions failed, and the authors describe safeguards for seeds, paired comparisons, and causality.
Appendix A. Experimental standards
- The partly autonomous research loop produced several wrong early conclusions, including effects that disappeared at five seeds and an effect that was mostly architectural.Other examples included synthetic dependencies outside model context and rules exploiting fixed generator structure.Source: Appendix A · p. 13
- The standards in program.md were created because breaking them produced incorrect results.Source: Appendix A · p. 13
- Trained-model comparisons use at least three seeds except two explicitly marked two-seed comparisons; BLT-1B comparisons use paired tests, and rules are checked for causality.Source: Appendix A · p. 13