PaperlayerView exact revision on arXiv

Source-linked AI summary

How Far Back Should a Transformer Look? Repetition and Copying in Music Sequence Models

Amir Fathi

arXiv:2610.02837v1cs.SD

TL;DR

The paper asks how much history symbolic-music predictors need and what information longer contexts provide. It sweeps trained context lengths and tests repeat-based copying; folk-corpus gains are much larger than those on MAESTRO and MusicNet, with representation-specific scope.

  • Problem

    The paper asks how predictive performance changes with available history and which properties of musical representation make longer context useful.

    Source: §I · p. 1

  • Method

    Separate token and binary-latent predictors are trained at context lengths from 6 to 336 and evaluated on the same target positions, with context interventions testing use of earlier repetitions.

    Source: §I · p. 1

  • Results

    Token-predictor NLL falls 71% on both folk corpora, versus 18% on MusicNet and 9% on MAESTRO; exact copying primarily explains the folk gains.

    Source: §VIII · p. 14

  • Takeaways & Limitations

    Exact-repeat copying is a principal source of long-context gains in these rendered folk scores, whereas the less repetitive MusicNet and MAESTRO results show smaller reductions.

    Source: §VIII · p. 14

    The copying conclusion is specific to rendered score representations with expanded written repeats, and short tunes are excluded when they lack a complete evaluation window.

    Source: §VIII · pp. 14–15

Abstract

from arXiv · show

We investigate how predictive performance depends on the maximum context available to an autoregressive model of symbolic music, and what information long-context models exploit. A small causal Transformer is trained separately at each context length T in {6, 18, 48, 96, 192, 336} and evaluated on the same target positions, both over raw tokens and over non-overlapping binary latent codes. On Nottingham folk tunes, the token predictor's test NLL decreases by 71% (0.986 bits per token) between 6 and 336 tokens, and by 71% on the O'Neill's tunes long enough for the same sweep (129 of 302 test tunes). Most of the long-context gain is explained by exact copying: the improvement appears when an earlier occurrence of the target's 16-token history enters the available context; overwriting that occurrence removes the gain, whereas equally large unrelated corruption does not; and a simple copy baseline recovers 94% of the reduction. A trained 336-token model likewise loses most of this gain when its history is restricted to recent tokens at test time. Because many repeats arise from written repeat signs expanded in the score, this result pertains specifically to these rendered score representations. By contrast, on MAESTRO performances and MusicNet scores, where exact repeats are substantially less frequent, the reduction is smaller (9% and 18%) and is largely attained by 96 tokens. Finally, an audit of an earlier draft that reported saturation at 16 tokens identified split leakage, overlapping latent receptive fields, an averaging predictor, and a per-file tempo grid; we present these as methodological checks for context-length measurements.

I. INTRODUCTION

The paper measures how available history affects symbolic-music prediction and investigates what long-context models use. Its folk-corpus gains are linked to exact repetition, while the introduction also audits methodological errors behind an earlier saturation result.

I. INTRODUCTION

  • The controlled sweep trains a separate predictor at each maximum context length and scores all lengths on the same target positions.The sweep uses raw tokens and binary latent codes, with latent steps representing disjoint six-token spans.Source: §I · p. 1
  • On Nottingham and O’Neill’s, token-predictor test NLL falls 71% from 6 to 336 tokens, while MAESTRO and MusicNet reductions are smaller at 9% and 18%.The folk-corpus reductions are 0.986 and 1.195 bits per token, respectively; the smaller-corpus reductions are 0.296 and 0.445 bits.Source: §I · p. 1
  • Exact copying of repeated passages primarily explains the folk-corpus gains, with results specific to scores whose written repeats are expanded note for note.The introduction reports targeted overwriting, a copy baseline, and test-time history restriction as evidence for this explanation.Source: §I · p. 1
  • An earlier draft’s reported saturation at 16 tokens is audited for methodological failures, and the authors propose checks for future context-length measurements.Additional analyses report abrupt, seed-dependent acquisition of copying and that the earlier averaging predictor fails to exploit long context.Source: §I · p. 2

II. RELATED WORK

  • Prior work studies long-range context in music and language, but the paper’s sweep trains separate models at each maximum context rather than relying only on history truncation.The related-work discussion also notes prior reports of useful distant context and copying from long histories.Source: §II · p. 2

III. DATA

The datasets pair rendered folk scores with recorded performances and classical scores, using source-aware splits and fixed-grid pitch-token representations. These choices shape which repeats are visible and which musical information is retained.

A. Corpora and splits

  • Nottingham and O’Neill’s expand written repeat signs into note-for-note passages, while only 133 of 154 Nottingham and 129 of 302 O’Neill’s test tunes provide a complete evaluation window.The study uses Nottingham melody-only MIDI and excludes O’Neill’s tunes that cannot be parsed or are too short.Source: A. Corpora and splits · p. 2
  • Near-duplicate tunes are grouped before splitting, and the resulting train/test overlap in verbatim non-constant 48-token blocks is at most 0.6% across the four corpora.The reported shares are 0.0% for Nottingham, 0.6% for O’Neill’s, 0.0% for MusicNet, and 0.2% for MAESTRO.Source: A. Corpora and splits · p. 3
  • MusicNet adds aligned classical scores, split by composer and movement; its tokenization follows each file’s tempo map, and only token predictors are trained.The MusicNet test split contains 218 windows, and 2.2% of steps are clamped to its vocabulary range.Source: A. Corpora and splits · p. 3

B. Tokenization

  • Each grid step retains the highest active pitch, yielding a monophonic representation that omits onsets and maps sustained pitches and repeated same-pitch attacks identically.MAESTRO therefore characterizes the upper voice rather than the full polyphonic performance.Source: B. Tokenization · p. 3

C. Transposition

The section describes the corpus-specific transposition augmentation, the disjoint-span binary autoencoder, and predictors designed to access an exact context length.

C. Transposition

  • Nottingham and O’Neill’s training windows receive random pitch transpositions, while validation and test windows are not transposed.Each sampled training window is shifted by a uniformly selected interval from −5 to +6 semitones that keeps pitches within the vocabulary.Source: C. Transposition · p. 3

A. Autoencoder with disjoint spans

  • Each autoencoder latent step represents and reconstructs only its corresponding six-token span, reducing sequence length sixfold without constituting bit-level compression.Test-token reconstruction accuracy is at least 99.923% for the listed corpora; the main sweep uses one autoencoder per corpus, so predictor-seed variation excludes representation variation.Source: A. Autoencoder with disjoint spans · pp. 3–4

B. Predictors with an exact context

  • Two band-limited attention layers give each predictor exactly T tokens of receptive field, with a short first layer supporting local matching and the second spanning the remaining history.The first-layer band is at most 18 tokens or 3 latent steps; this design lets the second layer attend directly to a distant matching position.Source: B. Predictors with an exact context · p. 4

C. The averaging predictor of our earlier draft

The earlier draft’s predictor averages positionwise representations, while the protocol fixes target windows, context lengths, and evaluation conventions. The repeat-lag analysis specifies when an earlier matching history becomes available.

C. The averaging predictor of our earlier draft

  • The earlier-draft predictor averages positionwise representations before its prediction head and has no direct mechanism to privilege recent or specific distant positions.It uses one relative-position attention layer over the context.Source: C. The averaging predictor of our… · pp. 4–5

V. PROTOCOL

  • For Nottingham and O’Neill’s, each 384-token window uses its final 48 tokens as targets, giving every context length the same target positions.The longest context is 336 tokens, exactly the history available to the first target; windows start every 48 tokens and are not padded.Source: §V · p. 5
  • Test NLL is reported in bits per token for token predictors and bits per latent bit for latent predictors, so their values are not numerically comparable.Reported values reflect the specified training protocol; differences across context lengths use paired bootstrap intervals over test pieces.Source: §V · p. 5
  • A 16-token repeat is in view when the nearest matching earlier history and its continuation fit within the context, equivalently when ℓ_t + 16 ≤ T.The analysis bins repeat lags so each bin becomes fully visible at one tested context length.Source: §V · p. 5

VI. RESULTS

The context sweep shows large long-context gains on Nottingham, while repeat-lag analyses associate most of the improvement with earlier matching histories becoming available. On Nottingham, targets with repeats account for most of the aggregate reduction; MAESTRO has fewer repeats and reaches most of its smaller gain by 96 tokens.

A. The context sweep

  • 0.986 bits per token and 0.392 bits per bit are the 6-to-336-token NLL reductions for Nottingham’s token and latent predictors, respectively.The token predictor reaches 0.405 bits per token at 336 tokens; the latent predictor reaches 0.465 bits per bit. Its token NLL initially worsens through 48 tokens, with both paired reduction intervals excluding zero.Source: A. The context sweep · pp. 5–6, Table II · p. 6
  • 0.299 bits is the clamped MAESTRO token predictor’s 6-to-336-token reduction, with 84% attained by 96 tokens.In the unclamped 89-token representation, the predictor reaches 82% of its total reduction by 96 tokens; the latent predictor changes by only 0.0004 bits per bit from 96 to 336 tokens, with an interval including zero.Source: A. The context sweep · p. 6

B. Where the gain comes from

  • 71% of Nottingham target tokens have an earlier occurrence of their 16-token history within 320 tokens, and 97% of these have the same following token.By contrast, 96.9% of MAESTRO targets have no such repeat.Source: B. Where the gain comes from · p. 6
  • NLL improvement emerges as repeat-lag bins enter view: at 336 tokens, Nottingham’s 177-to-320-token bin falls to 0.18 bits per token.At 192 tokens, that bin remains out of view and scores 0.96; the latent predictor shows a similar transition, falling from 0.81 to 0.37 bits per bit for that bin between 192 and 336 tokens.Source: B. Where the gain comes from · p. 6
  • 87% of the token predictor’s and 93% of the latent predictor’s aggregate NLL reductions come from targets with a repeat in view at 336 tokens.On MAESTRO, only 0.8% of targets have a repeat in view at 336 tokens, and 98% of the token predictor’s reduction comes from targets without one.Source: B. Where the gain comes from · p. 6
  • The token predictor reaches 84% of its eventual reduction at 192 tokens, as the share of targets with a repeat in view rises to 53.0%.At 96 tokens, those figures are 12% and 16.5%, respectively; at 336 tokens, the repeat share reaches 71.0%.Source: B. Where the gain comes from · p. 6

C. The predictors use the earlier occurrence

The intervention tests whether predictors depend on earlier occurrences by erasing matching histories while preserving the target’s immediate history and comparing against matched corruption elsewhere. Targeted erasure removes much of the repeated-history advantage, while a simple retrieval baseline recovers most of the Nottingham Transformer’s reduction.

C. The predictors use the earlier occurrence

  • Overwriting the target’s earlier matching histories and their continuations leaves its immediate 16-token history intact; control spans of equal length are overwritten elsewhere.Targets without sufficient non-overlapping positions are excluded; for the latent predictor, the edited span also includes six following tokens before re-encoding.Source: C. The predictors use the earlier… · p. 7
  • Targeted erasure raises Nottingham’s 336-token token-predictor NLL on repeated-history targets from 0.19 to 1.06 bits per token, while the matched control remains at 0.19.For the latent predictor, erasure raises NLL from 0.35 to 0.81 bits per bit while the control remains at 0.35.Source: C. The predictors use the earlier… · p. 7, Table IV · p. 7
  • Replacing the token following earlier occurrences with an injected pitch raises its mean predicted probability from 0.3% to 62%.Across five random draws, the Nottingham erased NLL ranges from 1.058 to 1.060 bits and the control from 0.187 to 0.188; O’Neill’s results show similarly narrow ranges.Source: C. The predictors use the earlier… · p. 7
  • 94% of the Nottingham Transformer’s NLL reduction is recovered by a copy baseline, which reaches 0.469 bits per token versus 1.391 for the 6-token predictor and 0.405 for the 336-token Transformer.The baseline uses no more history than the Transformer; on MAESTRO it recovers 11%.Source: C. The predictors use the earlier… · p. 7

D. One trained model with less history

Restricting a trained 336-token model’s history removes much of its long-context advantage on the folk corpora, especially for targets with visible repeats. On O’Neill’s, targeted erasure and a copy baseline likewise link much of the gain to exact continuation matches.

D. One trained model with less history

  • The test-time restriction interventions were not seen during training, so results at small history lengths reflect both lost distant information and robustness to out-of-distribution inputs.Truncation supplies only the most recent tokens; replacement preserves sequence length while substituting older tokens from another piece.Source: D. One trained model with less history · p. 8
  • On Nottingham, limiting a trained 336-token model to 96 recent tokens raises NLL on repeated-history targets from 0.190 to 0.951 bits, while targets without repeats rise from 0.933 to 1.146.Both truncation and replacement remove most of the model’s long-context advantage; small-R results also reflect robustness to histories unlike those seen in training.Source: D. One trained model with less history · p. 8
  • On O’Neill’s, restricting history to 96 tokens raises token-predictor NLL to 1.460 bits under truncation and 1.491 under replacement, versus 0.497 with full history.The restricted model performs worse than models trained directly at every shorter context length.Source: D. One trained model with less history · p. 8
  • On Nottingham, the latent predictor’s NLL rises from 0.465 bits per bit with full history to 0.837 when only the most recent 96 tokens are preserved.At 192 tokens, both the restricted and independently trained models score 0.571 bits per bit.Source: D. One trained model with less history · p. 8

E. A second folk corpus

  • On O’Neill’s, 73.3% of targets have an in-view repeat at 336 tokens, and repeated-history targets account for 90% of the token-predictor reduction.The repeat share is 71.0% on Nottingham; on O’Neill’s the latent-predictor reduction attributable to repeated-history targets is 91%.Source: E. A second folk corpus · p. 9
  • On O’Neill’s, targeted erasure raises the 336-token token predictor’s NLL on repeated-history targets from 0.22 to 1.24 bits per token, while the matched control remains at 0.22.The copy baseline recovers 97% of the token predictor’s reduction, and 97% of repeated-history targets exactly match the earlier continuation.Source: E. A second folk corpus · p. 9

F. A corpus in between

MusicNet has fewer exact repeats than the folk corpora, and most of its context-related NLL reduction is reached by 96 tokens. Additional analyses show seed-dependent acquisition of copying at 96 tokens and that an averaging predictor can miss useful long-range repetition.

F. A corpus in between

  • By 96 tokens, MusicNet reaches 97% of its 6-to-336-token NLL reduction; its 336-token model is 0.017 bits worse than the 192-token model.The reduction is 18%, from 2.512 to 2.067 bits per token, and only 10.5% of targets have an in-view 16-token repeat at 336 tokens.Source: F. A corpus in between · p. 9
  • On MusicNet, the copy baseline’s gain exceeds the reduction attributed to 16-token repeats because it also retrieves 8-token matches; this does not establish that the Transformer uses the same mechanism.The MusicNet sweep includes three seeds and 43 test pieces, so its bootstrap intervals are comparatively wide.Source: F. A corpus in between · pp. 9–10

G. Copying depends on training

  • At 96 tokens, 6 of 10 fixed-budget runs transition to lower validation NLL after plateaus, while four finish between 1.34 and 1.40 bits per token.All 10 runs achieve their best validation NLL between epochs 135 and 193, and several are still improving at epoch 200.Source: G. Copying depends on training · p. 10
  • At 192 tokens, all 10 fixed-budget runs transition between epochs 34 and 60 and finish between 0.55 and 0.56 bits per token.The authors conclude that acquisition of the copying solution depends strongly on initialization and training duration at 96 tokens, but all tested runs reach it at 192 tokens.Source: G. Copying depends on training · p. 10
  • Across three experimental setups, the Nottingham token predictor retains the qualitative pattern of small short-context changes and a larger reduction by 336 tokens.The setups vary tokenizer, window, autoencoder, and transposition together, so values are comparable within a row.Source: G. Copying depends on training · p. 10, Table V · p. 11

H. An averaging predictor does not use long context

  • On Nottingham, the averaging predictor degrades to 0.891 bits per bit at both 96 and 336 tokens, while the causal latent predictor improves to 0.465 at 336.The comparison indicates that the mean-pooled architecture fails to exploit long-range repetition; the authors suggest averaging makes retrieval of a specific distant continuation difficult.Source: H. An averaging predictor does not use… · pp. 10–11, Table VI · p. 11

I. Autoencoder seeds

Across Nottingham autoencoder seeds, latent NLL curves retain the same qualitative late-context decline, while generated streams increasingly repeat passages with longer context. Generation distances depend on the real-stream cut and sampling temperature, so they do not establish a match to real music or measure variety.

I. Autoencoder seeds

  • All three autoencoders retained the curve shape, with 6-to-336-token reductions of 45%, 48%, and 43%; representation-seed SD exceeded predictor-seed SD at five of six contexts.Representation-seed SD ranged from 0.008 to 0.024 bits per bit, versus 0.002 to 0.015 across predictor seeds.Source: §I · p. 11

J. Generation

  • Each of 100 test tunes primes a 480-token generation; temperatures are tuned on validation primes, with separate τ=1 runs for cross-context comparison.Latent streams are decoded by argmax, so all randomness comes from the predictor.Source: J. Generation · p. 11
  • At 192 and 336 tokens, token-stream distances are 0.25 and 0.29 with repetition rates 0.41 and 0.41; latent distances are 0.88 and 0.68.Long-context models generated recurring passages, unlike short-context models at their chosen temperatures.Source: Table VII · p. 12, J. Generation · p. 13
  • Encoded real streams score distance 0.07 with 99.94% token accuracy, inside the real band; bootstrap bands vary depending on where real streams are cut.The reported band for start-cut test tunes is 0.03 to 0.17; last- and random-segment bands differ.Source: J. Generation · p. 12
  • Only one of 36 generated sets fell within the end-cut real band, and τ=0.1 latent outputs emphasize likely continuations rather than continuation variety.The closest generated set scores 0.19; the authors do not claim a match because whether it counts as matching real music depends on the real-stream cut.Source: J. Generation · pp. 12–13
  • At τ=1, token distance falls from 2.83 at 6 tokens to 1.69 at 336, while latent streams remain far from the reference and almost never repeat passages.The token curve is not monotonic; only 192- and 336-token token models produce recurring passages at this temperature.Source: J. Generation · p. 13

VII. FAILURE MODES THAT CAN PRODUCE SPURIOUS CONTEXT SATURATION

An audit found several plausible sources of bias in an earlier result showing context saturation at 16 tokens. Because the audit changed multiple components jointly, it does not assign a specific share of that result to any one failure mode.

VII. FAILURE MODES THAT CAN PRODUCE SPURIOUS CONTEXT SATURATION

  • The audit identified four prediction-experiment failure modes and one generation-evaluation failure, but did not isolate their individual contributions to the earlier curve.The authors present them as plausible sources of bias to check independently.Source: §VII · p. 13

Leaky splits

  • In the earlier Nottingham split, 92.4% of test sequences had another rendering of the same tune in training, and 58.4% of non-constant 48-token test blocks occurred verbatim in training.Earlier MAESTRO and MusicNet test chunks also shared source performances or recordings with training chunks.Source: §VII · p. 13

Latents that overlapped their targets

  • Earlier latent steps encoded tokens that overlapped the target step, so added context could reveal target-overlapping information rather than strictly past history.At block starts, where context was strictly in the past, the earlier predictor did not improve with context.Source: §VII · p. 13

An averaging predictor

  • The earlier mean-pooled v5 readout failed to benefit from long context even on corrected data.Source: §VII · p. 13

A per-file tempo estimate

  • The per-file tempo estimate stretched the median Nottingham tune to 1.64 times its score representation and mapped the same melody to different grids across files.A separate receptive-field error in the first rerun limited effective copying to roughly half the advertised context, with the 336-token model strongest on repeats up to 168 tokens back.Source: §VII · pp. 13–14

A miscalibrated real-data baseline

  • The earlier generation baseline of 0.36 came from one unheld-out set at the 99.7th percentile of random 100-sequence draws.The revised evaluation uses a bootstrap reference band from held-out music.Source: §VII · p. 14

VIII. DISCUSSION AND LIMITATIONS

Long-context gains on the folk corpora track exact token-sequence repetition, but the relationship varies across datasets and representations. The conclusions are bounded by the tested context range, rendered-score settings, and evaluation subsets.

What determines the gain from long context

  • 94% and 97% of the long-context gain is recovered by the copy baseline on Nottingham and O’Neill’s, respectively, supporting exact-passage retrieval as the main source of folk-corpus benefit.The baseline adds three fitted interpolation weights to the 6-token predictor.Source: §VIII · p. 14
  • 71.0%, 73.3%, 10.5%, and 0.8% of targets have a repeat in view at 336 tokens for Nottingham, O’Neill’s, MusicNet, and MAESTRO, respectively; their reductions covary descriptively, not as a statistical test.A common rule across test positions with at least 336 preceding tokens gives similar repeat-prevalence estimates.Source: §VIII · p. 14
  • On MAESTRO, 98% of the 6-to-336-token reduction comes from targets without an exact in-view repeat, and most improvement is attained by 96 steps, or 12 seconds.The information responsible for that residual gain is not identified; MAESTRO gains beyond 336 tokens are outside this study’s evaluation.Source: §VIII · p. 14

Latents

  • Latent predictors benefit from long context on Nottingham and can benefit when earlier occurrences use different latent codes, but latent and token NLLs cannot be compared directly.The factorized binary-code predictor does not provide the token-level likelihood needed for that comparison.Source: §VIII · p. 14

Scope

  • The study tests four corpora, monophonic top-line representations, contexts up to 336 tokens, and approximately 2.5M-parameter predictors; it does not establish a generally sufficient or optimal context length.At 96 tokens, only about half of Nottingham runs discover the copying solution within 200 epochs; exact 16-token recurrence is measured, while approximate repetition is not.Source: §VIII · p. 14
  • The folk-corpus results characterize tunes long enough to fill the evaluation window, with stronger selection toward repetition for O’Neill’s.Many exact repeats come from written repeat signs expanded in rendered scores, so the comparison with MAESTRO also contrasts score and performance representations; 129 of 302 O’Neill’s test tunes contribute.Source: §VIII · pp. 14–15

IX. CONCLUSION

With controls for leakage and receptive-field mismatches, the study finds large long-context gains on folk rendered scores, where exact repetition explains much of the benefit. Gains are smaller on MAESTRO and MusicNet, and the findings do not establish a universal context requirement.

IX. CONCLUSION

  • 71% (0.986 bits per token) and 71% (1.195 bits) are the Nottingham and O’Neill’s token-predictor NLL reductions from 6 to 336 tokens; most folk-corpus gains are attributable to exact repetition.O’Neill’s result applies to test tunes long enough for evaluation; targeted erasure, a copy baseline, and test-time history restriction support the copying account.Source: §IX · p. 15
  • 0.296 and 0.445 bits are the total token-predictor gains within 336 tokens on MAESTRO and MusicNet, where gains arise primarily without exact 16-token repeats and are mostly attained by 96 tokens.The conclusion reports that contexts beyond 336 tokens and a universal context requirement are not established.Source: §IX · p. 15
  • The earlier 16-token saturation result coincided with split leakage, target-overlapping latent receptive fields, an averaging predictor, and a per-file tempo grid; the paper recommends testing for each failure mode.The experiments use source-disjoint splits with near-duplicate control and predictors matched to intended receptive fields.Source: §IX · p. 15
  • Table VIII covers attention-band widths and parameter counts by context length, while Table IX reports validation-fitted copy-baseline mixing weights.The captions specify that band widths sum to the context and note the MAESTRO token predictors’ 2.53M parameters at 336 tokens.Source: Table VIII · p. 15, Table IX · p. 15

APPENDIX REPRODUCIBILITY DETAILS

The reproducibility details specify split construction, random seeds, and procedures for copying baselines, random edits, and generation. They also identify the software and hardware environment and its reproducibility caveat.

Availability

  • Code, split manifests, and outputs from every experimental run are available from the author on request; one analysis script generates the reported numbers, tables, and figures.Source: APPENDIX REPRODUCIBILITY DETAILS · p. 15

Splits

  • Pieces are grouped before splitting, with linked groups merged by connected components; groups are then assigned 70%/15%/remainder to training, validation, and test using seed 42.MAESTRO uses the official split; Nottingham and MusicNet apply the near-duplicate rule again after final tokenization.Source: APPENDIX REPRODUCIBILITY DETAILS · p. 15

Predictors

  • Main configurations use seeds 42, 7, and 0; fixed-budget runs use 0–9, and Nottingham autoencoders use 42, 1, and 2.Each seed initializes both NumPy and PyTorch random-number generators.Source: APPENDIX REPRODUCIBILITY DETAILS · p. 15

Copy baseline

  • The copy baseline searches for the nearest earlier 32-token history match, backing off to 16 and then 8 tokens, and interpolates the copied next-token probability with the 6-token predictor.Each history-length interpolation coefficient is selected independently on validation targets matched at that length.Source: APPENDIX REPRODUCIBILITY DETAILS · pp. 15–16

Random edits

  • Random-edit sampling uses context-length-based seeds for token and latent predictors; history replacement uses one donor window per test window with seed 0.Additional independent draws add the draw index to the seed.Source: APPENDIX REPRODUCIBILITY DETAILS · p. 16

Generation

  • Generation temperatures are searched in different ranges for latent and token models, with the search extended until the best validation temperature has evaluated neighbors on both sides.The initial ranges are 0.3–0.8 plus 1.0 for latent models, and 0.6–1.2 for token models, in increments of 0.1.Source: APPENDIX REPRODUCIBILITY DETAILS · p. 16

Software and hardware

  • Experiments use Python 3.10.19, PyTorch 2.10.0, and NumPy 2.2.6 on an Apple M5 GPU through the MPS backend, which is not bit-reproducible.The listed environment is macOS 26.6 with 24 GB of memory.Source: APPENDIX REPRODUCIBILITY DETAILS · p. 16

Related papers

Loading 2610.02837v1…