PaperlayerView exact revision on arXiv

Source-linked AI summary

Proxy Compression for Language Modeling

Lin Zheng, Xinyu Li, Qian Liu, Xiachong Feng, Lingpeng Kong

arXiv:2602.04289v2cs.CLcs.LG

TL;DR

Compressed inputs make language-model training efficient but couple the model to a compressor at inference. Proxy compression jointly trains on compressed views and raw bytes, substantially outperforming pure byte-level baselines under fixed compute budgets and matching or surpassing tokenizer-based models at scale.

  • Problem

    Compressed representations reduce sequence length for efficient training, but the paper asks whether this efficiency can be retained without hard-wiring the compressor into the model interface.

  • Method

    Proxy compression jointly trains one language model on externally compressed views and raw UTF-8 bytes, discarding the compressor at inference so the model operates on raw bytes alone.

  • Results

    With only about 10% of training samples in raw UTF-8, proxy-trained models outperform pure raw-byte baselines under fixed compute budgets and match or surpass tokenizer-based baselines at scale.

  • Takeaways & Limitations

    Proxy compression decouples compressed-training efficiency from the inference interface while enabling transfer from compressed inputs to raw-byte inference.

  • Takeaways & Limitations

    Evaluation primarily focuses on code language modeling, while larger-scale natural-language, multilingual, and mixed-domain validation remains future work.

Abstract

from arXiv · show

Modern language models are trained almost exclusively on token sequences produced by a fixed tokenizer, an external lossless compressor often over UTF-8 byte sequences, thereby coupling the model to that compressor. This work introduces proxy compression, an alternative training scheme that preserves the efficiency benefits of compressed inputs while providing an end-to-end, raw-byte interface at inference time. During training, a single language model is jointly trained on raw byte sequences and compressed views generated by external compressors; through the process, the model learns to internally align compressed sequences and raw bytes. This alignment enables strong transfer between the two formats, even when training predominantly on compressed inputs that are discarded at inference. Extensive experiments on code language modeling demonstrate that proxy compression substantially improves training efficiency and significantly outperforms pure byte-level baselines given fixed compute budgets. As model scale increases, these gains become more pronounced, and proxy-trained models eventually match or surpass tokenizer approaches, all while operating solely on raw bytes and retaining the inherent robustness of byte-level modeling. Our code is available at https://github.com/LZhengisme/proxy-compression.

1. Introduction

Proxy compression trains one language model on mixed compressed and raw UTF-8 representations, preserving compressed-input efficiency while enabling raw-byte inference without a permanently hard-wired compressor. Experiments show strong transfer to raw-byte inference, outperforming byte-level baselines and closing the gap with tokenizer-based models as scale increases.

  • Motivation: Compressed inputs reduce sequence length and enable efficient training, but fixed external compressors couple the modeling stack to their interfaces and artifacts.Every input and output must pass through the compressor, preventing strictly end-to-end raw-data modeling.
  • Method: Proxy compression mixes externally compressed sequences with raw UTF-8 counterparts during training, then discards the compressor at inference.The scheme trains a single model with next-symbol prediction over both representations and uses representation-specific sentinels.
  • Results: As model scale increases, proxy-trained models progressively close the gap with tokenizer-based baselines and can match or surpass them.Byte-level models instead exhibit a persistent or widening gap, whereas proxy-based models improve relative performance across scales.
  • Mechanism: The model predicts in compressed and raw code spaces, implicitly learning an internal mapping that treats compressed sequences as informative hints rather than substitutes for bytes.This cross-representation alignment supports transfer from compressed training views to raw-byte inference.
  • Proxy compressors: Tokenizer-based and neural proxy compressors transfer effectively, whereas generic gzip compression fails to transfer effectively.The paper systematically studies tokenizer-based compression, arithmetic-coded neural proxies, and gzip.
  • Results: Models seeing only about 10% of training samples in raw UTF-8 outperform pure raw-byte baselines under fixed compute budgets while consuming substantially more compressed data.The reported advantage is attributed to strong transfer from compressed training to raw-byte inference.

2. Proxy Compression

Proxy compression jointly trains one autoregressive model on raw UTF-8 bytes and compressed representations, while reserving inference for raw bytes. The framework uses representation markers, optional paired views, and several offline compressors, including tokenization, neural arithmetic coding, and gzip.

  • Mixed-Representation Training: Mixed-representation training randomly presents each sample as raw bytes or a compressed sequence within fixed-length contexts, requiring only preprocessing changes.A single standard autoregressive model handles both formats, and contexts may contain both representations.
  • Byte-level Inference: At inference, the model uses exclusively raw UTF-8 bytes, so the proxy compressor can be discarded after training.This preserves a universal byte-level interface while using compression only for training efficiency.
  • Alignment Mechanisms: Format sentinel tokens distinguish raw and compressed sequences, while optional in-context translation pairing trains the model to predict one representation conditioned on the other.Pairing concatenates both views of the same sample with randomized ordering and is enabled by default during warm-up.
  • Proxy Compressor Instantiations: Standard tokenization serves as a proxy compressor because it produces discrete fixed-vocabulary sequences through entirely offline processing.This differs conceptually from conventional tokenizer-based language-model training because tokenization is used only as a training view, not as the inference interface.
  • Proxy Compressor Instantiations: Neural arithmetic-coding proxies use entropy-based segmentation to compress high-entropy-bounded segments independently in parallel, making large-scale processing practical.The neural compressor is a roughly 40M-parameter byte-level model, and its compressed representations are fuzzy: decoding is generally non-unique but typically varies in formatting details.
  • Mixing Training Schedule: The default schedule sets r = 0.9, presenting 90% of samples compressed and 10% as raw bytes, with pairing enabled during the first 10k warm-up steps.During warm-up, r increases linearly from 0.4 to 0.9; tokenizer-based and neural proxies achieve roughly 2.9× and 2.6× compression rates, respectively.

3. Experiments

Experiments on code show that proxy-compressed training transfers effectively to raw-byte inference, with gains that increase with model scale and preserve byte-level robustness. Results also depend on the proxy: neural and tokenizer proxies transfer strongly, whereas gzip fails to transfer.

  • Experimental setup: Experiments use EvaByte language models from 0.5B to 14B parameters trained primarily on Python and full GitHub code from RefineCode.The study centers on compute-matched comparisons across input representations and uses roughly 270 GB of Python source plus approximately 3.3 TB of multilingual code.
  • Transfer scales with model size: With raw bytes shown in only 10% of training samples, proxy-trained byte models track or outperform fully byte-level baselines above 1.5B parameters.The advantage of proxy training becomes more pronounced as model size increases.
  • Data versus compute efficiency: At 14B parameters, proxy compression performs comparably to tokenizer baselines under matched FLOPs while retaining byte-level data efficiency and substantially outperforming tokenizer baselines under matched data.At 0.5B–7B, proxy models generally lie between the byte-level and tokenizer baselines and progressively narrow the gap.
  • Longer training horizons: After 320B sequence symbols, proxy models outperform byte baselines at 1.5B parameters and close the gap with or often match or exceed tokenizer models at 7B.These results show that the transfer pattern remains effective under longer training horizons.
  • Transfer and translation: Mixed-representation training induces substantial in-context transfer without explicit pairs, while Always-on pairing yields over 95% oracle-translation pass@1 for tokenizer and neural compressors.Under No pairs, oracle-translation pass@1 reaches approximately 46% for tokenizer and approximately 33% for neural compression; Warmup-only retains transfer after pairing is removed.
  • Proxy comparison: Gzip proxies degrade downstream performance as their training proportion increases, whereas byte-level training with 0% gzip consistently outperforms gzip-mixed variants.The passage attributes this to gzip’s unstable and unstructured outputs; neural compression lies between stable tokenization and unstable gzip.
  • Robustness and inference interface: Proxy-trained models improve robustness: neural proxies achieve the highest robust pass@1, while tokenizer proxies improve over the tokenizer baseline and byte models score 18.7 versus 14.9 RP.Tokenizer-proxy inference can use raw bytes or compressed tokens, and byte-level inference often matches or outperforms token-level inference despite only 10% raw-byte training exposure.

4. Related Work

Prior work studies tokenization, alternative compressed representations, and byte-level models to improve language-model input efficiency. This work instead uses compressed representations as training-time proxies while retaining an unmodified byte-level interface at inference.

  • Input Representations for Language Models: Tokenization converts raw text into shorter discrete-token sequences using schemes such as Byte-Pair Encoding, WordPiece, UnigramLM, and SentencePiece.Related studies analyze tokenization pipelines, non-canonical behavior, vocabulary scaling, and compression improvements.
  • Input Representations for Language Models: Alternative compressed representations include gzip-based compressors, arithmetic coding with equal-information windows, semantic units, morphology-driven bytes, and pixel-rendered text.Morphology-driven byte representations are reported as improving efficiency and fairness in multilingual settings.
  • Byte-level Models: Byte-level models directly consume byte sequences, with prior work introducing architectural downsampling, adaptive compression, and multimodal applications.Adaptive methods dynamically adjust representation granularity based on input content.
  • Distinction from Prior Work: Unlike prior approaches, this work uses compressed representations as training proxies while retaining a byte-level inference interface without architectural modifications.The compressed views are used during training, whereas inference remains byte-level.

5. Conclusion · Appendix

Proxy compression decouples compressed-training efficiency from the inference-time interface by jointly training models on raw inputs and externally compressed views. The evaluation supports substantial gains over pure byte-level baselines, while limitations remain in broader validation and byte-level deployment efficiency.

  • 5. Conclusion: Proxy compression is a mixed-representation training scheme that separates compressed-training efficiency from the inference-time interface.It jointly uses raw inputs and compressed views produced by external compressors.
  • 5. Conclusion: Joint training teaches models to align raw and compressed representations.This alignment enables transfer from compressed inputs to raw-byte inference.
  • 5. Conclusion: Extensive code-language-modeling experiments show that proxy-trained models substantially outperform pure byte-level baselines.The supplied conclusion passage states this result without reporting a numerical value.
  • 5. Conclusion: The evaluation primarily focuses on code language modeling.Natural-language results provide initial evidence beyond code, but broader validation remains future work.
  • 5. Conclusion: Larger-scale natural-language, multilingual, and mixed-domain validation remains future work.The passage identifies these areas as limits of the current evaluation.
  • 5. Conclusion: Raw-byte inference makes deployment efficiency dependent on the underlying byte-level model architecture.EvaByte mitigates this cost in the reported experiments, but proxy compression itself does not remove the general challenge.

A. Token-level Compression Ratios for Proxy Compression · B. Implementation Details of Proxy Compressors

Proxy compression estimates token-level efficiency from sample-level mixing and compressor rates, then calibrates translation-pair warmups to preserve the intended compressed/raw fraction. Translation pairing reduces efficiency, so it is used only during warmup before being disabled.

  • A. Token-level Compression Ratios for Proxy Compression: Sample-level mixing does not directly determine the context’s compressed-token ratio, motivating a separate token-level estimate.The actual ratio varies because documents are compressed probabilistically while samples have different lengths.
  • A. Token-level Compression Ratios for Proxy Compression: The compressor rate is defined as C = Lb/Lt, comparing average raw-byte and compressed-sample lengths.The analysis fixes the total number of symbols per training step at M.
  • A. Token-level Compression Ratios for Proxy Compression: Compression-based models approximately consume C× more samples than raw-byte models under fixed-symbol training steps.This follows from comparing the number of samples required when average sample lengths differ by the compression rate.
  • A. Token-level Compression Ratios for Proxy Compression: At r = 0.9 with the OpenCoder tokenizer and C ≈3.7, the actual token-level compression rate is approximately 2.91.The effective rate accounts for both compressed and uncompressed samples in the mixture.
  • A. Token-level Compression Ratios for Proxy Compression: Translation pairs duplicate each input, further reducing the effective compression rate and requiring calibrated mixing rates.For r > 0.5, translation pairs mix with compressed samples; for r ≤0.5, they mix with raw samples.
  • A. Token-level Compression Ratios for Proxy Compression: With the stated configuration, translation pairing yields an actual compression rate of approximately N′ ≈2.62Nb, slower than without translation.The method therefore uses translation pairing only as an initial warmup and disables it later to maximize unique data per step.
  • A. Token-level Compression Ratios for Proxy Compression: The default warmup linearly anneals r(i) from 0.4 to 0.9 over the first 10000 steps.Averaged over this schedule, training processes approximately 1.38× as many samples per step as a pure raw-byte model.

B.1. Tokenizer-based Proxy Compression … C.1. Model Architectures

The paper compares tokenizer-based and neural proxy-compression designs, finding direct token indices offer the strongest tokenizer trade-off while entropy-segmented neural compression supports efficient offline processing but cannot decode. Architectural comparisons further show EvaByte achieves lower BPB while matching OpenCoder’s downstream performance at comparable token-based model sizes.

  • B.1. Tokenizer-based Proxy Compression: Direct token indices achieve the highest compression rate, 3.7× with OpenCoder, and the best downstream performance among tokenizer representations.Table 5 evaluates 1.5B-parameter models trained on 100B Python-subset sequence symbols.
  • B.1. Tokenizer-based Proxy Compression: Token bytes preserve both raw and compressed representations in byte space, using B = 3 bytes per OpenCoder token, but direct token indices perform better.For OpenCoder, V = 96,640 and the smallest B satisfying 256^B ≥ V is 3.
  • B.1. Tokenizer-based Proxy Compression: 3.5× compression from double-byte representations exceeds simple bytes’ 1.7×, yet both underperform direct token-index representations.The double-byte variant expands the model vocabulary to 65,536 so each token ID maps to one symbol.
  • B.1. Tokenizer-based Proxy Compression: Encoding strategies provide only marginal token-byte improvements, and all remain inferior to direct token-index representations.Gray coding slightly improves performance, while Huffman coding yields only marginal compression gains over fixed-length encoding.
  • B.1. Tokenizer-based Proxy Compression: Line-separated and SuperBPE pre-tokenization reach 2.3× and 2.9× compression, respectively, versus 1.7× for default, but do not match token-index performance.Unigram tokenization shows no significant transfer-performance difference at matched compression rates.
  • B.2. Neural Proxy Compression: Neural proxy compression combines a ∼40M-parameter byte-level Transformer with arithmetic coding and packs 16-bit chunks into symbols, producing vocabulary size V = 65,536.The compressor operates over a fixed alphabet of 256 bytes and produces conditional next-byte distributions for arithmetic coding.
  • B.2. Neural Proxy Compression: Entropy-based segmentation and GPU/CPU pipelining enable offline parallel compression at 0.57 GB/hour per process, compared with 0.005 GB/hour for naive sequential processing.The pipeline also uses equal-information windows, arithmetic coding, caching, and fixed-bit symbol packing.
  • C. Additional Architectural, Training, and Evaluation Details: Neural compression cannot decode because probability tables are absent, segment boundaries are unknown, and predicted compressed symbols do not uniquely determine raw bytes.This limitation distinguishes neural compression from invertible tokenization and motivates its use as a training-time proxy.

C.2. Training Configuration … D.1. Full Results of Downstream Transfer

The paper specifies representation-aware training configurations, evaluates code generation with pass@k metrics, and reports downstream transfer results across model sizes and input representations. Full results include HumanEval, MBPP, and EvalPlus variants, with HumanEval-Plus pass@10 following the pass@1 trend.

  • C.2. Training Configuration: Table 8 reports peak learning rates for different input representations and model sizes.The listed learning rates are peak values from each training schedule.
  • C.2. Training Configuration: Model parameters use truncated-normal initialization, except embeddings, with gradient clipping and AdamW optimization settings specified for stable large-scale training.Initialization uses standard deviation 0.02 for most parameters and 1.0 for embeddings; gradient norms are clipped to 1.0, with weight decay 0.1, β1 = 0.9, β2 = 0.95, and ϵ = 1e−15.
  • C.2. Training Configuration: Python-subset models use a fixed effective batch size of 2M sequence symbols across tokens, bytes, and compressed formats.Because representations consume different amounts of raw data per batch, equal sequence-symbol batch sizes do not imply equal raw-data consumption.
  • C.2. Training Configuration: Python-subset models train for 50000 steps with cosine decay, 500-step linear warm-up, and decay to 10% of peak learning rate.Full-GitHub-corpus models instead use 4M sequence symbols for 80000 steps with a constant learning rate and 2000-step warm-up.
  • C.2. Training Configuration: Stateful compressors process complete raw documents before packing, and compressor state never crosses document boundaries.Raw or compressed documents receive format sentinels before fixed-length packing; neural compression also resets state within documents at fixed bit budgets.
  • C.3. Evaluation Protocols: Evaluation emphasizes downstream code generation, reporting pass@k on HumanEval, MBPP, and their EvalPlus variants.Pass@1 uses greedy decoding, while pass@10 samples 20 outputs at temperature 0.2 with nucleus sampling.
  • D.1. Full Results of Downstream Transfer: The paper prioritizes downstream task performance over validation BPB because BPB comparisons across representations are biased, especially for mixed compressor, byte, and token training.The reported full results list pass@1 on HumanEval, MBPP, and EvalPlus variants, while HumanEval-Plus pass@10 shows a consistent trend with pass@1.

D.2. Additional Analyses of In-context Transfer

The analyses probe in-context transfer for tokenizer-based and neural compressors under controlled pairing schedules. Transfer remains stable with continual explicit pairing, whereas removing pairs causes decay, especially for neural compression, and larger models do not eliminate this dependence.

  • Experimental setup: The evaluation compares tokenizer-based and neural compressors at 1.5B and 7B across training checkpoints, including schedules with explicit compressed–raw translation pairs.All variants share an overall ratio r = 0.9 of compressed inputs, and oracle-translation pass@1 is tracked at 10k, 20k, 30k, 40k, and 50k steps.
  • Structural differences: Under Warmup-only, tokenizer-proxy accuracy falls from 90.9% to 31.1% before recovering to 45.7%, while neural-proxy accuracy falls to 14.6% before recovering to 38.4%.The differing stability reflects structural properties of the proxy compression types.
  • Structural differences: Tokenizer compression provides a stable global byte-pattern anchor, whereas neural compression produces context-dependent symbols without a fixed global mapping.The same neural-compressed pattern may yield different sequences depending on local context, introducing structured fuzziness.
  • Pairing supervision: Under ALWAYS-ON, both proxy compressors reach very high translation pass@1 because explicit translation pairing remains available, but Warmup-only removes that supervision after warmup.Figure 11 distinguishes continual translation pairs from abrupt removal after 10k steps.
  • Model scale: At 20k steps under Warmup-only, the 7B neural-proxy model retains 73.8% accuracy versus 39.0% for the 1.5B model, yet both rapidly decay after pair removal.The result indicates that larger models do not substitute for explicit pairing supervision.

D.3. Additional Results of Neural Proxy Compression · D.4. Additional Results of Robustness Evaluation on ReCode

Neural-compressor collisions are highly structured, with most colliding chunks sharing nearly identical prefixes and differing mainly in short suffixes or formatting. On ReCode robustness evaluation, formatting perturbations create the largest representation gap, while byte-level and proxy models remain stable or improve.

  • D.3. Additional Results of Neural Proxy Compression: Over 90% of neural-compressor collisions have LCP ratios above 0.8, indicating near-identical chunks that differ mainly in short suffixes.The LCP ratio is the shared-prefix length divided by average chunk length.
  • D.3. Additional Results of Neural Proxy Compression: Case 1 contains 61 variants, with nearly all differing only in trailing whitespace and indentation.The representative collision cases also include URL suffixes, boilerplate completions, and formatting differences.
  • D.3. Additional Results of Neural Proxy Compression: The collision examples include 4 URL-suffix variants, 11 main-function-boilerplate variants, and 4 function-call-formatting variants.These cases illustrate ambiguity caused by suffixes, completions, and formatting rather than arbitrary differences.
  • D.4. Additional Results of Robustness Evaluation on ReCode: ReCode evaluates robustness on HumanEval using Function, Format, Syntax, and Docstrings perturbation families, alongside nominal pass rate.Each original problem has s = 5 randomly perturbed variants per perturbation type; evaluation uses greedy decoding with n = 1 and k = 1.
  • D.4. Additional Results of Robustness Evaluation on ReCode: Robustness is summarized by RP, RD, and RR, where higher RP and lower RD indicate better worst-case stability, while negative RD values indicate perturbation gains.RDs@k = (pass@k − RPs@k)/pass@k measures relative degradation, and RRs@k captures stability through flip rate.
  • D.4. Additional Results of Robustness Evaluation on ReCode: Format perturbations exhibit the largest representation gap: tokenizer models degrade severely, whereas byte-level and proxy models remain stable or even improve.The results indicate that partial raw-byte exposure during training improves robustness to surface-level formatting noise.

D.5. Additional Analyses on Data Representations · D.6. On Document-boundary Attention Masking

The analyses compare representations from BPE tokens to individual bits under matched model size and compute, finding complementary compute- and data-efficiency trends. Attention masking experiments indicate that cross-representation transfer primarily comes from shared parameters rather than in-context interactions.

  • D.5. Additional Analyses on Data Representations: D.5. Additional Analyses on Data Representations examines BPE tokens, double-bytes, bytes, half-bytes, double-bits, and bits with matched parameter counts and training FLOPs.The representation spectrum ranges from token-level inputs to 1-bit inputs.
  • D.5. Additional Analyses on Data Representations: Under fixed compute, representations processing more data per FLOP achieve lower validation BPB, indicating greater compute efficiency.This is one of the two complementary trends reported in Figure 17.
  • D.5. Additional Analyses on Data Representations: Under fixed data, lower-level representations generally outperform higher-level ones, indicating greater data efficiency and aligning with prior byte-level findings.The passage notes that this trend does not extrapolate straightforwardly to the finest granularity.
  • D.5. Additional Analyses on Data Representations: The representation study therefore shows that compute efficiency and data efficiency favor different granularities.More data per FLOP helps under fixed compute, whereas lower-level inputs generally help under fixed data.
  • D.6. On Document-boundary Attention Masking: D.6. On Document-boundary Attention Masking asks whether cross-representation transfer comes from in-context attention between packed samples or shared model parameters.Proxy compression training concatenates and packs samples that may use different representations into fixed-length contexts.
  • D.6. On Document-boundary Attention Masking: Document-boundary attention masking improves performance by restricting attention to within-document tokens rather than allowing cross-document attention.The comparison contrasts standard packing with document-boundary masking.
  • D.6. On Document-boundary Attention Masking: The masking result suggests that cross-representation transfer primarily arises from shared parameters, not in-context interactions between representations.This conclusion follows from improved performance when cross-document attention is prevented.

D.7. Additional Results on Data Efficiency versus Compute Efficiency … D.11. Prompt Boundary Robustness

Across extended analyses, proxy compression becomes more competitive with scale, benefits from warmup-only randomized pairing and additional compressed data, and preserves robust raw-byte inference under boundary perturbations. Format sentinels help most at smaller scales, while tokenizer models remain highly sensitive to prompt boundaries.

  • D.7. Additional Results on Data Efficiency versus Compute Efficiency: Proxy compression becomes progressively more competitive with model scale, eventually achieving the best of matched-FLOPs and matched-data regimes at 14B.At 0.5B, proxy-trained models underperform baselines in both regimes.
  • D.7. Additional Results on Data Efficiency versus Compute Efficiency: Byte-level models do not consistently outperform tokenizer-based models under matched data during longer 1.5B- and 7B-model training runs.Final-checkpoint pass@1 remains the primary signal because intermediate comparisons can be confounded by learning-rate annealing.
  • D.8. Pairing Strategy Ablation: Warmup-only pairing with randomized order achieves the best downstream performance, reaching 20.1% pass@1.It provides early cross-representation alignment while avoiding the data-duplication cost of Always-on pairing.
  • D.9. Transfer Strength under Controlled Raw-Byte Exposure: Under matched raw-byte exposure, increasing total training bytes with compressed data improves byte-level inference through positive transfer from compressed to raw representations.The same pattern holds for token-level inference, with 15.2% versus 10.4% performance.
  • D.10. On the Effect of Format Sentinels: Format sentinels significantly improve performance at 1.5B scale, but their benefit becomes marginal at 7B.Larger models may distinguish representations without explicit markers.
  • D.11. Prompt Boundary Robustness: Tokenizer-based models show severe prompt-boundary sensitivity, with HumanEval pass@1 ranging from 0.00% to 34.15% across perturbations.Suffixes P2 and P8 collapse generation to near-zero performance, while standard deviations are 14.05% on HumanEval and 12.12% on HumanEval-Plus.
  • D.11. Prompt Boundary Robustness: The proxy-trained model maintains 32.39% mean HumanEval pass@1 with only 0.88% standard deviation across prompt variants.Its raw-byte interface avoids tokenization-boundary effects, and its average performance slightly exceeds the unperturbed baseline.

D.12. Natural Language Experiments

Natural-language experiments train 1.5B models on Nemotron-CLIMB under a compute-matched protocol, comparing tokenizer, raw-byte, and proxy-compression approaches. With raw-byte exposure on 10% of samples, proxy compression improves average accuracy over the byte baseline and narrows the tokenizer gap.

  • Setup: 1.5B models are trained on Nemotron-CLIMB for 50K steps and 25B symbols per run under the same architecture, optimizer, and compute-matched protocol.Each batch contains 0.5M symbols, where symbols are tokens or bytes depending on the input representation.
  • Inference: Proxy (Tokenizer) is evaluated on raw bytes at inference.This preserves the raw-byte inference interface for the proxy tokenizer configuration.
  • Results: +2.1 average points: proxy compression reaches 50.3 versus 48.2 for the byte-level baseline with raw-byte exposure on 10% of training samples.This closes roughly 64% of the gap between byte-level and tokenizer-based methods.
  • Results: 51.5: proxy compression slightly underperforms the tokenizer baseline while following the larger-scale trend of a gradually narrowing gap.The reported natural-language result is qualitatively consistent with the code experiments at 1.5B.
Loading 2602.04289v2…