PaperlayerView exact revision on arXiv

Source-linked AI summary

Going Deeper With Memories

Afshin Khadangi

arXiv:2611.05255v1cs.LGcs.AI

TL;DR

Byte-level models avoid tokenizer dependence, but processing every byte at full resolution is costly. DeepMem separates compressed semantic computation from exact byte preservation, and improves BPB over a matched byte-level Transformer on FineWeb-Edu, PG19, and the evaluated FLORES languages.

  • Problem

    Byte-level models avoid tokenizer dependence, but full-resolution byte processing is expensive and hierarchical compression must balance efficiency with fidelity.

    Source: §1 · p. 2

  • Method

    DeepMem combines a compressed semantic hierarchy with an invertible lossless carrier and detail sidebands, using fast-weight encoding, shared-state bottleneck refinement, and causal super-resolution decoding.

    Source: §1 · pp. 2–3

  • Results

    DeepMem improves FineWeb-Edu (1.0134 vs 1.0457 BPB) and PG19 BPB over a matched byte-level Transformer and achieves the best macro-average across eight FLORES languages.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results suggest lossless multiscale representations with shared-state bottleneck refinement offer a practical alternative to flat byte processing.

    Source: §6 · p. 10

    The study covers approximately 100M-parameter models and a single training budget, so larger-scale experiments are needed.

    Source: §5 · p. 10

Abstract

from arXiv · show

Byte level language models avoid tokenizer dependence and provide universal coverage, but processing every byte at full resolution is expensive. We introduce DeepMem, a causal multiscale architecture that separates semantic compression from exact byte preservation. A learned invertible subband transform constructs a hierarchy of shorter sequences, while a lossless carrier and explicit detail sidebands preserve byte identity. Fast weight memory blocks operate along the encoder, with iterative shared state computation at the compressed bottleneck, while an attention based super resolution decoder restores byte resolution representations. DeepMem achieves stronger language modeling performance than a matched byte level Transformer on FineWeb-Edu and PG19, and obtains the best macro average performance across the evaluated FLORES languages. It also improves PIQA accuracy while remaining competitive on code modeling, multiple choice reasoning, and byte level corruption robustness. Bottleneck refinement requires approximately $7{,}600\times$ fewer counted FLOPs than equivalent full resolution reasoning, while maintaining competitive evaluation throughput.

1 Introduction

Byte-level models avoid fixed-tokenizer dependence but make full-resolution processing costly; DeepMem separates compressed semantic computation from exact byte preservation and reports stronger performance than a matched byte-level Transformer on key language-modeling benchmarks.

1 Introduction

  • Byte-level modeling avoids fixed tokenizer vocabularies and segmentation, which can fragment rare forms or respond discontinuously to small character changes, but full byte sequences make computation expensive.Source: §1 · p. 2
  • Hierarchical methods reduce full-resolution computation, but compression must preserve exact byte information, use a separate local path, or reconstruct detail later.Source: §1 · p. 2
  • Because a span feature may summarize later positions, the analysis transform, synthesis path, and skip connections must all obey completed-span causality.Source: §1 · p. 2
  • DeepMem separates shortened semantic computation from exact byte preservation using learned invertible subbands, a lossless carrier, and explicit detail sidebands.Source: §1 · pp. 2–3
  • Fast-weight memory mixes progressively shorter encoder sequences, with two-step shared-state bottleneck refinement and attention-based super-resolution decoding from delayed parents, skips, and carrier features.Source: §1 · pp. 2–3
  • The decoder combines semantic states with locally available carrier features and encoder skips, unlike lossy compression that must retain all relevant local detail.Source: §1 · p. 3
  • At approximately 100 million parameters, DeepMem improves BPB over a matched raw-byte Transformer on FineWeb-Edu (1.0134 vs 1.0457), PG19 (1.4402 vs 1.4635), and eight-language FLORES macro-average (1.9694 vs 2.0082), with a modest Stack-smol tradeoff.Source: §1 · p. 3

2 Method

DeepMem maintains a predictive semantic stream alongside a lossless byte carrier, compresses semantic computation through a hierarchy, then causally restores byte-resolution representations for next-byte prediction.

2.1 Overview and Notation

  • Each dyadic level halves sequence length while maintaining a learned predictive semantic stream and a lossless carrier that preserves byte identity.Source: §2.1 · p. 3

2.2 Learned Lossless Subband Analysis

  • Learned lifting predicts odd carrier vectors from even vectors to form details, updates the even vectors into coarse coefficients, and permits exact real-arithmetic reconstruction.Source: §2.2 · p. 4

2.3 Semantic Analysis Path

  • The semantic stream is compressed by projecting adjacent vectors to the next level, with carrier information injected through a learned low-rank map and causal processing restricted to completed spans.Source: §2.3 · p. 4

2.4 Fast-Weight Memory Encoder

  • Gated fast-weight blocks provide recurrent key–value memory along the encoder, and chunk-parallel training changes execution scheduling without changing the recurrence.Source: §2.4 · p. 4

2.5 Shared-State Bottleneck Refinement

  • At the 16-position, width-128 bottleneck, a shared FastWeight mixer updates a latent state twice while holding the encoder representation fixed, preserving completed-span causality.Source: §2.5 · p. 5

2.6 Causal Super-Resolution Decoder

  • The decoder restores representations through shifted parent proposals fused with encoder skips and carrier features, so each restored position uses only available context.Source: §2.6 · p. 5

2.7 Byte Prediction and Training Objective

  • After super-resolution, causal byte-resolution attention produces logits over the 256-byte vocabulary and training minimizes next-byte negative log-likelihood, reported as BPB.Source: §2.7 · pp. 5–6

3 Experiments

The experiments train approximately 100M-parameter models under matched byte-level conditions, compare DeepMem with four byte-level baselines, and evaluate language modeling, reasoning, robustness, and throughput.

3.1 Experimental Setup

  • Models are trained from scratch on FineWeb-Edu UTF-8 bytes with a 2,048-byte context, three seeds, approximately 100 million parameters, and 50,000 optimizer steps.Source: §3.1 · p. 6

3.2 Baselines

  • Four byte-level baselines—Transformer, FastWeight, BLT, and AU-Net—use the same corpus, context length, training budget, batch size, schedule, precision, and hardware.Source: §3.2 · p. 6

3.3 Reference Architecture

  • The reference model uses seven analysis and synthesis transitions, two shared-state refinement steps, and 100,158,582 parameters; its hierarchy reaches 128 positions.Source: §3.3 · pp. 6–7

3.4 Evaluation

  • Evaluation covers BPB on general-domain, long-form, code, and eight-language multilingual text, plus multiple-choice accuracy, byte-perturbation robustness, and processed bytes per second.Source: §3.4 · p. 7

4 Results

DEEPMEM performs strongly across general-domain, long-form, multilingual, multiple-choice, and byte-perturbation evaluations, with code modeling and throughput tradeoffs. Its overall pattern favors the proposed hierarchy over the main comparison models.

4.1 Language Modeling Quality

  • DEEPMEM reaches 1.0134 BPB on FineWeb-Edu and 1.4402 BPB on PG19, outperforming the matched Transformer on both evaluations.The flat Transformer leads Stack-smol at 2.4474 BPB, while DEEPMEM records 2.4854 BPB; the all-scale-fusion variant reaches 2.4082 BPB on Stack-smol.Source: §4.1 · p. 7, Table 1 · p. 8

4.2 Multilingual Generalization

  • DEEPMEM obtains the best eight-language FLORES macro-average at 1.9694 BPB, with the lowest language-level BPB on English, German, Arabic, and Thai.The no-shared-state-refinement variant leads on Bengali and Vietnamese; Transformer leads on Swahili and flat FastWeight on Hindi.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 Evaluation Throughput

  • DEEPMEM averages 119,718 processed bytes/s, or 81.4% of Transformer throughput, while exceeding flat FastWeight, BLT, and AU-Net.It remains slower than the flat Transformer.Source: §4.3 · p. 8, Table 3 · p. 9

4.4 Multiple-Choice Evaluation

  • DEEPMEM records the highest HellaSwag accuracy (30.30%), ARC-Challenge accuracy (23.41%), and unweighted mean accuracy (37.76%) among the main comparison models.The no-shared-state-refinement variant scores 61.81% on PIQA versus 60.83% for the full model; flat FastWeight leads ARC-Easy at 37.72%.Source: §4.4 · p. 8, Table 4 · p. 9

4.5 Robustness to Byte-Level Perturbations

  • DEEPMEM averages 29.00% across clean and perturbed HellaSwag settings, above flat FastWeight at 28.87% and Transformer at 27.73%.It leads on clean inputs, repeated characters, and random casing, and remains close to the strongest baseline under deletion, Antspeak spacing, and uppercasing.Source: §4.5 · p. 9

4.6 Overall Tradeoff

  • Across tasks, DEEPMEM achieves the best FineWeb-Edu, PG19, and FLORES macro BPB and the highest mean multiple-choice and robustness accuracy, but trails on code and ARC-Easy.Its evaluation throughput is 119,718 bytes/s.Source: §4.6 · p. 9

5 Discussion and Limitations

The discussion frames DEEPMEM as separating semantic abstraction from exact byte preservation, while causal synthesis and task-dependent component effects qualify the design. The study is limited in scale and training budget, with additional performance and numerical caveats.

Discussion

  • DEEPMEM keeps exact byte information in the carrier and detail sidebands while a shared-state module iteratively refines the deepest semantic representation.The discussion associates this division with improved natural-language and multilingual BPB; auxiliary carrier injection and direct all-scale fusion have task-dependent effects.Source: §5 · pp. 9–10
  • Delayed parent proposals let the decoder recover high-resolution representations without exposing future bytes, preserving causal next-byte prediction.Coarse representations summarize completed spans and cannot be assigned directly to earlier positions within those spans.Source: §5 · p. 10
  • The semantic path allocates capacity to longer-range structure while the carrier hierarchy retains exact byte information, avoiding a forced choice between compression and local fidelity.Attention is concentrated in synthesis after much computation has occurred at shorter resolutions.Source: §5 · p. 10

Limitations

  • The study covers approximately 100M-parameter models and one training budget; DEEPMEM also trails the Transformer on Stack-smol and evaluation throughput.Invertibility is exact in real arithmetic, while finite precision may cause small reconstruction errors; the fixed dyadic hierarchy is less flexible than learned variable-length segmentation.Source: §5 · p. 10

6 Conclusion

DEEPMEM combines compressed semantic computation with an invertible byte carrier, detail sidebands, and causal super-resolution synthesis. The conclusion reports gains over a matched byte-level Transformer, while the appendix describes controlled configurations for examining architectural components.

6 Conclusion

  • The architecture uses a FastWeight encoder on progressively shorter sequences and two-step shared-state bottleneck refinement, with an invertible carrier hierarchy and causal decoder restoring byte-resolution representations.Explicit detail sidebands preserve byte information without future leakage.Source: §6 · p. 10
  • At approximately 100M parameters, DEEPMEM improves FineWeb-Edu, PG19, and multilingual BPB over a matched byte-level Transformer.The conclusion also reports strong multiple-choice accuracy and robustness to byte-level perturbations.Source: §6 · p. 10

A Ablation Studies

  • The ablation configurations hold data, context length, training budget, optimizer schedule, batch size, and hardware constant while varying encoder, carrier injection, fusion, or bottleneck refinement.The full model uses two shared-state updates without direct all-scale fusion; the no-shared-state variant removes those updates while retaining the analysis and synthesis architecture.Source: A Ablation Studies · p. 13
  • The all-attention variant replaces initial and down-path FastWeight blocks with causal self-attention, while the no-encoder-carrier-injection variant retains the carrier for synthesis and detail-sideband fusion.The no-all-scale-fusion configuration also omits iterative bottleneck updates.Source: A Ablation Studies · p. 13

A.1 Language Modeling and Throughput

Ablations show modest BPB improvements from shared-state refinement alongside lower evaluation throughput, and reveal task-dependent differences among encoder, carrier-injection, and fusion configurations. Multilingual variant results are exploratory because only one training run is available per configuration.

A.1 Language Modeling and Throughput

  • Replacing FastWeight with attention increases FineWeb-Edu BPB from 1.0145 to 1.0613, a relative degradation of 4.6%, while improving Stack-smol from 2.5021 to 2.4527 BPB.PG19 increases by 2.5% and FLORES macro BPB by 3.7%; average evaluation throughput rises by approximately 0.3%.Source: §A.1 · p. 13
  • Removing encoder-side carrier injection has only a small aggregate BPB effect: PG19 worsens by 0.0020 BPB, Stack-smol improves by 0.0176, and FLORES macro worsens by 0.0072.FineWeb-Edu is effectively unchanged, and the carrier remains available during synthesis and detail fusion.Source: §A.1 · p. 13
  • Removing all-scale fusion changes FineWeb-Edu BPB from 1.0142 to 1.0145, PG19 from 1.4413 to 1.4462, and Stack-smol from 2.4082 to 2.5021, while FLORES macro improves from 1.9851 to 1.9758.In this comparison, direct all-scale fusion benefits code modeling while the no-fusion configuration performs better on multilingual average BPB.Source: §A.1 · p. 13
  • Adding shared-state refinement lowers BPB on FineWeb-Edu from 1.0145 to 1.0134, PG19 from 1.4462 to 1.4402, Stack-smol from 2.5021 to 2.4854, and FLORES macro from 1.9758 to 1.9694.Average evaluation throughput decreases from approximately 138,052 to 119,718 bytes/s; the reported BPB gains are modest and need additional controlled runs to establish statistical reliability.Source: §A.1 · p. 13

A.2 Multilingual Effects

  • DEEPMEM has the lowest FLORES macro-average and lowest BPB on English, German, Arabic, Hindi, Thai, and Swahili among architecture variants; the no-refinement configuration leads on Bengali and Vietnamese.Without shared-state refinement, removing all-scale fusion improves Thai BPB by 0.0318, Vietnamese by 0.0268, Bengali by 0.0139, and Arabic by 0.0124; these comparisons are exploratory because each configuration has one run.Source: §A.2 · p. 14

A.3 Multiple-Choice Evaluation

DEEPMEM leads the main comparison on HellaSwag, ARC-Challenge, and unweighted mean accuracy, while its PIQA result trails the variant without shared-state refinement.

A.3 Multiple-Choice Evaluation

  • DEEPMEM achieves the highest HellaSwag accuracy (30.30%), ARC-Challenge accuracy (23.41%), and unweighted mean accuracy (37.76%) among the main comparison models.It exceeds the matched Transformer on all four tasks, though several gains are small relative to reported run-level variation.Source: §A.3 · p. 14
  • In the reported configurations, the variant without shared-state refinement has the highest PIQA accuracy at 61.81%; DEEPMEM has the strongest HellaSwag accuracy at 30.30% and highest mean accuracy at 37.76%.Removing encoder-side carrier injection changes mean accuracy from 37.44% to 37.53% relative to the no-shared-state configuration.Source: §A.3 · p. 14

A.4 Robustness to Byte-Level Perturbations

DEEPMEM has the strongest mean HellaSwag accuracy across clean and perturbed prompts, with gains over the no-shared-state configuration on several perturbations and a small Antspeak decline.

A.4 Robustness to Byte-Level Perturbations

  • 29.00% is DEEPMEM’s strongest mean HellaSwag accuracy across clean and perturbed settings, with the best clean, uppercase, and random-casing results.The no-carrier-injection configuration leads on deletion and repetition, while the no-shared-state configuration leads on Antspeak at 27.23%.Source: §A.4 · p. 14, Table A.4 · p. 15
  • 0.43 points on clean inputs, 0.32 on uppercase, and 0.38 on random casing are DEEPMEM’s gains over the no-shared-state configuration, raising mean accuracy by 0.15 points.Deletion and repetition improve slightly, while Antspeak accuracy decreases by 0.34 points.Source: §A.4 · pp. 14–15
  • The comparison does not establish that encoder-side carrier injection or all-scale fusion is individually necessary for robustness; effects vary by perturbation and overlap with other paths.Replacing FastWeight with attention lowers every perturbation score relative to the no-shared-state configuration and reduces its average by 1.43 points.Source: §A.4 · p. 15

A.5 Ablation Summary

The ablations show task-dependent tradeoffs among refinement, fusion, and encoding choices; the accompanying causality analysis constrains hierarchical features to positions whose context includes their source spans.

A.5 Ablation Summary

  • Removing shared-state refinement slightly worsens BPB across FineWeb-Edu, PG19, Stack-smol, and FLORES, while improving dense evaluation throughput.Direct all-scale fusion in the no-refinement configuration improves PG19 and Stack-smol BPB but worsens FLORES macro-average BPB; encoder-side carrier injection has smaller, task-dependent effects.Source: §A.5 · p. 15
  • The all-attention configuration generally has weaker natural-language and multilingual BPB than FastWeight-based comparisons, and the components show distinct task tradeoffs.The comparisons suggest different contributions from multiscale fusion, memory-based encoding, and iterative bottleneck refinement.Source: §A.5 · p. 15

B.1 Span-Based Causality

  • A hierarchy-level representation is valid only when it depends on bytes through the endpoint of its completed span; this constraint applies across analysis, refinement, synthesis, skips, and fusion.Shifted parent proposals keep completed-span features from reaching earlier child positions, and final logits depend only on bytes through their prediction position.Source: §B.1 · pp. 15–16

B.2 Future-Perturbation Protocol

  • Recorded perturbation tests use a randomly initialized, 2,048-byte all-scale-fusion configuration without shared-state bottleneck updates; they run on CUDA in FP32 with TF32 disabled, and pass only when δrep and every prefix difference are at most 10^-5.Both tests include a same-input repeatability control; suffix perturbations produce large changes in later logits.Source: §B.2 · pp. 16–17

B.3 Causality Results

Future-perturbation tests found no shared-prefix change in the midpoint test and kept all 2,048 exhaustive-cutoff discrepancies below the stated tolerance, validating the span-based argument for the tested configuration.

B.3 Causality Results

  • 0 was the maximum shared-prefix difference in the midpoint test, while the changed suffix produced a maximum logit difference of 4.3369.Identical-input repeated executions also produced exactly the same logits in both tests.Source: §B.3 · p. 17
  • 5.0664 × 10^-6 was the maximum shared-prefix discrepancy across 2,048 exhaustive cutoffs, below the 10^-5 threshold; no cutoff exceeded it.The changed suffix produced a maximum logit difference of 5.4749, showing the perturbations affected later predictions.Source: §B.3 · p. 17
  • The results empirically validate the span-based argument for the tested all-scale-fusion configuration, including its analysis, synthesis, skips, carrier features, and final causal attention.The shared-state bottleneck is argued to satisfy the same invariant by composition of causal FastWeight updates; the reported numerical tests concern the configuration without those updates.Source: §B.3 · p. 17

Related papers

Loading 2611.05255v1…