Lossy Compressive Text Autoencoders

1 Introduction

• The paper frames the research question as learning general-purpose lossy text reconstruction that preserves semantic information at a high compression rate.It describes compression and representation learning as disconnected tasks in prior approaches.Source: §1 · p. 1
• The authors report 2.24 bits per byte on web text using a 4.3B-parameter autoencoder, which, to the best of their knowledge, is the highest compression rate for a text autoencoder model in literature.Source: §1 · p. 1
• The proposed Transformer autoencoder uses residual time-axis downscaling and upscaling, a residual bottleneck projection, and quantization to form compressed latent representations.Source: §1 · p. 1
• The work evaluates quantization methods, training objectives, and datasets, and assesses reconstructions and representations on question-answering and semantic text similarity tasks.Source: §1 · p. 1
• The models favor text reconstruction over representation learning, revealing a reconstruction–representation trade-off.Source: §1 · pp. 1–2
• The authors argue that BLEU, Rouge, chrF, and BERTScore do not reliably measure semantic similarity, and introduce an LLM judge adapted from MQM.Source: §1 · p. 2

Lossless Text Compression

• Traditional lossless text compression combines dictionary methods, entropy coding, and transforms, while neural methods model sequences and use entropy coding.The passage describes neural lossless research as focused mainly on runtime efficiency and compression rates.Source: §2 · p. 2

Context Compression

• Context-compression research reduces sequence length through fewer latent tokens, with OCR and text autoencoding methods achieving up to 40x token-sequence compression.This work likewise compresses inputs along the time dimension and proposes an architecture for larger context-compression rates.Source: §2 · p. 2

Text Embedding Models

• Text embeddings provide latent representations for similarity, retrieval, clustering, and classification, and this work compares autoencoder representations with embedding models.The paper also compares their text-reconstruction capabilities.Source: §2 · p. 2

Text Reconstruction from Latent Representations

• Prior reconstruction work uses either iterative correction of embedding representations or autoencoders trained on text reconstruction; this paper explores compression and additional supervision objectives.The passage names Morris et al. (2023), (Omni-)SONAR, and Cideron et al. (2022) as examples.Source: §2 · p. 2

3.1 Experimental Setup

• The experimental setup uses 0.6B-parameter Transformers, cross-entropy training, and BLEU plus an LLM judge to evaluate reconstructions.Evaluation reports 95% confidence intervals on 1,000 validation documents sampled from the training-data distribution.Source: §3.1 · p. 3

3.2 Architecture

• The architecture progressively downsamples and upsamples representations along the time axis, retains a residual bottleneck projection, and applies next-token supervision to upscaled intermediates.The proposed changes remove the pooling operation used in the baseline architecture.Source: §3.2 · p. 3, Figure 1 · p. 4
• The multi-resolution autoencoder with upscaler loss performs best on BLEU and second-best on judge score among the reported architecture variants.The comparison uses 128-token sequences and a 384-dimensional bottleneck.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Latent Representation Quantization

• At approximately 9 bits per token, CRVQ has the highest reported performance; PRVQ improves on RVQ, while naive bottleneck narrowing reconstructs poorly.The comparisons cover FSQ, RVQ, PRVQ, CRVQ, and a narrow-bottleneck baseline.Source: §3.3 · p. 5, Table 2 · p. 5
• Stochastic CRVQ-depth training consistently degrades performance relative to fixed-depth training, while BLEU for both approaches rises toward its maximum near 17.2 bits per token.The passage attributes the near-uncompressed result to models primarily copying tokens stored in the latent representation.Source: §3.3 · p. 5, Figure 2 · p. 5

3.4 Training Supervision

• Most supervision variants do not beat unmasked cross-entropy on Copy; encoder masking and WeightedClass are exceptions, while WeightedClass worsens overall reconstruction.On RTT and Compress, decoder masking improves reconstruction relative to using the upscaler loss when input and target tokens differ.Source: §3.4 · p. 6, Table 3 · p. 6

4.1 Evaluation of Semantic Text Similarity

• Traditional similarity metrics can score verb-corrupted text highly, whereas the Qwen judge scores RTT and Compress highly and Corrupt lowly.The judge counts semantic information pieces and mismatches before assigning a score from 1 to 10.Source: Table 4 · p. 7, §4.1 · p. 7
• Judge scores increase as token compression ratio decreases and, within a model family, for larger summary models.The authors use token compression ratio as a proxy for retained information and report results on 1,000 documents.Source: §4.1 · p. 7, Figure 3 · p. 8

4.2 Text Compression Evaluation

• Reaching lower lossy-compression rates may require reinforcement learning, but multi-rollout training with an LLM-based semantic-similarity objective was too expensive for this work.A two-rollout approach with a trained binary scorer is proposed as future work.Source: §4.2 · p. 8
• At 128-token sequences, the autoencoders approach zstandard's compression rate and outperform SONAR on compression rate and reconstruction performance.The table compares compression rate in bits per byte with BLEU and judge score across traditional lossless, neural lossless, and neural lossy methods.Source: §4.2 · p. 8, Table 5 · p. 9

4.3 Downstream Task Evaluation

• On MS MARCO question answering, autoencoder reconstructions reach the unmodified-passage upper-bound score, while SONAR has lower accuracy.The evaluation uses reconstructed passages to extract answers and compares those answers with the true answers.Source: §4.3 · p. 8, Table 6 · p. 10
• The autoencoders reconstruct well but provide less useful semantic-similarity representations; SONAR and Qwen3-Embedding perform well on similarity with lower reconstruction performance.Similarity-matrix distillation raises the autoencoder STSBenchmark score above the random baseline, but it remains low.Source: §4.3 · p. 9, Figure 4 · p. 10

• The autoencoders achieve low bitrates comparable to traditional lossless compression methods while reconstructing semantically equivalent text.Source: §5 · p. 10
• The models prioritize copying information over high-quality representations and are better suited to moderate compression and reconstruction tasks than representation-learning tasks.Suggested examples include context compression and downstream tasks requiring reconstruction.Source: §5 · p. 10
• Rewarding semantically close but non-identical reconstructions might be necessary to match higher neural lossless compression rates.Source: §5 · p. 10

A LM Judge Scoring Examples

• Score 1 marks salmon-text reconstructions with severe semantic mismatches, factual contradictions, and altered entities and scientific details.The judge says these changes render the texts semantically unrelated.Source: A LM Judge Scoring Examples · pp. 16–17
• Score 3 reflects extensive historical distortions in the Megasthenes passage, including altered names, dates, places, and meanings.The judge cites 16 semantic differences and says they compromise historical accuracy and core meaning.Source: A LM Judge Scoring Examples · pp. 17–19
• Score 5 rates a profiling passage with eight semantic differences, many from substitutions that change technical meaning or readability.The judge says the overall structure and intent remain aligned despite the substantive changes.Source: A LM Judge Scoring Examples · p. 19
• Score 7 reflects four changes in the Critical Care passage, including its title and descriptions of collaboration and research outputs.The judge notes that over 60% of the semantic content remains verbatim.Source: A LM Judge Scoring Examples · p. 20
• Score 9 rates nearly identical game descriptions with one lexical substitution that the judge considers minor to the core meaning.The judge distinguishes this change from grammatical or stylistic variation it accepts.Source: A LM Judge Scoring Examples · p. 20

B Additional Experiments

• Figure 5 reports LLM-judge scores while varying latent bottleneck dimensions, with sequence length 128, scaling factor 128, and hidden dimension 768 fixed.The caption describes 0.6B-parameter multi-resolution autoencoders trained using the upscaler and decoder loss.Source: Figure 5 · p. 21
• Figure 6 reports LLM-judge scores while varying sequence lengths for 0.6B-parameter models with bottleneck dimension 48 and scaling factor 32.The models use the upscaler and decoder loss.Source: Figure 6 · p. 22
• Table 7 compares BLEU and LLM-judge scores across upscaler and decoder loss weights for 0.6B-parameter models.The caption fixes sequence length at 128, scaling factor at 128, and latent dimension at 384.Source: Table 7 · p. 22
• Table 8 compares BLEU and LLM-judge scores across CRVQ configurations at 9.63 bits per token.The caption specifies 0.6B-parameter models with sequence length 128.Source: Table 8 · p. 22
• Table 9 compares BLEU and LLM-judge scores across encoder mask ratios at matched non-masked token count.The caption specifies CRVQ multi-resolution models at 9.63 bits per token, with sequence length 128, scaling factor 8, and latent dimension 24.Source: Table 9 · p. 22
• Figure 7 reports judge scores for different summary models and target summary ratios across Copy, RTT, Compress, and Corrupt datasets.The caption identifies the Gemma4-31B-it judge.Source: Figure 7 · p. 22

C Round-trip Translation and Compress Dataset Generation

• RTT targets are created by translating input text into German and back into English while preserving inserted sentence boundaries.The process first asks the model to add sentence boundaries; the original text remains the input and the generated text is the target.Source: C Round-trip Translation and Compress… · p. 23
• Compress targets are generated by asking an LLM to remove redundant information while preserving semantics and sentence boundaries.The original data is used as input, and the compressed generation as target.Source: C Round-trip Translation and Compress… · pp. 23–24
• During autoencoder training, sentence boundaries help match original and synthetic tokens using the longest sentence-pair sequence that fits the context.The compression prompt also requires retaining each sentence's exact #eos_N# marker in its original position.Source: C Round-trip Translation and Compress… · pp. 23–24
• Sentence-boundary markers are appended after fully terminated sentences using a zero-based sequential counter, while unfinished final segments receive no marker.The instructions require preserving all other original characters and formatting.Source: C Round-trip Translation and Compress… · p. 23

D LM Judge Prompt for Evaluation of Semantic Matching

• The prompt defines semantic difference as a count of differing semantic information pieces, without penalizing different wording that preserves meaning.The judge counts a difference when one text contains information the other lacks or the information differs.Source: D LM Judge Prompt for Evaluation of… · pp. 24–25
• The global score runs from 1 to 10, with lower scores for greater difference and higher scores for closer semantic matches.The prompt assigns 1 to unrelated texts and 10 when there are no semantic information differences.Source: D LM Judge Prompt for Evaluation of… · pp. 24–25
• The guidelines distinguish factual or intent distortion, over-specification, under-specification, and unwarranted additions from acceptable variation.Acceptable variation includes synonymous phrasing, grammatical restructuring, and minor formatting changes when meaning and scope remain unchanged.Source: D LM Judge Prompt for Evaluation of… · pp. 25–26
• The judge must return JSON with information-piece counts, difference count, global score, and a reason.The required output example specifies the field names and forbids additional commentary.Source: D LM Judge Prompt for Evaluation of… · p. 26