PaperlayerView exact revision on arXiv

Source-linked AI summary

CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

Morayo Danielle Adeyemi, Ryan A. Rossi, Franck Dernoncourt

arXiv:2606.24083v1cs.CLcs.AIcs.LG

TL;DR

Whether linguistic compression lowers language-model inference cost depends on whether the prompt or response is compressed, but prior evaluations largely separate these channels and emphasize accuracy. CAVEWOM evaluates both channels using task accuracy, realized cost, and agreement with unconstrained generations across models, datasets, and reduction levels. It finds that output compression lowers realized cost, while input compression raises net cost and can produce surface-text divergence despite correct answers.

  • Problem

    Prior compression studies largely separate input and output channels and rely on accuracy at reduced token counts, missing realized-cost and reference-text-agreement measures.

  • Method

    CAVEWOM evaluates input and output compression across eight models, five benchmarks, and five reduction levels using accuracy, realized cost, and reference-text agreement.

  • Results

    Output compression cuts realized cost on most API models and all four open-weight models, whereas input compression raises net cost; under cost-saving settings, 51.9% of six-panel non-reasoning generations are correct yet diverge from unconstrained references.

  • Takeaways & Limitations

    Models should be ranked at the constraint level where they will be deployed, because output-compression robustness varies widely and is not predicted by unconstrained accuracy or parameter count.

  • Takeaways & Limitations

    The study makes no claim about content preservation in long-form generation, since its five benchmarks use short, structured answers.

Abstract

from arXiv · show

"Talk short. Drop grammar. Save token." This caveman style is widely promoted as a way to cut inference cost, but whether it actually saves anything depends on which channel (the user's prompt or the model's response) is being compressed. We present Cavewoman, a two-channel evaluation protocol that scores every generation on task accuracy, realized per-item cost, and reference-text agreement against the model's unconstrained reference. We evaluate eight models on five datasets at five reduction levels, with both channels measured on the same items. Output compression cuts realized cost on most API models (1.4-2.4x per model, up to 3x in the best case) and on all four open-weight models under public-tier pricing. Input compression has the opposite effect, a strict lose-lose: it raises net cost rather than lowering it (~1.15x on the five-benchmark mean, up to 1.8x on the worst dataset and 2.7x under stronger compression), because models compensate with longer responses even as accuracy collapses. Under the same setting, surface text diverges from the unconstrained reference: on the non-reasoning models, roughly half of all generations are correct yet their surface text no longer entails the model's own unconstrained baseline generation. The divergence survives length-controlled re-scoring, multiple-comparisons correction, and replication under complementary semantic measures. Code and data are available at https://github.com/danielle34/cavewoman.

1 Introduction

CAVEWOMAN evaluates input and output compression together on fixed models and items, measuring task accuracy, realized cost, and agreement with each model’s unconstrained generation. It addresses limitations of accuracy-only evaluation by exposing cost asymmetry and surface-text divergence.

  • Motivation: Inference cost depends on input and output token counts, with output tokens typically priced 4–8× higher than input tokens.Existing compression methods target either prompts or responses, rather than evaluating both channels together.
  • Motivation: Accuracy at a reduced token count misses realized cost when shorter prompts induce longer responses and cannot distinguish compressed answers that agree semantically.The introduction identifies these as two limitations of prevailing accuracy-focused evaluation.
  • CAVEWOMAN protocol: CAVEWOMAN scores every generation on task accuracy, realized per-item cost, and reference-text agreement with the model’s own unconstrained generation.It evaluates input compression and output compression at five reduction levels while holding model and item fixed.
  • CAVEWOMAN protocol: The protocol measures cost asymmetry between input and output compression on the same items and evaluates surface-text divergence between correct answers and unconstrained references.The evaluation includes audited answer-extraction rates, a twelve-metric semantic battery, and replication across complementary semantic measures.

2 Related Work

Prior work compresses either model inputs or outputs and typically reports task accuracy at a token budget. CAVEWOMAN evaluates both channels on identical items using realized cost, answer-extraction audits, and complementary semantic agreement criteria.

  • Input compression: Input compression uses hard token pruning or document summarization and soft prompt representations such as gist tokens or recursive summary vectors.
  • Output compression: Output compression constrains generation through decoding modifications, post-training, or prompting, but prior work reports task accuracy at a token budget.
  • Semantic fidelity, verbosity, and cost: Semantic-fidelity research separates propositional content from lexical form using bidirectional NLI entailment and examines verbosity, cost, and potentially disconnected reasoning traces.
  • Positioning: CAVEWOMAN measures input and output compression on the same items, reports realized per-item cost, audits answer extraction, and scores reference-text agreement.
  • Positioning: The input-channel divergence also reproduces under LLMLingua-2.

3 Methodology

CAVEWOMAN evaluates linguistic compression by applying the same deterministic reduction hierarchy either to model inputs or outputs, then measuring accuracy, realized cost, and agreement with unconstrained references. The protocol spans five datasets, five levels, and eight models under both conditions.

  • Two-channel design: CAVEWOMAN applies one nested, deterministic POS-filter hierarchy at two pipeline points: Condition A compresses the question, whereas Condition B constrains the response.The same level indexes both conditions, isolating whether the model needs grammatical input or expressive output freedom.
  • Reduction levels: Five levels form a monotone hierarchy: unconstrained baseline (L0), telegraphic register (L1), keyword-only register (L2), noun-phrase skeleton (L3), and 15-token budget (L4).Each level removes further word classes, while L4 truncates the result to its first 15 tokens.
  • Evaluation metrics: Every generation is scored on task accuracy, per-item realized token cost on the priced channel, and reference-text agreement with the same-item unconstrained generation.Agreement uses bidirectional NLI with a DeBERTa-based judge as the headline criterion and is replicated under eleven complementary semantic criteria.
  • Evaluation metrics: The protocol separates correct answers with surface-text divergence from reference agreement and incorrect answers with reference agreement, outcomes accuracy-only evaluation cannot distinguish.These outcomes are organized in a 2 × 2 table for L1–L3; L4 is excluded from semantic evaluation.
  • Experimental coverage: The study evaluates five datasets spanning four task types and three answer formats, with a single model evaluated on 11,465 items at five levels under both conditions.The datasets are GSM8K, BoolQ, ARC-Easy, CommonsenseQA, and MMLU-STEM.
  • Experimental coverage: All eight models are evaluated on every benchmark under both conditions, including seven named models alongside Qwen2.5-VL-7B in ordinary text-chat mode.The model set includes Qwen2.5-VL-7B, Qwen3.5-9B, DeepSeek-R1-Distill-Qwen-7B, Gemma-4-E4B, GPT-4o, GPT-5.4, Claude Haiku 4.5, and Claude Sonnet 4.6.

4 Results

Results show that output compression usually reduces realized cost while preserving classification accuracy, whereas input compression increases cost as accuracy declines. Output compression also causes substantial divergence from unconstrained reference text, and relative robustness varies widely across models.

  • Cost effects: 1.4–2.4× per-model output compression savings occur on most API models, with 17 of 20 model–dataset cells cheaper and savings across all four open-weight models.The best cell reaches up to 3× savings.
  • Cost effects: 1.15× higher mean cost under input compression grows to 1.8× on the worst dataset and 2.7× under deeper reductions as accuracy collapses.Compensatory output expansion outweighs the small input-token savings because output tokens are priced more heavily.
  • Channel differences: Input compression reduces accuracy and reference-text agreement together, while output compression preserves classification accuracy through deep levels but sharply reduces reference-text agreement at L1.Output-channel savings are largest for short-answer benchmarks and smaller for multi-step arithmetic such as GSM8K.
  • Reference-text divergence: 51.9% of L1 output-compressed generations are correct yet no longer match the unconstrained reference, rising to 80.4% after length-matched re-scoring.The divergence increases under length matching, survives Benjamini–Hochberg correction, and appears across all twelve semantic measures.
  • Model robustness: 1.5× to 3.7× relative robustness separates models, and the ratio is not predicted by parameter count or unconstrained accuracy.Gemma-4-E4B is the most output-robust overall, while Claude Sonnet 4.6 is the most output-robust API model.

5 Discussion

The discussion concludes that output, not input, is the channel to compress for real cost savings, while compression robustness and semantic agreement vary independently of unconstrained accuracy. Evaluating compression therefore requires jointly considering cost, accuracy, and reference-text agreement rather than relying on a single axis.

  • Cost and channel choice: Output is the channel to compress when models answer at length: it produces real cost savings on every API model whose billed output matches visible response, unlike input compression.This finding concerns realized cost on the priced channel, not prompt-token reduction alone.
  • Robustness: The highest-accuracy API model is least robust among the four API models, while a 7B open-weight model outranks two API models under output compression.Robustness to output compression does not follow from unconstrained accuracy.
  • Semantic agreement: Correctness and reference-text agreement can diverge because they compare against ground truth and the model’s own L0 generation, respectively.The divergence is a surface-text observation, affecting transcript, audit-trail, and reasoning-display uses but not deployments consuming only the final answer.
  • Evaluation design: Compression evaluation is underdetermined by a single axis because realized cost, observed accuracy, and reference-text agreement depend on different conditions and dissociate in the panel.A composite metric can therefore mis-rank methods on the dimension that ultimately determines deployment cost.

6 Conclusion

Cavewoman shows that language-model compression is a two-channel problem: output compression can reduce realized cost, while robustness and reference-text agreement vary substantially across models. The observed divergence concerns surface text rather than propositional content and persists under several robustness checks.

  • Conclusion: Output compression cuts realized cost on most API models by 1.4–2.4× per model, reaching up to 3× on the best cell.It also cuts cost on all four open-weight models under public-tier pricing.
  • Conclusion: 51.9% of generations on the six-non-reasoning panel are correct yet no longer match the model’s unconstrained reference surface text.This divergence strengthens under length-controlled re-scoring and replicates across complementary semantic measures.
  • Conclusion: Robustness to output compression varies widely across models and is not predicted by parameter count or unconstrained accuracy.Models should therefore be ranked at the constraint level where they will be deployed, rather than only at the unconstrained baseline.
  • Conclusion: The bidirectional NLI judge measures surface-text divergence rather than propositional drift, so the reported rate is not a claim about propositional content.Length-controlled re-scoring and eleven complementary measures address length-and-register confounds, although register changes between conditions may contribute to divergence.

A Implementation Details … B Per-Level Results Tables

The paper documents standardized implementation, pricing, token-accounting, and decoding procedures, then reports per-level accuracy and bidirectional NLI across all eight models. Detailed cost and token tables show that input compression can initially expand outputs and raise cost, while preservation may decline.

  • A Implementation Details: Implementation details cover pricing, dataset statistics and licenses, inference configuration, token accounting, cost estimates, decoder-truncation checks, prompts, POS-tag rules, and released artifacts.Verbatim system prompts and POS-tag rules are provided in Appendix H, with released artifacts in Appendix J.
  • A.1 Pricing Assumptions and Model Snapshots; A.2 Dataset Statistics and Licenses: May 2026 API prices define per-item cost as C = nin · pin + nout · pout, while datasets report sizes, splits, answer formats, mean L0 token lengths, and licenses.Tokens are counted with each model’s own tokenizer; open-weight models are excluded from dollar-cost analysis.
  • A.3 Inference Configuration: All eight models use greedy decoding under identical settings, with per-level decoder budgets as the only level-dependent parameter.The inference configuration is identical across Conditions A and B.
  • A.4 Token Accounting: At L1, output expansion makes Claude Haiku 4.5’s Condition A CommonsenseQA total tokens increase despite fewer input tokens.Input tokens fall at L1–L2, marginal savings appear at L2, and reductions emerge at L3–L4 after entailment has declined.
  • A.5 Cost Estimates: At L1, Claude Haiku 4.5’s Condition A CommonsenseQA cost rises from output expansion; reductions appear only at higher levels where entailment has fallen.Table 9 reports estimated inference cost per million items and cautions that lower cost does not imply higher preservation.
  • A.6 Open-Weight Cost Projection: 2.5× mean projected savings at L1 Condition B are measured for Qwen2.5-VL-7B, compared with 1.18× for Qwen3.5-9B, 1.17× for DeepSeek-R1-Distill, and 2.09× for Gemma.The projections average measured token counts across six tiers of May 2026 public pricing.
  • A.7 Decoder-Truncation Check at L1 Condition B: 0.9% of GPT-4o outputs, 7.3% of Haiku 4.5 outputs, and 0.9% of Qwen2.5-VL-7B outputs hit the L1 max_new_tokens=300 ceiling.The L1-B cost savings reflect natural stopping rather than truncation.
  • B Per-Level Results Tables: Per-level accuracy and bidirectional NLI are reported for all eight models, with colored deltas measured against each model’s L0 baseline.Green accuracy deltas with red NLI deltas on classification benchmarks under Condition B quantify Finding 2, while DeepSeek-R1 shows the inverse pattern.

B.1 Threshold and Task-Type Comparisons · C Judge Reliability and Semantic Robustness · C.1 NLI Judge Reliability by Compression Level

The paper defines compression-degradation thresholds using accuracy and strict bidirectional NLI, finding consistent semantic threshold ordering across evaluable cases while output compression often avoids accuracy degradation. Reliability checks show that NLI is dependable at L1, supplementary at L2, and descriptive only at L3.

  • B.1 Threshold and Task-Type Comparisons: Accuracy degradation is defined as at least 5 pp below L0, while semantic degradation is bidirectional NLI falling at least 15 pp below L0.Accuracy thresholds scan L1–L4; semantic thresholds scan L1–L3 because L4 has no NLI score.
  • B.1 Threshold and Task-Type Comparisons: 60 of 80 model–dataset–condition cells are evaluable because both degradation thresholds are crossed within the scanned levels.The remaining cells do not cross both criteria in range and are non-evaluable for threshold ordering.
  • B.1 Threshold and Task-Type Comparisons: Under strict bidirectional NLI, Lsem_c orders all 60 evaluable cells, whereas forward-only NLI violates ordering on 2 of 60 and cosine on 36 of 60.This full-panel result uses all eight models; the displayed table is only a representative three-model excerpt.
  • B.1 Threshold and Task-Type Comparisons: Output compression frequently leaves accuracy above the degradation threshold, producing many “—” cells where accuracy never falls 5 pp below L0.The frequent non-crossings under Condition B reflect the paper’s reported output-compression finding.
  • B.1 Threshold and Task-Type Comparisons: Table 13 reports mean L1 semantic non-preservation by benchmark across eight models, with the same task-type ordering under alternative semantic criteria.Figure 6 additionally aggregates the 2 × 2 dissociation by dataset across L1–L3 and eight models.
  • C Judge Reliability and Semantic Robustness: Calibration and eleven additional measures support the headline NLI result, whose rate lies near the conservative end of the metric family.The robustness analysis combines judge-reliability calibration with cross-metric replication.
  • C.1 NLI Judge Reliability by Compression Level: At L1 under Condition B, the Disagree-A rate is 17.1%, while judge false-negative rates are 2.9% at L1, 28.6% at L2, and 50.0% at L3.Calibration uses 70 POS-filtered synthetic positive pairs per level; L4 is excluded.

C.2 Robustness Across Alternative Semantic Measures · C.3 Length-Controlled NLI Re-Scoring

Alternative semantic measures consistently detect substantial divergence, while length-controlled NLI re-scoring shows that the headline rate is partly driven by output length. The effect persists across non-reasoning cells, with smaller changes for reasoning models and an unmeasured system-prompt noise floor.

  • C.2 Robustness Across Alternative Semantic Measures: The alternative-measure analysis combines NLI judges, faithfulness, learned and surface similarity, STS, and QA-based propositional checks.Continuous-similarity and answer-anchored variants provide additional context rather than belonging to the twelve headline measures.
  • C.3 Length-Controlled NLI Re-Scoring: Length-controlled scoring truncates each unconstrained output to the compressed output’s wordpiece-token length before applying the same bidirectional NLI judge.The procedure uses the same denominator as the Finding 2 headline and truncates to min(|L0|, |L1-B|).
  • C.3 Length-Controlled NLI Re-Scoring: +28.4 pp is the C2-rate increase on the 6-non-reasoning panel under length-matched scoring.The corresponding increase is +21.6 pp on the 8-model aggregate.
  • C.3 Length-Controlled NLI Re-Scoring: ≤7 pp is the movement for reasoning models DeepSeek-R1 and Qwen3.5-9B because their unconstrained outputs are already short.This contrasts with the larger aggregate shifts observed for non-reasoning models.
  • C.3 Length-Controlled NLI Re-Scoring: 18.3% →92.0% is GPT-4o’s largest GSM8K shift, followed by Sonnet (+55.8 pp) and Qwen2.5-VL-7B (+55.0 pp).The same direction appears on every non-reasoning model–dataset cell.
  • C.3 Length-Controlled NLI Re-Scoring: The C2 rate rises in the same direction on every non-reasoning model–dataset cell after length control.This establishes a consistent per-cell pattern rather than an effect confined to a small subset of evaluations.
  • C.3 Length-Controlled NLI Re-Scoring: The headline should be interpreted as a floor above unmeasured system-prompt register-shift noise because no paired L0-A versus L0-B baseline was scored.Such a baseline would have isolated the system-prompt register shift between conditions.
  • C.2 Robustness Across Alternative Semantic Measures: Twelve complementary semantic measures report substantial divergence at L1 Condition B on the 6-non-reasoning panel, spanning 41%–88%.Bidirectional NLI reports 51.9%, near the conservative end; DeBERTa-large reports 49.5% and BLEURT 87.6%.

D Comparison with LLMLingua-2

The LLMLingua-2 comparison shows that learned input compression remains a strict failure mode on short individual questions, outside its intended long-context and few-shot regime. Across models and datasets, divergence persists, accuracy collapses at default rates, and structural token pruning can destroy multiple-choice performance.

  • D Comparison with LLMLingua-2: C2 > 0 on every cell under both LLMLingua-2 and CAVEWOMAN input-compression methods, showing divergence beyond the POS filter.The comparison covered 3 models × 3 datasets, with τ=0.5 and a Qwen2.5-VL-7B τ=0.8 rate-sensitivity check.
  • D Comparison with LLMLingua-2: 20–35% GSM8K accuracy at LLMLingua-2’s default rate falls well below the ∼79% reported for few-shot demonstration compression.The comparison applies LLMLingua-2 outside its intended regime of compressing few-shot demonstrations and long-context inputs.
  • D Comparison with LLMLingua-2: 0.66 accuracy and 0.73 NLI rate at τ=0.8 on Qwen2.5-VL-7B GSM8K are comparable to CAVEWOMAN’s telegraphic level.The lower τ=0.8 values in Table 17 are the three-dataset Qwen2.5-VL-7B mean, while the cited GSM8K values illustrate rate-driven collapse.
  • D Comparison with LLMLingua-2: 6–7% ARC-Easy accuracy across all three models at τ=0.5 results from pruning single-letter MCQ labels, which CAVEWOMAN’s POS filter retains.This structural failure mode shows that LLMLingua-2 can remove low-information tokens that are essential to question interpretation.

E Reasoning-Token Accounting (DeepSeek-R1 and GPT-5.4)

Reasoning tokens consume constrained output budgets even when hidden, causing DeepSeek-R1 and Kimi-K2.6 to lose visible answers and GPT-5.4 to incur substantially higher billed output than visible text. These accounting differences make reasoning-model accuracy, cost, and reference-text comparisons unlike those for non-reasoning models.

  • DeepSeek-R1: At L4, DeepSeek-R1’s 20-subword decoder budget is exhausted before the visible answer begins, producing near-zero accuracy on every dataset.The shared budget counts both hidden traces and visible responses against max_new_tokens.
  • DeepSeek-R1: At L1–L3, partial DeepSeek-R1 traces bias bidirectional NLI against it relative to non-reasoning models.The trace counts as output_tokens and enters the NLI judge when it survives.
  • DeepSeek-R1: C2 is ∼19% and C3 is ∼21% on the DeepSeek-R1 L1 panel, while its reported output/input accuracy ratio is 2.4 but not apples-to-apples with non-reasoning models.The visible budget is effectively shorter because reasoning traces consume part of it.
  • GPT-5.4: At L0 Condition B on BoolQ, GPT-5.4 averages ∼56 visible tokens versus ∼198 billed output tokens, a 3.5× gap; the five-benchmark average is 2.9×.Serverside reasoning tokens are billed but absent from returned text, unlike GPT-4o, Haiku, and Sonnet, which bill exactly visible output.
  • Kimi-K2.6: Kimi-K2.6 returned empty visible text on 99–100% of constrained-output items at L1 Condition B and stricter, consuming the full max_new_tokens cap.Its reasoning-block protocol consumes the entire budget under output constraint, paralleling the DeepSeek-R1 accounting problem.

F Limitations in Detail … J Released Artefacts and Reproducibility

The paper details methodological limitations, extraction-rate artifacts, compression specifications, qualitative examples, and released materials for reproducibility. These qualifications show that apparent accuracy gains can reflect parsing effects, while compressed outputs may preserve answers without preserving the reference reasoning chain.

  • F Limitations in Detail: Judge reliability is strongest at L1; L2 is supplementary, L3 descriptive, and L4 excluded, while length and register differences confound semantic comparisons.Compressed outputs are shorter and more function-word-light than L0, outside DeBERTa-NLI’s training distribution; length-controlled rescoring and complementary judges provide corroboration.
  • F Limitations in Detail: 278 of 320 Wilcoxon cells remain significant after Benjamini–Hochberg correction, with per-cell bootstrap CIs spanning 1–7 pp.The divergence uses one greedy L0 reference and does not separately measure sampled-decoding stochasticity or the L0-A versus L0-B noise floor.
  • G.1 Extraction-Rate Audit: 45 of 80 cells have L0 parse rate below 0.95, making some apparent L1 accuracy gains extraction artifacts rather than substantive improvements.On MMLU-STEM Condition B, Gemma-4-E4B’s parse rate rises from 0.492 to 0.807 and GPT-4o’s shows a +16.8 pp parse-rate gap; gains are not reported when L0 parse is below 0.95.
  • G.2 L4 Output-Length Distribution: L4 uses a max_new_tokens = 20 ceiling for a 15-token target, so L4-A is retained only with a soft-constraint caveat.Condition A overshoots the target, whereas the same ceiling binds tightly under Condition B.
  • H Constraint-Level Specifications: The implementation releases verbatim prompts and POS-tag rules; Condition A keeps the system prompt fixed and deterministically filters user-message tokens across a monotone L0 ⊇ L1 ⊇ L2 ⊇ L3 ⊇ L4 ladder.Filtered tokens are rejoined with single whitespace, and empty outputs fall back to the original text.
  • I Qualitative Examples: A BoolQ example shows the correct answer surviving every level while NLI fails at L1, L2, and L4 and unexpectedly passes at L3.The noun-phrase skeleton preserves enough propositional structure for entailment, illustrating why answer accuracy alone misses reasoning-chain collapse.
  • J Released Artefacts and Reproducibility: Each configuration releases inference, entailment, embedding, and run-manifest artifacts recording outputs, costs, NLI or cosine measures, software versions, hardware, and provenance.The repository also contains the full per-cell extraction-rate audit and released implementation materials.
Loading 2606.24083v1…