PaperlayerView exact revision on arXiv

Source-linked AI summary

Brevity Constraints Reverse Performance Hierarchies in Language Models

MD Azizul Hakim

arXiv:2604.00025v1cs.CLcs.AI

TL;DR

The paper investigates why larger language models sometimes underperform smaller ones under standard prompting. Evaluating models across benchmark problems and validating alternative explanations, it finds that scale-dependent verbosity masks large-model competence and that brevity constraints reverse these performance hierarchies.

  • Problem

    Standard prompting can mask large-model performance on specific problem types, limiting evaluation of scale-dependent prompt sensitivity.

  • Method

    The authors systematically evaluate 31 models from 0.5B to 405B parameters across 1,485 problems and use contamination tests plus response-length and error-pattern analyses.

  • Results

    On 7.7% of benchmark problems, smaller models outperform larger models by 28.4 percentage points, while contamination tests support genuine scale-dependent capability differences rather than memorization artifacts.

  • Takeaways & Limitations

    Scale-aware prompt engineering can reveal masked large-model capabilities and support deployment strategies that improve accuracy while identifying when smaller models suffice.

  • Takeaways & Limitations

    The analysis uses greedy decoding and focuses primarily on knowledge and reasoning benchmarks, so its findings may not generalize to sampled decoding or generative capabilities.

Abstract

from arXiv · show

Standard evaluation protocols reveal a counterintuitive phenomenon: on 7.7% of benchmark problems spanning five datasets, larger language models underperform smaller ones by 28.4 percentage points despite 10-100x more parameters. Through systematic evaluation of 31 models (0.5B-405B parameters) across 1,485 problems, we identify the mechanism as spontaneous scale-dependent verbosity that introduces errors through overelaboration. Causal intervention experiments demonstrate this reflects correctable prompt design rather than fundamental capability limitations. Constraining large models to produce brief responses improves accuracy by 26 percentage points and reduces performance gaps by up to two-thirds. Most critically, brevity constraints completely reverse performance hierarchies on mathematical reasoning and scientific knowledge benchmarks, with large models achieving 7.7-15.9 percentage point advantages over small models -- direct inversions of the original gaps. These reversals prove large models possess superior latent capabilities that universal prompting masks. We validate findings through three independent contamination tests and demonstrate inverse scaling operates continuously across the full parameter spectrum, with dataset-specific optimal scales ranging from 0.5B to 3.0B parameters. Our results establish that maximizing large model performance requires scale-aware prompt engineering rather than universal evaluation protocols, with immediate implications for deployment: prompt adaptation simultaneously improves accuracy and reduces computational costs.

1 Introduction

The paper tests whether larger models always outperform smaller ones and finds systematic inverse scaling on standard benchmark problems. It attributes these reversals to scale-dependent verbosity and shows that brevity constraints can recover large-model performance.

  • Research question and contribution: 31 models spanning 0.5B–405B parameters were evaluated on 1,485 problems across five benchmarks, identifying inverse scaling on 7.7% of problems.Small models outperformed large models by an average of 28.4 percentage points on these problems.
  • Research question and contribution: Inverse scaling appeared across mathematical reasoning, reading comprehension, scientific knowledge, and commonsense reasoning benchmarks, unlike prior cases centered on artificial tasks.The effect was reported on standard benchmarks measuring practical capabilities.
  • Evaluation gap: Standard prompting systematically underestimates large-model performance on specific problem types because scale-dependent prompt sensitivity affects problem-level accuracy.The paper frames this as masked capability rather than a fundamental limitation of scale.
  • Causal intervention: Brevity constraints improved large-model accuracy by 26.3 percentage points and reduced the inverse-scaling gap by 67%, from 44.2% to 14.8%.The intervention used control, brief, and direct-answer conditions across all 115 inverse-scaling problems.
  • Validation: Three contamination tests found response diversity, natural length variation, and error patterns consistent with genuine scale-dependent capability differences rather than memorization.Large-model failures predominantly involved over-reasoning.

2 Related Work

Prior work established scaling laws and a limited set of inverse-scaling cases, while benchmark research examined contamination and informativeness. This paper extends that literature by analyzing scale-dependent performance and discriminative efficiency on standard benchmarks.

  • Scaling laws and inverse scaling: Scaling laws describe performance improvements with model size, training compute, and downstream performance, while emergent abilities reinforce the scaling paradigm.The paper positions its question as whether these improvements are universal across problems.
  • Scaling laws and inverse scaling: Earlier inverse-scaling studies reported limited cases involving rare-pattern memorization, distractor reasoning, spurious correlations, or task-specific artifacts.Those studies focused primarily on constructed or adversarial examples.
  • Model evaluation and benchmark design: Existing evaluation research addressed contamination, robustness, benchmark saturation, item response theory, and evaluation informativeness without examining scale-dependent performance patterns.The paper identifies discriminative efficiency as an unaddressed dimension.
  • Model evaluation and benchmark design: The analysis finds that 55.9% of benchmark problems discriminate between models, while 7.7% exhibit inverse scaling, creating implications for efficient evaluation design.These findings connect model evaluation to deployment research that often assumes smaller models are degraded versions of larger ones.

3 Methods

The study evaluates models across five benchmarks using deterministic decoding, task-specific answer extraction, and problem-level performance comparisons. It then measures response length, intervenes on verbosity, and tests contamination through response diversity, variability, and error patterns.

  • Model evaluation: 31 language models from 0.5B to 405B parameters were evaluated on GSM8K, BoolQ, ARC-Easy, CommonsenseQA, and MMLU-STEM using greedy decoding.The base prompts did not elicit chain-of-thought reasoning.
  • Model evaluation: Accuracy was computed from task-specific extracted answers compared with ground truth, using an indicator function for correctness.Models were categorized as small at N ≤10B and large at N >70B parameters.
  • Inverse-scaling detection: Inverse-scaling problems were identified by comparing accuracy averaged across small and large models, with positive performance gaps indicating small-model advantage.Cohen’s d was used to assess effect magnitude and statistical significance.
  • Verbosity analysis: Response length was measured in tokens for each model–problem pair and compared across model-size categories using Welch’s t-test.This operationalized the overthinking hypothesis while accounting for unequal variances.
  • Causal intervention: Seven models were tested under unrestricted control, brief, and direct-answer prompts across all 115 inverse-scaling problems.Brief prompts constrained responses to under 50 words for mathematics and 10 words for reading comprehension.
  • Contamination analysis: Contamination validation combined unique-response rates, response-length coefficient of variation, and error-pattern classification.High variation was interpreted as natural response behavior rather than memorized templates.

4 Results

Problem-level evaluation reveals substantial discriminative inefficiency and robust inverse scaling across five benchmarks and multiple model families. Brevity interventions substantially improve large-model performance, though benefits vary by dataset, while contamination tests support the robustness of the phenomenon.

  • Discriminative inefficiency: 27.1% of problems were non-discriminative because of ceiling or floor effects, yielding limited information about relative model performance.Filtering them could reduce evaluation costs by approximately 28% while maintaining discriminative power.
  • Inverse scaling discovery: 7.7% of problems showed inverse scaling, with small models outperforming large models across all five benchmarks.The inverse-scaling category comprised 115 of 1,485 problems.
  • Inverse scaling discovery: Inverse-scaling prevalence ranged from 3.9% on MMLU-STEM to 11.3% on BoolQ, and every one of the 115 problems favored small models.The mean performance gap was 28.4 percentage points.
  • Inverse scaling discovery: Cohen’s d was 1.34, indicating a very large and reliable separation between small- and large-model performance distributions on inverse problems.This exceeded the conventional large-effect threshold of d = 0.8.
  • Scale and architecture: Inverse scaling persisted across Llama, Qwen, Gemma, and Mistral families, supporting consistency across architectures.Within-family analyses also reported degradation among larger variants.
  • Scale and architecture: Dataset-specific optimal scales ranged from 0.5B to 3.0B parameters, with four of five datasets showing significant negative size–accuracy correlations.Reported correlations ranged from ρ = −0.50 to −0.66.
  • Causal mechanism: Brevity constraints improved large-model accuracy by 26.3 percentage points and reduced the gap from 44.2pp to 14.8pp, while direct answers reduced it to 7.8pp.The brief intervention was highly significant across 96 problems.
  • Causal mechanism: Brevity benefits varied by dataset: ARC-Easy showed 73.8% gap reduction, CommonsenseQA 61.5%, while BoolQ showed a 3.1% gap increase.The paper attributes BoolQ’s divergence to functional cross-sentence passage integration.

5 Discussion

The paper argues that inverse scaling reflects scale-dependent prompt sensitivity rather than absent large-model capability. Brevity constraints expose masked competence, while optimal prompting and routing must vary by problem and model size.

  • 5.2 The Overthinking Mechanism and Its Mitigation: 67% gap reduction under brevity constraints supports overthinking as a causal, scale-specific failure mode rather than a capability deficit.Large-model accuracy rose by 26.3 percentage points while the inverse-scaling gap fell from 44.2% to 14.8%.
  • 5.1 Reframing Scaling Laws: 7.7–15.9pp advantages for large models under brevity constraints show that standard prompts can reverse the apparent scaling relationship.Problems classified as inverse scaling under standard prompts become normal scaling when brevity constraints are applied.
  • 5.1 Reframing Scaling Laws: Large models require more careful prompt engineering to access capabilities that standard prompts mask on inverse problems.Llama-3.1-405B improved from 41.5% under control conditions to 67.2% under brevity constraints.
  • 5.2 The Overthinking Mechanism and Its Mitigation: Brevity is most beneficial for mathematical and scientific reasoning, whereas reading comprehension shows more modest gains because elaboration can support passage integration.Intervention effectiveness varies substantially by task type, so a single brevity policy is not universally appropriate.
  • 5.3 Practical Implications: Problem-aware routing with scale-specific prompts can improve deployment performance while addressing predictable underestimation of large-model capability.The paper recommends identifying problems prone to overthinking and applying brevity constraints selectively.
  • 5.4 Limitations and Future Directions: The conclusions are bounded by greedy decoding, five primarily knowledge-and-reasoning benchmarks, unresolved causes of overthinking, and contamination concerns that are reduced but not eliminated.The intervention also selected large models partly because they showed stronger overthinking tendencies in control conditions.

A.1 Contamination Validation Details

Three independent tests—response diversity, length variability, and error patterns—converge against contamination explanations for inverse scaling. Large-model failures predominantly involve over-reasoning rather than memorization-like behavior.

  • Response Diversity: 89–100% unique responses across datasets contradict template-based memorization patterns.Three datasets showed 100% uniqueness, while BoolQ and MMLU-STEM showed 94.7% and 89.3%, respectively.
  • Length Variability: 0.31–1.21 coefficient-of-variation values indicate natural response-length variability beyond memorization thresholds.All datasets exceeded the memorization threshold of CV < 0.15, and 3/5 exceeded the natural-variation threshold of CV > 0.30.
  • Error Patterns: 41–82% of large-model failures were classified as over-reasoning, making verbose incorrect logic the dominant failure mode.Dataset-specific over-reasoning rates ranged from 41% to 82%, while memorization avoidance accounted for 12–24%.
  • Convergent Evidence: All three validation tests support genuine capability differences rather than contamination artifacts.The evidence combines high response diversity, natural length variability, and over-reasoning failure patterns.

A.2 Dataset-Specific Analysis

Inverse scaling varies across benchmarks in prevalence, family susceptibility, and response generation, yet small models consistently outperform large models on inverse problems. The shared pattern spans mathematical, comprehension, scientific, and commonsense tasks.

  • Problem-Level Accuracy: Inverse problems appear across all five benchmarks, with dataset-specific clustering and prevalence patterns.GSM8K concentrates inverse problems early, BoolQ distributes them more broadly, and ARC-Easy, CommonsenseQA, and MMLU-STEM show distinct clustering or sparse distributions.
  • Model Family Performance: 50+ percentage point gaps favoring Gemma models over DeepSeek on inverse problems reveal strong within-benchmark scale effects.Gemma models at 2B–9B achieve 78–85% accuracy, while DeepSeek at 67B achieves 8–35%.
  • Response Length: 104 vs 121 tokens is the only statistically significant response-length difference between inverse and normal BoolQ problems.MMLU-STEM shows a larger but marginally significant difference of 52 vs 81 tokens; other datasets are non-significant.
  • Cross-Dataset Generalization: 4–11% prevalence, systematic large-family underperformance, and elevated inverse-problem response lengths recur across task domains.The shared characteristics span mathematical reasoning, reading comprehension, scientific knowledge, and commonsense reasoning.

A.3 Architecture-Independence Analysis

Inverse scaling persists across Llama, Qwen, Gemma, and Mistral, with each family covering multiple parameter scales. Cross-family comparisons therefore attribute the pattern to scale rather than a particular architecture.

  • Cross-Family Consistency: 5/5 cross-dataset consistency across Llama, Qwen, Gemma, and Mistral demonstrates architecture-independent inverse scaling.All four families exhibit the phenomenon despite differing attention mechanisms, training objectives, and design optimizations.
  • Size Range Fairness: 0.5B–405B parameter coverage across families enables within-family comparisons across small and large scales.Llama spans 1B–405B, Qwen 0.5B–32B, Mistral 7B–24B, and Gemma 1B–9B.
  • Size Comparisons: 200× fewer parameters still yielded higher inverse-problem accuracy for Gemma-2B than Llama-405B.This controlled cross-family comparison supports scale-dependent failure rather than a single-model anomaly.
  • Implications for Generalization: Scale-dependent degradation exceeds 50 percentage points on 7.7% of problems, sometimes outweighing architectural advantages.The result challenges scaling-law assumptions of uniformly monotonic performance improvement with compute.

A.5 Complete Statistical Validation

Statistical analyses provide convergent validation of inverse scaling, causal brevity effects, contamination independence, and continuous size dependence. Primary tests remain significant under multiple-comparison correction.

  • Inverse Scaling Validation: Cohen’s d = 1.34 and uniformly significant Mann–Whitney tests establish a large, reliable inverse-scaling effect across datasets.Small versus large model comparisons on inverse problems yielded p < 0.001 across all datasets.
  • Causal Mechanism: t = 7.80, p = 7.89 × 10^-12 across 96 problems supports a strong causal effect of brevity constraints on large-model performance.Paired within-problem comparisons contrasted control and brief conditions.
  • Contamination Independence: p = 0.230 with ϕ = 0.12 indicates no significant association between contamination indicators and inverse-scaling occurrence.The small effect size supports interpreting inverse scaling as genuine capability differences rather than memorization artifacts.
  • Continuous Size Effects: r = −0.388, p = 0.0035 confirms that inverse scaling varies continuously across the 0.5B–405B parameter spectrum.The negative correlation links increasing model size with lower inverse-problem accuracy.
  • Multiple Comparisons: 16 of 18 tests achieved p < 0.05, and Bonferroni correction preserves significance for the primary confirmatory tests.Convergent parametric, nonparametric, and correlation-based evidence strengthens confidence in the findings.

A.4 Complete Model Specifications

The study evaluates 31 diverse language models spanning 0.5B–405B parameters under deterministic decoding and finds severe scale-dependent degradation on inverse problems.

  • Model coverage: 31 models spanning 0.5B–405B parameters represent diverse architectural families and were evaluated using greedy decoding.The evaluated families include Llama, Qwen, Gemma, Mistral, DeepSeek, Yi, and StableLM.
  • Performance by scale: 78.9% versus 68.2%: small models achieve higher accuracy on inverse problems than on normal problems.This is a +10.7 percentage-point advantage for small models on inverse problems.
  • Performance by scale: 39.9 percentage points separate small and large model performance on inverse problems, with a large effect size of Cohen’s d = 1.34.The result demonstrates substantial scale-dependent degradation rather than a minor performance fluctuation.
  • Performance by scale: The 405B-parameter Llama-3.1-405B shows the most severe degradation, while mid-sized models exhibit intermediate patterns.Its gap is −42.2 percentage points, and the relationship between scale and inverse-problem performance is monotonic.

Appendix B: Evaluation Methodology

The evaluation uses identical deterministic prompts, task-specific answer extraction, and explicit criteria for classifying problem-level scaling patterns. Controlled brief and direct conditions test whether response length and reasoning quantity affect performance across model sizes.

  • Base evaluation: 31 models are evaluated with greedy decoding and identical prompts containing no chain-of-thought elicitation.Multiple-choice tasks use bare Question/Answer formats, while mathematical reasoning uses bare Problem/Solution formats.
  • Answer extraction: 98.5% extraction accuracy is achieved in manual validation of 200 randomly sampled responses.Responses that fail all extraction patterns are scored as incorrect.
  • Problem classification: A 5 percentage-point threshold separates normal scaling from inverse scaling and prevents categorization from minor random variation.Small models are defined as ≤10B parameters and large models as ≥70B, with mid-sized models excluded from inverse-scaling calculations.
  • Causal interventions: Three intervention conditions compare standard control prompts with brief and direct answer formats across seven models spanning the size spectrum.Brief prompts constrain response length, whereas direct prompts request only the answer, allowing reasoning quantity to be distinguished from reasoning absence.
  • Causal interventions: 60% versus 15%: large models reduce response length much more than small models under identical brevity instructions.Large-model median output decreases from 197 to 78 tokens under brief constraints, validating scale-dependent instruction sensitivity.
  • Causal interventions: 66.5% versus 61.7%: large-model accuracy is higher under Brief than Direct conditions, indicating that concise reasoning can outperform eliminating reasoning.The comparison supports testing whether reasoning is beneficial but excessive under standard prompts.
  • Statistical analysis: Paired t-tests compare control and brief conditions within each problem, testing whether the mean accuracy change is greater than zero.Acc_cond(p) denotes mean accuracy across large models under condition cond.
Loading 2604.00025v1…