PaperlayerView exact revision on arXiv

Source-linked AI summary

When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction

Feiyang Ren, Shengtao Wen, Lingbing Guo, Yu Tian, Yuanning Cui, Xiang Chen

arXiv:2608.15592v1cs.AI

TL;DR

Entropy-only pooling misses semantically important tokens, limiting reliable LLM output-length prediction for efficient serving. ESTP combines token entropy with attention-based semantic importance and improves prediction and end-to-end serving performance across most evaluated scenarios.

  • Problem

    Entropy-only pooling does not reliably capture token semantic importance, limiting accurate output-length prediction for length-aware LLM serving.

  • Method

    ESTP combines token-level entropy with semantic importance derived from prefill self-attention and uses soft-label regression for length prediction.

  • Results

    ESTP reduces average MAE by over 9 points versus EGTP, with reductions exceeding 20 points in reinforcement-learning scenarios, and improves accuracy across most benchmark scenarios.

  • Takeaways & Limitations

    ESTP provides a lightweight building block for length-aware serving that improves throughput and reduces padding ratios in end-to-end tests.

  • Takeaways & Limitations

    ESTP’s attention-based semantic-importance proxy is correlational rather than causal, and closed-source applicability is limited by inaccessible activations.

Abstract

from arXiv · show

Efficient LLM serving is often bottlenecked by the need to pad sequences to a fixed maximum length, and this wastes compute and degrades throughput. Predicting output lengths in advance makes it possible to adopt length-aware scheduling, and this reduces the overhead. This advantage is especially pronounced in long-context reasoning and reinforcement learning applications. Existing approaches, such as entropy-guided token pooling, use token-wise entropy as their primary signal, but they tend to ignore differences in semantic content across tokens. So, important tokens are often underweighted, and tokens carrying little information receive disproportionate emphasis. This hurts the reliability of length prediction. We introduce ESTP (Entropy-and-Semantic Token Pooling), a lightweight framework that addresses this issue by combining entropy with attention-based importance scores. These scores are derived directly from the self-attention weights computed during the LLM prefill phase, and this allows ESTP to capture both uncertainty and semantic importance with minimal additional computation. Since the framework reuses prefill activations, it adds almost no extra memory overhead and introduces only minimal latency. On the ForeLen benchmark, ESTP outperforms baseline methods, achieves better prediction accuracy and lower error rates in most scenarios. When integrated with a length-aware scheduler in end-to-end system tests, it further helps improve overall throughput and reduce the padding ratio. Our results offer a practical and effective building block for length-aware LLM serving systems.

Introduction

Efficient LLM serving is hindered by padding waste, especially in high-length-variance workloads, while accurate output-length prediction enables length-aware scheduling. ESTP addresses entropy–semantic misalignment by combining token entropy with attention-based semantic importance at negligible additional inference cost.

  • Motivation: Padding shorter sequences to the batch maximum wastes computation, particularly in long-context reasoning and dynamic reinforcement-learning sampling.These workloads exhibit high length variance, making the barrel effect especially severe.
  • Motivation: Accurate pre-generation output-length prediction enables length-aware scheduling that reduces padding waste and improves hardware utilization.Existing approaches include external auxiliary models and LLM-intrinsic methods such as TRAIL and EGTP.
  • Entropy–Semantic Misalignment: A diagnostic on 512 RL-sampled instances found substantial divergence between token entropy and semantic importance estimated from self-attention weights.When tokens were split at the 60th percentile, many high-entropy tokens carried little semantic importance.
  • Method: ESTP combines entropy with attention-based semantic weights and applies a temperature-scaled softmax to produce pooling weights.The framework reuses self-attention weights from LLM prefill and uses a soft-label regression head, adding negligible inference cost.
  • Evaluation: ESTP is evaluated on ForeLen across long-sequence generation, complex reasoning, and dynamic RL sampling using Qwen2.5-3B/7B and Llama3.2-1B/3B backbones.The introduction reports generally reduced prediction error across most benchmark scenarios.
  • Entropy–Semantic Misalignment: Semantically critical tokens can have high prediction confidence and therefore receive insufficient weight, showing that uncertainty alone is an unreliable relevance signal.This mismatch affects entities, numbers, constraints, and key instructions, whereas entropy-only weighting emphasizes transitional, reasoning-boundary, and filler words.

Related Work

Prior LLM serving work improves throughput through dynamic request management, scheduling, memory management, and parallel computing. Output-length prediction has used external auxiliary models, while TRAIL instead leverages the target LLM to reduce overhead and maintain accuracy.

  • Continuous batching, batch prompting, and PagedAttention improve LLM serving throughput by dynamically managing requests.
  • Recent serving studies advance throughput through scheduling strategies, memory management, and parallel computing techniques.
  • SARATHI improves decoding throughput with chunked prefill and decode piggybacking.
  • Conventional output-length predictors use external auxiliary models such as DistilBERT-based classifiers, which process prompts and require substantial computation and training time.
  • TRAIL predicts output length with the target LLM, achieving low overhead and accurate predictions.

Preliminaries

Entropy correlates with output-length prediction but can underrepresent semantically central tokens. Attention-based importance provides complementary evidence, with high-attention tokens aligning with semantic importance and token contribution.

  • Entropy limitations: Entropy-weighted pooling supports lightweight length prediction, but high-entropy weighting can dilute semantically core tokens and lose critical information.The limitation arises because entropy primarily reflects model uncertainty rather than semantic content.
  • Semantic motivation: Key entities and core instructions often appear in low-entropy tokens with high attention weights.Attention weights indicate how much a token contributes to the overall contextual representation.
  • Empirical alignment: 80% of high-attention tokens are identified as semantically important.This supports alignment between high attention weights and semantic importance.
  • Attribution evidence: r=0.424 links attention scores and token importance, supporting high-attention tokens as informative for length prediction.A gradient-based attribution analysis measures each token’s contribution to the final prediction.

Method

ESTP combines attention-derived semantic importance with token entropy to form pooled representations for sequence-length prediction. Its prediction head uses joint classification and regression objectives to improve robustness and accuracy.

  • Semantic weighting: ESTP computes each token’s semantic importance by averaging final-layer, multi-head self-attention values across the attention matrix’s column dimension.Padding tokens and self-similarity are handled explicitly to preserve the rationality and accuracy of the weights.
  • Entropy estimation: Token entropy is derived from each hidden state’s next-token probability distribution over the full vocabulary.The entropy calculation follows the scheme introduced in EGTP.
  • Semantic-and-entropy pooling: ESTP weights and combines semantic scores S_i with entropy H_i into final token scores T_i, then normalizes them with temperature-controlled softmax weights.The resulting semantic-entropy weights are used to aggregate hidden states into a representation h.
  • Joint optimization: The model is optimized with a combined cross-entropy and mean squared error loss to stabilize distribution learning while reducing predicted-length error.A balancing hyperparameter λ_3 weights the two loss components.
  • Length prediction head: The prediction head converts continuous length targets into soft distributions over K bins and jointly predicts a K-dimensional classification distribution and an expected regression length.The regression output is computed as the expectation of the predicted bin distribution, using each bin center c_i.

Experiments

Experiments across four LLMs and multiple prediction and serving scenarios show that ESTP improves length-prediction accuracy and robustness while maintaining low inference overhead. Its entropy–semantic pooling also improves end-to-end scheduling outcomes, including padding, throughput, and JCT.

  • Experimental setup: Experiments evaluate ESTP on ForeLen across long-sequence reasoning, reinforcement learning, and other challenging output-length prediction scenarios.ForeLen is constructed specifically to assess predictor capabilities under complex and realistic conditions.
  • Experimental setup: ESTP is compared across Qwen2.5-3B/7B and Llama3.2-1B/3B against SSJF-Reg, SSJF-MC, TRAIL, LTR-C, and EGTP.Evaluation uses MAE, RMSE, length-bin accuracy, inference time, GPU memory, Throughput, JCT(Job Completion Time), and Padding Ratio.
  • Prediction accuracy: Across all evaluated models, ESTP achieves the lowest average MAE and reduces Avg MAE by over 9 points versus EGTP, especially over 20 points in RL scenarios.The method provides broad gains across a variety of models and most scenarios.
  • Prediction robustness: On Qwen2.5-7B, ESTP achieves the best RMSE in both RL and Reasoning scenarios and the lowest average RMSE.RMSE evaluation examines sensitivity to large prediction errors.
  • End-to-end serving: With an SJF scheduler and vLLM, ESTP reduces Reasoning Padding Ratio from 1.18 to 0.37 versus EGTP and improves LongSeq throughput while reducing average JCT.ESTP achieves the best end-to-end results across Reasoning and LongSeq scenarios.

Conclusion and Future Outlook

The paper concludes that ESTP addresses the semantic limitations of entropy-only pooling and improves prediction and end-to-end performance, while identifying causal attribution and black-box adaptation as future directions.

  • Conclusion: ESTP combines attention-based semantic representations, token-level entropy, and soft-label regression to address entropy-only pooling’s semantic shortcomings and heavy-tailed generation lengths.The framework is designed as a lightweight approach to reclaim semantic information that entropy-only pooling misses.
  • Conclusion: Experiments across several ForeLen tasks and multiple open-source LLMs show that ESTP outperforms prior methods and substantially improves end-to-end system performance.The reported empirical benefits span prediction tasks and system-level evaluation.
  • Future Outlook: ESTP’s attention-based proxy is correlational rather than causal, limiting how directly its semantic importance estimates can be interpreted.This limitation motivates future work on causal attribution and semantic–positional disentanglement.
  • Future Outlook: Closed-source applicability is limited because the method depends on internal activations that are inaccessible in black-box settings.Future work therefore targets black-box adaptation without internal activations.

Reproducibility Checklist

The reproducibility checklist instructs authors to replace response placeholders with valid answers while preserving the document’s structure. It may be included in the main LaTeX file or compiled separately, depending on conference requirements.

  • Completion instructions: Authors should replace only “Type your response here” with an applicable response such as yes, no, partial, or NA.The checklist illustrates this process using a proof-inclusion question answered with “yes.”
  • Completion instructions: Authors must not modify other parts of the question command or other lines in the document.
  • Submission options: The checklist can be included before \end{document} or compiled as a standalone document, subject to conference-specific submission instructions.

1. General Paper Structure

The paper clearly presents its methods, distinguishes opinions and hypotheses from objective results, and provides pedagogical references to support unfamiliar readers.

  • The paper includes a conceptual outline and/or pseudocode description of the introduced AI methods.
  • The paper clearly separates opinions, hypotheses, and speculation from objective facts and results.
  • The paper provides well-marked pedagogical references giving less-familiar readers background needed to replicate it.

2. Theoretical Contributions

The paper is assessed as making theoretical contributions with clearly stated assumptions and formally presented novel claims. The review also indicates that proofs, intuitions, citations, empirical demonstrations, and experimental code are provided.

  • The paper makes theoretical contributions, and its assumptions and restrictions are stated clearly and formally.
  • Novel claims are stated formally, with proofs and proof sketches or intuitions included for complex or novel results.
  • The paper provides appropriate citations to the theoretical tools it uses.
  • Theoretical claims are demonstrated empirically, and all experimental code used to eliminate or disprove claims is included.

3. Dataset Usage

The paper relies on datasets and provides a motivation for conducting experiments on the selected datasets. Existing datasets are appropriately cited and publicly available, while novel or unavailable datasets are not applicable.

  • Dataset Usage: The paper relies on one or more datasets and motivates conducting experiments on the selected datasets.Both dataset reliance and motivation are marked yes.
  • Dataset Usage: Novel datasets are not introduced, so data-appendix inclusion and public-release requirements are not applicable.Both criteria are marked NA.
  • Dataset Usage: Datasets drawn from existing literature are accompanied by appropriate citations and are publicly available.Both criteria are marked yes.
  • Dataset Usage: The requirement to describe non-public datasets and justify the lack of suitable public alternatives is not applicable.This criterion is marked NA.

4. Computational Experiments

The experiments provide substantial reproducibility and evaluation documentation, including code, infrastructure, metrics, runs, variation measures, and final hyperparameters. Statistical significance testing is only partially addressed, as is reporting the development-time hyperparameter search and selection criterion.

  • Reproducibility: The paper includes preprocessing and experiment-analysis code in the appendix, with plans for public research use and implementation comments linked to the paper.The reproducibility checklist marks items 4.3–4.6 as yes.
  • Reproducibility: The paper describes randomness seeding and specifies the hardware, software, operating system, and relevant library and framework versions used for experiments.The checklist marks both replication-related items 4.7 and 4.8 as yes.
  • Evaluation methodology: Evaluation metrics are formally defined and motivated, and the paper reports the number of algorithm runs used for each result.Checklist items 4.9 and 4.10 are marked yes.
  • Evaluation methodology: The experimental analysis includes variation, confidence, or other distributional information beyond averages or medians.Checklist item 4.11 is marked yes.
  • Statistical analysis: Judging the significance of performance improvements or decreases with appropriate statistical tests is reported only partially.Checklist item 4.12 is marked partial.
  • Experimental configuration: Final hyperparameters for each experimental model or algorithm are listed, while development-time search ranges and selection criteria are reported only partially.Checklist items 4.13 and 4.2 are marked yes and partial, respectively.
Loading 2608.15592v1…