PaperlayerView exact revision on arXiv

Source-linked AI summary

ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping

Shuang Chen, Yue Guo, Yimeng Ye, Shijue Huang, Wenbo Hu, Haoxi Li, Manyuan Zhang, Jiayu Chen, Song Guo, Nanyun Peng

arXiv:2510.08457v1cs.CL

TL;DR

MLRMs often overthink simple problems while under-exploring difficult ones, creating an efficiency–accuracy imbalance. ARES addresses this with difficulty-aware cold-start training and AEPO, which uses HWE tokens to trigger exploration and dynamic entropy/KL control to set its depth. Experiments report superior performance and reasoning efficiency across diverse mathematical, logical, textual, and multimodal benchmarks.

  • Problem

    MLRMs can generate unnecessarily long reasoning for simple tasks while providing insufficient exploration for challenging tasks, increasing cost and risking missed solutions.

  • Method

    ARES uses adaptive cold-start training with difficulty-proportional reasoning traces, followed by AEPO for HWE-triggered exploration and difficulty-aware entropy and KL control.

  • Results

    ARES achieves superior performance and reasoning efficiency across diverse benchmarks, including +19.0 on MathVision and +11.5 on MMMU-Pro over the best open-source models, plus 61.7 on AIME25.

  • Takeaways & Limitations

    ARES adaptively allocates reasoning effort by encouraging deeper exploration on hard tasks and shorter reasoning on easy tasks.

Abstract

from arXiv · show

Recent advances in multimodal large reasoning models (MLRMs) have substantially improved their ability to solve complex textual and visual tasks. However, these models tend to overthink on simple problems, producing unnecessarily lengthy reasoning traces, while under-exploring on challenging ones, leading to missed solutions. To address this imbalance, we propose ARES, a unified open-source framework for adaptive reasoning that dynamically allocates exploration effort based on task difficulty. Our approach is motivated by two key empirical findings: (i) while single-token entropy is noisy, high window-entropy (HWE) tokens (token-level entropies averaged under a sliding window) can reliably capture reasoning-critical moments; and (ii) reducing HWE usage benefits easy problems, while increasing it is essential for solving hard ones. Building on these insights, ARES introduces a two-stage training pipeline. In the Adaptive Cold-Start stage, we curate multimodal and textual data paired with reasoning traces of length proportional to problem difficulty, equipping the model with initial difficulty awareness. In the second stage, we develop Adaptive Entropy Policy Optimization (AEPO), which uses HWE tokens as exploration triggers to decide when to explore, and a hierarchical entropy reward with dynamic KL control to decide how much to explore. Extensive experiments demonstrate that ARES achieves superior performance and reasoning efficiency across diverse mathematical, logical, and multimodal benchmarks, while closing the gap to leading commercial systems under significantly lower inference costs.

1 INTRODUCTION

ARES addresses inefficient multimodal reasoning by adapting exploration to problem difficulty, using HWE tokens to trigger exploration and entropy-aware control to allocate its depth. Its experiments report improved performance and inference efficiency across diverse reasoning benchmarks.

  • Adaptive reasoning targets the cost of unnecessarily long chains of thought on easy tasks, which increases inference latency and limits real-world usability.
  • High-window-entropy tokens identify distinct exploration behaviors across easy and hard tasks, motivating difficulty-aware reasoning control.The analysis finds that reducing HWE usage benefits easy problems, whereas harder problems require greater exploration.
  • ARES combines adaptive cold-start data with Adaptive-Entropy Policy Optimization to align reasoning length and exploration with problem complexity.The cold-start stage pairs reasoning length with difficulty, while AEPO uses HWE regions and hierarchical rewards to control exploration depth.
  • ARES achieves superior performance and inference efficiency across mathematical, general-knowledge, textual, and multimodal reasoning benchmarks.

2 RELATIONS BETWEEN TOKEN ENTROPY AND REASONING DIFFICULTY

The analysis shows that window entropy provides a more reliable signal of reasoning-critical uncertainty than single-token entropy, while exploration should vary with task difficulty. Easy problems benefit from suppressing excess exploration, whereas hard problems benefit from sustained high-entropy reasoning.

  • 2.1 WINDOW ENTROPY SERVES AS A TRIGGER FOR EXPLORATION IN MLRMS: Window entropy aggregates uncertainty across consecutive tokens to capture persistent reasoning bifurcations rather than transient lexical ambiguity.The window size controls how many consecutive tokens are included, producing a smoother and more semantically aligned indicator.
  • 2.1 WINDOW ENTROPY SERVES AS A TRIGGER FOR EXPLORATION IN MLRMS: Moderate windows of 4–8 tokens achieve the best trade-off for detecting reasoning-critical tokens, outperforming single-token entropy and avoiding dilution from 16–32-token windows.Single-token measures are sensitive to lexical artifacts, while long windows incorporate low-entropy tokens.
  • 2.2 EXPLORATION DIFFERS ACROSS PROBLEM DIFFICULTY: For easy problems, below-threshold high-entropy counts yield higher accuracy and shorter responses, whereas hard problems gain accuracy from above-threshold exploration at the cost of longer responses.
  • 2.2 EXPLORATION DIFFERS ACROSS PROBLEM DIFFICULTY: Correct easy-task solutions use fewer high-entropy tokens and shorter responses, while correct hard-task solutions typically use more high-entropy tokens and longer responses.
  • 2.2 EXPLORATION DIFFERS ACROSS PROBLEM DIFFICULTY: These findings establish an entropy–difficulty interaction in which exploration should be suppressed for easy tasks but encouraged for hard tasks.This interaction motivates adaptively modulating entropy-based triggers across difficulty buckets.

3 METHOD

ARES adapts reasoning depth to task difficulty through a cold-start curriculum followed by entropy-guided policy optimization. Its design suppresses unnecessary exploration on easy problems while allocating more exploratory reasoning to hard problems.

  • Overview: ARES combines a difficulty-aware cold-start curriculum with KL-regularized AEPO to adapt reasoning depth in multimodal policies.The cold-start stage preserves length-controllable modes, while AEPO couples high-entropy-window detection with a difficulty-aware KL budget.
  • Motivation: Easy problems benefit from fewer high-entropy tokens and shorter responses, whereas hard problems achieve higher accuracy with additional high-entropy exploration.Figure 4 reports the entropy–difficulty interaction that motivates difficulty-aware exploration.
  • Adaptive Cold-Start Fine-Tuning: The cold-start stage pairs problem difficulty with target response lengths, sampling short reasoning for easy problems and longer reasoning for difficult ones.Target lengths are determined from pass-rate brackets using median response lengths for the easiest and hardest problems.
  • When to Explore: AEPO triggers branching when windowed entropy exceeds a dynamically updated batch threshold, concentrating exploration at sustained high-uncertainty regions.Per-trajectory 95th-percentile thresholds are averaged across trajectories to produce a batch-level cutoff.
  • How Much to Explore: AEPO regulates exploration with difficulty-aware hierarchical rewards and token-adaptive KL control, penalizing over-exploration on easy tasks and under-exploration on hard tasks.The reward uses batch statistics to target high-entropy-token counts, while KL constraints are relaxed inside validated high-entropy windows.

4 EXPERIMENTS

ARES is evaluated across diverse multimodal and textual reasoning benchmarks, where it improves over open-source baselines and adapts reasoning effort to task difficulty.

  • 4.2 MAIN RESULTS: MRLMs outperform chat models across complex reasoning, knowledge-intensive, and general-purpose benchmarks, although many open-source MRLMs show weaker textual reasoning without cold-start fine-tuning.
  • 4.2 MAIN RESULTS: ARES-7B exceeds the best open-source models by +19.0 on MathVision and +11.5 on MMMU-Pro, while scoring 61.7 on AIME25 versus below 3.3 for most 7B baselines.
  • 4.2 MAIN RESULTS: ARES achieves strong gains at both 3B and 7B scales and substantially outperforms open-source baselines on textual reasoning benchmarks.
  • 4.2 MAIN RESULTS: ARES modulates response length by difficulty, extending reasoning on OlympiadBench and AIME25 while shortening it on GSM8K and MathVista.
  • 4.3 ADDITIONAL VISUALIZATIONS: Figure 7 compares ARES-CS-Vanilla, ARES-CS-7B, and ARES-RL-7B across six benchmarks grouped as easy, medium, and hard.

5 VISUALIZATION RESULTS

Visualization and ablation results show that entropy-aware reinforcement learning improves accuracy while adapting response length, with dynamic KL and entropy reward jointly supporting efficiency.

  • 5 VISUALIZATION RESULTS: RL fine-tuning consistently improves accuracy over both cold-start variants across all benchmarks and difficulty categories.
  • 5 VISUALIZATION RESULTS: RL reduces response length on easier datasets but increases it on OlympiadBench and AIME25, allocating more reasoning to difficult tasks.
  • 5.1 ABLATION STUDIES: Together, dynamic KL and entropy shaping regulate reasoning depth and exploration allocation to improve accuracy and training efficiency.
  • 5.1 ABLATION STUDIES: The hierarchical entropy reward alone improves average accuracy by +1.8 points over GRPO while steadily reducing response length.
  • 5.1 ABLATION STUDIES: The dynamic KL component alone improves average accuracy by +1.3 points over GRPO and functions as an efficient exploration-budget allocator.
  • 5.1 ABLATION STUDIES: The full ARES model achieves the highest average accuracy of all configurations at 55.7 and the largest response-length reduction during training.

6 RELATED WORK

Related work addresses excessive or insufficient reasoning through fixed budgets, early exiting, adaptive training, and entropy-based exploration strategies.

  • 6 RELATED WORK: Multimodal large reasoning models extend chain-of-thought fine-tuning to heterogeneous visual and textual environments.
  • 6 RELATED WORK: Large reasoning models often generate excessive reasoning for trivial questions while providing insufficient exploration for challenging ones.
  • 6 RELATED WORK: Training-free approaches use token budgets, budget forcing, or dynamic early exiting, while training-based methods seek more strategic adaptive reasoning.
  • 6 RELATED WORK: Entropy has long supported exploration–exploitation trade-offs in reinforcement learning, but global entropy bonuses show mixed evidence for language models.

7 CONCLUSION

ARES addresses overthinking on simple problems and under-exploration on complex ones through difficulty-aware, entropy-guided adaptive reasoning.

  • 7 CONCLUSION: ARES combines Adaptive Cold-Start difficulty awareness with AEPO, which uses high window entropy to trigger exploration and hierarchical rewards to control reasoning depth.
  • 7 CONCLUSION: Experiments report superior performance and significantly improved reasoning efficiency, validating the adaptive entropy-guided approach.

A IMPLEMENTATION DETAILS

ARES is trained with supervised cold-start followed by AEPO, which uses entropy signals to identify coherent uncertainty and allocate exploration. The implementation combines token-level uncertainty, window aggregation, difficulty-aware sampling, and KL-controlled policy updates.

  • Training setup: ARES uses two training stages: supervised cold-start fine-tuning followed by Adaptive Entropy Policy Optimization.The cold-start stage uses two epochs of SFT; AEPO then performs reinforcement-learning policy updates.
  • Entropy signals: Token entropy measures local uncertainty from the policy’s next-token distribution rather than from the realized token.Higher entropy indicates multiple plausible continuations, while identical tokens can have different entropy at different timesteps.
  • RL baselines: GRPO normalizes rewards within prompt groups, whereas DAPO adds token-level updates, dynamic sampling, and overlong-reward shaping without an explicit KL penalty.DAPO is adopted as the primary RLVR baseline because of its empirical performance and widespread use.
  • Entropy signals: Sliding-window entropy smooths noisy single-token fluctuations and identifies persistent high-uncertainty regions associated with reasoning-critical phases.A window is flagged using an online dynamic threshold, and validated high-entropy tokens receive relaxed KL budgets in AEPO.

E KL PENALTY INFLATES GRPO ADVANTAGE VARIANCE COMPARED TO KL LOSS

The section analyzes how incorporating KL penalties into GRPO returns affects variance and contrasts this with actor-only KL loss. It also describes AEPO’s difficulty-aware entropy and KL-control pipeline.

  • Variance analysis: The implementation compares GRPO with KL penalty, which merges KL into returns, against GRPO with actor-only KL loss.In the KL-loss formulation, the KL gradient is not multiplied by the group advantage and is applied per token.
  • Variance analysis: Under the stated covariance conditions, Proposition 1 shows that a nonzero KL penalty increases GRPO advantage variance.The comparison assumes weak correlation between the advantage and score-function terms, with a broader covariance-based conclusion also noted.
  • Variance analysis: Merging per-token KL into the return injects task-agnostic high-frequency noise into group competition, amplifying advantage and policy-gradient variance.Actor-only KL regularization decouples trust-region control from task and entropy signals, allowing a dedicated KL-budget controller.
  • AEPO pipeline: AEPO estimates difficulty from rollout pass@G accuracy and recomputes difficulty buckets each iteration to parameterize entropy and KL budgets.The algorithm then computes window entropy, applies entropy shaping, forms group-centered advantages, and updates the actor and KL controller.
  • Ablation result: Figure 9 shows that GRPO and DAPO baselines plateau or lack stable improvement, whereas ARES variants achieve higher accuracy rewards, with combined KL regularization and entropy shaping most stable.The figure reports moving-average accuracy reward across training steps under ablation settings.

I VISUAL ANALYSIS OF ENTROPY REWARD DESIGN

ARES shapes entropy rewards according to task difficulty and deviation from a difficulty-dependent high-entropy target. The design suppresses overthinking on easy tasks, regulates reasoning depth on medium tasks, and encourages exploration on hard tasks.

  • Reward design: The reward curves use deviation from a difficulty-dependent target, where deviation compares detected high-entropy-token count with that target.This makes the shaping mechanism interpretable across difficulty levels.
  • Easy tasks: Easy-task rewards heavily penalize excess high-entropy activity while mildly encouraging exploration after incorrect answers.This favors short, efficient correct responses while preserving limited exploration when the initial attempt fails.
  • Medium tasks: Medium-task shaping is approximately symmetric around the target, tolerating moderate deviations while penalizing excessive deviation in either direction.Incorrect responses are encouraged to increase reasoning length, balancing efficiency with robustness.
  • Hard tasks: Hard-task rewards positively reinforce excess high-entropy activity for both correct and incorrect responses.The design encourages longer exploratory reasoning chains for difficult problems.
  • Reward design: Overall, entropy shaping suppresses unnecessary verbosity on easy problems, regulates depth on medium problems, and encourages exploration on hard problems.The strategy provides a smooth difficulty-aware mechanism for adaptive reasoning.

J WHY HIGH–ENTROPY TOKENS PREDICT REASONING RESPONSE LENGTH

The section models reasoning and answer emission as entropy-linked stopping processes to explain response length. Its theory predicts an approximately linear relationship between high-entropy activity and reasoning length, supported by diagnostics involving stopping, information, and accuracy.

  • Latent-process model: Under the two-state latent-process model, expected response length is affine in expected high-entropy-token count: E[L] = a′ + b′ E[NHE].The model distinguishes exploratory high-entropy reasoning from declarative low-entropy output and assumes stopping occurs in the latter state.
  • Entropy-dependent stopping: If answer-emission probability decreases with entropy, higher-entropy trajectories have stochastically larger stopping times and longer expected responses.Each high-entropy step contributes a lower stopping hazard under the stated monotonicity assumption.
  • Information-theoretic bound: An information-theoretic argument gives a linear lower bound on expected response length in terms of expected high-entropy activity.The argument assumes low-information non-reasoning steps and bounds per-step answer information by entropy.
  • Synthesis and predictions: The synthesized theory predicts a stable, monotone, near-linear relationship between reasoning length and high-entropy activity.The diagnostics include increasing E[L | NHE], decreasing stopping hazard with entropy, and concentration of per-step information in high-entropy windows.
  • Synthesis and predictions: The analysis treats high-entropy-token counts as a proxy for reasoning effort because they grow approximately with response length and accuracy in Figure 3.The reported relationship supports using entropy activity to diagnose reasoning depth, while the section’s assumptions define its theoretical scope.

K WHY KL LOSS IS A VALID Thinking Budget

The paper formalizes KL regularization as an operational thinking budget that limits policy deviation from a reference model and controls expected reasoning cost. AEPO extends this budget with token- and difficulty-aware allocation while preserving global trust-region control.

  • KL-as-budget: A global KL budget bounds changes in expected reasoning length or entropy-based thinking cost.Pinsker- and Donsker–Varadhan-type inequalities connect bounded policy divergence to explicit limits on expected costs.
  • Tokenized control: The budget becomes token-additive in autoregressive models, measuring cumulative exploration through per-token deviations.The trust-region radius controls allowable policy movement and corresponding growth of thinking cost.
  • Adaptive allocation: AEPO relaxes KL selectively inside validated high-entropy reasoning windows while maintaining a separate global control signal.Window weighting permits local exploration, whereas non-window KL control prevents underestimating total deviation and risking drift.
  • KL-as-budget: KL regularization is an operational thinking budget, not merely a regularizer.Strong duality identifies the KL loss with the multiplier of a reference-deviation constraint; dual updates track the target budget.
  • Adaptive allocation: Difficulty-specific multipliers control exploration scale, while token weights determine where deviations are encouraged.This produces a two-time-scale scheme with slow budget tracking and faster policy ascent.

M CASE STUDY

The case study contrasts reasoning behavior across easy, medium, and hard geometry problems, including examples of both correct and incorrect exploratory solutions. The examples illustrate that excessive or insufficient exploration can produce inefficient or unresolved reasoning.

  • Hard problems: The hard circle problem resolves the radii as rK = 16, rH = 4, and rJ = 8, then verifies the circumference condition exactly.The reasoning rejects interpreting the labels as center distances before treating 4x, x, and 2x as radii.
  • Easy problems: The easy circle-chord example reaches the correct length RS = 11 through a short congruent-arc argument.The solution sets equal chord expressions, obtaining x = 4 and matching both chord lengths at 11.
  • Easy problems: The easy triangle example also confirms cos B = 7/25 using both a direct trigonometric ratio and the Law of Cosines.The side lengths satisfy 14^2 + 48^2 = 50^2, confirming the right-triangle interpretation.
  • Medium problems: The medium examples show that exploratory reasoning can remain incorrect: one solution is indeterminate, while another depends on an uncertain side-label assumption.The cube-net attempt lacks enough geometric information, and the triangle attempt notes that the side labeled 15 may not be opposite 30°.
Loading 2510.08457v1…