PaperlayerView exact revision on arXiv

Source-linked AI summary

Hermes: Learning Contextual Reasoning Unlocks Test-Time Scaling

Xinyu Li, Mononito Goswami, Hao Liu, Nikos Kanakaris, Langlin Huang, Prithwish Jana, Patrick Blöbaum, Purak Jain

arXiv:2609.38332v1cs.LGcs.AIcs.CL

TL;DR

Effective test-time scaling across multiple contexts requires models to decide how to allocate fresh contexts and what information to carry forward, choices often prescribed by existing harnesses. Hermes exposes these choices to models, and Hermes-Learn trains them to make them; trained models adapt their strategies and generalize beyond the training settings.

  • Problem

    Using additional compute across context windows requires deciding how to allocate fresh contexts and what information to carry forward, decisions existing approaches largely prescribe through their harnesses.

    Source: §1 · p. 2

  • Method

    Hermes provides configurable delegation and digestion, while Hermes-Learn combines supervised fine-tuning and curriculum reinforcement learning to teach strategy selection and composition.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Trained models adapt contextual strategies to the problem and reasoning progress, generalize across benchmarks and model families, extrapolate beyond training compute, and transfer to complementary scaling methods.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The results establish contextual reasoning as a learnable model capability and an important component of long-horizon test-time scaling.

    Source: §7 · p. 12

    Hermes uses a hand-designed strategy space and focuses on learning delegation, relying on the base model’s zero-shot ability to summarize and resume reasoning.

    Source: A Limitations and Future Work · p. 16

Abstract

from arXiv · show

Test-time scaling improves model performance by allocating additional compute during inference. Using this compute effectively across multiple context windows requires deciding how to allocate fresh contexts and what information to carry between them. We call a model's ability to make these decisions contextual reasoning. Existing approaches largely prescribe these decisions through their harness; we instead shift them to the model. We introduce 1) Hermes, a family of simple, configurable harnesses that progressively varies model control over context allocation and reuse, and 2) Hermes-Learn, a two-stage framework for learning these capabilities. We find that capable models can exploit this flexibility to scale with additional inference-time compute, while smaller open-source models initially struggle to do so. Training with Hermes-Learn closes this gap, inducing adaptive contextual reasoning strategies that vary with both the problem and the progress of reasoning. These gains generalize across benchmarks and models, extrapolate beyond the inference-time compute seen during training, and transfer to complementary test-time scaling methods beyond Hermes.

1 Introduction

The paper frames reasoning across multiple contexts as a distinct capability: models must decide how to allocate fresh contexts and what information to carry forward. It introduces Hermes and Hermes-Learn, reporting that learning helps smaller models use additional inference-time compute and generalizes across settings.

1 Introduction

  • The paper distinguishes mathematical reasoning within a context from contextual reasoning, which coordinates separate contexts as reasoning unfolds.It focuses on the latter capability.Source: §1 · p. 1
  • Existing approaches largely encode context-allocation and carry-forward decisions in their harnesses, although the best use of fresh contexts can change during problem solving.A proof may benefit from parallel exploration, then verification, and later continuation of a promising line.Source: §1 · p. 2
  • Hermes shifts decisions about allocating and reusing contexts from the harness to the model through configurable levels of steering.Its mechanisms are delegation and digestion.Source: §1 · p. 2
  • Hermes-Learn closes the contextual reasoning gap, and learned strategies adapt to problem structure and reasoning progress, generalize across benchmarks and model families, extrapolate compute, and transfer to other scaling methods.The reported gains do not degrade mathematical reasoning.Source: §1 · pp. 2–3
  • Hermes-Learn combines supervised fine-tuning and curriculum reinforcement learning to teach models to select and compose delegation strategies.The curriculum progressively reduces harness steering.Source: §1 · p. 2

2 Related Work

Prior work often prescribes how contexts are allocated or reused, including through delegation and compaction. Hermes treats contextual strategy selection as a model capability, and Hermes-Learn trains models to select and compose strategies.

2 Related Work

  • Prior multi-context approaches generally prescribe delegation strategies or compress prior reasoning for reuse, while Hermes studies adaptive strategy choice as a model capability.Hermes can also benefit complementary methods that use search with self-reflection.Source: §2 · p. 3
  • Existing training methods align models with particular inference harnesses but typically leave contextual strategy choices encoded in the harness.Hermes-Learn instead teaches models to autonomously select, instantiate, and compose strategies.Source: §2 · pp. 3–4
  • Table 1 contrasts methods by model-driven strategy selection, context compaction, and recursive delegation.Hermes-Learn trains model-driven selection through the steering curriculum.Source: Table 1 · p. 3, §2 · p. 4

3 HERMES: A Family of Simple, Flexible Harnesses

Hermes provides delegation and digestion to extend reasoning across context windows, while varying how much the harness steers strategy choice. Its evaluations show that capable models can use the added context effectively, whereas Qwen3-4B degrades.

3 HERMES: A Family of Simple, Flexible Harnesses

  • Hermes is designed to remain simple, flexible, and configurable, with steering that ranges from prescribing a strategy to letting the model choose.Its two generic mechanisms allocate fresh contexts and carry useful information across them.Source: §3 · p. 4
  • Delegation assigns work to fresh contexts, while digestion carries accumulated progress into a fresh context for continued reasoning.The two operations extend reasoning across contexts in parallel and sequentially.Source: §3 · p. 4, Figure 1 · p. 4
  • Each agent can reason, answer, or launch subagents that work on specified tasks in fresh contexts and return compact reports.Delegation may recur to a configured depth, and agents return their best current answer when their step limit is reached.Source: §3 · p. 4
  • The L3–L1 hierarchy progressively shifts delegation control from harness-prescribed strategies to model-selected strategies as reasoning unfolds.At L2, the harness specifies a search or verification category and the model selects strategies within it.Source: Figure 1 · p. 4, §3 · p. 5
  • Claude Sonnet 5 and DeepSeek-V3.2 match or exceed inline performance with Hermes, while Qwen3-4B degrades despite reasonable inline mathematical reasoning.Table 2 reports Avg@16 on pooled and synthetic benchmarks, comparing inline reasoning with Hermes L1 and L2; the caption describes 8K-token inline contexts and up to 56K tokens across contexts.Source: Table 2 · p. 5, §3 · p. 6

4 Learning Contextual Reasoning

Hermes-Learn focuses on learning root-level delegation, using supervised fine-tuning followed by reinforcement learning to develop more autonomous strategy selection. It relies on the model’s existing digestion ability rather than explicitly training digestion.

4 Learning Contextual Reasoning

  • Hermes-Learn first teaches delegation with supervised fine-tuning, then uses reinforcement learning to develop autonomous strategy selection.The framework relies on the model’s existing ability to summarize and continue from digests rather than explicitly training digestion.Source: §4 · p. 6
  • SFT warm-starts the model on successful teacher trajectories from different steering levels, retaining root-agent trajectories that delegate at least once and answer correctly.The SFT stage provides an initialization for reinforcement learning.Source: §4 · p. 6
  • Training optimizes root-agent trajectories only; joint root–subagent optimization and credit assignment are left for future work.The authors note that extending training would require optimizing policy-generated subtasks and assigning credit to individual subagent outcomes.Source: §4 · p. 7
  • RL trains on root-agent rollouts with binary correctness rewards, using a curriculum that gradually shifts from more-steered L2 toward less-steered L1.Training retains some L2 exposure as the model takes greater control over delegation strategy.Source: §4 · pp. 6–7
  • Synthetic problems vary whether splitting, chaining, verification, or additional attempts are useful, broadening strategy coverage in training and a held-out evaluation set.Splitting combines independent problems whose joint answer creates opportunities for parallel delegation.Source: §4 · p. 7

5 Experimental Setup and Main Results

Across the mathematical benchmarks, Hermes-Learn improves performance when models use additional contexts, and its strategy choices vary with both the problem and reasoning progress. Evaluation uses constrained 8K-token contexts and includes inline and larger-context baselines.

5 Experimental Setup and Main Results

  • Evaluation covers pooled AIME/HMMT, synthetic problems, BeyondAIME, IMO-AnswerBench, and FrontierScience, with an 8K-token context per agent.The reported accuracy is Avg@16 under inline reasoning and Hermes L1.Source: §5 · p. 8
  • The baselines include 56K-token inline reasoning and 8K-token inline GRPO, which separates cross-context reasoning gains from larger context or RL alone.With T = 3 and B = 3, the 56K reference matches the maximum total context of a three-step Hermes run.Source: §5 · p. 8
  • Hermes-Learn’s SFT+RL under L1 beats the base model’s 56K inline reference on all four mathematical benchmarks, while inline performance changes relatively little.The table reports Avg@16 accuracy; the base model performs substantially worse under L1 than inline reasoning, and SFT and RL progressively improve L1 performance.Source: §5 · pp. 8–9, Table 3 · p. 9
  • After training, Split is less common on the pooled benchmark but remains prevalent on synthetic problems designed to benefit from decomposition.At the second delegation step, the model often switches to verification or continues its first-step strategy.Source: §5 · p. 9, Figure 3 · p. 9

6 Analysis and Discussion

The ablations identify teacher-data composition and curriculum as factors in Hermes-Learn’s strongest results, while tests show generalization across compute budgets, scaling methods, and model families.

6.1 (RQ3) Teacher diversity, curriculum, and synthetic data shape HERMES-LEARN.

  • The ablations vary teacher source and demonstration volume for SFT, then L1/L2 exposure and order and data augmentation for RL.Mix and Curriculum use the same number of L1 and L2 episodes, differing in order.Source: §6.1 · p. 9
  • The full trajectory set from both teachers gives the best SFT performance, but token-matched mixed data performs similarly to DeepSeek-V3.2 alone.The authors suggest much of the full-set gain comes from more high-quality demonstrations rather than teacher diversity alone.Source: §6.1 · p. 10
  • Curriculum paired with synthetic problems yields the strongest L1 performance; rephrasing the same number of existing problems does not produce a comparable gain.Inline performance remains largely unchanged, with improvements concentrated under L1.Source: §6.1 · p. 10

6.2 (RQ4) HERMES-LEARN generalizes beyond its training setting

  • After RL, Hermes-Learn improves performance with longer sequential reasoning, more subagents, and deeper delegation, including configurations unseen during training.SFT benefits primarily from additional context windows, with little gain from more subagents or deeper delegation.Source: §6.2 · p. 11
  • Under RSA, performance improves over the base model, with further gains when combined with L1; under DSM, L1 recovers and substantially improves sharply degraded performance.DSM alone degrades sharply under the 8K-token context limit.Source: §6.2 · p. 11
  • On Olmo-3-7B-Instruct, SFT reverses L1 degradation and RL further improves performance while inline reasoning remains unchanged.This applies the same training pipeline to a different model family and scale.Source: §6.2 · p. 11

7 Conclusion

Hermes shifts context allocation and reuse from the harness toward the model, and Hermes-Learn addresses smaller open-source models’ contextual reasoning gap. The learned strategies scale with additional inference compute, adapt as reasoning unfolds, and transfer across model families and scales.

7 Conclusion

  • Hermes uses delegation and digestion to shift control over context allocation and reuse from the harness to the model.The harness family supports test-time scaling beyond a single context window.Source: §7 · p. 11
  • Hermes-Learn improves contextual reasoning for smaller open-source models while preserving in-context reasoning performance.Frontier models can exploit Hermes’ flexibility, whereas smaller open-source models exhibit a substantial contextual reasoning gap.Source: §7 · p. 11
  • Trained models scale with additional inference-time compute, adapt strategies as reasoning unfolds, and transfer across model families and scales.The authors conclude that contextual reasoning is a learnable model capability and an important component of long-horizon test-time scaling.Source: §7 · p. 12

A Limitations and Future Work

The section’s scope includes limits in evaluation, learned contextual reasoning, and root-only training, along with future directions concerning broader evaluation, delegation and digestion, and inference cost. Some supplied passages are procedural rather than substantive limitations or future-work findings.

A Limitations and Future Work

  • Evaluation is limited to AIME, HMMT, and science tasks under an artificial 8K-token context, leaving performance on native long contexts and evolving, tool-using tasks untested.The paper names SWE-bench, scientific discovery, and deep research as natural future evaluation settings.Source: A Limitations and Future Work · p. 16
  • Hermes relies on manually specified strategies and learns delegation but not digestion, making jointly learning useful summaries and reasoning from them an important next step.The strategy menu, search–verification distinction, and L3–L1 hierarchy are hand-designed; the base model supplies zero-shot summarization and resumed reasoning.Source: A Limitations and Future Work · p. 16
  • Training optimizes only root-agent trajectories, avoiding the added sample-complexity and credit-assignment burden but preventing subagents from adapting to delegated tasks.Joint root-and-subagent optimization is proposed, potentially with process-level supervision or learned judges; RAO [6] is cited as an example of recursive-tree optimization.Source: A Limitations and Future Work · pp. 16–17
  • Hermes exchanges additional inference computation for improved reasoning, motivating evaluation of accuracy, token use, latency, and cost and learning when another context is worthwhile.The longer-term aim is for models to learn the contextual-reasoning interface itself, rather than merely control a fixed one.Source: A Limitations and Future Work · p. 17

B Additional Discussion of Related Work

  • Related training methods use distinct credit-assignment schemes but generally train models to execute contextual-reasoning strategies prescribed by their harnesses.Examples include SPIRAL, PaCoRe, RAO [6], Context-Folding, and Reasoning Cache.Source: B Additional Discussion of Related Work · p. 17

C HERMES Harness Details

  • Experiments use the L1 harness with an 8K-token limit, three steps, delegation depth one, and up to three subagents, with digestion disabled.The root and subagents share the same underlying model and context limit; bounded contexts also permit parallel branches and may reduce wall-clock latency.Source: C HERMES Harness Details · p. 17

C.1 Harnesses in the HERMES Hierarchy

  • The hierarchy varies model control: L1 exposes all six search and verification strategies at each delegation step, while L2 fixes a category per step and offers search–verify or search–search variants.Search includes independent solving, diverse approaches, and decomposition; verification checks candidate answers, their supporting steps, or disagreements.Source: §C.1 · p. 18

C.2.1 System Prompt; C.2.4 Return Step; C.2.5 Subagent Prompts; C.2.6 Digestion across Context Windows

  • The prompts implement delegation followed by a non-delegating return, with subagents reporting compact findings and cross-window continuation using digests composed from prior summaries and the current root trajectory.A continued window starts with the original problem and generated digest; digest instructions preserve attempted approaches, verified results, the current answer, confidence, and remaining checks.Source: §C.2.1 · p. 18, §C.2.4 · p. 21, §C.2.5 · pp. 21–22

C.2.6 Digestion across Context Windows

  • The prompt asks for independently executable tasks with relevant context and specifies a concise first-person output format.It directs delegation to one or more subagents and says each task should be written on its own line beginning with “TASK:”.Source: §C.2.6 · p. 23

D Data Augmentation Details

The appendix describes synthetic problems designed to elicit varied delegation strategies, how they are composed and rewritten, and how training and evaluation sets are constructed. Rewriting is checked for solvability, and the final training set balances problem types.

D Data Augmentation Details

  • Five synthetic problem types are designed to encourage splitting, sequential delegation, verification, multiple independent attempts, or a single attempt, so delegation can adapt to the problem.The types are Splitting, Chaining, Verifying, Multi-attempt, and Single-attempt.Source: D Data Augmentation Details · p. 23

D.1 Synthetic Problem Generation Pipeline

  • Each synthetic problem is composed from seed problems with a type-specific answer rule, then rewritten by Claude Sonnet 5 and rejected if post-rewrite solvability falls below the stated threshold.The rewrite is rejected if no post-rewrite attempt is correct or its solve rate is at most one third of the pre-rewrite rate.Source: §D.1 · pp. 23–24

D.2 Synthetic Problem Construction

  • The training set contains 500 synthetic problems covering 505 distinct seeds, with 60% composite Splitting or Chaining problems and matched correct/incorrect Verifying pairs.These are mixed with AceReason-5K for an RL training set of 5,739 problems, of which 8.7% are synthetic.Source: §D.2 · p. 26
  • The held-out synthetic evaluation set contains 88 AIME- and HMMT-derived problems, distributed across the same five problem types.Golden answers are recomputed independently from the seed problems' reference answers.Source: §D.2 · p. 26

E.1 Pooled Benchmark Details

The pooled benchmark contains 238 audited AIME and HMMT problems, with exclusions and corrections made to address duplication, missing diagrams, grading scope, and answer or transcription errors. Hermes evaluations use constrained contexts and report average accuracy over 16 sampled solutions.

E.1 Pooled Benchmark Details

  • Five HMMT problems are excluded for duplication, a missing diagram, or proof-style format incompatible with final-answer grading.The remaining 148 HMMT answers were independently derived or checked, and 19 transcription or answer corrections were applied.Source: §E.1 · p. 26
  • The final pooled benchmark has 238 problems: 90 AIME and 148 HMMT, after auditing, decontamination, and corrections to problem statements and answers.The collection is graded with Math-Verify4 against ground-truth answers.Source: §E.1 · pp. 26, 28

E.2 Inference under HERMES Harnesses

  • Hermes evaluations limit agents to an 8,192-token context and 1,024 generated tokens per step, and report average accuracy over 16 sampled solutions.At maximum delegation depth, subagents may use up to 7,680 generated tokens for their first solve-and-report step.Source: §E.2 · p. 28

E.3 Training Experiments Details

The training setup builds a cleaned AceReason dataset, uses successful teacher trajectories for supervised fine-tuning, and then applies a harness curriculum during reinforcement learning. Multiple teachers and steering levels broaden the delegation patterns represented in the SFT data.

Training Dataset

  • AceReason-5K is a cleaned 5,239-problem subset of AceReason-Math, and 1,000 problems are uniformly sampled from it to form AceReason-1K.Cleaning includes deduplication, decontamination, and reference-answer verification.Source: §E.3 · p. 28

Stage I: Supervised Fine-Tuning

  • Successful teacher trajectories from Claude Sonnet 5 and DeepSeek-V3.2 provide diverse delegation demonstrations, with complementary preferences for exploring different approaches and making independent attempts.Across harnesses, L2-SS favors continued search, L2-SV favors verification, and L1 gives the model more control over delegation.Source: §E.3 · pp. 28–29
  • The final SFT dataset balances strategy occurrence across teachers and contains 5,466 trajectories, split evenly between the two teachers.Trajectories are retained when they include delegation and reach the correct final answer.Source: §E.3 · pp. 28–29

Stage II: Reinforcement learning

  • RL uses a curriculum that shifts from L2 harnesses early in training toward mostly L1 later, while maintaining equal overall exposure to both harness levels.Training mixes AceReason-5K with 500 synthetic problems, for 5,739 total problems.Source: §E.3 · p. 29

E.4 Labeling Executed Contextual Reasoning Strategies

The strategy-labeling procedure asks a judge to label the work requested by each delegation step while limiting its access to harness and model cues. The rubric permits multiple labels for a step, and the supplied agreement results caution against treating all label distinctions as equally reliable.

E.4 Labeling Executed Contextual Reasoning Strategies

  • Two independent judge passes on a small subset of trajectories achieved 79.4% agreement; because the rubric is multilabel, prior single-label figures do not transfer and agreement must be re-measured under this schema.Source: §E.4 · pp. 30, 32
  • The judge labels executed strategies from the problem, delegation instructions, and prior subagent results, without access to harness identity or model identity.Tasks are sorted before labeling, though the number of delegated tasks may still leak information about the harness.Source: §E.4 · pp. 29–31
  • A delegation step may receive multiple strategy labels, and a mixed fanout turn can qualify as both same_solve and different_methods.The rubric distinguishes checking existing work from solving from scratch and allows co-occurring labels where both apply.Source: §E.4 · p. 31
  • Raters' disagreement is concentrated within the four verify sub-tags rather than between search and verification, supporting verify as one category rather than four.The cited result reports 78.8% of disagreement internal to verify labels and none between search and verify.Source: §E.4 · p. 32

E.5 Additional Delegation Behavior Analysis

Hermes-Learn makes delegation strategies depend on both problem structure and reasoning progress. Verification becomes more prominent at the second step, with its form varying according to the initial search strategy.

E.5 Additional Delegation Behavior Analysis

  • Split remains common on decomposition-friendly synthetic problems but becomes much less frequent on the pooled benchmark after training.This reflects increasingly problem-dependent strategy selection.Source: §E.5 · p. 33, Figure 5 · p. 34
  • The second delegation step is often used for verification; otherwise, the model typically continues its first-step search strategy.Source: §E.5 · p. 33, Figure 5 · p. 34
  • Correct-answer trajectories perform verification more often than incorrect-answer trajectories.The figure reports this association without establishing that verification causes correctness.Source: §E.5 · p. 33, Figure 5 · p. 34
  • After Same or Different search, verification spans answer, intermediate-step, and disagreement checks; after Split, disagreement checks decrease while intermediate-step checks increase.The authors relate this pattern to the different structures of independent attempts and decomposition-based reasoning.Source: §E.5 · p. 34, Figure 5 · p. 34
  • Together, the findings suggest that Hermes-Learn strategies adapt to problem structure and reasoning progress.Source: §E.5 · p. 34

F Example Rollouts

  • The following section presents example evaluation rollouts sampled from the Hermes-Learn model after both SFT and RL.Source: F Example Rollouts · p. 34

F.1 L1 Rollout

The rollouts illustrate delegation under L1 and L2-SV, including verification after conflicting solutions and use of learned delegation on a chemistry problem outside the mathematical training domain.

F.1 L1 Rollout

  • Under L1, three independent solution attempts produced conflicting answers; the model initially followed the incorrect majority, then verified and corrected its answer.Simple majority voting over those same solutions would have been wrong.Source: §F.1 · p. 35

F.2 L2-SV Rollout

  • Under L2-SV, the second delegation step is restricted to verification, while the model must still interpret and synthesize subagent reports.The rollout's return reports a second-smallest value of 420261 after subagents identified 202606 as the smallest qualifying number.Source: §F.2 · pp. 37–38

F.3 L1 Rollout on a FrontierScience Problem

  • On a FrontierScience chemistry problem, the model uses delegation similarly to its mathematical rollouts, suggesting learned contextual reasoning transfers beyond the training domain.Source: §F.3 · p. 39

F.4 L1 Rollouts under Larger Inference-time Compute Budgets

The larger-budget rollouts show Hermes-Learn applying delegation, digestion, and recursive delegation beyond the settings used in training. They illustrate distinct-task assignment, correction across windows, and delegation at greater depth.

F.4 L1 Rollouts under Larger Inference-time Compute Budgets

  • Hermes-Learn trains with B=3 subagents, W=1 context window, and D=1; the example rollouts increase these to B=4, W=3, or D=2.Source: §F.4 · p. 40
  • With four subagents, the model assigns the extra slot to a distinct task and mixes Same and Different search strategies.Two tasks specify explicit solution techniques and two do not.Source: §F.4 · p. 40
  • Across three digestion-linked windows, the model revisits an incorrect first-window answer, corrects it in the second, and builds on the updated digest in the third.Source: §F.4 · p. 42
  • At delegation depth two, subagents recursively delegate through the same harness, extending beyond the depth-one delegation seen during training.Source: §F.4 · p. 44

Related papers

Loading 2609.38332v1…