PaperlayerView exact revision on arXiv

Source-linked AI summary

NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness

NeoHorse Team, Guoliang Cao, Guohao Dai, Tianyu Guo, Kai Han, Hailin Hu, Zihan Jiang, Xiang Kuang, Boxun Li, Yulong Li, Zehua Pei, Yuchuan Tian, Jiamin Wang, Yu Wang, Yunhe Wang, Yihong Wu, Haiyang Xu, Shuo Zhang, Hang Zhou, Siyang Cheng, Jiayu Fan, Wei He, Qingrui Jiao, Hongguang Li, Zhiyuan Li, Runke Liu, Xi Liu, Xinchen Liu, Sinno Jialin Pan, Yi Ren, Liuyang Song, Chenyu Wang, Bei Yu, Quanlu Zhang, Xiangyu Zhang, Mengyu Zheng, Yingjie Zong

arXiv:2609.08183v1cs.CL

TL;DR

Recursive self-improvement needs a mechanism for a system to observe its capabilities and use that evidence to shape its next round of learning. NeoHorse-1 explores this through agentic post-training with routing-harness signals, raising macro-average scores at both 4B and 9B scales.

  • Problem

    Recursive self-improvement requires a concrete mechanism for a system to observe its capabilities and convert that evidence into its next round of learning.

    Source: §1 · p. 2

  • Method

    NeoHorse-1 uses a multi-model routing harness to record capability demand and subsequent interactions, then uses routing estimates to organize examples for curriculum training.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    58.94 to 64.87 at 4B and 65.60 to 69.04 at 9B: agentic post-training raises macro-average scores across the evaluation suite.

    Source: §1 · p. 3, §6 · p. 18

  • Takeaways & Limitations

    NeoHorse-1 provides preliminary validation of feedback-driven, harness-mediated recursive self-improvement within the evaluated capabilities.

    Source: §6 · pp. 18–19

    The study evaluates one loop iteration, so whether gains accumulate across successive iterations remains untested.

    Source: §6 · p. 18

Abstract

from arXiv · show

Recursive self-improvement (RSI) requires a concrete mechanism through which an AI system observes its capabilities and converts that evidence into the next round of learning. We present NeoHorse-1, a family of agent-native models developed to explore this path through agentic post-training. Our system combines a heterogeneous model pool with intelligent routing, recording the predicted capability demand, selected service tier, and subsequent interaction for each user turn. These records are converted into training examples that preserve interleaved reasoning, tool calls, and harness context, and are admitted through structural validation, six-dimensional semantic evaluation, and subscene-level labeling. Routing signals organize supervised fine-tuning into a three-stage curriculum and extend to routing-guided on-policy distillation, where a teacher supervises student-generated responses under the same progression. Capability-guided allocation then converts evaluation feedback into the next training mixture, closing an evaluation-selection-update loop in which what the system learns to do shapes what it learns from next. Across eleven benchmarks covering harness-based agents, tool use, coding, and instruction following, post-training raises the macro-average from 58.94 to 64.87 at 4B and from 65.60 to 69.04 at 9B, substantially narrowing the aggregate gap between the post-trained 4B model and the 9B base model. NeoHorse-1 provides an initial prototype of this feedback-driven process and a path toward harness-mediated RSI across successive iterations.

1. Introduction

NeoHorse-1 explores recursive self-improvement by turning routed agent interactions and capability feedback into training data and subsequent updates. Its post-training combines routing-guided curricula and on-policy distillation, with reported macro-average gains at both model scales.

Routing-guided agentic training

  • NeoHorse-1 turns interaction trajectories into evidence for learning by combining a heterogeneous model pool and multiple harnesses across diverse tasks.Routing records estimated capability demand, the model used, and the subsequent interaction, while agent activity supplies task experience and observable evidence of strengths and limitations.Source: §1 · p. 2

Feedback loop

  • Capability feedback shapes each subsequent training mixture, while continued harness execution supplies new experience and updated models reveal evolving strengths and limitations.The loop connects multi-model experience with the continued improvement of individual models.Source: §1 · pp. 2–3

Post-training methods

  • Training examples preserve historical and harness context; routing-derived scores guide a curriculum, and on-policy distillation has a teacher supervise student-generated responses [29].The curriculum progressively introduces higher-scored examples while retaining lower-scored coverage; on-policy supervision follows the student’s evolving behavior.Source: §1 · p. 3

Results

  • Macro-average rises from 58.94 to 64.87 at 4B and from 65.60 to 69.04 at 9B after agentic post-training.The largest gains appear on harness-based and execution-intensive evaluations, and the post-trained 4B model substantially narrows the aggregate gap to the 9B base model.Source: §1 · p. 3

2. Related Work

Prior work develops agent post-training through trajectory imitation, on-policy supervision, and environment feedback, while routing signals and recursive feedback loops offer ways to organize capability improvement.

2.1. Agentic Model Post-Training

  • Trajectory-based supervised fine-tuning transfers planning and tool use into model parameters, with data composition, scale, and iterative synthetic-data training shaping the approach.FireAct, AgentTuning, Agent-FLAN, and AgentBank study trajectory learning and data composition [8] [71] [12] [57]; Llama 3 adds synthetic multi-step tool-use data, iterative SFT, rejection sampling, and direct preference optimization [19].Source: §2.1 · p. 4
  • On-policy distillation has students generate trajectories while teachers provide token-level logits on student-visited states, aligning denser process supervision with evolving behavior.It reduces the distribution gap of fixed-teacher trajectory SFT and differs from conventional teacher-trajectory distillation [22].Source: §2.1 · p. 4
  • Harness choices shape the tools, observations, and feedback in training, while executable-environment agentic RL replaces fixed labels with rewards from tool execution and task outcomes.Work studies harness design and structured environment-grounded trajectories [68] [27] [69], and agentic RL for search, tool use, long-horizon interaction, deployment harnesses, and joint harness-model updates [25] [17] [64] [21] [13].Source: §2.1 · p. 4

2.2. LLM Routing and Curriculum Learning

  • LLM routing balances response quality and inference cost, from cost-aware cascades and preference-based routing to agentic selection of models or sub-agents.FrugalGPT studies cascades [9], RouteLLM routes between stronger and weaker models [41], and agentic routing selects models or sub-agents for requests [33].Source: §2.2 · p. 4
  • Routing predictions of request- and context-conditioned capability demand can order curriculum examples without explicit difficulty labels or dataset-specific heuristics.The approach uses predicted demand rather than the served model's identity as a difficulty label, since executed routes can reflect user overrides, service availability, and deployment policy; curriculum learning and its LLM post-training use are discussed in [66] [29].Source: §2.2 · p. 4

2.3. Recursive Self-Improvement

  • RSI describes iterative improvement of a model, scaffold, or improvement procedure, with research spanning what is improved and how generation, evaluation, and updating connect.System-level efforts also automate harness design, serving infrastructure, and training pipelines [11] [56].Source: §2.3 · pp. 4–5
  • Recent RSI work evolves skills and harnesses, supports reset-free online adaptation, and tests agents that improve training algorithms for later runs.Examples include MetaSkill-Evolve, AREX, Self-Harness, Agentic Harness Engineering, Retrospective Harness Optimization, Continual Harness, and AI4AI-Bench [37] [72] [30] [14].Source: §2.3 · p. 5

3. Data from Routing Harness

NeoHorse-1 turns deployment-harness trajectories into training data that preserves execution context and links user intent, routing decisions, quality, and outcomes. Evaluation feedback and new trajectories guide subsequent data mixtures, forming a harness-mediated RSI loop.

3.1. Data Composition

  • Training records link complete harness trajectories to user-turn units and goal-sharing subscenes, retaining current-turn reasoning, tool calls, observations, and prior interaction context.Earlier-turn reasoning is omitted, while records remain linked to their parent trajectory and subscene.Source: §3.1 · p. 5

3.2. Data Quality

  • Admission combines rule-based structural checks and six evidence-grounded semantic dimensions, assigning trajectories complete, recoverable, or quarantined states with explicit verdict coverage.The semantic dimensions are goal attainment, instruction adherence, tool use, evidence consistency, error recovery, and termination; structural validity alone does not establish task success.Source: §3.2 · p. 6

3.3. Data Characterization

  • Subscenes are characterized by Scene, Goal, and Outcome attributes that connect user context and objectives to verifiable task results.Goal attributes include acceptance criteria and cross-turn relations; annotations retain derivation method and confidence, while deterministic structural facts cannot be overwritten by a semantic judge.Source: §3.3 · pp. 6–7, Figure 3 · p. 7

3.4. Agentic Routing Signals

  • The router assigns each user turn to C0–C3 and records predicted demand, policy-adjusted choice, and served tier separately; linked outcomes inform curriculum ordering and data allocation.Routing-derived scores order examples in the three-stage SFT curriculum and schedule starting contexts for on-policy distillation.Source: §3.4 · pp. 7–8

3.5. Capability-Guided Data Allocation

  • Evaluation profiles model deficiencies across task, quality, outcome, and routing regions to shift the next training mixture while retaining broad coverage.Verified successes provide positive supervision, while informative failures identify regions needing more or rebalanced data; new trajectories are reallocated with existing data in subsequent iterations.Source: §3.5 · p. 8

4. Agentic Post-Training

The section defines agentic supervision over user-turn trajectories, then uses routing-estimated capability demand to schedule SFT examples and student-generated distillation contexts.

4.1. Agentic Supervision

  • Agentic SFT treats each user request and ensuing assistant/tool exchange as one example, preserving interleaved thinking and masking loss to current-turn assistant targets.Historical messages remain context, but earlier-turn reasoning is omitted; current-turn targets include reasoning when present, tool calls and arguments, visible responses, and end-of-response tokens.Source: §4.1 · p. 9

4.2. Routing-Guided Curriculum Learning

  • Routing-estimated capability demand orders complete user-turn examples into a three-stage SFT curriculum that progressively adds higher-scored examples while retaining some lower-scored examples later.Demand is re-estimated from the request and available interaction history, serving as a turn-level ordering proxy rather than an action-level difficulty label; routing scores schedule examples, not reweight the SFT loss.Source: §4.2 · pp. 9–11

4.3. Routing-Guided On-Policy Distillation

  • Routing schedules recorded starting contexts across three stages, where students generate responses that a fixed teacher supervises with response-normalized reverse-KL over shared top-K token bins.The rollout checkpoint is refreshed as training proceeds, and optimization uses the collected generated prefixes while holding generated tokens, candidate IDs, and teacher scores fixed.Source: §4.3 · pp. 11–12

5. Results and Analysis

Post-training improves benchmark performance at both model scales, while the larger model shows particular strengths in interaction-heavy tasks. Agent traces and controlled data studies examine execution behavior and how trajectory source and supervision quantity relate to performance.

5. Results and Analysis

  • NeoHorse-1-4B outperforms Qwen3.5-4B on every benchmark with results for both; at 9B, training remains effective, with benefits concentrated on interactive execution and instruction-following largely stable despite one minor metric decrease.Source: §5 · pp. 14–15
  • NeoHorse-1-9B consistently outperforms NeoHorse-1-4B across available comparisons, with larger gains on harness-based agents, function calling, and challenging coding than on instruction following.NeoHorse-1-4B already matches or exceeds Qwen3.5-9B on several benchmarks.Source: §5 · p. 15

5.1. Agent Trace Analysis

  • In a project-scheduling trajectory, NeoHorse-1-4B retrieves an overlooked updated constraint, recomputes and verifies the schedule, and saves the artifact to the required path, unlike Qwen3.5-4B.Qwen3.5-4B plans from outdated information and saves its invalid schedule to an unintended location.Source: §5.1 · p. 16
  • In WorkBuddy, 9B repeats edit–test–inspect–repair until verification passes; in PinchBench, it switches around unavailable pandas, reducing requests, execution time, and tokens by approximately 70.8%, 76.7%, and 83.6%.The traces illustrate iterative error repair and strategy recovery under environmental constraints.Source: §5.1 · p. 16

5.2. Data Analysis

  • Under matched routing-guided training, routing-harness trajectories outperform Toucan on all five benchmarks, raising their unweighted average by 6.26 points, with HumanEval +8.54 and 𝜏2-Bench +11.31.The comparison uses the same Qwen3.5-4B initialization, curriculum schedule, and closely matched training budgets.Source: §5.2 · p. 17, Table 3 · p. 17
  • The five-benchmark development-suite average rises from 69.31 for the base model to 71.45 at the largest routing-harness data scale shown in Figure 7.The experiment uses strictly nested subsets and holds initialization, optimization, packing, and passes fixed to isolate the effect of adding unique supervision.Source: §5.2 · pp. 17–18, Figure 7 · p. 17

6. Conclusion and Discussion

NeoHorse-1 uses a routing harness to turn interaction evidence and capability feedback into staged post-training and updated training mixtures. Its results show gains across evaluated benchmarks, while the authors characterize the work as an initial prototype whose broader scope and multi-iteration effects remain untested.

Conclusion and Discussion

  • The harness turns interaction trajectories, capability-demand routing signals, and outcomes into validated training data and routing-guided learning.Training examples preserve interleaved reasoning, tool calls, and harness context, with structural validation, six-dimensional semantic evaluation, and subscene labeling. Routing scores structure a three-stage supervised fine-tuning curriculum and on-policy distillation of student-generated responses; capability-guided allocation uses evaluation feedback to select the next training mixture.Source: §6 · p. 18
  • 4B macro-average rose from 58.94 to 64.87, and 9B from 65.60 to 69.04 across ten benchmarks; 9B retained advantages on iterative debugging, failure recovery, and long-sequence state maintenance.The post-trained 4B model also substantially narrowed the aggregate gap to the 9B base model; trajectory analyses showed more complete evidence-gathering, revision, verification, and delivery workflows.Source: §6 · p. 18
  • Returning post-trained checkpoints to the routing harness generates new trajectories, routing records, and capability feedback that can seed the next training iteration.The evaluation–selection–update loop was run for a single pass in this validation.Source: §6 · p. 18
  • The findings are preliminary: evaluation covers agentic and coding capabilities, tool use, and instruction following, while broader harness capabilities and gains across successive iterations remain untested.The authors present the work as an initial attempt at recursive self-improvement rather than a definitive demonstration.Source: §6 · p. 18
  • Future work will test successive model generations, extend data collection and curriculum design to more tasks and environments, and improve routing estimates of difficulty and deficiency.The proposed routing work includes calibrating prediction–action–outcome signals and training the router itself to guide data selection, allocation, and what a model should attempt next.Source: §6 · pp. 18–19

Appendix · B. Case Study · Case A: Daily Ticket Reporting under Temporal and Audit Constraints

Case A tests whether an agent can produce an auditable daily ticket report under strict matching and temporal constraints. NeoHorse-1-9B matches independently recomputed counts and excludes a next-day update, but retains a status-label error.

Case A: Daily Ticket Reporting under Temporal and Audit Constraints

Case B: Implementing a Time-Leakage Auditor from Repository Requirements

Case B tests whether repository requirements are followed when implementing a time-leakage auditor. NeoHorse-1-9B applies the specified five-minute tolerance and preserves the required output schema, unlike Qwen3.5-9B.

Case B: Implementing a Time-Leakage Auditor from Repository Requirements

  • The README requires a five-minute tolerance, and clean records must preserve input order with sample_id, cutoff_time, and feature_count.The user asks the agent to implement the empty audit() function in app/audit_leakage.py using the supplied repository materials.Source: Case B: Implementing a Time-Leakage… · p. 21
  • The auditor must record genuinely late features in the rejection output while continuing to export clean samples.The rejection output must contain summary and rejected.Source: Case B: Implementing a Time-Leakage… · p. 21
  • NeoHorse-1-9B applies the five-minute tolerance and preserves the required schema, while Qwen3.5-9B uses two seconds, rejects valid sample s1, and changes the output structure.The required outputs include clean samples and a rejection output with summary and rejected fields.Source: Case B: Implementing a Time-Leakage… · p. 21

Case C: Supporting Sustained Two-Player Gomoku Interaction

In a 26-click replay, NeoHorse-1-9B sustained Gomoku play with the expected stones and turn order, while Qwen3.5-9B accepted no moves.

Case C: Supporting Sustained Two-Player Gomoku Interaction

Related papers

Loading 2609.08183v1…