Source-linked AI summary
Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration
Qifan Zhang, Dongyang Ma, Tianqing Fang, Jia Li, Jing Tang, Nuo Chen, Haitao Mi, Yan Wang
TL;DR
Existing agents depend on human-defined workflows and rewards, limiting autonomous evolution in unseen environments. This paper trains intrinsic meta-evolution through training-only outcome-based rewards that assess the downstream utility of self-generated world knowledge. The resulting agents improve web-task performance by approximately 20%, and generated knowledge enables Qwen3-14B to outperform unassisted Gemini-2.5-Flash.
Problem
Current self-evolving agents rely on human-defined workflows and verified rewards, and lack the ability to choose their own growth direction in new environments.
Method
The method trains agents to explore unseen environments and distill structured world knowledge using an outcome-based reward based on downstream utility, with external guidance removed at inference.
Results
Approximately 20% absolute performance improvement was achieved for Qwen3-30B and Seed-OSS-36B, while generated knowledge enabled Qwen3-14B to outperform unassisted Gemini-2.5-Flash.
Takeaways & Limitations
The results support a meta-evolution paradigm in which agents spontaneously explore and adapt to unknown environments without human guidance or inference-time rewards.
Takeaways & Limitations
Experience-Driven Evolution remains bottlenecked by the massive human labor required to engineer its tasks and rewards.
Abstract
from arXiv · showhide
Most agents today ``self-evolve'' by following rewards and rules defined by humans. However, this process remains fundamentally dependent on external supervision; without human guidance, the evolution stops. In this work, we train agents to possess an intrinsic meta-evolution capability to spontaneously learn about unseen environments prior to task execution. To instill this ability, we design an outcome-based reward mechanism that measures how much an agent's self-generated world knowledge improves its success rate on downstream tasks. This reward signal is used exclusively during the training phase to teach the model how to explore and summarize effectively. At inference time, the agent requires no external rewards or human instructions. It spontaneously performs native self-evolution to adapt to unknown environments using its internal parameters. When applied to Qwen3-30B and Seed-OSS-36B, this shift to native evolution yields a 20% performance increase on WebVoyager and WebWalker. Most strikingly, the generated world knowledge even enables a compact 14B Qwen3 model to outperform the unassisted Gemini-2.5-Flash, establishing a new paradigm for truly evolving agents.
1 Introduction
Existing self-evolving agents depend on human-defined workflows, instructions, or rewards, whereas this work trains agents to explore unfamiliar environments and create reusable world knowledge without inference-time guidance. The approach uses training-only outcome-based rewards and improves downstream web-task performance.
- Existing self-evolving agents stop evolving when human-defined workflows, instructions, or verified rewards are removed.
- Human intelligence proactively builds an internal map of unfamiliar environments, but current agents wait for instructions and rewards before evolving.
- Native Agency lets agents autonomously explore novel environments and distill observations into structured, reusable world knowledge.
- The outcome-based reward measures how much self-generated knowledge improves downstream task success and is used only during training.
- Approximately 20% absolute performance improvement was achieved by Qwen3-30B and Seed-OSS-36B over standard baselines on WebVoyager and WebWalker.
- Generated world knowledge enabled compact Qwen3-14B to outperform unassisted Gemini-2.5-Flash.
2 Related Works
Prior self-evolution paradigms rely on human-designed tasks, rewards, or engineered workflows. The proposed meta-learning-driven paradigm instead explores environments and compresses observations into world knowledge, avoiding inference-time parameter updates required by Test-Time Training.
- Existing self-evolving agents rely heavily on human-defined workflows and verified environment-specific rewards to guide improvement.
- Experience-Driven Evolution: Experience-Driven Evolution uses human-crafted tasks and predefined rewards, then updates prompts, memories, tools, or model parameters from scored trajectories.
- Experience-Driven Evolution: This paradigm is bottlenecked by human labor, with agents passively adapting by studying human-provided tasks and rewards.
- Adversarial Evolution: Adversarial Evolution reduces manual task and reward design but shifts the burden to orchestrating complex challenger-solver workflows.
- Meta-Learning-Driven Evolution: Meta-Learning-Driven Evolution autonomously explores environments and compresses observations into structured world knowledge that enhances downstream performance.
- Test-Time Training: Unlike Test-Time Training, which requires gradient-based inference-time updates, this method feeds distilled world knowledge into the prompt as external context.
3 Methodology
The method trains agents to explore unfamiliar environments, compress observations into World Knowledge, and use that knowledge for downstream execution. It evaluates knowledge by downstream utility and trains the capability through supervised fine-tuning followed by rejection-sampling refinement.
- Native Evolution: World Knowledge is a structured representation generated by exploring an environment before a downstream task and later used to guide task actions.The framework separates task-free native evolution from knowledge-enhanced execution.
- Outcome-Based Reward Design: The outcome-based reward measures the success gain from using generated knowledge over the zero-knowledge baseline on downstream tasks.For each environment, success is estimated across labeled tasks using the agent’s predicted answers.
- Outcome-Based Reward Design: The reward is applied only during training, while inference-time evolution requires no external rewards, predefined task sets, or human verification.This trains the model to identify and compress high-value information before deployment in new environments.
- Training Framework: The training framework combines teacher-model supervised fine-tuning with reinforcement-based rejection sampling to cultivate planning, exploration, refinement, and summarization.The teacher generates structured knowledge and trajectories, after which candidate trajectories are selected using the outcome-based reward.
- Training Framework: 10.72% average absolute accuracy improvement was achieved by teacher-generated knowledge for Qwen3-30B-A3B on training tasks versus the zero-knowledge baseline.The selected expert trajectories averaged 374.8 steps and 3,322.4 tokens per step; two rejection-sampling iterations then refined the policy.
4 Experiments
The experiments evaluate effectiveness, efficiency, transferability, training-stage contributions, token-length sensitivity, and qualitative behavior on WebWalker and WebVoyager. World knowledge improves success rates, reduces execution steps, transfers across models, and benefits from staged training and appropriately sized contexts.
- Experimental Questions: The evaluation uses WebWalker and WebVoyager to test effectiveness, efficiency, transferability, ablations, and sensitivity to world-knowledge length.The experiments assess success rate and execution steps, cross-model transfer, SFT/RFT stages, and token-length effects.
- Effectiveness: 40.91% success on WebWalker: Ours (RFT) outperforms the Without baseline by nearly 19% absolute and surpasses Prompt-only (Gemini) at 29.85%.Prompt-only (Base) reaches 19.50%, below the Without baseline at 22.04%, indicating that untrained generated guidance can be noisy or hallucinated.
- Efficiency: 17% average efficiency improvement: integrating world knowledge reduces execution steps across domain websites.The authors describe world knowledge as a cognitive map that supplies structural priors for navigating relevant pages and extracting answers.
- Cross-Model Transfer: 18.3% average-accuracy gain for Qwen3-14B and 21.0% for Kimi-K2-Turbo: Seed-36B’s world knowledge transfers across model families and scales.The transferred knowledge also lets Qwen3-14B exceed unassisted Gemini-2.5-Flash on Conference (35.6% vs. 31.3%) and Game (30.5% vs. 25.7%).
- Training Stages: SFT and the first RFT round provide substantial performance boosts, while the second RFT round generally adds marginal gains or slight fluctuations.Performance is compared across base, SFT, rft1, and rft2 stages for world-knowledge generation and downstream task solving.
- Token-Length Sensitivity: 30.74 to 39.71: increasing world-knowledge length from 4k∼8k to 8k∼16k sharply improves game-site success, but 16k∼32k to 32k∼64k declines from 41.56 to 40.72.The authors attribute diminishing or negative returns at longer lengths to redundant noise, whereas overly short knowledge causes information loss.
5 Conclusion
The paper proposes intrinsic meta-evolution, enabling agents to explore environments and distill structured world knowledge without human guidance or inference-time rewards. This native evolution improves performance and transfers knowledge across models.
- Intrinsic meta-evolution enables agents to spontaneously explore environments and distill structured world knowledge without human guidance or inference-time rewards.
- 20% absolute performance improvement is reported for Qwen3-30B and Seed-OSS-36B under native evolution.
- Generated world knowledge enables compact Qwen3-14B to outperform unassisted Gemini-2.5-Flash.
A Details of Input Processing
The input-processing procedure reduces noisy web data by scoring webpages in a directed graph and clustering them by shared URL prefixes. This produces a structured representation that is more interpretable and easier to navigate.
- Importance Scoring: Webpages are modeled as a directed graph and assigned importance scores from their linkage topology.
- Importance Scoring: Importance(v) = 0.7 · din(v) + 0.3 · dout(v).
- Clustering: Webpages are recursively grouped by shared URL prefixes until each cluster satisfies a size constraint.
- Clustering: The resulting clusters organize complex websites into coherent categories for downstream processing.
B Example Chowcase
The appendix showcases the processed website input and the resulting world knowledge. The processed input records clustered URLs and importance scores, while the generated knowledge summarizes a website’s main information.
- Examples B.1: An Example of the Processed Input: The processed-input example contains 5 clusters covering 221 URLs, with cluster sizes of 40, 10, 130, 21, and 20.
- Examples B.1: An Example of the Processed Input: The processed input lists URL prefixes together with webpage importance scores.
Overview
The ACL 2024 website serves as an official information hub for the conference. Its summarized content covers key participant information and opportunities for involvement.
- The ACL 2024 website is the official information hub for the 62nd Annual Meeting of the Association for Computational Linguistics.
- The website provides registration, program schedules, venue information, and calls for participation.
- The summarized website contains 2 categories and 31 analyzed pages.
Category: ACL 2024 Main Website
This category covers the ACL 2024 website’s main attendee information, including conference details, registration, visa guidance, travel, and related participant resources.
- The ACL 2024 main website provides conference information and attendee resources for the event in Bangkok, Thailand.The listed resources include registration, visa, travel, and participant guidance.
- Registration information covers in-person, virtual, and tutorial/workshop-only passes, with paper-registration and financial-need details.
- Visa guidance directs attendees to Thailand’s official e-visa resources and describes exemption and visa-on-arrival eligibility.
Category: ACL 2024 Program Information
This category describes the ACL 2024 program and related paper directories, awards, workshops, and website-analysis instructions for producing structured world knowledge.
- The ACL 2024 program includes schedules, tracks, workshops, tutorials, presentation formats, keynotes, and paper listings.
- Program pages list accepted papers, award winners, outstanding papers, and workshops spanning multiple natural-language-processing topics.
- The teacher-agent prompt directs a web-intelligence agent to analyze clustered URLs and produce structured World Knowledge within a target token range.
Phase 0: Initialization & Planning
Initialization requires parsing clustered-URL statistics and creating a token-allocation plan before category processing begins.
- The agent must parse total clusters, total URLs, and per-cluster URL counts from the queue-file header.
- Token allocations should scale with effective URL count and keep the total plan within the specified minimum and maximum token limits.The prompt suggests separate budgets for page entries and category headers or summaries.
Phase 1: Category-by-Category Processing (Loop)
The processing pipeline iterates through URL categories, allocates and enforces token budgets, scrapes selected pages, writes formatted sections, evaluates answers, and finalizes the guidebook.
- Phase 1: Category-by-Category Processing (Loop): Each category is loaded, checked against a saved token plan, selectively scraped, summarized, appended, and marked complete only after successful writing.
- Phase 1: Category-by-Category Processing (Loop): Category output follows a fixed structure containing a category heading, URL prefix, summary, scraped-page entries, and a note about additional pages.
- Phase 1: Category-by-Category Processing (Loop): The guidebook is compressed or expanded according to token counts, then given an overview and saved as the final complete document.
- Phase 1: Category-by-Category Processing (Loop): The trained-agent prompt organizes a single website’s clustered URLs into a concise, categorized Guidebook using link metrics such as inbound, outbound, and composite scores.
- Phase 1: Category-by-Category Processing (Loop): Selected page summaries must come from web-agent calls, include full URLs, remain on the site’s domain, and avoid fabricated content.
- Phase 1: Category-by-Category Processing (Loop): The evaluation prompt accepts answers with equivalent meaning despite differences in wording, format, order, or detail, and outputs only 1 or 0.