Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Tong Zheng; Xidong Wu; Zheng Zhang; Zhankui He; Chaoyi Zhang; Benjamin Coleman; Ruoqiao Wei; Di Bai; Haolin Liu; Rui Liu; Xue Wang; Yue Zhuan; Wang-Cheng Kang; Renkai Xiang; Heng Huang; Xinwu Cheng; Yunsong Guo

Source-linked AI summary

Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Tong Zheng, Xidong Wu, Zheng Zhang, Zhankui He, Chaoyi Zhang, Benjamin Coleman, Ruoqiao Wei, Di Bai, Haolin Liu, Rui Liu, Xue Wang, Yue Zhuan, Wang-Cheng Kang, Renkai Xiang, Heng Huang, Xinwu Cheng, Yunsong Guo

arXiv:2609.14858v1Sep 14, 2026cs.CL

TL;DR

Dream-RSI targets the difficulty of improving exploration when fixed strategies do not adapt and online policy optimization receives delayed, expensive feedback. It replays accumulated discovery histories to evaluate and revise exploration policies cheaply, then redeploys them online; across three domains, it achieves competitive or improved quality while reducing discovery cost in several settings.

Problem
Fixed exploration strategies do not adapt from accumulated experience, while online policy optimization requires costly long-horizon feedback across a vast meta-policy space.
Method
Dream-RSI uses a lightweight programmable orchestration layer and historical discovery trees as replay simulators for dreaming-based policy improvement before online redeployment.
Results
Across 8 tasks spanning algorithm engineering, mathematical optimization, and GPU kernel engineering, Dream-RSI achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.
Takeaways & Limitations
Reusing discovery history as a replay simulator makes delayed exploration feedback available for efficient meta-exploration policy evaluation.

Abstract

from arXiv · show

Recursive self-improvement is becoming increasingly vital for autonomous AI agents, where progress hinges on discovering high-value solutions across complex domains. The driver of this process is effective exploration, however, managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization requires navigating vast meta-search spaces under delayed and expensive feedback over long-horizon rollouts. We introduce \textsc{Dream-RSI}, a framework for scalable and recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying coding agent unchanged. Our key insight is that accumulated discovery history can serve as a replay simulator over the realized search space. By performing dreaming in the replay simulator constructed from historical discovery trees, \textsc{Dream-RSI} secures immediate, low-cost off-policy feedback to evaluate and refine exploration policies without invoking repetitive, expensive online evaluations. The improved policy is subsequently redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, \textsc{Dream-RSI} achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.

1. Introduction

Dream-RSI addresses the cost of improving exploration by turning completed discovery histories into replay simulators for rapid policy evaluation. It combines this mechanism with a recursive loop that redeploys improved policies, achieving competitive or improved discovery quality while reducing discovery cost across several domains.

1. Introduction: Fixed exploration strategies cannot learn from accumulated experience, while online meta-policy optimization faces delayed, expensive feedback and a vast policy space.Each candidate policy may require a long online rollout before receiving useful feedback.
1. Introduction: Completed discovery histories provide structured replay simulators that reuse recorded decisions and execution outcomes for low-cost off-policy exploration-policy evaluation.Alternative policies can navigate recorded branches, orders, parallel groupings, and stopping decisions without rerunning every execution.
1. Introduction: Dream-RSI makes exploration programmable through a lightweight orchestration layer while leaving the underlying coding agent unchanged.The framework controls exploration behavior at the meta-exploration layer.
1. Introduction: The framework alternates online exploration, simulator construction, and dreaming-based policy improvement, then redeploys the updated policy to expand its simulator pool.This closes a recursive self-improvement loop over exploration strategies.
1. Introduction: Across 8 tasks in algorithm engineering, mathematical optimization, and GPU kernel engineering, Dream-RSI achieves competitive or improved quality while reducing discovery cost in several settings.Reported examples include up to 162× fewer agent calls over SimpleTES, over 50× budget savings in mathematical optimization, and up to 2.09× kernel-performance improvement under identical budgets.

2. Motivation: Discovery History as a Replay Simulator

Discovery history can be reused as an empirical simulator of the explored portion of a search space. This lets alternative exploration policies evaluate recorded trajectories without repeatedly executing the underlying discovery system.

2. Motivation: Discovery History as a Replay Simulator: An agent’s recorded traversal can become a reusable map for planning, allowing new policies to reconsider routes without revisiting every location.The analogy motivates reusing accumulated experience for exploration-policy evaluation.
2. Motivation: Discovery History as a Replay Simulator: Model-based reinforcement learning similarly evaluates policies through simulated experience rather than repeated interaction with the real environment.Dream-RSI applies this model-based perspective to long-horizon discovery.
2. Motivation: Discovery History as a Replay Simulator: Thousands of candidate policies can be tested by traversing different recorded branches, orders, parallel groupings, and stopping decisions at zero execution cost.A single costly online run therefore supports many rapid off-policy evaluations.
2. Motivation: Discovery History as a Replay Simulator: Online exploration records branches, decision points, computational costs, and outcomes, forming an empirical replay simulator for the observed discovery space.The recorded tree is grounded in realized discovery trajectories rather than newly generated outcomes.

3. Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Dream-RSI uses a shared exploration-policy interface for online discovery and offline replay, where recorded trees support iterative policy revision. Replay scores balance solution quality, attempted generations, and parallelism before the best revised policy is redeployed.

3. Dream-RSI: Recursive Self-Improvement through Evolving Worlds: Online and offline phases share a batch-based interface in which policies select eligible tree nodes for up to W parallel workers.Online execution generates new children, whereas replay reveals recorded outcomes from selected branches.
3. Dream-RSI: Recursive Self-Improvement through Evolving Worlds: Online rollout applies a fixed policy to stochastic discovery attempts, expanding and recording a new tree over at most K1 decision rounds.Selected nodes are assigned to workers, and completed attempts become new children.
3. Dream-RSI: Recursive Self-Improvement through Evolving Worlds: Offline evaluation keeps historical trees fixed while separately evaluating successive policy versions on every replay world.The policy-development agent uses replay trajectories and scores to revise executable policy code.
3. Dream-RSI: Recursive Self-Improvement through Evolving Worlds: Replay deterministically reveals recorded children, supports alternative branch, ordering, batching, and stopping choices, and generates no outcomes beyond the recorded tree.Replay terminates at an empty batch, the K2 round limit, or full tree revelation.
3. Dream-RSI: Recursive Self-Improvement through Evolving Worlds: The replay objective combines best solution quality with penalties for attempted generations and rewards for average attempts per decision round.This favors high-quality discovery with lower cost and useful batching.
3. Dream-RSI: Recursive Self-Improvement through Evolving Worlds: After revision, the highest-scoring candidate is selected for the next online rollout and is no worse than the current policy on average replay score over the fixed history.Redeployment then expands the history available for subsequent offline improvement.

4. Experiments

Dream-RSI is evaluated against fixed exploration and domain baselines across algorithm engineering, mathematical optimization, and GPU kernel engineering. It generally improves discovery quality or achieves comparable results with substantially less discovery compute.

Algorithm Engineering: Dream-RSI reduces average Lasso runtime from 3587.1 ms to 2931.0 ms with Gemini-3.1 Pro using 317 rather than 550 calls, and from 2516.7 ms to 2350.6 ms with Gemini-3.7-Flash using 1879 rather than 3200.
Algorithm Engineering: Dream-RSI outperforms sklearn and glmnet on all six held-out Lasso datasets while using roughly two orders of magnitude fewer calls than SimpleTES.
Mathematical Optimization: Dream-RSI achieves 1.145427 on Sum–Difference, matches the strongest Circle Packing result at 2.635983, and remains competitive on Autocorrelation at 1.456375.
Mathematical Optimization: Dream-RSI reaches these mathematical results within fewer than 1,000 generations, whereas SimpleTES requires 51,200 generations for state-of-the-art Autocorrelation performance.
GPU Kernel Engineering: On GPU kernels, Dream-RSI reaches comparable VGG16 and LayerNorm performance with 2.43× and 1.79× fewer generations, respectively.
GPU Kernel Engineering: On ConvDiv and ConvMax, Dream-RSI achieves 2.09× and 1.44× higher performance under comparable discovery budgets, respectively.

5. Further Analysis

Further analysis examines how Dream-RSI uses historical discovery information and adapts exploration effort across recursive rounds. Interactive replay outperforms guidance-only history, while exploration effort changes in response to progress.

Analysis of Historical Inductive Biases in Long-Horizon Discovery: Explicit directional guidance consistently underperforms unguided history under equivalent discovery budgets, suggesting that strong semantic biases can over-constrain diverse exploration.
Analysis of Historical Inductive Biases in Long-Horizon Discovery: Using history as an interactive replay simulator outperforms using it only as guidance on ConvDiv.
Analysis of Evolution of Exploration Behavior: As ConvDiv performance improves, the learned policy initially reduces evaluated attempts from 110 to 50, then increases exploration effort after progress plateaus and performance gains resume.

6. Related Work

Related work frames Dream-RSI within agentic discovery, self-evolving agents, and experience reuse. Its distinguishing move is to turn historical exploration into a replay simulator for cheap meta-level policy evaluation.

AI-Driven Scientific and Algorithmic Discovery: LLM-based discovery systems iteratively generate, evaluate, and refine candidate solutions, while newer systems increasingly optimize exploration itself.
Self-Evolving Agents: Self-evolving-agent research improves components including model weights, harnesses, contexts, skills, behavior, and rubrics.
Memory, History, and Experience Reuse: Prior work reuses experience as history, context, memory, skills, or training signals, whereas Dream-RSI organizes it as a replay simulator for alternative exploration controllers.
Memory, History, and Experience Reuse: This replay-simulator view turns collected discovery experience into reusable feedback and alleviates the scarcity and cost of training signals for improving exploration strategies.

7. Conclusion

Dream-RSI turns accumulated discovery histories into replayable simulators for recursively improving exploration policies. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, it achieves competitive or improved discovery quality while reducing discovery cost in several settings.

Dream-RSI achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.The evaluated domains are algorithm engineering, mathematical optimization, and GPU kernel engineering.
Dream-RSI evaluates candidate exploration policies in replay simulators built from historical discovery trees, providing rapid, negligible-cost feedback.Dreaming reuses accumulated discovery history instead of repeatedly invoking expensive online evaluations.
Improved exploration policies are redeployed online to drive further discovery and expand the simulator pool.This redeployment closes the recursive self-improvement loop.
Dream-RSI converts accumulated discovery history from static context into an active, replayable simulator for meta-exploration policy evaluation.The framework uses this conversion to make delayed exploration feedback reusable.

A. Detailed Task Description

The detailed task descriptions define benchmark objectives across Lasso paths, sum–difference sets, circle packing, and autocorrelation inequalities. They specify the relevant optimization targets, correctness checks, and geometric constraints.

Problem 1 (Lasso Regularization Path): The Lasso benchmark seeks efficient complete regularization-path implementations while preserving numerical correctness.The task uses a feature matrix, response vector, and decreasing regularization sequence; candidate solvers return approximate coefficients subject to objective-value checks.
Problem 1 (Lasso Regularization Path): Correctness is checked on fresh instances, and any failed required check makes the search score zero.Timing is measured separately on designated timing instances.
Problem 2 (Sum–Difference Problem): The Sum–Difference Problem seeks a finite integer set whose normalized sumset is large relative to its normalized difference set.The sumset and difference set are defined by pairwise addition and subtraction, respectively.
Problem 3 (Circle Packing in a Unit Square): For n ∈ {26, 32}, circle packing places circle centers in the unit square with nonnegative radii, requiring containment and non-overlap.The supplied passage states the geometric feasibility conditions but not the associated optimization objective.
Problem 4 (Autocorrelation Inequalities): The autocorrelation tasks define objectives through maxima of convolution expressions over t ∈ [−1/2, 1/2].The supplied formulations include Φ1(f) using (f∗f)(t) and Φ3(f) using its absolute value.

B. Prompts

The prompts operationalize exploration as history-aware, diversity-seeking proposal generation and as prefix-only replay-policy improvement under quality, work, and parallelism constraints.

B.1. Exploration Prompt: Online exploration requires reading complete historical proposals and measured outcomes, including failures, before proposing a new solution.The prompt instructs agents to trust eval/score.json and error.txt evidence over proposal claims.
B.1. Exploration Prompt: The exploration prompt distinguishes flawed ideas from repairable implementation failures and permits retries only after locating a concrete bug and specifying its fix.It also requires reasoning over both successful and failed attempts.
B.1. Exploration Prompt: Agents must resist locally saturated directions by favoring structurally different mechanisms or untried combinations over marginal refinements.The prompt treats exploration diversity as comparable in importance to the next incremental gain.
B.2. Replay-Based Policy Improvement Prompt: Replay-based improvement edits a prefix-only exploration policy rather than solving the scientific task or modifying unrelated programs.The replay agent operates through the designated policy implementation and uses revealed observations rather than hidden future outcomes.
B.2. Replay-Based Policy Improvement Prompt: The replay environment is a frozen irregular branch×attempt grid in which each revealed cell costs one probe and policies observe only revealed cells.The runtime grid bounds available parallelism, while unrevealed scores remain unavailable during decisions.
B.2. Replay-Based Policy Improvement Prompt: Replay evaluates policies by trading off attainment, total probes, and parallel execution, with pareto.reward combining pareto.auc and a parallelism penalty.Batching useful independent probes is favored because sequential execution incurs higher effective-round cost.
B.2. Replay-Based Policy Improvement Prompt: Replay decisions rank roots and frontiers using prefix-derived trajectory and failure evidence, retaining repairable failures while closing only cumulatively unrecoverable or repeatedly unpromising branches.The dynamic portfolio mixes exploitation, exploration, and at most one recovery candidate, subject to max_parallelism.

C. Discovered Programs

Dream-RSI’s discovered program is a performance-oriented Lasso-path solver combining screening, pruning, active-set management, lazy Gram computation, and hardware-aware optimization.

C. Discovered Programs: The discovered solver combines strong-rule screening, adaptive Cauchy–Schwarz KKT pruning, disjoint active-set bookkeeping, lazy Gram-matrix construction, and hardware-aware optimizations.These components are presented as the solver’s central algorithmic and implementation mechanisms.
C. Discovered Programs: The solver uses 64-byte aligned allocation and alignment assumptions for data structures and SIMD accesses.The code defines ASSUME_ALIGNED for compiler-supported builds and falls back to the original pointer otherwise.
C. Discovered Programs: Cauchy–Schwarz KKT pruning activates only when p >= 500 and n >= 150, making the pruning strategy workload-aware.The implementation uses a boolean condition to enable this optimization.
C. Discovered Programs: The implementation maintains screened and unscreened feature partitions alongside active-feature indices and related gradient state.These structures support disjoint screening bookkeeping and active-set updates.
C. Discovered Programs: A SIMD 4x register-blocked lazy Gram precomputation reduces column loads by 75% and runs in parallel for sufficiently large active workloads.The parallel path requires old_k >= 64 and n_padded * old_k >= 150000.