CURIO: Curiosity-Driven Test-Time Learning for Open-Ended Discovery

Motivation

• Discovery requires exploiting successful attempts while exploring directions whose eventual value may not be reflected in current reward.Strongly prioritizing high-reward trajectories can leave low-reward but promising directions with too little future sampling.Source: §1 · p. 1
• Frozen-LLM search can reuse prior attempts in context but cannot update model parameters from successes and failures on the test problem.RL-based discovery updates the proposing policy using experience from that same problem.Source: §1 · p. 1
• On Erdős and Circle Packing, CURIO has higher code diversity than task-only RL at every displayed stage, a structural diagnostic rather than evidence of semantic diversity or causality.Figure 1 reports the comparison using the Appendix F metric.Source: Figure 1 · p. 2, §1 · p. 2

Contribution

• CURIO uses an ICWM to model token-level transitions in policy hidden states and adds normalized prediction-residual bonuses at sampled tokens outside top-k choices, with an annealed weight.The task evaluator still determines candidate quality; the bonus supplements rather than replaces task feedback.Source: §1 · p. 2

Results

• Three-run means improve over matched task-only RL on five mathematical objectives, match Circle Packing’s best reported performance, and improve denoising Score and MSE on both corpora at every tested scale.Relative improvements reach 18.3% on 235B Hadamard and 10.8% on 8B Tabula Score.Source: §1 · p. 2

2.1 OPEN-ENDED DISCOVERY

• A discovery problem specifies a description, candidate solutions, and an evaluator; an archive stores evaluated candidates and programs, and the aim is the best feasible solution within a finite budget.A new record improves on the prior best-known value.Source: §2.1 · p. 2
• CURIO’s overview shows archive parents expanded into rollouts, an ICWM modeling token-level transitions in encoded policy hidden states, and prediction residuals used as a selective curiosity bonus combined with task feedback.The diagram is schematic; Equations (5)–(12) specify the representations, normalization, and masking.Source: Figure 2 · p. 3

2.2 RL FOR DISCOVERY

• In RL discovery, exponential reward weighting emphasizes high-reward completions, with β controlling that emphasis; larger β may reduce exploration of currently low-reward but promising directions.The prompt is archive-conditioned and the evaluator supplies the reward.Source: §2.2 · p. 3

3.1 TASK FEEDBACK AND TOKEN NOTATION

• Task coefficients are sampled per rollout group, while a centered token-local reference correction is held fixed during the policy update.The correction is not the exact gradient of a sequence-level KL-regularized population objective.Source: §3.1 · pp. 3–4

3.2 INTRINSIC CURIOSITY WORLD MODEL

• The ICWM predicts token-level transitions in the policy’s representation space, and prediction residuals become an exploration signal rather than a model of the external environment.It uses actor hidden states and token embeddings; no ICWM gradient is propagated into the actor.Source: §3.2 · p. 4
• The normalized residual is retained only for sampled tokens outside the policy’s top-k choices, a heuristic that does not guarantee semantic novelty or an unbiased intrinsic advantage.Selected bonuses can be positive or negative; unselected tokens still receive task feedback and can train the ICWM. Prediction error also does not establish future task improvement, and the formulation has no anti-collapse guarantee.Source: §3.2 · pp. 4–5

3.3 JOINT OPTIMIZATION AND TRAINING PROCEDURE

• The joint update combines task coefficients with a normalized, token-local curiosity bonus whose weight is annealed toward a floor over training.Setting the initial curiosity weight to zero recovers the task-only control; constant-reward groups are filtered before either loss.Source: §3.3 · pp. 5–6

4.1 EXPERIMENTAL SETUP

• Evaluation spans six mathematical problems and PBMC and Tabula denoising, comparing frozen-policy search and trainable test-time RL with three-run means and held-out corpora excluded from checkpoint selection.The 235B comparison includes only TTT-Discover and CURIO.Source: §4.1 · p. 6

4.2 TASK PERFORMANCE AND CODE DIVERSITY

• CURIO generally maintains higher code diversity D than TTT-Discover across six task trajectories, although the curves cross or approach one another at some stages.The metric captures code-level structural variation, not semantic diversity.Source: Figure 3 · p. 7, §4.2 · p. 7
• Across scales, CURIO beats TTT-Discover on five mathematical objectives, matches best Circle Packing, and improves denoising Score and MSE; the Hadamard gain reaches 18.33% at 235B.These results are three-run means; the tables omit variability.Source: §4.2 · p. 6, Table 1 · p. 8, Table 2 · p. 8

4.3 ABLATION STUDIES OF THE ICWM

• On Qwen3-14B, η0=0.3 performs best on autocorrelation tasks, while η0=0.5 is preferable on Erdős, Hadamard, and aggregated single-cell; 0.8 underperforms task-only control on some tasks.The ablation varies initial curiosity weight against η0=0, which recovers TTT-Discover.Source: §4.3 · p. 8, Figure 4 · p. 9
• The MLP is competitive but the best architecture varies by task; the policy-coupled representation leads on most plotted tasks, with frozen representations winning on individual tasks.The ablations compare MLP, GRU, and Transformer predictors, and policy-coupled, frozen-actor, and frozen external-LLM representations.Source: §4.3 · p. 9, Figure 4 · p. 9

• Frozen-policy search reuses prompts, histories, and populations, whereas test-time RL learns from current-problem experience; the paper contrasts these approaches with SimpleTES and test-time RL baselines [Ye et al., 2026; Wang et al., 2025; Yuksekgonul et al., 2026].Source: §5 · p. 9
• Exploration methods range from randomness and entropy regularization to directed bonuses based on visitation counts or prediction error [Amin et al., 2021; Yu et al., 2024; Pathak et al., 2017; Burda et al., 2018].Source: §5 · p. 9
• CURIO predicts encoded actor-state transitions and applies normalized residuals at non-top-k sampled tokens, complementing task-driven policy learning with selective exploration.Source: §5 · p. 9

6 CONCLUSION

• CURIO uses an ICWM to model token-level transitions in encoded policy hidden states and add normalized, selectively masked prediction-residual bonuses under an annealed weight.Source: §6 · p. 9
• Reported runs improve over the matched task-only control on several mathematical and denoising objectives, match the best reported Circle Packing performance, and show higher code-level structural diversity in the displayed analysis.Source: §6 · pp. 9–10
• Prediction error and code diversity are proxies; further controls are needed to establish semantic exploration, token-selection contribution, and robustness across runs.Source: §6 · p. 10

A IMPLEMENTATION DETAILS

• 50 epochs with a target of 8 parent groups and 64 rollouts per group configure 25,600 generation slots, not equal token counts, wall-clock time, or training cost.The archive may initially supply fewer than the target number of unrelated parents, so the configured budget does not guarantee the realized count.Source: A IMPLEMENTATION DETAILS · p. 14
• The initial curiosity weight is task-dependent, while other listed hyperparameters are shared across tasks and backbone scales.CURIO uses η0 = 0.5 for Erdős, Hadamard, and denoising; η0 = 0.3 for AC1–AC3 and Circle Packing, with LoRA rank 32.Source: A IMPLEMENTATION DETAILS · p. 14, Table 3 · p. 15
• Parents are selected greedily by PUCT while excluding selected candidates’ ancestors and descendants, so a round can return fewer than the target number.Feasible, non-duplicate children are inserted; the archive retains up to two highest-valued children per parent and is capped at 1,000 candidates while keeping the seed.Source: A IMPLEMENTATION DETAILS · p. 14
• Temperature calibration targets a task-coefficient divergence of log 2, but tied maxima can make that target unattainable and require a finite cap or fallback.The described calibration uses upper-bracket expansion and 60 bisection steps when a finite bracket exists.Source: A IMPLEMENTATION DETAILS · p. 14
• The ICWM uses three-layer MLPs for its encoder and forward predictor, runs in fp32, and trains on mini-batches of 8 rollouts.Cached residuals are normalized across the epoch’s retained completion positions and the ICWM weights and optimizer state persist between epochs; this does not guarantee decreasing prediction error.Source: A IMPLEMENTATION DETAILS · pp. 15–16
• Candidate programs run in a separate sandbox; malformed, infeasible, failing, or time-exceeding candidates receive zero reward and are not added to the archive.The evaluator environment is pinned per task, and each rollout’s evaluation seed is its global index, though hardware and scheduling can still affect wall-clock-limited searches.Source: A IMPLEMENTATION DETAILS · p. 16

B TRAINING PROCEDURE DETAILS

• At each epoch’s start, the actor is snapshotted; generation, ICWM feature extraction, and reference correction use that snapshot before the final policy update.The base weights remain frozen, and disabling the adapter yields the initial reference policy.Source: B TRAINING PROCEDURE DETAILS · p. 16
• The policy loss treats cached residuals and epoch statistics as constants, while only the current-policy numerator receives an actor gradient.The ICWM loss does not backpropagate through actor hidden states or action embeddings; mini-batches with no trainable tokens are skipped.Source: B TRAINING PROCEDURE DETAILS · p. 16
• Setting η0 = 0 removes the curiosity block and yields the matched TTT-Discover task-only actor update.Constant-reward groups are filtered before curiosity, so they receive no intrinsic-only update; the equivalence does not extend to ThetaEvolve’s implementation.Source: B TRAINING PROCEDURE DETAILS · p. 16

C INFERENCE PROCEDURE

• Each rollout is a single user turn containing the task prompt, which already includes state context; no system message is rendered.The assistant turn opens with a <think> tag, and generation stops at <|im_end|>.Source: C INFERENCE PROCEDURE · p. 16, Table 4 · p. 17
• Sampling uses temperature 1.0 without top-p or top-k truncation, and the exploratory mask is computed from returned log-probabilities during generation.The fixed request seed does not ensure unique completions or bitwise reproducibility across execution backends.Source: C INFERENCE PROCEDURE · p. 17
• If phase 1 hits its cap inside the think block, a forced suffix opens the answer fence and phase 2 resumes with the remaining context budget.Forced-suffix tokens are excluded from the trainable mask and receive no intrinsic reward; the suffix branch is selected by token subsequence matching.Source: C INFERENCE PROCEDURE · pp. 17–18, Table 5 · p. 18
• The evaluator scores the last fenced Python block; completions without one receive zero.Taking the last block scores the program committed to at the end when reasoning contains several speculative programs.Source: C INFERENCE PROCEDURE · p. 18

D TASK DESCRIPTIONS

• All seven discovery problems ask the policy for a Python program, execute it under a wall-clock limit and CPU allocation, and return a scalar score.Tasks differ in what the program returns, how its score is computed, and how that score maps to the RL reward.Source: D TASK DESCRIPTIONS · p. 18
• A candidate that fails a task’s feasibility check receives reward 0 regardless of its raw objective; the sandbox’s hard wall-clock limit applies per candidate.The prompt may advertise a separate soft compute budget, but the hard limit binds when the two differ.Source: Table 6 · p. 18

Erdős’ minimum-overlap problem

• Erdős’ minimum-overlap task minimizes C5 over discretized step-function samples constrained to [0,1] and a fixed sum, with 0.38080 set as the target.The prompt states a best published bound of C5 ≤ 0.38092.Source: §D.1 · p. 18

Autocorrelation inequalities

• AC1 and AC2 optimize autoconvolution norm ratios over nonnegative sequences, whereas AC3 minimizes a related ratio over sequences that may take negative values.The scorer does not clamp negatives; AC3 uses a cubic shaping transform to concentrate reward differences near its target.Source: §D.1 · pp. 18–19

Circle packing

• Circle Packing asks for 26 circles in the unit square that maximize the sum of radii, with a validator rejecting invalid or overlapping circles.The passage gives a best-known value of approximately 2.635983.Source: §D.1 · p. 19

Hadamard matrix

• The Hadamard-labeled task searches over 29 × 29 sign matrices to maximize the absolute determinant, normalized by a fixed benchmark constant.The passage clarifies that an order-29 sign matrix is not a Hadamard matrix with mutually orthogonal rows.Source: §D.1 · p. 19

D.2 SINGLE-CELL DENOISING

• PBMC and Tabula are held out from the Pancreas discovery loop, and reported Score averages benchmark-normalized MSE and Poisson scores against no- and perfect-denoising references.Observed molecules are split into input and target counts by binomial sampling within each dataset.Source: §D.2 · p. 19
• Each run reports both held-out corpora at the epoch selected by in-domain Pancreas MSE, avoiding independent test-set selection because transfer is non-monotone across epochs.Held-out evaluation uses random_state=42; repeating evaluation of a fixed candidate is distinct from repeating the stochastic discovery run.Source: §D.2 · p. 19
• On Pancreas, normalized Poisson score is a hard acceptance gate and accepted candidates are rewarded by 1/MSE; MSE is measured in log-normalized space after total-count normalization to 104.The held-out Poisson gate is not applied to PBMC or Tabula; their normalized Poisson score is clamped to [0, 1], and candidates with negative Poisson NLL are excluded.Source: §D.2 · p. 19
• The reversed-normalization MAGIC seed baseline scores 0.6416 on PBMC and 0.6414 on Tabula.It denoises in square-root space before normalizing.Source: §D.2 · p. 19

E.2 MATHEMATICAL DISCOVERY PROMPTS

• The mathematical-discovery prompt recommends a compact-support window containing a smooth, symmetric oscillatory function, with negative values allowed for phase cancellation.It suggests exploring support width, amplitude, modulation, and frequency, and discretizing the continuous form on a grid around 400–800 points.Source: §E.2 · p. 26

E.3 SINGLE-CELL DENOISING PROMPT

• The denoising prompt emphasizes denoising in raw or square-root space before normalization, reporting Poisson values of about 0.98 versus 0.55 for reversed versus standard normalization order.It also notes that the square-root transform stabilizes Poisson variance and that low nonzero values strongly affect Poisson loss.Source: §E.3 · pp. 28–30
• The denoising prompt targets held-out single-cell RNA-seq, minimizes log-normalized MSE, and imposes a hard Poisson constraint; its stated MAGIC baseline is MSE 0.2316.The prompt defines Poisson loss as a hard constraint and rejects solutions with poisson_norm below 0.97.Source: §E.3 · p. 29

F CODE-LEVEL SOLUTION DIVERSITY METRIC

• The pairwise score combines code length, newline count, and distinct-character-set differences, while the island statistic averages selected unordered program pairs.The candidate-level routine uses a separate average over reference programs, so the two routines aggregate the same heuristic differently.Source: F CODE-LEVEL SOLUTION DIVERSITY METRIC · pp. 30–31
• Larger Figure 1 values indicate greater code-level structural variation across early, middle, late, and final epochs on Erdős and Circle Packing.The passage says these stage-level summaries are consistent with Figure 3’s epoch-level trajectories.Source: F CODE-LEVEL SOLUTION DIVERSITY METRIC · p. 31
• The score is not semantic or execution-based: formatting can change it without changing behavior, and distinct programs can receive zero dissimilarity.Accordingly, the paper interprets it as code-level variation and compares methods within each task.Source: F CODE-LEVEL SOLUTION DIVERSITY METRIC · p. 31