Source-linked AI summary

Learning to Discover at Test Time

Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, Jed McCaleb, Xiaolong Wang, Jan Kautz, Yejin Choi, James Zou, Carlos Guestrin, Yu Sun

arXiv:2601.16175v2cs.LGcs.AI

TL;DR

Scientific discovery requires solving problems beyond an AI model’s training data. TTT-Discover trains an open LLM at test time on experience from the specific problem, achieving state-of-the-art results in almost all tested domains.

  • Problem

    AI systems struggle with scientific discovery problems requiring ideas beyond both their training data and existing human knowledge.

    Source: §1 · p. 2
  • Method

    TTT-Discover performs reinforcement learning on a single test problem, favoring the most promising solutions through its learning objective and search routine.

    Source: §1 · p. 2
  • Results

    TTT-Discover achieves state-of-the-art results in almost all tested mathematics, GPU engineering, algorithm design, and biology problems using only an open model.

    Source: §1 · p. 2
  • Takeaways & Limitations

    Test-time training can produce leading solutions across diverse scientific and engineering discovery problems with an open model.

    Source: §1 · p. 2

    The current method applies only to problems with continuous rewards, not sparse or binary rewards or non-verifiable domains.

    Source: §6 · p. 19

Abstract

from arXiv · show

How can we use AI to discover a new state of the art for a scientific problem? Prior work in test-time scaling, such as AlphaEvolve, performs search by prompting a frozen LLM. We perform reinforcement learning at test time, so the LLM can continue to train, but now with experience specific to the test problem. This form of continual learning is quite special, because its goal is to produce one great solution rather than many good ones on average, and to solve this very problem rather than generalize to other problems. Therefore, our learning objective and search subroutine are designed to prioritize the most promising solutions. We call this method Test-Time Training to Discover (TTT-Discover). Following prior work, we focus on problems with continuous rewards. We report results for every problem we attempted, across mathematics, GPU kernel engineering, algorithm design, and biology. TTT-Discover sets the new state of the art in almost all of them: (i) Erdős' minimum overlap problem and an autocorrelation inequality; (ii) a GPUMode kernel competition (up to $2\times$ faster than prior art); (iii) past AtCoder algorithm competitions; and (iv) denoising problem in single-cell analysis. Our solutions are reviewed by experts or the organizers. All our results are achieved with an open model, OpenAI gpt-oss-120b, and can be reproduced with our publicly available code, in contrast to previous best results that required closed frontier models. Our test-time training runs are performed using Tinker, an API by Thinking Machines, with a cost of only a few hundred dollars per problem.

1 Introduction

TTT-Discover trains an open language model at test time on experience from the single scientific problem it is solving, rather than relying only on search with a frozen model. Its objective and search procedure prioritize one best solution, achieving state-of-the-art results across mathematics, GPU engineering, algorithms, and biology.

  • Scientific discovery requires ideas beyond both model training data and existing human knowledge, making out-of-distribution generalization especially difficult [48] [22] [54] [34].The introduction motivates learning from failed attempts as a way to develop new ideas for hard problems.Source: §1 · p. 2
  • Prior test-time methods, including AlphaEvolve, prompt a frozen LLM to search through attempts, so the model cannot internalize new ideas [50] [37].Source: §1 · p. 2
  • TTT-Discover performs reinforcement learning on the single test problem, using problem-specific attempts as training data while optimizing for one best solution rather than average policy reward.Its policy need not generalize to other problems, and its learning objective and search subroutine favor the most promising solutions.Source: §1 · p. 2
  • TTT-Discover sets the state of the art in almost all attempted problems across mathematics, GPU kernel engineering, algorithm design, and biology using only an open model.The reported problems have continuous rewards, and results are provided for every attempted problem.Source: §1 · p. 2
  • Compared with ThetaEvolve, TTT-Discover yields significant improvements under the same model and compute budget because of its specialized learning objective and search subroutine [78].Source: §1 · p. 2

2 Preliminaries

The preliminaries frame scientific discovery as finding a state-of-the-art solution with an LLM policy in a problem-induced environment. They formalize candidate solutions, rewards, and discovery events, then describe search initialization and state reuse.

2 Preliminaries

  • The paper models test-time scientific discovery as using a pretrained LLM policy to find a new state-of-the-art solution in a problem-specific environment.The environment is formalized as a Markov Decision Process supporting search and learning.Source: §2 · p. 3

2.1 Discovery Problem

  • Each scientific problem supplies a description d, candidate solution state s, and continuous reward R(s), such as inverse kernel runtime, following AlphaEvolve [50].Actions are generated from πθ(· | d,s); valid coding actions produce new states by parsing code, while mathematical problems also execute the parsed code.Source: §2.1 · p. 3
  • A discovery occurs when R(s) exceeds the best-known reward r_sota, with larger improvements defining more significant discoveries.The discovery problem seeks a state with large R(s) − r_sota; the best-known solution may be the empty string when no prior solution exists.Source: §2.1 · p. 3

2.2 Search Methods

  • Search usually starts from the empty solution to avoid over-exploiting the current best solution, while state reuse can warm-start exploration from prior solutions.Reuse favors high-reward states but retains nontrivial probability for lower-reward states; prior work calls related state-action reuse evolutionary search [50] [37] [42].Source: §2.2 · p. 4

3 Learning to Discover at Test Time

TTT-Discover trains the policy online on accumulated test-problem search attempts rather than keeping it frozen. It addresses discovery’s objective, horizon, and exploration differences from standard reinforcement learning by combining an entropic training objective with PUCT-based state reuse.

3 Learning to Discover at Test Time

  • The policy is updated online using its accumulated search attempts in buffer H_i, allowing experience from the test problem to improve π_θ rather than only the next prompt.Algorithm 1 separates the method into reuse and train subroutines, with the final output taken as the highest-reward state encountered.Source: §3 · p. 4, §3.1 · p. 5

3.1 Naive RL at Test Time

  • Discovery seeks one state-of-the-art solution, unlike standard RL’s average reward and repeated deployment objective, so naive RL is misaligned with the desired maximum outcome.Naive RL also starts attempts from scratch, limiting the effective horizon, and can collapse toward safe actions or over-exploit a few promising states.Source: §3.1 · p. 5

3.2 TTT-Discover

  • TTT-Discover combines the entropic objective Jβ(s), which favors maximum-reward actions, with PUCT reuse, which selects initial states using maximum child rewards and exploration bonuses.The objective uses adaptive per-state β to avoid instability and vanishing advantages, while PUCT prioritizes promising yet under-visited states rather than relying on mean rewards.Source: §3.2 · pp. 5–6

3.3 Implementation Details

  • Each run uses gpt-oss-120b with LoRA for 50 training steps and 512 rollouts per step; with typical token usage, a run costs around $500 on Tinker.Rollouts use high reasoning effort, a 32,768-token context window, and one gradient step over each batch.Source: §3.3 · p. 6

4 Applications

This section evaluates TTT-Discover across mathematics, GPU kernel engineering, algorithm engineering, and single-cell denoising, reporting results for every attempted task against human and AI baselines or existing methods. It describes benchmark performance, expert review, and ablation findings across these applications.

4 Applications

  • The evaluation compares TTT-Discover with human results, matched-budget Best-of-25600 sampling, and OpenEvolve using the same model, context budget, and Tinker client.OpenEvolve rollouts were often truncated by expanding prompts, but its prompts and rollouts were not modified.Source: §4 · p. 7

4.1.1 Erdős’ Minimum Overlap Problem

  • 0.380876: TTT-Discover improves Erdős’ minimum-overlap upper bound over AlphaEvolve’s 0.380924, using a 600-piece asymmetric step function.The improvement over AlphaEvolve is reported as 16 times larger than AlphaEvolve’s improvement over the previous state of the art.Source: §4.1.1 · p. 8, Table 2 · p. 8, Figure 2 · p. 8

4.1.2 Autocorrelation Inequalities

  • C1 ≤1.50286: TTT-Discover sets the first autocorrelation inequality state of the art with a 30000-piece step function, but its second-inequality bound of 0.959 trails AlphaEvolve’s 0.961.The Qwen3-8B variant also certified tighter bounds than ThetaEvolve using a worse model and smaller sampling budget.Source: Table 2 · p. 8, §4.1.2 · p. 9

4.1.3 Circle Packing

  • TTT-Discover with Qwen3-8B matches the best known circle-packing constructions for n = 26 and n = 32, making no improvement.Its solutions use staggered or hexagonal initialization followed by sequential least squares refinement and are simpler than recent work.Source: §4.1.3 · p. 10, Table 3 · p. 10

4.1.4 Expert Review

  • Experts found the Erdős and AC1 constructions straightforward to verify as valid piecewise-constant bounds that improve the state of the art.Verification evaluates the target quantity over discrete points and checks the corresponding norm constraints.Source: §4.1.4 · p. 11

4.2 Kernel Engineering

  • TriMul achieves state of the art across H100, A100, B200, and MI300X, including a kernel 50% faster than the top human A100 kernel and over 15% improvement for every GPU type.Training used H100 runtime, while generated kernels generalized to other GPU types; B200 and MI300X could not be submitted because of an infrastructure problem.Source: §4.2 · p. 11, Table 4 · p. 12

4.2.1 Expert Review

  • GPUMode organizers attributed the TriMul advantage to fusing pointwise operations, reducing memory traffic and launch overhead, using FP16 activations, and delegating matrix multiplications to cuBLAS.They noted that this strategy resembles the best human solutions but executes more effectively, while FP16 may create numerical-stability risks in full workloads.Source: §4.2 · pp. 11–12, §4.2.1 · p. 13

4.3 Algorithm Engineering

  • The algorithms would have placed first in both AtCoder competitions: ahc039 marginally surpasses the best human and improves the ALE-Agent starting solution from fifth to first, while ahc058 outscored all submissions from scratch.ShinkaEvolve reached second place on ahc039 using significantly more capable frontier models.Source: §4.3 · p. 14, Table 6 · p. 14

4.4 Single Cell Analysis

  • The denoising environment trains on Pancreas and reports performance on held-out PBMC and Tabula Muris Senis Lung using MSE and Poisson negative log-likelihood metrics.Rewards use MSE, with zero reward for violating Poisson-score constraints or exceeding the 400-second limit.Source: §4.4 · p. 15
  • TTT-Discover consistently improves denoising performance on held-out PBMC and Tabula Muris Senis Lung datasets, outperforming existing methods on the OpenProblems benchmark.The method starts from MAGIC and adds gene-adaptive transform ensembling, low-rank SVD refinement, and log-space polishing that directly optimize benchmark metrics.Source: §4.4 · p. 15, Table 7 · p. 15

4.4.1 Expert Review

  • Prof. Eric Sun judged the MAGIC improvement simple, aligned with MAGIC’s smoothing approach, and empirically better on key metrics, while recommending evaluation on biologically relevant tasks.Benchmark metrics may not transfer to improved biological insights, which are difficult to quantify.Source: §4.4.1 · p. 16

4.5 Ablations

  • Only the full TTT-Discover algorithm achieves the best TriMul competition performance; constant-β or expected-reward training, no training, and naive RL or no reuse improve less or stagnate.The ablations compare training objectives and reuse strategies, with PUCT fixed or varied depending on the experiment.Source: §4.5 · pp. 16–17, Table 8 · p. 16
  • The ablation study prioritizes broadly reliable design choices with minimal task-specific tuning, leaving possible gains from additional schedules and hyperparameter interactions untested.Learning rate, batch size, and LoRA rank were fixed after initial project iterations.Source: §4.5 · p. 17

5 Related Works

This section situates TTT-Discover within continual learning and test-time training, emphasizing its per-instance training objective and distinction from methods that generalize across problems or optimize test-set averages.

5.1 Continual Learning

  • Conventional continual learning updates models as data distributions change over time while emphasizing retention of past knowledge when learning from present data.Its formal setting samples training and test data from the current distribution and evaluates on test data accumulated through the current timestep.Source: §5.1 · p. 17

5.2 Test-Time Training

  • Test-time training formulates a potentially different learning problem for each test instance, thereby adapting training to the specific instance rather than using one fixed model.This framework connects to personalized learning and removes the strict separation between training and testing.Source: §5.2 · p. 18

5.2.1 TTT on Nearest Neighbors: Larger Effective Capacity

  • Nearest-neighbor test-time training finds related training examples and fine-tunes on them before prediction, increasing effective capacity enough for a linear model to fit nonlinear ground truth.This family includes locally weighted regression, local learning, and KNN-SVM, with later extensions to language, reasoning, and visual-motor tasks [18] [25] [26] [4].Source: §5.2.1 · p. 18

5.2.2 TTT for Novel Instances: Better Generalization

  • For novel unlabeled test instances, test-time training can generate relevant data or self-supervised pairs on the fly, improving generalization under distribution shifts [68] [13].Recent examples include AlphaProof, few-shot reasoning with supervised augmentations [2], and policy-gradient methods that use the policy as evaluator [38].Source: §5.2.2 · pp. 18–19
  • TTT-Discover combines per-instance RL updates with replay or reuse, but tailors the learning objective and reuse rule to discovery, outperforming ThetaEvolve under matched model and compute budgets (Table 2).The paper attributes this improvement to an entropic objective and PUCT-based reuse rather than more complicated evolutionary heuristics.Source: §5.2.2 · p. 19

5.3 RL on One Example

  • One Example RL trains on a single training problem and seeks generalization to other problems, whereas TTT-Discover trains directly on the test problem to solve that specific problem [79].The cited method uses an example from datasets such as MATH and reports cross-problem generalization.Source: §5.3 · p. 19

5.4 RL on the Test Set

  • TTRL trains across an entire test set using majority-vote pseudo-labels, whereas TTT-Discover trains on one test problem with a continuous verifiable reward to find one exceptional solution [88].The objectives therefore differ between improving average performance over problems and discovering a single high-quality solution.Source: §5.4 · p. 19

6 Future Work

Future work should extend the method beyond continuous-reward problems to settings with sparse or binary rewards and non-verifiable domains.

6 Future Work

  • The method currently applies only to problems with continuous rewards, motivating test-time training for sparse or binary rewards and non-verifiable domains.The authors identify this extension as the most important direction for future work.Source: §6 · p. 19

A Training details

Experiments used fixed hyperparameters, with the KL penalty coefficient set to 0.1 for most applications and 0.01 for algorithm engineering.

Training details

  • The KL penalty coefficient was 0.1 for almost all applications and 0.01 for algorithm engineering, with hyperparameters otherwise fixed across experiments.The paper presents the objective function and reuse algorithm separately.Source: A Training details · p. 25

A.1 Entropic utility objective · A.2 PUCT Prioritization · B Mathematics

The method combines an adaptive entropic utility objective with PUCT-inspired archive prioritization. The entropic update controls reweighting through a KL budget, while archive search favors promising states and maintains diversity.

A.1 Entropic utility objective

  • The entropic objective yields Aβ as a mean-baselined advantage; setting β is tricky because later-training advantages can become very small, while large early β can cause instabilities.Concurrent work [29] recommends β = 2, but the authors found setting β tricky.Source: §A.1 · p. 26
  • Adaptive β controls entropic reweighting through a KL budget, preventing updates from moving too far from the current policy.β is increased only until the KL budget is exhausted, and γ = ln2 is fixed in experiments.Source: §A.1 · p. 26
  • Batch estimation solves the weight-concentration constraint by bisection over β ≥0 and computes leave-one-out entropic advantages with reward-maximum stabilization.The estimator uses N rollouts from the same state and an ϵ denominator for numerical stability.Source: §A.1 · p. 26
  • Adaptive β permits larger values when improvements are consistently small and smaller values when rare large improvements would concentrate weights on outlier trajectories.The estimator is invariant to positive reward scaling and additive reward shifts.Source: §A.1 · p. 26

A.2 PUCT Prioritization

  • PUCT-inspired archive selection scores discovered states using visitation counts, archive reward range, rank-based priors, and the best one-step reachable reward.The archive state with rank 0 has the highest reward, and the exploration coefficient satisfies c > 0.Source: §A.2 · p. 27
  • The archive retains the top two children per expanded parent, then keeps at most 1000 highest-reward states while preserving initial seeds.After expansion, the best child reward updates the search statistics and increments T.Source: §A.2 · p. 27
  • Compared with AlphaZero PUCT, this search uses maximum-child values, rank-based archive priors, ancestor-wide visitation backpropagation, and lineage blocking for diversity.AlphaZero instead uses mean edge values, learned action priors, and virtual loss as a temporary penalty.Source: §A.2 · p. 27

B.1 Circle Packing

The circle-packing approach initializes structured layouts and optimizes circle centers and radii to maximize their total size under square-boundary and non-overlap constraints. It considers both a staggered five-row layout and a 30-circle hexagonal arrangement with two additional corner circles.

Staggered-grid initialization

  • One initialization uses a staggered grid with 5 rows, varying the number of circles per row, row-specific horizontal spacing, and refined vertical spacing for denser packing.The initial centers are generated row by row before optimization.Source: §B.1 · p. 28

Optimization objective and constraints

  • The method maximizes the sum of circle radii while enforcing containment within the unit square and pairwise non-overlap.The constraints require each center-radius pair to remain inside the square and each inter-center distance to be at least the sum of the corresponding radii.Source: §B.1 · pp. 28–30

Numerical optimization

  • The initialized variables are optimized with Sequential Least Squares Programming, after which the optimized centers, radii, and total radius are extracted.The objective is implemented as minimizing the negative total radius.Source: §B.1 · pp. 28–30

Hexagonal initialization

  • A second initialization generates a hexagonal arrangement for 30 circles using 6 rows with 5 columns each, then adds two extra circles near the top-right and bottom-right corners.Its starting radius is set from the vertical constraint before the joint optimization of centers and radii.Source: §B.1 · p. 30

B.2 Autocorrelation Inequalities

The autocorrelation-inequality tasks use randomized or state-of-the-art initial sequences and verifier-defined objective bounds. Algorithms are evaluated under fixed computational resources and a timeout of up to 1100 seconds.

B.2 Autocorrelation Inequalities

  • Initial sequences are generated by repeating a random value in [0,1] 1,000–8,000 times or loading a state-of-the-art construction when available.The first verifier rejects invalid inputs and returns ∞ for invalid constructions.Source: §B.2 · p. 31
  • The first inequality minimizes an upper bound based on discrete autocorrelation, while the second computes C2 from the L2, L1, and L∞ norms of the autocorrelation.The second inequality estimates the L2 norm using piecewise-linear integration with a Simpson-like rule and endpoint zeros over [−1/2,1/2].Source: §B.2 · p. 31
  • Each algorithm runs with 1 GB of memory, 2 CPUs, and a timeout of up to 1100 seconds.Source: §B.2 · p. 31

B.3 Erdős’ · C Kernel engineering

The paper describes initialization and resource settings for Erdős’ experiments, alongside problem-specific kernel seeds for trimul and MLA-Decode. These procedures use random or example-based starting states before test-time discovery.

B.3 Erdős’

  • Erdős’ experiments initialize TTT-Discover with random constructions of 40–100 samples around 0.5 and random perturbations.Source: §B.3 · p. 31
  • Sequences with more than 1000 values in the verifier are filtered out before running the algorithms.Source: §B.3 · p. 31
  • Each Erdős’ algorithm runs with 1 GB of memory, 2 CPUs, and a timeout of up to 1100 seconds.Source: §B.3 · p. 31

C Kernel engineering

  • For trimul, kernel engineering starts from the matrix multiplication kernel provided in Triton’s README, primarily as a syntax example.Source: C Kernel engineering · p. 31
  • For MLA-Decode, the process first prompts the base model with a softmax kernel to generate a correct but unoptimized kernel.Source: C Kernel engineering · p. 31
  • The generated MLA-Decode kernel then becomes the initial state after removing the earlier softmax example.Source: C Kernel engineering · p. 31

C.1 Kernel evaluation details

Kernel evaluation follows the official GPU Mode competition’s correctness and runtime procedures, with target-hardware verification and task-specific reporting and selection protocols.

Verifier setup

  • The verifier reproduces official competition checks, comparing custom-kernel outputs with PyTorch references before multi-iteration runtime benchmarking on H100s for TriMul and H200s for MLA-Decode.Training verification uses official test cases, precision settings, timing iterations, and related hyperparameters, with minor codebase integration changes.Source: §C.1 · p. 32

Final-report environments

  • Final reports use the official TriMul A100/H100 leaderboard when possible, while other TriMul and MLA-Decode tasks rely on GPU Mode-reviewed local environments replicating the official setup.Official submission was blocked for TriMul B200/MI300X and MLA-Decode MI300X by a GPU Mode server infrastructure problem.Source: §C.1 · p. 32

Best-kernel selection

  • For final submission, kernels are selected from verifier or training candidates, reverified three times on target hardware, and the kernel with the smallest average runtime is reported.TriMul H100 uses the 20 best verifier-scoring kernels; other tasks combine 20 best training-scoring kernels with 20 random correct kernels every 10 training steps.Source: §C.1 · p. 32

C.2 Analysis of best generated kernels

This section analyzes TTT-Discover’s generated TriMul H100 kernel, focusing on its fused computation and handling of memory-I/O bottlenecks. It describes the kernel’s no-gradient forward pass for float32 inputs, optional masks, module weights, and configurable dimensions.

TriMul H100 kernels

  • The best TriMul H100 kernel fuses input LayerNorm, input gating, output LayerNorm, and output gating, while converting the O(N^3) matmul inputs to fp16 for cuBLAS TensorCores.This design targets heavy memory I/O from elementwise operations while using cuBLAS for the compute-heavy matrix multiplication.Source: §C.2 · p. 32
  • The final kernel outperforms earlier training kernels by fusing more operations and optimizing memory access patterns more deeply, especially for output LayerNorm, gating, and projection.An early kernel fused LayerNorm but not input gating; a middle-stage kernel matched the final fusion set but had less efficient output-side memory access.Source: §C.2 · p. 32
  • Compared with the best human kernel, TTT uses similar input LayerNorm and gating fusion but additionally fuses output LayerNorm and gating with output projection.The TTT kernel performs less block-size auto-tuning, which may limit it, while its extra output fusion could explain its moderate advantage.Source: §C.2 · p. 32
  • The implementation includes row-wise LayerNorm, fused projection and gating with an optional scalar mask, and fused hidden-dimension LayerNorm, output gating, and final linear projection.The listed fused stages use FP16 outputs or matmuls with FP32 accumulation where specified.Source: §A.2 · p. 27, §C.2 · pp. 33, 34, 36

Kernel interface

  • The outgoing TriMul forward pass accepts float32 input tensors, optional boolean/float masks, float32 module weights, and configuration with dimension and hidden dimension.The configuration may also include a nomask option.Source: §C.2 · p. 37

Fused projection and gating

  • The kernel performs row-wise LayerNorm, prepares column-major fp16 projection and gate weights, and conditionally applies the input mask before fused projection and gating.The fused projection/gating stage is launched over tiled row and hidden-dimension grids.Source: §C.2 · pp. 37–38

Pairwise multiplication and output projection

  • After projection and gating, the kernel computes left @ right^T with batched fp16 matrix multiplication, then applies hidden-dimension LayerNorm, an output gate, and a final linear projection.The final projection uses fp16 weights with fp32 accumulation.Source: §C.2 · p. 38

C.3 TTT MLA-Decode kernels filtered with Triton kernels

This section reports results for TTT MLA-Decode kernels filtered with Triton kernels.

C.3 TTT MLA-Decode kernels filtered with Triton kernels

  • Table 10 reports results for TTT MLA-Decode kernels filtered with Triton kernels.Source: Table 10 · p. 39

D Algorithm Engineering

The algorithm-engineering experiments train on 150 generated AtCoder cases with correctness and time-limit gating, then evaluate the three best local programs on hidden official tests. AHC039 and AHC058 use reduced token limits, shared KL regularization, and a slightly improved learning rate for AHC058.

  • Training uses 150 generated test cases, requiring every case to pass correctness checks and finish within the 2-second limit before receiving non-zero reward.Per-case scores follow the problem-specific AtCoder scoring; ahc039 uses ShinkaEvolve’s relative-placement metric, while ahc058 directly uses the contest score.Source: D Algorithm Engineering · p. 39
  • The final evaluation submits the three highest-scoring local programs to AtCoder, where hidden tests determine the final score under official validation and scoring.Submissions use C++23 (GCC 15.2.0) and are checked for incorrect output, time-limit violations, compilation errors, and runtime errors.Source: D Algorithm Engineering · pp. 39–40
  • For AHC039 and AHC058, prompt-length and thinking-token limits are reduced to 22000 and 25000, respectively, while both use a KL coefficient of 1 × 10^-2.AHC058 performed slightly better with a learning rate of 2×10−5; other hyperparameters remain standard.Source: D Algorithm Engineering · p. 40

E Single cell analysis

This section presents an adaptive, gene-wise denoising pipeline for single-cell analysis, combining diffusion, variance stabilization, imputation, ensemble aggregation, and distribution-preserving refinements. It evaluates normalized MSE and Poisson losses on the OpenProblems Pancreas dataset and tests generalization on PBMC and Tabula.

Evaluation setup

  • The Pancreas dataset supplies normalized MSE and Poisson scores, while PBMC and Tabula assess generalization; verification excludes normalized Poisson outside 0.97–1.The benchmark’s main score is the mean of normalized MSE and normalized Poisson, but filtering focuses optimization on MSE.Source: E Single cell analysis · p. 40
  • The evaluation increases memory to 3GB and limits execution time to 400 seconds to encourage generalization.The prompt also permits solutions inspired by the OpenProblems paper’s Supplementary Materials.Source: E Single cell analysis · p. 40

Denoising method

  • The pipeline applies variance-stabilizing transforms, selects highly variable genes, and can match each gene’s diffused mean and variance to the raw data.It also includes residual diffusion smoothing and low-rank post-processing controls.Source: E Single cell analysis · pp. 40, 41, 43, 44
  • The method uses adaptive gene-specific diffusion weights based on dropout, variance reduction, correlation, and expression, with larger weights favoring diffusion.It imputes zeros with diffusion-weighted neighbor averages and applies weighted multi-scale diffusion with baseline smoothing for every gene.Source: E Single cell analysis · pp. 41–42
  • The denoiser supports a balanced mode mixing MSE and Poisson objectives and an mse mode tuned for minimum MSE while retaining the Poisson constraint.The method exposes `mode` values `balanced` and `mse`.Source: E Single cell analysis · p. 44
  • Optional final diffusion directly smooths the log-normalised representation used for MSE, with final_smooth_t controlling the number of polishing steps.Typical final polishing uses 2-4 diffusion steps.Source: E Single cell analysis · pp. 44–45
  • The pipeline preserves raw library sizes, applies gene-wise dropout handling, removes extremely sparse highly variable genes, and falls back to all genes if filtering removes everything.These steps support subsequent denoising while retaining a valid gene set.Source: E Single cell analysis · pp. 45–46
  • A shared diffusion operator is built and then applied separately across the selected variance-stabilizing transforms to produce denoised cell-by-gene matrices.The transform stage performs forward variance-stabilizing transformation and library-size normalisation, with rows summing to 1.Source: E Single cell analysis · pp. 46–47

Gene-wise ensemble

  • The pipeline ensembles denoised outputs from multiple variance-stabilizing transforms using gene-wise dropout-dependent weights.The weighted outputs are summed across transforms, with one weight defined as 2.0 * dropout_frac * (1.0 - dropout_frac).Source: E Single cell analysis · p. 48

Global post-processing

  • Optional global post-processing applies exponent shrinkage, low-rank SVD refinement, residual diffusion, and log-space smoothing.Low-rank refinement is conditional on a matrix-size limit, while log-space smoothing uses gene-specific blending based on reliance on the raw signal.Source: E Single cell analysis · pp. 48–49
  • Additional optional blending mixes the result with scaled raw counts for very highly expressed genes and re-normalizes rows to preserve library sizes.The raw-count blend is explicitly intended to help very high-expression genes, followed by row-wise rescaling.Source: E Single cell analysis · p. 49

Final cleanup

  • Final polishing performs extra and final log-space smoothing before a cleanup stage thresholds small values and returns float64 denoised counts.The pipeline includes configurable extra and final smoothing steps, then applies zero-threshold cleanup before returning the result.Source: E Single cell analysis · pp. 49–50

F Prompts

This section presents prompts for diverse test-time optimization and discovery tasks, spanning mathematical inequalities, GPU kernel implementations, heuristic algorithm contests, and single-cell RNA-seq denoising. Across these domains, the prompts define task specifications, implementation or feasibility constraints, evaluation criteria, and encouragement to explore diverse or improved solutions.

First autocorrelation inequality

  • The first autocorrelation prompt reduced the upper bound from 2.0000000000 to 1.5172973712 using a 1000-item nonnegative step-function construction.The task allowed arbitrary search methods and repeated calls to evaluate_sequence() within a 1000-second limit.Source: F Prompts · p. 51

Second autocorrelation inequality

  • The second autocorrelation prompt improved the lower bound from 0.6666666667 to 0.9235566275 with a 1024-item nonnegative construction.It presented a coarse-to-fine stochastic-search, projected-gradient, and upscaling procedure as prior inspiration while inviting alternative searches.Source: F Prompts · p. 53

Erdős’ minimum overlap problem

  • The Erdős’ prompt asks for a step function h:[0,2]→[0,1] minimizing the overlap constant C5 under integral normalization, with lower values providing tighter upper bounds.The stated current record is C5 ≤0.38092, with a goal of finding a construction showing C5 ≤0.38080.Source: F Prompts · p. 55

Triton TriMul kernel

  • The Triton prompt asks for a forward-only custom_kernel implementing the outgoing TriMul operator for AlphaFold3 on [B,N,N,C] tensors.The task emphasizes fusion across projections, gating, normalization, and an N^3 pairwise contraction under varied sizes, masks, and input distributions.Source: F Prompts · p. 56

Kernel optimization

  • The kernel is tested under configurations varying sequence length, batch size, dimension, masking, and normal or Cauchy input distributions, including sequence lengths 256, 512, 768, and 1024.Source: F Prompts · p. 58
  • 1000000.0000 microseconds current runtime leaves a 999000.0000-microsecond gap to the 1000-microsecond target.The runtime is lower-is-better, and the prompt reports no previous implementation attempt.Source: F Prompts · p. 58
  • The Triton optimization prompt requires correctness across multiple shapes while targeting an H100, Triton 3.3.1, float32 output, and at least partial kernel implementation.Mixed precision and PyTorch operations are allowed, but the final code must be contained in one Python block with a short algorithm docstring.Source: F Prompts · p. 59

MLA-Decode

  • The MLA-Decode prompt asks for an optimized Triton forward pass covering Q/KV down-projections, KV-cache updates, up-projections, RoPE, multi-head attention, and the final output linear layer.The function receives a configuration, a single-token bfloat16 input, and a KV cache, then returns the attention output and updated cache.Source: F Prompts · pp. 60, 62
  • The MLA implementation separates latent and RoPE streams, applies cached cosine and sine tables, projects the no-PE query into latent space, and sums latent and RoPE attention scores.The prompt updates the cache before processing the current query and computes attention over all cached positions.Source: F Prompts · p. 63

Triton kernel prompt

  • The Triton prompt specifies an H200 environment, Triton 3.4.0, bfloat16 inputs, a `custom_kernel` entrypoint, and a single final Python block.It encourages optimizing multiple operations, including RoPE and attention, and permits `torch.compile()`.Source: F Prompts · p. 64
  • The Triton prompt documents compiler-specific restrictions for `tl.load`, dtypes, `tl.arange`, `tl.range`, and SIMD indexing, with compliant examples.The examples show typed constants, power-of-two ranges, masked loads, broadcast-based scalar loads, and stable loop-index handling.Source: F Prompts · pp. 64–65

AHC039 prompt

  • The AHC039 task asks contestants to construct a simple axis-aligned polygon maximizing mackerels minus sardines, subject to vertex, perimeter, integer-coordinate, and non-self-intersection constraints.The score is max(0, a − b + 1), where a and b are the numbers of mackerels and sardines inside the polygon.Source: F Prompts · p. 66

AHC058 prompt

  • The task uses a hierarchical machine system in which strengthening consumes apples, production propagates upward toward apple generation, and the objective is maximizing total apples after T turns.If S is the number of apples at the end of T turns, the score is round(10^5 × log_2 S).Source: F Prompts · p. 68
  • 5,626,752.9267 current performance remained below the 6,500,000 target, leaving a gap of 873,247.0733.Source: F Prompts · p. 70
  • The denoising task predicts true expression from noisy single-cell RNA-seq counts and evaluates outputs using log-normalized-space MSE and Poisson loss.The task targets technical dropout and low capture efficiency while requiring denoised output with the same shape as the input.Source: F Prompts · p. 71
  • Reversed normalization—denoising raw or log counts before normalization—achieved Poisson approximately 0.98 versus approximately 0.55 for standard ordering.The prompt also identifies square-root transformation as variance-stabilizing for Poisson distributions.Source: F Prompts · p. 71
  • Poisson loss is especially sensitive to low nonzero values, so values below 1 should be pushed toward zero; reversed-normalization MAGIC performs best.Poisson is a hard constraint, with rejection below poisson_norm 0.97, while reward is based only on MSE after passing it.Source: F Prompts · pp. 71–72
Loading 2601.16175v2…