PaperlayerView exact revision on arXiv

Source-linked AI summary

Goal-Persistent Coding Agents as Scientific Performance Engineers: A Fixed-Radius Nearest-Neighbor Case Study

Xiangyang Ju

arXiv:2609.31980v1cs.AIcs.PFcs.SE

TL;DR

Evidence remains limited on whether general-purpose coding agents can conduct rigorous scientific performance engineering. This paper studies persistent-goal agents optimizing FRNN in a real repository, and reports a standalone implementation with a 1.58-fold speedup over the original GPU-resident PyTorch path. A separate rerun followed a different optimization path and found a workload-specialized variant with even better performance.

  • Problem

    Whether general-purpose coding agents can sustain credible scientific performance engineering across a real repository remains an open empirical question.

    Source: §1 · p. 1

  • Method

    Codex and Claude Code pursue persistent goals with correctness constraints and evaluation procedures, selecting and testing successive optimization hypotheses without a prescribed transformation sequence.

    Source: §1 · p. 2

  • Results

    1.58-fold speedup over the original GPU-resident PyTorch path was achieved by the standalone C++/CUDA libFRNN’s synchronous NumPy interface.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The study demonstrates feasibility of iterative scientific software optimization by general-purpose agents, guided by persistent objectives, correctness tests, repository state, and benchmark feedback.

    Source: §6 · p. 10

    The optimized K ≥256 path omits heap maintenance and is exact for the evaluated workload’s density, but general nearest-K semantics require heap maintenance when more than K neighbors fall within the radius.

    Source: §5 · p. 10

Abstract

from arXiv · show

Coding agents can pursue persistent objectives across many tool-use turns, but evidence that general-purpose agents can conduct rigorous scientific performance engineering remains limited. We present a repository-scale case study in which off-the-shelf Codex and Claude Code agents optimize fixed-radius nearest-neighbor (FRNN) search for particle tracking. Starting from a PyTorch-dependent CUDA implementation, the agents follow an executable goal that specifies exact-correctness tests, profiling requirements, and acceptance criteria without prescribing code transformations. In the primary sequential trajectory, they autonomously remove the PyTorch dependency and conduct hypothesis-driven optimization experiments. The resulting standalone C++/CUDA library exactly reproduces the targeted reference result. Its synchronous NumPy interface achieved 1.6-fold speedup over the original GPU-resident PyTorch interface, despite including host transfers. Similar speedups were observed across different GPU architectures and software stacks. An independent optimization rerun followed a different sequence of hypotheses and reached even better performance on the target workload. These results show that goal-persistent coding agents can act as experimental performance engineers, and that executable scientific contracts are needed both to guide and to validate their optimization.

1 Introduction

The paper asks whether general-purpose, goal-persistent coding agents can conduct credible scientific performance engineering, using fixed-radius nearest-neighbor search as a test case. It reports a repository-scale optimization study and identifies the limits of the evidence.

1 Introduction

  • Scientific performance engineering is iterative: engineers profile bottlenecks, test hypotheses, and use measured results to retain, revise, or reject changes.Later experiments build on earlier outcomes, and many cycles may be needed for a satisfactory solution.Source: §1 · p. 1
  • General-purpose agents can sustain a complete repository-scale optimization loop, but whether they can do so credibly as scientific performance engineers remains an open empirical question.Agents receive correctness constraints, evaluation procedures, and candidate optimization areas, but no prescribed transformation sequence.Source: §1 · pp. 1–2
  • FRNN preserves a physical or learned length scale while allowing neighbor counts to vary with local density, supporting applications including ExaTrkX particle-tracking graph construction.The paper also cites molecular and colloidal simulation, smoothed particle hydrodynamics, and learned point-cloud models.Source: §1 · pp. 1–2
  • Exact FRNN search trades performance against correctness: faster spatial methods depend on workload conditions, while small floating-point or boundary-screening changes can silently alter neighbors near the radius threshold.Brute force costs O(N^2D); grids, cell lists, trees, and spatial sorting reduce candidates but depend on dimension, density, radius, and maximum neighbor count.Source: §1 · p. 2
  • The study contributes a repository-scale case study of hypothesis-driven optimization by general-purpose Codex and Claude Code agents.It also analyzes the executable goal, repository artifacts, profiler feedback, and handoff state, and evaluates an exact standalone C++/CUDA FRNN implementation.Source: §1 · p. 2
  • The primary trajectory establishes feasibility, while the continuation rerun illustrates an alternative path; controlled comparisons across repeated runs and optimization approaches remain future work.The authors do not claim these trajectories estimate the causal effect of goal persistence.Source: §1 · p. 2

2 Related work

Related work spans FRNN implementations, agentic code optimization, and autonomous research frameworks. The paper situates its study as repository-scale scientific CUDA optimization using general-purpose agents and executable correctness constraints.

2 Related work

  • The study builds on Xue’s uniform-grid CUDA FRNN and libFRNN, which retains PyTorch, host synchronization, and array-of-structures layout.FRNN is also used in ExaTrkX graph construction, a production particle-tracking pipeline.Source: §2 · pp. 2–3

2.1 Agentic code optimization

  • Agentic optimization systems use population-based search or performance-engineer-like agent loops, while broader autonomous-science systems include AI Scientist-v2 and ScienceFlow.Purpose-built search and evaluation harnesses can make research more reliable and traceable, but make it harder to isolate the underlying agent’s contribution.Source: §2.1 · p. 3, §2.2 · p. 3

2.2 Autoresearch Frameworks

  • Autoresearch repeatedly modifies code, runs fixed-duration experiments, and keeps or discards changes; SynAgent instead conducts hypothesis-driven materials synthesis in a physical laboratory.These frameworks encode specialized research control structures, whereas this study applies agents to a software repository under executable correctness constraints.Source: §2.2 · p. 3

3 Autonomous optimization methodology

The optimization campaign uses persistent goals to guide a sequential, hypothesis-driven process across Codex and Claude Code. The repository preserves the contract and experimental state through the handoff.

3 Autonomous optimization methodology

  • The campaign first removed PyTorch and Python dependencies, then optimized FRNN latency while preserving brute-force-equivalent results; a later Claude rerun began from the Codex handoff.Persistent-objective mechanisms kept the completion contract active across tool-use turns and required auditing against repository evidence.Source: §3 · pp. 3–4

3.3 Executing and handing off the goals

  • The agents ran goals from repository artifacts with limited human intervention; Codex’s first goal produced a standalone C++/CUDA API, then Claude continued from its handoff.Codex’s optimization goal stopped at a spend cap before declaring completion; a separate Claude rerun from the same checkpoint ended after 91.9 minutes at the provider’s session-usage limit. The 200 ms continuation threshold was chosen arbitrarily to encourage further hypothesis testing.Source: §3 · p. 4, §3.3 · pp. 5–7

3.1 From a human meta-prompt to machine-executable goals

  • Codex drafted two goal documents from a human meta-prompt, defining the starting state, required outcomes, correctness contract, tests, acceptance rules, and preserved artifacts.The optimization goal specifies an immutable baseline, profiling before redesign, and hypothesis-driven experiments, including structural improvements rather than only parameter tuning.Source: §3.1 · pp. 4–5

3.2 Autonomous optimization loop

  • The goal’s outer loop tests optimization hypotheses; its inner loop tunes a hypothesis’s settings, retaining faster correct variants and discarding exhausted ones without speedup.Experiments are logged whether kept or discarded.Source: §3.2 · p. 5

4 Results

On the representative particle-tracking workload, the standalone implementation outperformed the original FRNN at the public interface and across tested A100 configurations. The optimization trajectories also show distinct intermediate gains and a rerun that did not reach its target threshold.

4.1 Experimental configuration

  • The representative event had 271,663 points in 12 dimensions, radius 0.12, and K = 1000; the NumPy binding included host transfers and compact host output unlike the device-resident PyTorch interface.Primary measurements used an RTX 2070 SUPER; the cross-hardware evaluation also used 40 GB and 80 GB A100 GPUs.Source: §4.1 · p. 7

4.2 Primary agent-directed optimization trajectory

  • The initial Codex phase improved warm latency on five synthetic acceptance cases by 1.80-fold and reduced temporary edge capacity on the real workload from about 4.35 GB to 148 MB through count-first exact allocation.Latency rose as well as fell across experiments, so the trajectory was non-monotonic.Source: §4.2 · p. 7
  • 299.6 ms for the frnn_cuda NumPy path versus 473.0 ms for the original GPU-resident PyTorch path; the median ratio was 1.58.The comparison includes different API boundaries and is an end-to-end interface result, not a kernel-only attribution.Source: Table 1 · p. 8, §4.2 · p. 8
  • The final audited 267.829 ms result was 25.7% faster than the Claude baseline; profiling attributed 98.6% of compute to neighbor search and identified dependent-load latency and divergent query work as bottlenecks.The kernel showed 46.4% SM throughput and 9.1% DRAM throughput; adding grid dimensions or cooperative search did not help.Source: §4.2 · p. 8
  • On A100 GPUs, frnn_cuda speedups were 1.47 and 1.50 on the 40 GB and 80 GB GPUs with PyTorch 2.13.0/CUDA 13.0, and 1.38 with PyTorch 2.8.0/CUDA 12.9.Table 2 reports median latencies and paired speedups across GPU generations; every row reproduces the complete reference edge set and favors frnn_cuda.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 Rerun agent-directed optimization

  • The independent Claude rerun reduced warm-device latency from 307.861 ms to 207.299 ms, a 32.7% reduction, but did not reach the requested sub-200 ms threshold before its provider session limit ended the run.Its largest step deferred the indirect original-index lookup until after radius rejection.Source: §4.3 · p. 9, Table 3 · p. 9

5 Limitations

The evaluation demonstrates two optimization trajectories, not a controlled agent study, and its results apply to a limited workload and execution setting. The rerun’s heap-free optimization is exact for the evaluated workload but requires further testing for general FRNN use.

5 Limitations

  • The study contains one primary trajectory and one continuation-stage rerun with unequal budgets, preventing estimates of repeated-run outcomes or isolation of goal persistence, model choice, profiler access, environment, and provider effects.The primary run accumulated 5 h 30 min of active-turn time; the rerun ended after 91.9 min because of a provider session limit.Source: §5 · pp. 9–10
  • The heap-free K ≥ 256 path was exact for the evaluated workload, where every query had fewer than K in-radius neighbors; denser inputs require heap maintenance for general nearest-K semantics.Generalizing the optimization requires tests of the new dispatch and adversarial high-density cases.Source: §5 · p. 10
  • Evaluation was limited to single-GPU CUDA execution and one representative scientific workload; newer GPU generations, broader workload and density regimes, power efficiency, and multi-GPU scaling remain unevaluated.The implementation was faster than the original on every tested Turing and Ampere GPU.Source: §5 · p. 10
  • The case study optimizes one algorithmic library rather than an end-to-end scientific workflow; workflow-level choices and end-to-end scientific metrics remain future evaluation targets.The paper identifies data preparation, algorithm selection, resource placement, and stage boundaries as possible workflow-level choices.Source: §5 · p. 10

6 Conclusion

General-purpose coding agents sustained hypothesis-driven optimization, producing a standalone FRNN implementation with a faster synchronous NumPy interface. An independent rerun found a faster workload-specialized variant, while underscoring the need to make its density assumption explicit.

6 Conclusion

  • The agents sustained iterative optimization by using a persistent objective, correctness tests, repository state, and benchmark feedback to test hypotheses, keep improvements, reject regressions, and repair an audited correctness issue.Source: §6 · p. 10
  • The standalone C++/CUDA implementation’s synchronous NumPy interface achieved a 1.58-fold speedup over the original GPU-resident PyTorch path, despite including host transfers.The retained design combines four-axis gridding, structure-of-arrays screening, roundoff-safe early rejection, adaptive top-K maintenance, workspace reuse, and compact output.Source: §6 · p. 10
  • An independent rerun explored different hypotheses and found a faster workload-specialized variant, but its density assumption holds for the target workload, not arbitrary FRNN inputs.The authors note that optimization may reveal assumptions that correctness tests and scientific contracts need to make explicit and test.Source: §6 · p. 10
  • The study concludes that general-purpose agents can conduct empirical optimization of a real GPU algorithm and repository, while identifying reproducibility, broader workloads, hardware, and end-to-end workflows as future research.Source: §6 · p. 10

Related papers

Loading 2609.31980v1…