PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomous Research Project Management as an Agent Skill: A Case Study in Exact Spectral Spatial Regression

Alexander Chen, Jeffrey Meng, Bram Hoex, Tong Xie

arXiv:2609.31683v1cs.LGcs.AIcs.MA

TL;DR

Dense KRR scales prohibitively, while the closed-form application of circulant and FFT methods to regular-grid regression remains under-characterised. The paper evaluates an exact FFT-based KRR formulation for regular grids and reports lower pooled-cell RMSE than coordinate-only baselines.

  • Problem

    Dense KRR scales prohibitively as O(N^3) time and O(N^2) memory, while its closed-form application to regular-grid regression remains under-characterised.

    Source: §1.1 · p. 2

  • Method

    The paper uses BCCB embeddings and 2D-FFT spectral shrinkage for exact KRR on regular grids, with PCG for masked free-boundary systems.

    Source: §1.3 · p. 2, §2.2.6 · p. 8

  • Results

    RMSE 0.0999 on 2,000 pooled test cells for spectral reconstruction, versus 0.820–0.824 for coordinate-only exact/Nyström/RFF KRR.

    Source: §1.5.6 · p. 4

  • Takeaways & Limitations

    The study presents a transparent, resource-bounded algorithmic benchmark for structured spatial regression, including disclosed empirical limitations and refuted transfer hypotheses.

    Source: §1.7 · p. 4

    Spectral efficiency applies to embedded-torus kernels; free-boundary masked training requires iterative PCG at ∼6× higher error.

    Source: §1.6 · p. 4

Abstract

from arXiv · show

This work presents an end-to-end demonstration of autonomous machine learning research conducted by an agent skill on consumer hardware. The demonstration evaluates an FFT-based Kernel Ridge Regression (KRR) solver for regular spatial grids using 2005 monthly NOAA Kaplan SST v2 anomaly fields on a $36 \times 72$ grid. This was autonomously executed by DeepSeek V4 Flash, orchestrated by our agent skill suite within DeepSeek Harness (DSH). Experiments were executed on CPU-only hardware (Apple M2 Pro; 78.7 s solver time, 1.57 GB peak RSS). Long-horizon state was decoupled into a file-based epic- and issue-tracking substrate. Across 74 sub-agent sessions, the agent demonstrated closed-loop scientific resilience: routing two failed hypothesis review gates back to literature retrieval, patching bootstrap indexing bugs, and executing with only four discrete human steering events. Finally, we reflect on autonomous research governance, arguing that scientific credibility requires inspectable state, falsifiable review gates, and transparent reporting of negative results, urging the machine learning community to favour agent-accessible structured formats over static PDF manuscripts.

1 Autonomous research result

Dense KRR is computationally expensive, and the paper presents an exact FFT-based formulation for regular grids. It situates that approach among approximate kernel methods and describes its torus and free-boundary implementations.

1.1 Introduction

  • The paper frames its contribution as addressing under-characterised closed-form regular-grid regression while evaluating against Nyström and RFF approximations.Dense KRR scales as O(N^3) time and O(N^2) memory; the paper also proposes a leakage-verified evaluation protocol.Source: §1.1 · p. 2

1.2 Related work

  • On regular grids, the method extends established BCCB and FFT techniques to exact regression; unlike sketch-circulant KRR on non-grid data, it matches dense references to ∼10^-12.The passage identifies RFF, Nyström, and other approximate alternatives as trading exactness for scale.Source: §1.2 · p. 2
  • For serially dependent residuals, the evaluation calibrates split-conformal intervals on calendar-month blocks and treats transfer as a benchmark, not an operational forecast claim.It also uses blocked cross-validation and block bootstrap resampling for autocorrelated errors.Source: §1.2 · p. 2

1.3 Method: exact torus spectral KRR

  • The formulation uses BCCB embeddings and 2D-FFT spectral shrinkage to solve exact grid KRR in O(N log N) time and O(N) memory.The spectral solve is exact for the embedded-torus kernel; masked free-boundary training instead uses PCG.Source: §1.3 · p. 2

1.4 Data and evaluation protocol

The study evaluates spectral KRR on NOAA SST anomaly fields using future-blind temporal splits, valid-cell-only metrics, and several reconstruction, functional-regression, transfer, and scaling tests. Results establish torus exactness and strong masked reconstruction, while transfer forecasting fails its pre-registered win rule.

1.4 Data and evaluation protocol

  • 78.7 s wall-clock and 1.57 GB peak RSS on an Apple M2 Pro; the study evaluates 2,005 monthly 36 × 72 fields with valid-cell-only metrics and 2016–2022 held out.Training covers 1856–2015; missing cells are imputed for training but excluded from evaluation targets.Source: §1.4 · p. 2, §1.5.7 · p. 4

1.5.1 P0: torus exactness on a real field

  • 2.62 × 10^-12 and 1.12 × 10^-12 relative ℓ∞-norm errors match dense floored-torus references for Matérn-3/2 and RBF, respectively, passing the 10^-6 gate.The reported free-boundary gaps are 0.95 and 0.12, respectively.Source: §1.5.1 · p. 3

1.5.3 Genuinely masked training (T1b)

  • 0.529–0.541 masked RMSE for free-boundary PCG is less accurate than full-field spectral T1a, despite convergence to 10^-8 across 10/30/50% observed runs.Mean iteration counts are 245–281.Source: §1.5.3 · p. 3

1.5.4 Nino3.4 functional index (T2)

  • 0.0856 RMSE (corr 0.997) for Nino3.4 box-functional regression outperforms raw-feature ridge at 0.2696 and the zero baseline at 0.7955.Evaluation uses 5-fold temporal-block cross-validation over all 2,005 months.Source: §1.5.4 · p. 3, Figure 2 · p. 4

1.5.2 Reconstruction under leakage-aware masks (T1a)

  • 0.0816–0.0880 RMSE for Matérn-3/2 beats RBF’s 0.2356–0.3155 across 84 test months; global coverage is 0.877–0.898 against a 0.90 target.The reported local coverage violations include 0.747 for halo w=2 decay and 0.739 for halo w=4 decay.Source: Table 1 · p. 3, §1.5.2 · p. 3, Figure 1 · p. 3

1.5.5 Transfer forecast (T3): REFUTED under pre-registered win rule

  • 0/4 horizons support transfer under the pre-registered win rule; transfer loses to persistence and AR(1) at h=1, and its h=6 improvement over persistence is insignificant.At h=12, skill is −0.427.Source: Table 2 · p. 3, §1.5.5 · p. 4

1.5.6 Solve-cost control on a fixed pooled-cell task (B)

  • 0.0999 RMSE on 2,000 pooled test cells for spectral reconstruction contrasts with coordinate-only exact/Nyström/RFF KRR at 0.820–0.824.Coordinate baselines are near field-standard deviation, including zero at 0.801 and climatology at 0.813.Source: §1.5.6 · p. 4

1.5.7 Resolution sensitivity (R) and scaling (S)

  • 0.0853 to 0.1667 reconstruction RMSE and 0.0856 to 0.3825 functional RMSE as grids coarsen from 2 × 2 to 10°; peak CPU-only use is 78.7 s and 1.57 GB RSS.Fits remain 0.2–0.5 ms through N=10,368.Source: §1.5.7 · p. 4

1.6 Limitations and discussion

The discussion bounds the spectral efficiency claim to embedded-torus kernels; free-boundary masked training requires iterative PCG at approximately six times higher error.

1.6 Limitations and discussion

  • Spectral efficiency applies to embedded-torus kernels; free-boundary masked training instead requires iterative PCG at approximately 6× higher error.Other disclosed boundaries include monthly-mean imputation for 53.4% missing entries and degraded skill under nonuniform coverage.Source: §1.6 · p. 4

1.7 Conclusion

The conclusion presents exact circulant-block KRR as a CPU-only route to regular-grid regression, while emphasizing transparent reporting of limitations and refuted hypotheses.

1.7 Conclusion

  • Exact circulant-block KRR solves regular-grid regression in O(N log N) time and O(N) memory, using 78.7 s and 1.57 GB with masked errors near 14% of field standard deviation.The conclusion frames the study as a resource-bounded benchmark that discloses empirical limitations, conformal breakdowns, and refuted transfer hypotheses.Source: §1.7 · p. 4

2 System Design and Meta-Analysis

The section is described as a system-design and meta-analysis account of the autonomous research pipeline and its verification protocols.

2 System Design and Meta-Analysis

  • The section covers agent architecture, file-based issue tracking, execution loops, human interventions, and verification protocols.It presents the system design and meta-analysis for the pipeline producing Section 1’s results.Source: §2 · p. 5

2.1 Autonomous Research as an Agent Skill

The system organizes autonomous research through modular skills and a file-based substrate that keeps project state inspectable outside conversational memory. A specialized project manager coordinates staged work, while the same substrate can track its own development and can be adapted to other harnesses.

2.1 Autonomous Research as an Agent Skill

  • The system packages end-to-end scientific workflows as modular, composable agent skills.Long-horizon execution state is maintained in plain Markdown files rather than LLM conversational memory.Source: §2.1 · p. 5

2.1.1 Core Skills: Epic and Issue Tracking

  • Plain Markdown artifacts decouple research state from conversational memory, making the project trajectory inspectable, diffable, persistent across resets, and auditable.Core tracking skills manage epics, issues, comments, and orchestration; worker agents return structured outputs while the issue manager alone mutates tracking records.Source: §2.1.1 · p. 5, §2.1 · p. 5

2.1.2 Epic Manager Skill: Research Project Manager

  • A specialized epic manager coordinates five research phases, creating issues just in time and routing failed review gates back to earlier stages.It validates tracking and cost records before execution, then stages findings and costs and synthesizes the manuscript under reporting-integrity checks.Source: §2.1.2 · pp. 5–6

2.1.3 Cross-Harness Portability

  • The skills can run on harnesses other than DSH when their discovery roots and native sub-agent dispatch, messaging, and process-collection wrappers are adapted.The system requires nested sub-agent dispatch and local file-system access.Source: §2.1.3 · p. 6

2.1.4 Comparison to Other Systems

  • Compared with prior systems’ transient, unstructured, or opaque state, the file-based substrate makes research history human-inspectable, git-diffable, and editable with standard tools.The same representation supports “system-as-subject” development, but autonomous protocol refinement without human oversight remains open.Source: §2.1.4 · p. 6

2.2 Run-Specific Settings

This section reports the harness, tools, and human involvement used in the real-data study, along with its review and verification process. The run produced an exact grid-based KRR result and retained an inspectable operational record.

2.2.1 Agent and Harness

  • DeepSeek V4 Flash ran within DSH as an issue manager, governing task progression, state changes, and sub-agent dispatch through local file artifacts.The subsection describes the experiment’s settings and verification as instantiated under the Section 2.1 architecture.Source: §2.2 · p. 6, §2.2.1 · p. 7

2.2.2 Tools

  • The run used a CPU-only Apple M2 Pro with 12 cores and an approximately 2 GB container limit, with scientific computing executed offline.The environment included Python scientific libraries and read-only climate-data ingestion; web retrieval used Tavily MCP.Source: §2.2.2 · p. 7

2.2.3 Research Loop

  • After two hypothesis-gate failures, the manager routed work back to literature review; reviewers caught and corrected two bugs before analysis passed its gate.The two-iteration research loop lasted 13 h 52 min, and the run used 74 sub-agent sessions, 1,625 LLM invocations, and 203.3M metered tokens.Source: §2.2.3 · p. 7

2.2.4 Human Interventions

  • Four human steering events redirected the research, authorized h5py, clarified the page budget, and prompted presentation revisions.A human-prompted post-generation audit also adapted formatting and cross-checked citations.Source: §2.2.4 · pp. 7–8

2.2.5 Verification

  • An independent verification issue checked method-code alignment, superseded-number leakage, and claim traceability, reproducing 77/77 quantitative claims and compute footprints.Automated verification suites audited algorithmic correctness, while manual proofreading focused on presentation fidelity.Source: §2.2.5 · p. 8

2.2.6 Significance of Result

  • Exact KRR on regular grids uses BCCB embeddings and 2D-FFT spectral shrinkage for O(N log N) time and O(N) memory, avoiding dense-solve costs without approximate basis truncations.The study also reports a 0.95 relative error for the Matérn-3/2 free-boundary approximation gap and demonstrates inference on commodity CPU hardware.Source: §2.2.6 · p. 8

2.2.7 Interpretability

  • Stage transitions, code diffs, reviewer scores, and execution logs were committed to disk, leaving the operational narrative inspectable without transient model activations.The retained record includes the failed hypothesis gates and reviewer bug reports.Source: §2.2.7 · p. 8

2.3 Limitations

The system and its empirical results remain bounded by statistical, computational, and human-steering constraints; autonomous protocol refinement is not yet achieved.

2.3 Limitations

  • The empirical findings are limited by the free-boundary PCG gap, localized conformal under-coverage, and refuted multi-horizon forecasting.An exploratory DST-II Woodbury and randomized-Nyström attempt stalled at 0.135× speedup versus PCG, while full-support boundary effects exceeded the approximately 2 GB container cap.Source: §2.3 · p. 8
  • Execution required four human steering events and post-generation editorial auditing, while closed-loop protocol refinement without human steering remains open.The account also notes vulnerabilities in managing complex linear-algebra trade-offs under tight container budgets.Source: §2.3 · p. 8

3 Broader Impact

The paper situates autonomous research amid verification concerns and argues for publication practices that preserve inspectable, machine-readable evidence. It also notes that physical experimentation remains constrained by laboratory throughput.

3 Broader Impact

  • The work builds on prior autonomous-paper demonstrations by showing that end-to-end scientific workflows can be dynamically orchestrated on modular agent-skill substrates.The cited prior work demonstrated workshop-calibre papers for under $15 per manuscript.Source: §3 · p. 9
  • Autonomous agents can generate hypotheses, optimize code, and synthesize analyses at near-zero marginal cost, while physical validation remains bottlenecked by laboratory processes.The cited examples include chemical synthesis, materials characterization, and in vitro assays.Source: §3 · p. 9
  • Audits of autonomous discovery frameworks report verifiability failures, including citation hallucination, method-code divergence, and unreproducible empirical claims.The paper describes robust automated verification as an open challenge as synthetic paper output accelerates.Source: §3 · p. 9
  • The paper argues that static PDFs can suppress negative or anomalous results and sever execution provenance through lossy extraction.It advocates moving beyond traditional PDF manuscripts as one step toward addressing these concerns.Source: §3 · p. 9
  • The paper proposes machine-readable execution deliverables, dataset-first evaluation rubrics, and granular artifact-level micro-citations.These proposals include independently re-executable pipelines, review of data provenance and empirical information gain, and attribution to primary data collectors.Source: §3 · p. 9

A Reproducibility Statement

The experiments used publicly available NOAA Kaplan SST v2 anomalies and ran entirely on CPU hardware. Reproduction is supported by documented verification steps and available run artifacts.

A Reproducibility Statement

  • 78.7 s wall-clock time and 1.57 GB peak RSS were recorded on an Apple M2 Pro in a local Docker container.The experiments were CPU-bound under a ∼2 GB memory limit.Source: A Reproducibility Statement · p. 14
  • Reproducibility checks include future-blind temporal splits, dense Cholesky equivalence within 2.62 × 10−12, and metric traceability to programmatic keys.The statement points to validate_execution.py and results/iter2/*.json.Source: A Reproducibility Statement · p. 14
  • The codebase and Section 1 run artifacts are available in the Research Epic Manager Agent Skills suite repository.The passage provides the repository URL.Source: A Reproducibility Statement · p. 14

B Disclosure Statement

DeepSeek V4 Flash orchestrated the pipeline in DeepSeek Harness across 74 sub-agent sessions. Tavily MCP mediated external literature search, and Gemini 3.8 Flash was used for post-generation editorial changes, including citation auditing.

B Disclosure Statement

Related papers

Loading 2609.31683v1…