PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Numerical Intelligence via Auto-Diagnosis and Skill Discovery

Peter Chen, Wotao Yin

arXiv:2610.03872v1cs.AIcs.CLcs.LG

TL;DR

Numerical solvers can perform poorly even when they run, while execution feedback alone may not identify the underlying numerical cause or suitable remedy. ADSD links diagnosis of solver behavior to discovery and packaging of reusable numerical methods. Across four domains, it improves solver performance, reducing mean solver error by nearly 71× on GOC-500.

  • Problem

    Execution feedback and literature search are often poorly coordinated, so agents may observe poor solver performance without identifying its numerical cause or a method suited to the condition.

    Source: §1 · p. 1

  • Method

    ADSD uses a diagnosis-first process to identify numerical conditions, guide literature-based method discovery, and package verified methods as reusable solver skills.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 5

  • Results

    Across four numerical domains, ADSD improves solver accuracy, robustness, and efficiency; it reduces mean solver error by nearly 71× on GOC-500.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results identify diagnosis as a critical component of numerical intelligence, connecting observed solver behavior to actionable numerical knowledge.

    Source: §5 · p. 9

    ODE trajectory accuracy is evaluated using a 10^-4 error threshold, following the original benchmark criterion.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

AI agents are becoming increasingly capable of generating scientific code, but generating code is not the same as improving the algorithms behind it. For numerical solvers, execution feedback can expose poor performance, but rarely reveals its underlying cause and how to address it. We introduce Auto-Diagnosis and Skill Discovery (ADSD), a framework that links numerical diagnosis to reusable solver self-improvement. ADSD follows a diagnosis-first paradigm that first explains why a solver performs poorly, then uses this diagnosis to guide the discovery of appropriate numerical methods. The resulting knowledge is packaged into reusable solver skills, turning solver improvement from trial-and-error editing into a structured process of diagnosis, discovery, and implementation. Across four challenging numerical domains--power flow equation, AC optimal power flow control, stiff ordinary differential equations, and heterogeneous diffusion PDEs--ADSD consistently improves solver accuracy, robustness, and efficiency. On GOC-500 power flow, for example, ADSD reduces mean solver error by nearly $71\times$, with improvements further transferring to unseen grid topologies and operating regimes.

1 INTRODUCTION

Numerical agents need to diagnose why a solver performs poorly before selecting a method suited to the underlying numerical condition. ADSD structures this process into diagnosis and reusable skill discovery, improving solver performance across diverse numerical tasks.

1 INTRODUCTION

  • Existing workflows rarely coordinate execution feedback, literature search, and reusable tools, leaving agents to detect poor performance without identifying its cause or the best-matched methods.Numerical symptoms can have different causes, and candidate methods carry assumptions and trade-offs.Source: §1 · p. 1
  • ADSD separates observable solver behavior, diagnostic knowledge linking symptoms to numerical conditions, and executable solver knowledge that packages algorithms for integration.The framework operationalizes these stages in a diagnosis-first loop without updating model weights.Source: §1 · pp. 1–2
  • Across four numerical domains, ADSD consistently improves solver accuracy and efficiency; on GOC-500, it reduces mean solver error by nearly 71×.The evaluation covers power flow, AC optimal power flow control, stiff ODEs, and heterogeneous diffusion PDEs.Source: §1 · p. 2
  • The paper’s contributions are a diagnosis-first paradigm for verified diagnoses and targeted skills, and a self-improvement framework evaluated across model families and unseen regimes.The evaluation includes power flow, AC-OPF control, stiff ODE integration, and heterogeneous diffusion PDEs.Source: §1 · p. 2

2 PRELIMINARIES

The paper studies four numerical solver settings, from power-system equations and optimization to stiff ODEs and heterogeneous diffusion PDEs. Figure 1 sketches how ADSD combines diagnosis and reusable skills to generate an improved solver.

Power flow and AC optimal power flow

  • Power flow solving seeks a voltage state with a small power-mismatch residual, while AC-OPF minimizes generation cost subject to power balance and operational limits.AC-OPF performance depends on both constraint satisfaction and objective quality.Source: §2 · pp. 2–3

ADSD overview

  • Figure 1 depicts ADSD combining baseline-solver feedback and literature-derived diagnostic skills, then using verified hypotheses to build solver skills for a fresh agent.The fresh agent receives the reusable skills together with the baseline solver N0 to generate B.Source: Figure 1 · p. 3

ODEs

  • The ODE setting approximates initial-value trajectories in systems-biology models that may be stiff, making stability-related step-size limits relevant to explicit methods.Accuracy and computational work depend on the numerical scheme, step-size selection, and error control.Source: §2 · p. 3

PDEs

  • The PDE setting uses two-dimensional steady-state diffusion with heterogeneous, high-contrast coefficients, which can make discrete systems ill-conditioned and slow iterative convergence.The diffusion coefficient k(x) is positive and f(x) is the forcing.Source: §2 · p. 3

3 MAIN RESULTS

ADSD is a diagnosis-driven pipeline that converts observed solver behavior into evidence-grounded improvements. It verifies hypotheses about numerical conditions, then discovers, tests, and packages methods as reusable solver skills.

3 MAIN RESULTS

  • ADSD separates auto-diagnosis, which identifies numerical conditions explaining solver behavior, from skill discovery, which finds and builds reusable methods from literature.Public cases support development, while private cases remain hidden for final evaluation.Source: §3 · p. 3

3.1 AUTO-DIAGNOSIS

  • The auto-diagnosis stage freezes a baseline solver, turns literature-supported symptom-condition relationships into executable probes, and verifies hypotheses against held-out public observations.Diagnosis proceeds in rounds, and hypotheses passing more than 90% of the verification pool are retained as common illnesses.Source: §3.1 · p. 4
  • Table 1 covers IEEE-118 and GOC-500 validation performance, reporting mean ± standard deviation for the N0 baseline and its +ADSD counterpart.Source: Table 1 · p. 5

3.2 SKILL DISCOVERY

  • Skill discovery screens for methods relevant to verified hypotheses, audits their theoretical claims and compatibility, empirically tests implementations, then packages verified methods for integration with N0.The agent receives the original solver and frozen skill packets to generate improved solver B.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate power-flow solving, AC optimal power-flow control, stiff-ODE integration, and 2D diffusion PDE solving on challenging numerical tasks. Across these settings, ADSD improves solver performance, with evaluations also testing generalization and direct or recursive self-improvement.

4.1 POWER FLOW (PF) EQUATION SOLVER

  • Approximately 71× lower solver error on GOC-500 for the Qwen3.8 baseline, with comparable speed; learned skills also improve performance on unseen grid topologies and close-to-infeasible cases.The power-flow evaluations include in-domain validation, transfer from N2 to unseen N1/N topologies, and transfer from feasible to close-to-infeasible cases.Source: §4.1 · p. 6, Figure 2 · p. 6

4.2 AC OPTIMAL POWER FLOW (AC-OPF) SOLVER

  • On 7,000 IEEE-57 edge cases near maximum loadability, ADSD improves robustness and efficiency while maintaining low objective error.The cases comprise 2,000 training and 5,000 hidden evaluation examples with high demand and transmission congestion.Source: §4.2 · p. 7

4.3 ODE INTEGRATOR

  • Across all agent baselines, ADSD produces stiff-ODE integrators with lower trajectory error and faster runtime on hidden evaluation models.The evaluation uses 72 held-out source families with 117 models, and Table 3 reports pass rate, median normalized per-case error, and wall-clock runtime.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 ELLIPTIC PDE SOLVER

  • On 2,400 held-out ConDiff cases, ADSD effectively improves solver performance among the evaluated baselines.The benchmark uses heterogeneous diffusion problems on 128×128 grids; each solver-case pair is evaluated in three fresh-process repetitions.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 ADDITIONAL ANALYSES

  • Direct self-improvement yields inconsistent gains—error decreases in four comparisons and increases in five—while ADSD achieves significantly lower error in all settings; extending revisions does not sustain reductions or close the ADSD gap.Figure 4 reports mean GOC-500 error across recursive iterations for GPT-5.5, Claude-Sonnet-5, and Qwen3.8-2.4T; GPT initially worsens then partially recovers before plateauing above its N0 baseline, while Claude-Sonnet-5 remains nearly unchanged.Source: §4.5 · p. 9, Figure 4 · p. 9

5 CONCLUSION

ADSD turns observed solver behavior into verified numerical conditions and reusable executable skills. Across four numerical domains it improves accuracy, robustness, and efficiency, with skills transferring to unseen regimes.

5 CONCLUSION

  • Across four numerical domains, ADSD improves solver accuracy, robustness, and efficiency across different base models, with learned skills transferring to unseen problem regimes.The conclusion identifies diagnosis as the link between observed solver behavior and actionable numerical knowledge.Source: §5 · p. 9
  • ADSD grounds solver improvement in literature-derived diagnostic probes, hypothesis verification, and final skill construction.Source: §5 · p. 9
  • Within the evaluated budgets, direct or recursive self-improvement using a model’s inherent knowledge does not recover ADSD’s gains.Source: §5 · p. 9

A FURTHER RELATED WORKS

ADSD differs from work on power-system knowledge, model discovery, numerical methods, retrieval, and reusable skills by focusing on numerical conditions inferred from solver behavior. Its diagnostic front end links failures to retrieved methods and transferable implementations.

Power-system knowledge and applications

Model discovery and formulation

  • Model-discovery and formulation work determines governing equations or optimization models, whereas ADSD addresses numerical solver behavior.Source: A FURTHER RELATED WORKS · p. 14

Numerical methods and solver evaluation

  • Numerical-method and solver-evaluation studies emphasize that successful execution alone does not establish numerical quality.Source: A FURTHER RELATED WORKS · p. 14

Scientific retrieval

  • ADSD connects literature with computation by checking a method’s assumptions against an existing solver before constructing a transferable implementation.Source: A FURTHER RELATED WORKS · p. 14

Agentic skills aligned from experience

  • Unlike general skill-discovery frameworks, ADSD first converts frozen-solver behavior into verified numerical hypotheses that specify methods to retrieve and implementation assumptions to satisfy.Source: A FURTHER RELATED WORKS · p. 14
  • Prior agentic-skill work retains knowledge through memory, executable tools, and environment interactions; the authors propose extending ADSD toward self-supervised learning and language-model alignment.Source: A FURTHER RELATED WORKS · pp. 14–15

B.1 GPT BASELINE OVER PF∆

The GPT power-flow case study starts from a dense polar Newton baseline and uses execution traces to diagnose iteration and factorization costs. Verified hypotheses guide literature search and the construction of two solver skills.

B.1 GPT BASELINE OVER PF∆

  • The baseline is a dense polar Newton power-flow solver with an analytic Jacobian, safeguarded step, and bounded regularized fallback for singular systems.Source: §B.1 · p. 15
  • Each accepted baseline iteration uses one Jacobian assembly, dense factorization, and triangular solve, while traces record iteration, solve, fallback, failure, residual, step, and scale data.Source: §B.1 · p. 15
  • The agent builds ten deterministic diagnosis skills; ac-jacobian-criticality converts weak-Jacobian behavior into a feature evaluated on the state returned by the baseline.Source: §B.1 · p. 15
  • A 0.16 Jacobian-singular-value cutoff identifies 50 cases averaging 5.78 iterations, versus 5.01 for the remaining cases; the relationship is checked at nearby cutoffs and on a 4,000-case verification pool.Source: §B.1 · pp. 15–16
  • The resulting search screens 312 papers to 82 candidates and six fully read selections, yielding 40 verified claims and nine method reviews before two skills proceed to implementation.The skills are multistep-factorization-reuse and seventh-order-method9.Source: §B.1 · p. 16
  • Multistep reuse fixes the Jacobian at an outer iterate and reuses its factors across internal corrections, while Method 9 uses two reusable factorizations for a higher-order update.Table 5 describes a skill ablation on the IEEE-118 validation set, but the supplied passage does not provide its outcome values.Source: §B.1 · pp. 16–17, Table 5 · p. 16

B.2 QWEN BASELINE OVER AC-OPF

In the Qwen AC-OPF case study, diagnosis motivates scaling and a reweighted progress measure for a stalling baseline. The combined method significantly improves pass rate, and the case-study account reports nearly 84% improvement over N0.

B.2 QWEN BASELINE OVER AC-OPF

  • Nearly 84% pass-rate improvement over N0 is reported for the Qwen AC-OPF case study, where the learned scaling strategy changes solver behavior qualitatively.Source: §B.2 · p. 17
  • The baseline is a dense primal-dual interior-point method with explicit slacks, analytic derivatives, fraction-to-boundary line search, regularized KKT fallback, best-iterate retention, and stall exit.Source: §B.2 · p. 17
  • Its scalar progress measure aggregates four differently unit-scaled quantities and normalizes stationarity by 1 + ∥∇f∥∞; termination uses relative improvement of the running best with patience 12.Source: §B.2 · p. 17
  • On failing rows, the stationarity residual is nearly invariant because the IEEE-57 generator cost vector is linear and unchanged across contingencies and load draws.Source: §B.2 · p. 17
  • The paper provides topology- and feasibility-generalization evaluations, with results summarized over 10,000 and 5,000 cases respectively.The supplied captions specify the evaluation scope but do not report the comparison values.Source: Table 6 · p. 18, Table 7 · p. 19
  • On held-out verification, the diagnosed condition selects 833 rows, of which 832 fail, versus 4 failures among 167 controls.The reported difference is +97.48 percentage points with 95% interval [95.15, 99.82].Source: §B.2 · p. 18
  • The generated gmax-rowscale-kkt-measure skill combines one-time automatic row scaling with a normalized first-order error measure; the combined method significantly improves pass rate.The scaling leaves variable bounds untouched and is an equivalence transformation; the reweighted measure replaces meas for best-state selection and stall detection, while acceptance gates and reported residuals remain in exact checker units.Source: §B.2 · p. 18

C FURTHER DISCUSSIONS

The discussion explains why residual accuracy matters to engineering analyses and argues that diagnosis, rather than unguided revision or broad retrieval, makes numerical knowledge actionable.

On the practical essence of solver residue

  • Near regular operating points, monitored-quantity error follows δg ≈ ∇g^T J^-1 r, so tighter residuals improve sensitivity-weighted security-margin assessments.The passage ties the residual to the power-flow Jacobian and monitored-quantity sensitivity.Source: C FURTHER DISCUSSIONS · p. 19
  • Lower residuals also reduce numerical contamination in finite-difference sensitivities and support consistent dynamic-simulation initial conditions, especially when downstream decisions are sensitive.The passages describe error amplification by inverse perturbation size and potential artificial transients from initialization mismatches.Source: C FURTHER DISCUSSIONS · p. 19

On the role of diagnosis

  • Diagnosis turns stalling in an existing primal-dual interior-point solver into targeted retrieval of row scaling and normalized KKT progress measures.The baseline already uses a primal-dual interior-point method but lacks these scaling and progress-measure mechanisms.Source: C FURTHER DISCUSSIONS · p. 19
  • Diagnosis narrows 312 papers to 82 candidates and six full-text papers; five rounds of solver revisions alone still do not close ADSD's GOC-500 gap.Source: C FURTHER DISCUSSIONS · p. 20
Loading 2610.03872v1…