Source-linked AI summary
Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints
Xinge Liu, Terry Jingchen Zhang, Bernhard Schölkopf, Zhijing Jin, Kristen Menou
TL;DR
AI research agents need dynamic benchmarks for iterative, scientifically grounded workflows with feedback. STARGAZER provides such an environment through scalable RV model-fitting tasks and evaluates frontier agents on statistical and physical criteria. Across the benchmark, agents often fit the data without recovering the correct physical systems, with performance falling sharply on harder and real-data tasks.
Problem
Existing benchmarks provide limited evidence about whether agents can perform dynamic, multi-step scientific reasoning over raw empirical data with physical constraints.
Method
STARGAZER evaluates agents in an iterative RV workflow where submissions are forward-modeled and graded on statistical fit and physical recovery.
Results
Performance exceeds 70% on Easy tasks but falls to 17–35% on Medium and at most 6% on Hard; all eight models achieve 0% passes on the 20 real-data tasks.
Takeaways & Limitations
STARGAZER exposes a persistent gap between numerical optimization and physical reasoning in agentic scientific model fitting.
Takeaways & Limitations
Skills improve efficiency on relatively simple tasks but do not fundamentally improve physical reasoning on Hard tasks, and simulation-to-real transfer remains a primary consideration.
Abstract
from arXiv · showhide
The rise of autonomous AI agents suggests that dynamic benchmark environments with built-in feedback on scientifically grounded tasks are needed to evaluate the capabilities of these agents in research work. We introduce Stargazer, a scalable environment for evaluating AI agents on dynamic, iterative physics-grounded model-fitting tasks using inference on radial-velocity (RV) time series data. Stargazer comprises 120 tasks across three difficulty tiers, including 20 real archival cases, covering diverse scenarios ranging from high-SNR single-planet systems to complex multi-planetary configurations requiring involved low-SNR analysis. Our evaluation of eight frontier agents reveals a gap between numerical optimization and adherence to physical constraints: although agents often achieve a good statistical fit, they frequently fail to recover correct physical system parameters, a limitation that persists even when agents are equipped with vanilla skills. Furthermore, increasing test-time compute yields only marginal gains, with excessive token usage often reflecting recursive failure loops rather than meaningful exploration. Stargazer presents an opportunity to train, evaluate, scaffold, and scale strategies on a model-fitting problem of practical research relevance today. Our methodology to design a simulation-driven environment for AI agents presumably generalizes to many other model-fitting problems across scientific domains. Source code and the project website are available at https://github.com/AIPS-UofT/Stargazer and https://aips-uoft.github.io/Stargazer/, respectively.
1 Introduction
STARGAZER addresses the need for dynamic, feedback-driven benchmarks that test agents on scientifically grounded, multi-step workflows rather than static question answering. It uses radial-velocity exoplanet analysis to evaluate whether agents can obtain statistically good fits while recovering physically correct planetary systems.
- Existing agentic benchmarks often simplify scientific work, leaving open whether models can reason over raw empirical data as researchers do.
- Radial-velocity analysis requires periodograms, iterative Keplerian fitting, model selection, and submission in a structured workflow.
- RV success is objectively verifiable because proposed planetary configurations can be compared with ground truth.
- 120 tasks span three difficulty levels, including 20 real archival cases, while synthetic difficulty is controlled by six physical factors.
- Evaluation reports that statistical fit quality does not imply physical recovery, while token usage does not predict performance and failed agents often repeat.
2 Related Work
Prior physics and astronomy benchmarks show that model performance declines on sophisticated reasoning tasks, but they largely emphasize coursework, factual recall, retrieval, or isolated workflow stages. This leaves empirical, multi-step scientific reasoning over raw observations insufficiently tested.
- Physics benchmarks report sharp performance degradation as problems require more sophisticated physical theorems and derivations.
- Astronomy benchmarks extend evaluation to expert knowledge retrieval and multimodal image interpretation.
- Existing benchmarks largely draw from coursework, competitions, or factual recall rather than raw empirical data.
- Physical-science agent benchmarks cover narrow workflow slices, including a single astronomy stage or simulation-based discovery without scalable real observations.
3 STARGAZER
STARGAZER constructs reproducible synthetic and archival RV tasks, gives agents iterative analysis and submission tools, and evaluates both statistical fit and physical recovery. A task passes only when all four criteria are simultaneously satisfied, preventing low-residual but physically incorrect solutions.
- Physics-Grounded Environment: The environment uses irregular observation times, Gaussian uncertainty, correlated noise, and optional N-body integrations to reflect observational and dynamical complexity.
- Physics-Grounded Environment: RV time series combine Keplerian planetary contributions, a systemic velocity offset, and measurement or stellar noise.
- Task Construction: Synthetic tasks are reproducible from a single seed controlling orbital parameters, scheduling, noise, and N-body signal generation, enabling on-demand held-out suites.
- Task Construction: Difficulty sums six physics-based components: multiplicity, SNR, resonances, period coverage, observation count, and correlated-noise amplitude.
- Agent Interaction: Agents receive RV data and operate in a ReAct-style loop using Python analysis and submissions, with per-criterion diagnostic feedback and multiple attempts.
- Evaluation Protocol: The evaluator forward-models submissions and applies RMS, ΔBIC, Match, and Count criteria, requiring all four to pass simultaneously.
4 Results and Discussion
STARGAZER reveals that frontier agents can fit RV data statistically while failing to recover the underlying physical systems, especially as task difficulty increases. More attempts or procedural skills provide limited relief, while successful behavior depends on using diagnostics to revise hypotheses and escalate model complexity.
- Baseline comparison: 95.0% Easy-tier performance from both deterministic baselines exceeds the best LLM agents, while Classical Pipeline performance falls to 5.0% on Hard tasks.LLM agents have not yet outperformed traditional methods on simple single-planet tasks.
- Difficulty scaling: 80.0% is GPT-5.3-codex’s best Easy-tier pass rate, but no model exceeds 6% on Hard tasks as difficulty rises.Top performers exceed 70% on Easy and fall to 17–35% on Medium; lower success correlates with SNR and planet multiplicity.
- Real-data generalization: 0% of all eight models pass the 20 real-data tasks, despite universal ∆BIC passes among submitted episodes and 0% Match Score passes.The closest cases recover correct orbital periods but overestimate semi-amplitudes; the reference solutions were verified against published values within reported uncertainties.
- Statistical fitting vs. physics reasoning: Above 70% statistical pass rates contrast with below 40% physical pass rates on Hard tasks, separating curve fitting from recovery of physically correct configurations.The statistical criteria are ∆BIC and RMS; the physical criteria are Match Score and Planet Count.
- Self-generated skills: +28.3 pp is the largest reported Easy-tier skills gain, but Hard-tier Match Score remains below 33% for all models.Episode analysis attributes gains largely to efficiency: skills move episodes from Budget Exceeded to Env Done without improving underlying Hard-tier physical reasoning.
- Failure modes and resource use: 68 K tokens sufficed for a successful two-planet recovery, whereas an alias-period failure consumed 730 K tokens through repeated submissions.The successful agent used residual diagnostics to identify a 31 d signal and revise the model; the failed agent did not escalate beyond its wrong two-planet hypothesis.
5 Conclusion
STARGAZER evaluates agents on iterative radial-velocity exoplanet discovery and reveals a persistent gap between numerical fitting and physical reasoning. Its scalable, physically grounded setting provides a target for developing agents that interpret fitted models scientifically.
- STARGAZER evaluates AI agents on the iterative, multi-step workflow of exoplanet discovery via radial-velocity analysis.
- Eight frontier models fit numerical data effectively but struggle with the physical reasoning needed to distinguish curve fitting from scientific discovery.
- The statistical–physical dissociation persists across models, difficulty tiers, domain-expert skills, and additional compute budget.
- STARGAZER provides a physically grounded, infinitely scalable setting for developing agents that fit data and interpret what the fit means.
LLM Usage Statement
The authors used large language models to assist with prose drafting, revision, and minor LaTeX editing. They state that all technical content and research decisions were produced and verified by the authors.
- Large language models assisted with drafting and revising prose and with minor LaTeX editing.
- The authors produced and verified all technical content, experimental design, code, analyses, and results.
A Task Construction Details
Synthetic tasks receive integer difficulty scores from 1 to 10 based on six physically motivated factors. The total is formed by adding component terms and clipping the result to that range.
- Each synthetic task receives an integer difficulty level d ∈[1, 10] based on six physically motivated factors.
- The difficulty score is computed as d = clip(d_base + d_SNR + d_res + d_cov + d_obs + d_GP, 1, 10).
- Each component of the difficulty score is defined in Table 3.
A.2 Synthetic Task Generation Details
Synthetic RV tasks are generated from random seeds by sampling physically motivated planetary, noise, and observation properties. Tasks are organized into calibrated difficulty tiers using factors that challenge standard RV analysis.
- Each synthetic task is fully determined by a single random seed, which controls its generated parameters and observations.
- Planetary systems: Planetary systems contain 1–4 planets, with higher counts at higher difficulty levels and near-resonant period pairs inserted with 25% probability.
- Planetary systems: Orbital parameters are sampled using log-uniform periods from [2, 300] days, minimum masses from [0.01, 1.0] MJup, and a Kipping Beta eccentricity distribution.
- Noise and orbital parameters: Angular parameters are uniform on [0, 2π), while white noise, optional jitter, and correlated stellar Gaussian-process noise are sampled over specified physical ranges.
- Observation schedule: Observation schedules use irregular timestamps over 2–4× the shortest planetary period, with 30–100 observations.
- Difficulty tiers: Tasks are grouped into Easy (1–2, 20 tasks), Medium (3–6, 40 tasks), and Hard (7–10, 40 tasks), with scoring calibrated around factors that destabilize standard RV workflows.
A.3 Real-World RV Dataset
STARGAZER includes 20 anonymised real-world RV tasks from published exoplanet datasets, spanning simple high-amplitude systems to interacting multi-planet configurations. The benchmark evaluates these tasks against literature-derived ground truth, including cases where standard Keplerian fitting is intentionally inadequate.
- Dataset scope: 20 real-world tasks use published radial-velocity datasets spanning single hot Jupiters to multi-planet systems with sub-m s−1 signals in correlated noise.The real-data set complements 100 synthetic tasks and covers a broad range of RV-analysis complexity.
- Ground truth: Ground-truth orbital parameters come from discovery or characterisation papers, with real-data tasks independently checked by refitting each system with RadVel.For contested detections, the benchmark adopts the most widely accepted published solution.
- Anonymisation: Task files remove target names, instrument names, literature references, and known planetary parameters, leaving anonymised time-series data and host-star mass.Instrument labels are replaced with generic identifiers such as inst A and inst B.
- Special cases: GJ 876 tests a known boundary of the standard workflow because its four-planet Laplace resonance invalidates Keplerian superposition and requires an N-body ground truth.Agents observed residual structure but continued escalating within the Keplerian model family rather than switching to a dynamical model.
- Real-data outcome: No submission simultaneously matched all literature periods and semi-amplitudes within reported uncertainties across the real-data evaluation.The closest five submissions matched literature-consistent periods but had at least one semi-amplitude outside its published uncertainty interval.
B.1 Pass Rate Breakdown by Criterion
Criterion-level results show that agents usually achieve statistical fits but struggle with physical recovery, especially on Hard tasks. The main bottlenecks are selecting the correct planet count and accurately recovering orbital parameters.
- Statistical versus physical criteria: On Easy tasks, ∆BIC exceeds nearly 90% for almost all models while RMS remains uniformly high, indicating that basic signal detection and fitting are not bottlenecks.The standard periodogram-to-Keplerian pipeline is within frontier-model capabilities on these tasks.
- Statistical versus physical criteria: On Hard tasks, Planet Count exceeds 25% for several models while Match Score falls below 5% for most, separating planet-number inference from accurate orbital recovery.GPT-5.2 reaches 59.5% Planet Count and Gemini 58.2%, while Match Score collapses to single digits for most models.
- Model-specific patterns: Claude-Sonnet-4.6 reaches 96.6% on both Hard-tier ∆BIC and RMS but only 10.3% on Planet Count, showing strong fixed-model fitting alongside weak model selection.Its statistical success does not translate into selecting the correct number of planets.
- Model-specific patterns: GPT-5.3-codex achieves 100% on both Hard-tier statistical criteria but only a 4.2% overall Hard pass rate because Match and Count must both pass.The conjunction gate requires simultaneous statistical and physical success.
- Interpretation: The core physical bottleneck is choosing when to increase model complexity, recognising aliases, and distinguishing missing planets from correlated-noise residuals.These tasks require physical reasoning beyond numerical optimisation.
- Metric caveat: The current Match Score averages only successfully paired planets, so unmatched truth planets do not reduce that score directly.A stricter alternative would normalise by the number of true planets, but the current formulation remains primary because the edge case is reportedly uncommon.
B.4 Per-Criterion Effect of Skills Injection
Skills injection improves procedural performance on Easy tasks and can improve physical recovery for stronger models on Medium tasks, but effects weaken or reverse on Hard tasks. Much of the overall gain comes from helping agents reach submission before exhausting their budgets.
- Easy tier: All four models show improved or stable Easy-tier ∆BIC, RMS, and Count after skills injection, with Gemini-3.1-Pro reaching 100% on all four criteria.The result indicates that skills encode the standard periodogram-to-fit workflow effectively on Easy tasks.
- Medium and Hard tiers: Gemini-3.1-Pro’s Medium Match Score rises from 39.3% to 81.9% (+42.6 pp), while GPT-5.3-codex rises from 40.0% to 63.8% (+23.8 pp).Both models also gain on Medium Planet Count: +17.9 and +21.4 pp, respectively.
- Medium and Hard tiers: GPT-5-mini and Qwen-3.5-Plus lose Hard-task RMS performance by −12.7 and −7.3 pp, while their Match Scores remain near zero.The procedural template may interfere with exploratory strategies needed for complex multi-planet systems.
- Efficiency: Skills shift episodes from Budget Exceeded to Env Done without changing behaviour after submission; for Gemini-3.1-Pro on Hard tasks, Budget Exceeded falls from 93.5% to 83.3% (−10.1 pp).The reported pass-rate gains are therefore attributed almost entirely to workflow efficiency.
- Metric sensitivity: Match-score threshold variation changes overall pass rates by less than 5 pp and preserves model rankings because most submissions score above 0.9 or below 0.5.Only 17% of submissions fall in the 0.70–0.90 boundary region.
- Skill content: The injected skills cover alias detection, phase-folding, narrow period refinement, robust Keplerian fitting, eccentricity checks, and multi-instrument offsets.They also prescribe residual-based multi-planet detection and repeated submission to avoid budget exhaustion.
- Skill content: The system prompt imposes a six-step workflow from periodogram analysis and a sine baseline through model gating, Keplerian fitting, fit checks, and submission.Budget constraints and fit-quality thresholds scale with task difficulty.
E Case Study Trajectories
The case studies contrast successful iterative model revision with failure to escape alias solutions in a resonant three-planet system. Good statistical fits can therefore coexist with incorrect physical recovery when agents do not revise their model hypothesis.
- Success Case: Medium GPT-5.2 recovered both planets in 11 steps, achieving match = 0.932 and RMS = 1.13 m s−1.The two-planet fit reduced RMS from 9.16 to 1.13 m s−1 and passed BIC, RMS, match, and count criteria.
- Failure Case: The properly formatted alias submission passed the RMS check but had match = −0.008 and failed the planet-count criterion.Its low residual therefore did not indicate recovery of the physical system.
- Failure Case: The failure agent resubmitted the same alias solution four times, never attempted a three-planet model, and explored no new periods during its final 20 steps.Two earlier submissions were also wasted on formatting errors involving l rad and inc rad.
- Failure Case: The failure consumed 730 K tokens versus 68 K for success—10.7× more—yet produced a negative match score.The trajectory illustrates that additional computation did not compensate for failure to revise the model hypothesis.