AstroMLab 1: Who Wins Astronomy Jeopardy!?

1. INTRODUCTION

• Astronomical research requires broad logical thinking, cross-domain connections, and keeping pace with technological and statistical developments.The authors describe LLMs as potential aids for recommender systems, research agents, and knowledge graphs.Source: §1 · p. 2
• General benchmarks often lack astronomy focus or depth and remain tangential to evaluating LLMs for astronomical research.The paper identifies a shortage of precise benchmarks for broad scientific Q&A, partly because human annotation in PhD-level domains is costly.Source: §1 · p. 2
• Astronomy-specific benchmarking matters because models can be overtrained on familiar tests and vary substantially on astronomy despite similar performance on established benchmarks.The authors cite cost differences of up to three orders of magnitude among models performing equally on benchmarks such as MMLU.Source: §1 · p. 2
• The study evaluates proprietary and open-weights LLMs on astronomical knowledge recall and inference to support more informed deployment decisions.The benchmark is intended to assess capabilities in astronomical research and inform future development.Source: §1 · p. 2
• The authors test native model capabilities without RAG, reserving embedding, retrieval, fine-tuning, and summarization choices for dedicated study.They describe effective RAG deployment as involving complex choices that merit a separate paper.Source: §1 · p. 2
• This paper begins a study series covering dataset details, comparisons with astronomy-specific models, and battles between base and fine-tuned models.The series aims to establish an astronomy LLM evaluation standard and support AI integration in astronomical research.Source: §1 · p. 2

2. BENCHMARKING MCQ DATASETS

• The benchmark tests recall of astronomical facts and community consensus, as well as connections across subfields and interdisciplinary understanding.The authors also developed Q&A components, with their detailed evaluation deferred to a subsequent paper.Source: §2 · p. 3
• 4,425 multiple-choice questions were generated from 885 Annual Review of Astronomy and Astrophysics articles published from 1963 to 2023.Gemini-1.5-Pro proposed five questions per article, each with four answer choices and an explanation with supporting review paragraphs.Source: §2 · p. 3
• Most questions target key domain knowledge and were written to stand alone rather than depend on specific research articles.Human experts reviewed a subset and judged the quality adequate based on the explanations.Source: §2 · p. 4
• The authors treat reported accuracy as a lower limit because generated questions may be inaccurate, outdated, or too vague.They state that restricting analysis to questions correctly answered by the strongest model would not significantly change their conclusions.Source: §2 · p. 4
• The authors plan further validation of question correctness and difficulty, and report that Gemini-1.5-Pro did not visibly favor Gemini models.The benchmark details and validation efforts are deferred to a companion paper.Source: §2 · pp. 4–5

3. INFERENCE METHODOLOGY

• The study evaluates MCQ performance and reserves open-ended astronomical research questions for future work.The authors note that open-ended Q&A may require more careful human curation through an interactive platform.Source: §3 · p. 5
• The authors use instruct/chat models and request an answer plus explanation rather than relying on base-model next-token probabilities.They argue that earlier or weaker models may not follow direct instructions consistently, and that base-model first-token performance may not translate to practical research-agent use.Source: §3 · p. 5
• Models were evaluated with default documented settings, without extensive model-specific prompt or hyperparameter optimization.The authors aim to provide a broad comparison under generic conditions rather than maximize each model’s capability.Source: §3 · p. 5
• Chain-of-thought prompting generally improved most models’ MCQ accuracy, according to the authors’ observations.The prompt asks models to provide an explanation of their rationale.Source: §3 · p. 5
• A preliminary regex extracted answers in about 99% of cases; GPT-4o resolved the remaining instances where extraction failed.The fallback used the model’s answer and explanation to identify its intended choice.Source: §3 · p. 5
• Refusals were rare, typically below 1% per model, and unanswered questions were discarded individually, introducing some model-to-model variability.For stronger models, at most 0.2% of questions were affected; the authors say this decision did not significantly affect overall results.Source: §3 · pp. 5–6

4. BENCHMARKING PROPRIETARY LARGE LANGUAGE MODELS

• Accuracy is the percentage of questions answered correctly, with typical Wilson Score Interval noise of about ±0.6–0.8% at 1σ across 4,425 questions.The Wilson interval is described as a confidence interval for binomial proportions; the authors state that most discussed model differences are statistically significant.Source: §4 · p. 6
• The section begins by presenting proprietary-model performance.The accuracy measure is used throughout the comparison.Source: §4 · p. 6
• The proprietary-model analysis focuses on leading representative series from OpenAI, Anthropic, Google, and ZhiPu.The listed series include GPT, Claude, Gemini, and GLM models.Source: §4 · p. 6

• ByteDance’s Doubao series includes Doubao-Lite and Doubao-Pro.Source: §7 · p. 6
• MiniMax AI’s ABAB series includes ABAB-5.5 and ABAB-6.5.Source: §7 · p. 6
• 01.AI’s Yi series includes Yi-Medium and Yi-Large.Source: §7 · p. 6

4.1. Overall Performance Comparison Across Model Series

• 85.0% accuracy made Claude-3.5-Sonnet the top model, 4.6 percentage points ahead of GPT-4o; scores ranged from 60.5% for Doubao-Lite to 85.0%.Popular models also differed: Gemini-1.5-Pro scored 77.6%, while GPT-4o scored 80.4%.Source: §4.1 · pp. 6, 10, Table 1 · p. 7, Figure 1 · p. 8

4.2. Cost Efficiency Consideration

• About a 10-fold cost increase bought a 3.5-point accuracy improvement within many proprietary model series, while costs for a target performance could vary by more than three orders of magnitude.As of June 2024, Claude-3.5-Sonnet was the cost-efficiency leader among proprietary models; GLM-4-Air’s price implied USD 420 to process roughly 3B tokens.Source: §4.2 · pp. 10–12

4.3. How Fast is the Cost Efficiency Improving for Astronomical Tasks?

• About a 10-fold improvement in cost-efficiency arrived every 3–12 months on average across the four model series studied.Claude improved about 1000-fold from Claude-2.0 to Claude-3.5-Sonnet over 11 months; Gemini improved about 100-fold in three months, GPT about 30-fold, and GLM about 10–100-fold. These are rough guidelines based on release dates and pricing assumptions.Source: §4.3 · p. 12

4.4. Why Are the Weaker Proprietary Models Weaker?

• Weaker models showed greater deficits in Earth and Planetary, Solar and Stellar, and Instrumentation and Methods astrophysics, while non-English-focused models also lagged in specialized topics.The authors suggest limited training data, newer research areas, and missing historical context may contribute. Claude-3.5-Sonnet led five of six topic categories; Gemini-1.5-Pro scored 75.9% in Cosmology and 75.4% in Solar and Stellar Astrophysics, while Yi-Large scored 77.3% on average versus Claude-3.5-Sonnet’s 83.0%–87.0% range.Source: §4.4 · pp. 13, 19
• Non-English-focused models showed particular weakness in Historical and Theoretical Knowledge, and sometimes in Current Research and Advanced Topics.GLM-4-0520 scored 75.0% in Historical and Theoretical Knowledge versus Claude-3.5-Sonnet’s 84.3%; Yi-Large and Step-2 each scored 72.9% in that category. Questions were classified by topic and tested ability using GPT-4o.Source: §4.4 · p. 13
• 77.8% accuracy in Analytical & Reasoning Skills contrasts with Claude-3.5-Sonnet’s 86.4% in Understanding Fundamental Concepts and 89.4% in Advanced Topics.Analytical skills here mean logical deductions from observations, not complex mathematical problems.Source: §4.4 · p. 19
• ∼78 −79% scores place Step-1 and Step-2 among the strongest models in Analytical & Reasoning Skills.Step-2 is a trillion-parameter model developed by a team led by experts formerly associated with Microsoft Research Asia.Source: §4.4 · p. 19
• 9 gold and 2 silver medals make Claude-3.5-Sonnet the leader, ahead of Claude-3.0-Opus with 1 gold, 7 silver, 2 bronze, and 1 notable mention.GPT-4o earned 1 silver, 2 bronze, and 7 notable mentions; other medalists included Yi-Large, Step-1, Step-2, Gemini-1.5-Pro, Claude-3.0-Sonnet, and Claude-3.0-Haiku.Source: §4.4 · p. 19

5. BENCHMARKING OPEN-WEIGHTS LARGE LANGUAGE MODELS

• The evaluation covers open-weights models from 2B to 176B parameters, separately compares open and proprietary versions of hybrid models, and targets settings where GPU access or customization may favor open weights.The paper identifies continual pretraining and astronomy-specific fine-tuning as possible uses, with detailed benchmarking of those models deferred to future work.Source: §5.1 · pp. 19–20, §5 · p. 19

5.1. Comparative Analysis of Leading Open-Weights Model Series

• Recent model generations improve quickly: LLaMA-3-8B (72.9%) surpasses LLaMA-2-70B (70.7%), while Mistral-7B-v0.3 improves 15.8 points over v0.1 and Qwen-2-7B improves 10.6 points over Qwen-1-7B.Phi and Gemma series also improve across the compared versions and sizes.Source: §5.1 · pp. 20–21
• 80.6% accuracy for LLaMA-3-70B exceeds GPT-4o (80.4%), while Qwen-2-70B and Mixtral-8x22B-v0.1 each score 77.7%, below Claude-3.5-Sonnet (85.0%).The authors note that proprietary models may benefit from more rigorous customization and testing.Source: §5.1 · pp. 21, 23, Figure 6 · p. 22
• 75.6% accuracy from Gemma-2-27B and Phi-3-14B approaches Gemini-1.5-Pro (77.6%), while LLaMA-3-8B leads the under-10B group at 72.9%.The authors note that smaller models generally score below larger counterparts.Source: §5.1 · p. 22

5.2. Analyzing Performance Discrepancies in Open-Weights Models

• Open-weights models generally score lower in Earth and Planetary Astrophysics, Solar and Stellar Astrophysics, and Instrumentation and Methods for Astrophysics, where recent developments or historical context may be less represented in training data.Claude-3.5-Sonnet scores consistently across topics, from 77.8% to 89.4%.Source: §5.2 · p. 24
• Open-weights models also tend to lag in Historical and Theoretical Knowledge and Current Research and Advanced Topics, including a 10.4-point Phi-3-14B gap on Advanced Topics versus Claude-3.5-Sonnet.The authors report that even LLaMA-3-70B trails by about five points in both categories.Source: §5.2 · p. 24
• The topic weaknesses appear across development regions and model types; analytical-reasoning gaps are less pronounced and also occur for Claude-3.5-Sonnet.Non-English-focused models show additional weaknesses in the same astronomy topics and in historical and advanced knowledge.Source: §5.2 · pp. 24–25
• LLaMA-3-70B earns 2 silver, 4 bronze, and 4 notable mentions across categories, while Qwen-2-70B earns 6 notable mentions.Mixtral-8x22B-v0.1 earns 4 notable mentions; Phi-3-14B earns 1 bronze and 1 notable mention, and Gemma-2-27B earns 2 notable mentions.Source: §5.2 · p. 25

6.1. Limitation and rationale behind this benchmarking dataset

• The MCQ benchmark gauges foundational and current astronomical knowledge but is necessary, not sufficient, for evaluating actual research capability.The authors recommend considering it alongside other metrics; they identify open-ended research questions and rigorous analytical tests as future benchmark directions.Source: §6.1 · pp. 25–26

6.2. Model Self-Awareness: Are You Sure?

• Claude-3.5-Sonnet reaches a 0.98 confidence–performance correlation, while GPT-4o improves to 0.91 from GPT-4’s 0.69.Calibration generally improved in recent models; Table 7 identifies Claude-3.5-Sonnet as having the highest correlation and Step-1 the lowest mean absolute offset.Source: §6.2 · pp. 27–28, Table 7 · p. 28
• Open-weights calibration varies with model size: LLaMA-3-70B and Qwen-2-72B score 0.92 correlation, compared with -0.46 for LLaMA-3-8B and 0.28 for Qwen-2-7B.Source: §6.2 · p. 28
• Most models show slight underconfidence; Claude-3.5-Sonnet has a mean absolute offset of 0.10 despite its high accuracy.The authors note that question ambiguity or inaccurate formulation could contribute to underconfidence and propose more human curation and data self-cleaning.Source: §6.2 · p. 28

6.3. Assessing Potential Data Leakage and Temporal Performance

• Models show no sudden performance drop on recent questions, and all perform better on recent than older reviews, including GPT-4 on post-cutoff articles.Source: §6.3 · pp. 29–30
• Non-English-focused models tend to degrade more on older questions, but steeper historical degradation also appears for Mixtral-8x22B-v0.1 and GPT-4.The authors say the exact reason is difficult to trace, particularly for proprietary models.Source: §6.3 · p. 30

6.4. Balancing Performance and Affordability in LLM Deployment

• Qwen-2-72B and LLaMA-3-70B perform at GPT-4o level on this task and can run at an order of magnitude lower cost.Source: §6.4 · p. 31
• Approximately 10-fold price-performance improvements every 3–12 months let proprietary models reach a given performance level at lower cost on this benchmark.The authors project that, if the trend continues, some simple LLM-agent tasks could become feasible at about O(1)% of telescope build cost.Source: §6.4 · p. 31

7. CONCLUSION

• Non-English-focused proprietary models underperformed especially in historical and theoretical knowledge, current research, and several astronomy subfields; open-weights models also showed regional discrepancies.GLM-4-0520 scored 75.0% on Historical and Theoretical Knowledge versus Claude-3.5-Sonnet’s 84.3%.Source: §7 · pp. 31–32
• Competitive proprietary models improved at a rate equivalent to a 10-fold price reduction for similar performance every 3–12 months.Source: §7 · p. 31
• Recent open-weights models reached 77.7–80.6% accuracy, while pre-2024 models lagged proprietary models by as much as 20–30 percentage points.LLaMA-3 scored 80.6%, and Qwen-2 and Mixtral 8x22B each scored 77.7%; older models under 30B typically reached 50–65%, and those over 30B around 70%.Source: §7 · p. 32
• Top-performing recent models had confidence-correctness correlations above 0.9, but models generally showed slight underconfidence and smaller open-weights models had weaker calibration.Mean absolute offsets ranged from 0.04 for Step-1 to 0.13 for Qwen-2-7B.Source: §7 · p. 32
• 85.0% accuracy gave Claude-3.5-Sonnet a 4.4-percentage-point lead over the best open-weights model, LLaMA-3-70B.LLaMA-3-70B scored 80.6%.Source: §7 · p. 32
• Improving open-weights performance, lower inference costs, and model calibration offer a potential path to more affordable astronomy deployment.The authors suggest current computational and financial barriers may be overcome sooner than anticipated.Source: §7 · p. 32

A. INFERENCE METHODOLOGY DETAILS

• The prompt asks models to select one multiple-choice answer and provide a detailed astrophysical explanation in the requested output structure.Source: A. INFERENCE METHODOLOGY DETAILS · p. 35
• When a separate system prompt was available, the astrophysics-expert instruction became the system prompt; GPT-4o was used to parse responses when models did not follow the format.Source: A. INFERENCE METHODOLOGY DETAILS · p. 35
• Expert role-play, an answer-plus-explanation request, and a clear output format consistently improved performance, while sampling parameters such as temperature played a minor role.Source: A. INFERENCE METHODOLOGY DETAILS · p. 35

B. PERFORMANCE ANALYSIS ON POST-1990 QUESTIONS

• Older questions may reflect outdated information or consensus, and historical-context differences may affect some models’ scores without necessarily reflecting their research-agent abilities.The authors identify temporal knowledge as important when evaluating LLMs for specialized domains.Source: B. PERFORMANCE ANALYSIS ON POST-1990… · pp. 35, 37
• The Claude-3.5-Sonnet–GPT-4o gap shrinks from 4.6 to 3.4 percentage points, while model rankings remain largely consistent.Figure 12 restricts the comparison to Annual Review articles published after 1990.Source: Figure 12 · p. 36, B. PERFORMANCE ANALYSIS ON POST-1990… · p. 36
• Across 2,245 post-1990 questions, most models score higher than on the full 4,425-question set.Claude-3.5-Sonnet improves from 85.0% to 87.6% (+2.6 points), and GPT-4o from 80.4% to 84.2% (+3.8 points).Source: B. PERFORMANCE ANALYSIS ON POST-1990… · p. 36

C. CLASSIFICATION OF ASTROPHYSICS QUESTIONS

• Questions are grouped into six arXiv astro-ph subclasses to support more nuanced comparisons across astrophysics specializations.The classes span solar and stellar, planetary, galactic, cosmological and nongalactic, high-energy, and instrumentation and methods topics.Source: C. CLASSIFICATION OF ASTROPHYSICS… · p. 37

C.1. Solar and Stellar Astrophysics

• Cool white dwarfs have slightly higher mean masses than hot white dwarfs, while planetary-nebula progenitors are described as typically having 1.0 to 1.5 solar masses.The white-dwarf comparison is consistent with older objects originating from higher-mass progenitors.Source: §C.1 · pp. 37–38

C.2. Earth and Planetary Astrophysics

• Directly imaged exoplanet spectra often offer higher resolution and signal-to-noise, while Jupiter’s same-latitude vortices are explained as Kármán vortex streets.The imaging advantage comes with unknown orbital phase and degeneracies; the vortex arrangement prevents merging through mutually repulsive interactions.Source: §C.2 · p. 38

C.3. Astrophysics of Galaxies

• Radio-source axes align with host-galaxy angular momentum axes, and integrated-spectrum absorption lines reveal which stellar types contribute light across wavelengths.The spectral features can support a qualitative picture of a galaxy’s Hertzsprung-Russell diagram and spectrum-luminosity function.Source: §C.3 · p. 39

C.4. Cosmology and Nongalactic Astrophysics

• A power-law column-density distribution suggests a common origin for many Lyman alpha absorbers, while weak lensing measures total mass including dark matter.Unlike light-based methods, weak lensing traces mass regardless of luminosity.Source: §C.4 · p. 40

C.5. High Energy Astrophysics

• Black-hole spin precession modulates gravitational-wave phase and amplitude, while SN 1987A’s luminosity after three months primarily followed Cobalt-56 decay.The modulation breaks parameter degeneracies and can improve measurements of binary properties.Source: §C.5 · p. 41

C.6. Instrumentation and Methods for Astrophysics

• GLONASS transmissions interfered with the 1610.6–1613.8 MHz radio-astronomy band, while speckle interferometry can achieve diffraction-limited resolution.The technique extracts high-resolution information from short-exposure stellar speckle patterns.Source: §C.6 · p. 42

D. CLASSIFICATION OF ASTROPHYSICS QUESTIONS BY ABILITY

• The ability-based scheme groups questions by the skills they require to assess model performance across astrophysical knowledge and reasoning.The categories include fundamental concepts, techniques, reasoning, historical and theoretical knowledge, and current research.Source: D. CLASSIFICATION OF ASTROPHYSICS… · p. 42, §D.5 · p. 45

D.1. Understanding Fundamental Concepts

• Radio-emission polarization helps infer magnetic-field structure, with low observed polarization suggesting the field is not uniform in direction.Source: §D.1 · p. 43

D.2. Technical and Observational Techniques

• At low frequencies, interstellar dispersion forces narrower receiver bandwidths to avoid signal smearing, reducing pulsar-search sensitivity.The effect becomes more pronounced with increasing dispersion measure and distance.Source: §D.2 · p. 43

D.3. Analytical and Reasoning Skills

• Quasar-galaxy correlations suggest low-redshift quasars have more companion galaxies than average but do not typically reside in rich clusters.High-redshift environments remain less clear across the studies described.Source: §D.3 · p. 44

D.4. Historical and Theoretical Knowledge

• Claims of substantial electron-neutrino mass in the early 1980s spurred interest in non-baryonic dark matter models and their implications for CMB anisotropies.Source: §D.4 · p. 44

D.5. Current Research and Advanced Topics

• Gravitational screening suppresses modified-gravity effects in dense regions while allowing them on cosmological scales; assigning each question one ability category simplifies overlapping skills.The authors identify the ability classification as subjective and potentially ambiguous.Source: §D.5 · pp. 44–45