PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models

Vanessa Lama, Sanjay Das, Emily Herron, Yuan-Sen Ting, Tijmen de Haan, Junqi Yin, Tirthankar Ghosal, Feiyi Wang

arXiv:2609.17644v1astro-ph.IMcs.AIcs.CYcs.LG

TL;DR

Open-ended astronomy reasoning raises the question of when domain specialization offers value over strong general-purpose models. The paper evaluates model types on a curated Olympiad-style benchmark using judge-based and reference-based measures. Strong general-purpose models establish a high correctness baseline, while metric, judge, benchmark, and modality analyses show variation beyond a single ranking.

  • Problem

    Open-ended astronomy answers require evaluation beyond exact match or shallow overlap, and evidence is limited on how domain-specialized models compare with strong general-purpose systems on free-response reasoning.

    Source: §I · p. 1

  • Method

    The study evaluates models on curated astronomy free-response questions using judge-based correctness, complementary reference metrics, and analyses across evaluation and benchmark characteristics.

    Source: C. Evaluation Splits · p. 3, E. Evaluation Protocol and Metrics · p. 4

  • Results

    Strong general-purpose models establish a high correctness baseline, while reference similarity does not consistently reflect judged correctness and small score differences can depend on judge choice and benchmark composition.

    Source: §X · p. 8

  • Takeaways & Limitations

    The value of domain specialization should be established through task- and deployment-specific evaluation, since scientific model selection involves capability and deployment requirements beyond aggregate accuracy.

    Source: §VII · p. 7

    The LLM-as-judge results are sensitive to judge selection and rubric design, and the second-judge check does not replace expert adjudication or broader multi-judge evaluation.

    Source: §VIII · p. 8

Abstract

from arXiv · show

Domain-specialized language models are widely used for scientific question answering, but stronger general-purpose systems raise a sharper question: when does domain-specific fine-tuning remain valuable for open-ended scientific reasoning? We study this in astronomy with a curated QA benchmark from publicly available 2017--2026 Olympiad-style materials. The free-response subset contains 300 questions, including 204 text-only and 96 image-linked examples. We compare open-weight and API-served general-purpose, multimodal, and astronomy-specialized models using judge-based correctness and complementary reference metrics. Strong general-purpose models establish the highest correctness baseline in this testbed, while analyses of metric agreement, judge sensitivity, benchmark composition, and modality reveal variation not captured by a single leaderboard. These results motivate treating domain specialization as a task- and deployment-dependent property and highlight the role of domain-specific evaluation in determining which models, capabilities, and evaluation criteria are appropriate for scientific workflows.

I. INTRODUCTION

The paper asks when astronomy specialization offers measurable advantages over strong general-purpose models, especially for open-ended reasoning. It introduces a free-response benchmark and evaluates models across capabilities and evaluation views.

I. INTRODUCTION

  • The study targets a gap in astronomy evaluation: free-response reasoning with explanations, intermediate steps, and quantitative or conceptual conclusions remains underrepresented.Earlier astronomy-focused evaluations often emphasize multiple-choice QA or similar formats.Source: §I · p. 1
  • The benchmark uses curated Olympiad-style astronomy questions from 2017–2026, including text-only and image-linked examples, while reserving multiple-choice materials for future expansion.The evaluation compares open-weight and API-served models spanning general-purpose, multimodal, reasoning, and astronomy-specialized capabilities.Source: §I · p. 1
  • The paper contributes a structured free-response corpus and conversion pipeline that preserve subpart context, reference solutions, and associated figures.It also presents a unified framework for comparing general-purpose, multimodal, and astronomy-specialized models on open-ended astrophysics reasoning tasks.Source: §I · p. 1
  • The evaluation characterizes model behavior across question properties, evaluation metrics, judge selection, modality, and benchmark robustness.Fig. 1 summarizes the pipeline from benchmark construction through response generation, evaluation, and exploratory analysis.Source: §I · p. 1, Figure 1 · p. 2

II. BACKGROUND AND RELATED WORK

  • Scientific QA evaluation is evolving toward specialized assessments of domain knowledge, complex reasoning, and multimodal understanding as models and scientific use cases change.The background calls for systematic frameworks to identify meaningful capability and reliability measures for particular scientific workflows.Source: §II · p. 2

A. Scientific Question Answering and Benchmarking

The paper situates its benchmark among broad, expert, multimodal, and domain-specific science evaluations. It constructs an open-ended astronomy dataset and pairs judge-based assessment with complementary response-similarity measures.

A. Scientific Question Answering and Benchmarking

  • The benchmark targets open-ended astronomy problems requiring physical modeling, derivation, numerical estimation, and explanation, complementing multiple-choice astronomy evaluation.Related benchmarks cover broad academic knowledge, multimodal science, expert questions, and olympiad-style reasoning [1], [2], [3], [4], [5], [6], [7].Source: A. Scientific Question Answering and… · p. 2

B. Evaluation of Open-Ended Model Outputs

  • Because exact match and multiple-choice accuracy poorly fit open-ended answers, the study combines LLM-as-judge evaluation with reference metrics and robustness analyses.Lexical metrics can miss scientific correctness, while semantic metrics may still score scientifically incorrect responses highly; judge reliability also depends on selection and criteria –[14].Source: B. Evaluation of Open-Ended Model… · p. 2

C. Domain-Specialized and Multimodal Models for Science

  • Astronomy domain adaptation motivates evaluating specialized models alongside general-purpose systems and practical considerations such as open-weight access, local deployment, and data governance.The study also considers multimodal capability because astronomy problems often include diagrams, plots, and sky maps.Source: C. Domain-Specialized and Multimodal… · p. 2

A. Dataset Source and Construction

  • The dataset derives from publicly available USAAAO Olympiad materials spanning 2017–2026, converted from annual question and solution files into structured JSONL records.The resulting free-response dataset contains 300 examples.Source: A. Dataset Source and Construction · pp. 2–3

B. Record Construction

  • Each record represents a standalone problem or evaluated subpart and stores question, reference answer, year, length category, and image paths when applicable.Length categories are heuristic, and shared context is retained for multi-part problems to preserve the interpretation of equations, constants, diagrams, and coordinate conventions.Source: B. Record Construction · p. 3

C. Evaluation Splits

The benchmark separates a shared text-only evaluation from a full multimodal setting. Models span general-purpose, multimodal, and astronomy-specialized systems, evaluated through a common inference and scoring pipeline.

C. Evaluation Splits

  • The free-response benchmark has 204 text-only and 96 image-linked examples; text-only models use the 204-example shared set, while image-capable models use all 300 examples.The multiple-choice track is reserved for future development, and the two evaluation settings support comparisons without disadvantaging text-only models.Source: C. Evaluation Splits · p. 3

A. Dataset

  • Fig. 2 summarizes dataset composition by question format, year and modality, and question-length category.The 301 multiple-choice questions are reserved for a future benchmark track.Source: A. Dataset · p. 3

B. Models

  • The evaluated systems include open-weight Llama, GPT-OSS, Gemma, AstroLLaMA, and AstroSage models, alongside API-served GPT-5.5, Claude Sonnet 4.6, and GPT-OSS 120B.The systems span general-purpose, multimodal, and astronomy-specialized capabilities.Source: B. Models · p. 3

C. Inference Configuration

  • Each response is generated once and scored against its reference using automatic metrics and a fixed GPT-5 judge.GPT-5.5 uses no additional reasoning effort, GPT-OSS 120B uses its provider-required low-reasoning setting, and higher-reasoning configurations are reserved for future ablations.Source: C. Inference Configuration · p. 3

D. Computational Platform

Local and API-served inference use the benchmark’s shared evaluation pipeline, with some infrastructure measurements unavailable for provider-hosted systems. The evaluation uses a fixed judge score alongside reference-based metrics treated as diagnostics.

D. Computational Platform

  • Local open-weight inference ran on an 8-GPU NVIDIA A100 system, with models assigned GPUs according to memory needs and independent evaluations run concurrently when possible.The study is inference-only, with no training or fine-tuning; local runs retained wall-clock and per-example execution records.Source: D. Computational Platform · p. 3
  • API-served models used ORNL Azure OpenAI and OLCF/AmSC-compatible endpoints, but provider-side GPU utilization and energy measurements were unavailable.The study therefore does not provide provider-side resource measurements.Source: D. Computational Platform · p. 3

E. Evaluation Protocol and Metrics

  • Free-response answers receive a fixed GPT-5 judge score and complementary BLEU, ROUGE-L, BERTScore, and sentence-embedding cosine-similarity measures.The study treats reference-based metrics as diagnostic measures rather than direct measures of scientific correctness.Source: E. Evaluation Protocol and Metrics · pp. 3–4
  • Fig. 2 describes 601 questions divided between free-response and multiple-choice formats, with distributions of the 300 free-response examples by year, modality, and length.The multiple-choice questions are reserved for a future benchmark track.Source: Figure 2 · p. 4

F. Score Uncertainty and Robustness

The section quantifies uncertainty in aggregate scores with nonparametric bootstrap resampling and examines whether comparisons remain stable across year, modality, and benchmark composition.

F. Score Uncertainty and Robustness

  • Bootstrap confidence intervals come from resampling N examples with replacement and taking the empirical distribution of replicate mean scores.Each replicate samples N examples; the intervals reflect example-resampling variation only, because each model generates one response per question.Source: F. Score Uncertainty and Robustness · p. 4
  • Year-level variation is summarized by the dispersion of year-specific mean judge scores around the overall mean.The modality gap is the text-only mean minus the image-linked mean; positive values indicate lower performance on image-linked questions.Source: F. Score Uncertainty and Robustness · p. 4
  • These analyses assess whether aggregate model comparisons are stable across benchmark composition.Bootstrap intervals capture resampling variation, not variability from repeated model generations.Source: F. Score Uncertainty and Robustness · p. 4

V. RESULTS

On the 2017–2026 free-response benchmark, API-served general-purpose models set the strongest judge-score baseline. Shared text-only comparisons and metric analyses show that judged correctness and reference similarity do not always align.

A. Aggregate Model Performance

  • GPT-5.5 scored 0.6700 on the full benchmark, while Gemma 4 31B IT was strongest among open-weight multimodal models at 0.3933.Table II reports judge performance on each model’s evaluation setting and the shared text-only subset.Source: A. Aggregate Model Performance · pp. 4–5, Table II · p. 5

B. Text-Only Comparison

  • On the shared text-only subset, GPT-5.5, Claude Sonnet 4.6, and GPT-OSS 120B scored 0.7083, 0.6814, and 0.6544, respectively.AstroSage 70B and GPT-OSS 20B scored 0.3333 and 0.3284 among locally evaluated open-weight models.Source: B. Text-Only Comparison · p. 5

C. Metric Disagreement

  • ROUGE-L and sentence-level cosine similarity correlated moderately with judge scores (r = 0.68 and r = 0.66), while BERTScore showed little association (r = −0.11) and BLEU was negatively correlated (r = −0.60).Fig. 3 shows that reference-based metrics do not consistently track judged correctness.Source: C. Metric Disagreement · p. 5, Figure 3 · p. 5
  • Reference-based metrics are complementary diagnostics rather than stand-alone measures of scientific correctness.Domain terminology can raise similarity without establishing correctness, while correct responses may use different wording or derivations.Source: C. Metric Disagreement · p. 5

D. Judge Sensitivity

Judge choice changes some score magnitudes, but the highest-performing models remain stable across evaluators. Robustness analyses show that large differences persist under resampling, while close rankings and performance across years or question lengths require qualification.

D. Judge Sensitivity

  • Agreement between judges ranged from 0.608 for GPT-OSS 20B to 0.882 for GPT-OSS 120B, with score differences ranging from −0.0147 to +0.1593.The highest-performing models remained stable across both evaluators, although score magnitude shifted more for some models.Source: D. Judge Sensitivity · p. 5, Table III · p. 6

E. Performance Robustness

  • Large performance differences remain separated under 95% bootstrap intervals, but intervals overlap for GPT-5.5 versus Claude Sonnet 4.6 and AstroSage 70B versus GPT-OSS 20B.Small aggregate-score differences should not be treated as definitive rankings.Source: Figure 4 · p. 6, E. Performance Robustness · p. 6
  • Performance varies across exam years, while broader differences among model groups are generally preserved.Fig. 5 plots mean judge scores by exam year for selected models.Source: E. Performance Robustness · p. 6, Figure 5 · p. 6
  • Judge scores decline from short to medium and long problems across all selected models.The length categories also reflect problem structure and expected reasoning depth, making the trend a coarse indication of increasing difficulty for longer problems.Source: E. Performance Robustness · p. 6, Figure 6 · p. 6

F. Modality Effects

Image-linked questions receive lower judge scores than text-only questions for several high-performing models, although modality is confounded with problem content and difficulty. The broader analyses show why specialization and model performance must be assessed across task, deployment, and evaluation dimensions.

F. Modality Effects

  • Modality gaps cannot be attributed solely to visual reasoning because image-linked questions also differ in content and difficulty.A broader controlled comparison across vision-capable models would be needed to isolate the effect.Source: F. Modality Effects · p. 6
  • GPT-5.5 scores 0.708 on text-only versus 0.589 on image-linked questions; Gemma 4 31B scores 0.431 versus 0.312, and Gemma 4 26B scores 0.390 versus 0.286.Fig. 7 covers the 204 text-only and 96 image-linked questions in the full benchmark.Source: F. Modality Effects · p. 6, Figure 7 · p. 7

VI. ERROR ANALYSIS

  • Targeted response annotation identifies incomplete derivations, numerical or unit errors, visual grounding failures, and context fragmentation.Table IV summarizes these recurring failure patterns.Source: §VI · p. 7, Table IV · p. 7
  • These errors explain why reference-based similarity is a diagnostic complement rather than a stand-alone measure of scientific correctness.Appropriate terminology or a correct starting relation can coexist with an incomplete derivation or incorrect numerical result; visual and context errors may also escape lexical or embedding similarity.Source: §VI · p. 7
  • The targeted error analysis covers only a response subset, so it does not estimate failure prevalence across the benchmark.Broader annotation across models and question categories would allow more systematic comparison of failure-mode variation.Source: §VI · p. 7

VII. DISCUSSION

  • Strong API-served general-purpose models set a high correctness baseline, while specialized open-weight models offer different local-deployment, reproducibility, controllability, and data-governance properties.The systems also operate under different computational and serving environments, so model choice involves more than aggregate accuracy.Source: §VII · p. 7
  • Scientific capability conclusions depend on measurement: reference-metric agreement, judge choice, benchmark composition, question length, and modality can all affect observed performance.Complementary evaluation views help locate model successes and failures and test whether differences are robust to task characteristics.Source: §VII · p. 7
  • Domain-specific evaluation can guide workflow choices across domains involving specialized knowledge, quantitative reasoning, multimodal evidence, or computational constraints, without assuming one approach is universally preferred.Repeated characterization can help determine where general models suffice and where specialization or additional scaffolding provides measurable value.Source: §VII · pp. 7–8

VIII. LIMITATIONS

The study is limited to free-response questions, and its model comparisons do not capture every deployment tradeoff. It also notes uncertainty from possible training-data contamination, judge sensitivity, and a targeted rather than comprehensive error analysis; future work expands task formats and evaluation approaches.

VIII. LIMITATIONS

  • The study evaluates the free-response benchmark only; text-only and full-setting results use different subsets, and API-versus-open-weight comparisons do not represent the full deployment tradeoffs.The multiple-choice track is reserved for future evaluation.Source: §VIII · p. 8
  • Potential training-data contamination cannot be ruled out for publicly available source materials, particularly for broadly web-trained models [22].Year-to-year variation is only a coarse temporal diagnostic and cannot establish individual problem exposure or distinguish contamination from content and difficulty changes.Source: §VIII · p. 8
  • LLM-as-judge results are sensitive to judge selection and rubric design, while the second-judge check does not replace expert adjudication or broader multi-judge evaluation.The targeted error analysis covers only a response subset and does not estimate failure-mode prevalence across the benchmark.Source: §VIII · p. 8

IX. FUTURE WORK

  • Future work will add the multiple-choice track and more vision-capable models to compare constrained-answer accuracy with open-ended reasoning and better isolate visual reasoning from problem differences.It also plans evaluation across adaptation strategies and scientific reliability properties, including topic robustness, physical consistency, multi-step reasoning, and uncertainty calibration.Source: §IX · p. 8
  • Planned retrieval- and agent-based evaluations will combine language-model reasoning with astronomy resources, task decomposition, scientific tools, and numerical or unit verification.The comparisons will test whether adaptation, retrieval, tools, or agentic scaffolding offer measurable benefits over stronger general-purpose baselines and whether benefits justify additional costs.Source: §IX · p. 8

X. CONCLUSION

The paper presents an Olympiad-style astronomy benchmark and uses multiple evaluation views to characterize model behavior. Strong general-purpose models set a high correctness baseline, while specialization's value depends on capability, task requirements, and deployment constraints; the paper also documents its LLM-assisted workflow and reproducibility artifacts.

X. CONCLUSION

  • The benchmark uses Olympiad-style astronomy problems from 2017–2026 and evaluates aggregate correctness, modality, metric agreement, judge sensitivity, and benchmark robustness.These views provide a broader characterization of model behavior than a single leaderboard.Source: §X · p. 8
  • Strong general-purpose models establish a high correctness baseline, while specialization's value depends on model capability, task requirements, and deployment constraints.Reference-based similarity does not consistently reflect judged scientific correctness, and small differences can be sensitive to evaluator choice and benchmark composition.Source: §X · p. 8
  • Domain-specific evaluation can help identify which models, capabilities, and evaluation criteria fit scientific workflows and where general models or additional scientific scaffolding provide measurable value.Source: §X · p. 8

XI. LLM USAGE DISCLOSURE

  • The authors report that LLMs assisted with code generation, data-format validation, figure generation, and drafting, while authors reviewed the materials and remain responsible for the final analysis and conclusions.Source: §XI · p. 8

XII. ARTIFACT DESCRIPTION

  • Code and analysis workflows support benchmark construction, evaluation, and reproduction of tables and figures; structured dataset release is pending documentation and licensing review.Model regeneration requires suitable checkpoints, compute resources, or API access.Source: §XII · p. 8

APPENDIX: ADDITIONAL BENCHMARK DETAILS

The appendix describes a comparison of astronomy-specialized models with nearby general baselines, the judge prompt and rubric, and examples of benchmark records.

Specialization Deltas

  • Specialization deltas compare each astronomy-specialized model's text-only judge score with that of the closest evaluated general model at similar scale.The comparison is intended to use nearby general baselines rather than only frontier systems.Source: Specialization Deltas · p. 10
  • The judge receives the question, reference answer, and model answer, then returns a strict JSON verdict, score, and brief factual rationale.Verdicts are correct, partial, or incorrect, with scores restricted to 1.0, 0.5, or 0.0.Source: Prompt Template and Judge Rubric · p. 10

Example Benchmark Questions

Loading 2609.17644v1…