Multi-Agent System for Cosmological Parameter Analysis

1 Introduction

• Multiple specialized LLM agents divide complex scientific tasks into subtasks, retrieve information, write and execute code, and coordinate transitions toward a shared objective.The authors say this strategy can apply across physical sciences involving quantitative data analysis.Source: §1 · p. 1
• The example system, cmbagent, is implemented with the open-source autogen framework and uses gpt-4o-2024-05-13 through the OpenAI API.Although the framework supports several language models, the presented cases use GPT-4 alone.Source: §1 · p. 1
• The authors situate cmbagent alongside prior scientific multi-agent systems and identify retrieval-augmented generation as a key feature of their approach.They cite sciagents for materials discovery and mephisto for interpreting multi-band galaxy observations.Source: §1 · p. 2
• The authors target generic automation of state-of-the-art cosmological analysis pipelines, focusing here on the question of whether such pipelines can be automated generically and extended to other domains.They also pose the separate question of whether a multi-agent system can find more optimal pipeline solutions than humans.Source: §1 · p. 2

2.1 Main task

• cmbagent is designed around a known task—deriving cosmological parameter constraints from ACT DR6 CMB lensing data—while keeping code and instructions free of task-specific details.The design uses the ACT likelihood with research software including classy_sz and cosmocnc.Source: §2 · p. 2, §2.1 · p. 3
• The example combines ACT likelihood information with community and research software, including classy_sz for theory calculations and cosmocnc for galaxy-cluster number-count likelihoods.The ACT Collaboration used Bayesian inference with MCMC, camb for signal prediction, and cobaya and GetDist for sampling and posterior estimation.Source: §2.1 · p. 3

2.2 Agent types

• Experiment and software RAG agents retrieve information from experiment papers and software tutorials, while a memory RAG agent can reuse saved summaries of prior tasks.Memory is intended to help apply past solutions to similar tasks, speed work, and reduce calls to other RAG agents.Source: §2.2 · pp. 3–4
• Python coder agents divide implementation and execution: an engineer writes code from retrieved suggestions, and an executor runs it after human validation.Manager agents organize the workflow through a chat manager, an admin for human input, and a planner that divides the main task into subtasks.Source: §2.2 · p. 4

2.3 System and workflow

• The planner assigns one agent to each subtask, while the admin can revise the plan and provides feedback throughout the workflow.Only the admin communicates with the planner, RAG agents, engineer, and executor; low temperature and TopP settings are used to reduce stochasticity.Source: §2.3 · pp. 4–5

3.1 Performance on Main Task

• The cmbagent contours overlap almost perfectly with the original ACT chains, with statistically insignificant differences.The remaining difference is attributed to differing precision settings for the theory-code neural-network emulators and CAMB.Source: Figure 2 · p. 6, §3.1 · p. 6
• The full ACT analysis took 40 minutes and required no human-written code; the MCMC itself converged in 8 minutes.The MCMC used four chains across 10 CPU cores and reached R −1 = 0.01; setup and preliminary tasks took another 8–10 minutes.Source: §3.1 · pp. 6–7
• The ACT reproduction alone does not demonstrate usefulness beyond the task used to develop the system.The authors therefore tested it on unrelated tasks.Source: §3.1 · p. 7

3.2 Generalization and application to research software

• On tasks not considered during development, cmbagent produced CMB spectra for ten fEDE values and a cluster mass-bias likelihood consistent with the simulated input.The spectra were computed with classy_sz; cosmocnc evaluated the likelihood for a Simons-Observatory-like catalogue.Source: §3.2 · p. 7, Figure 3 · p. 7

Human input and setup

• The system requires human feedback at every planner step, and users must install packages and software beforehand.The authors plan to reduce required input while retaining user flexibility.Source: §4 · p. 7

Reliability and appropriate use

• Current LLMs can produce plausible but physically incorrect cosmology responses, so the authors say the system may currently suit experienced researchers who can catch them.They also suggest using MAS as a cross-check of human analysis.Source: §4 · p. 8

Cost and efficiency

• The main task used 15 agent calls, 274483 tokens, and $1.55; a less demanding plot used six calls and $0.1.The authors identify token usage and cost as limitations and point to exploring models and caching.Source: §4 · p. 8

Retrieval challenges

• Scientific-document retrieval remains difficult for complex tables and equations, and current parsing pipelines do not yet provide robust answers.The authors plan to explore agentic RAG tools such as PaperQA2.Source: §4 · p. 8

Future directions

• The examples use GPT-4o exclusively; exploring other models and combining fine-tuning with RAG are future directions.The authors also mention minimal-human-input systems as a continuation of the work.Source: §4 · pp. 8–9

Verification and benchmarking

• Robust benchmarking is identified as urgent for both RAG question-answering and agentic tasks; outputs also cannot be confirmed without independently redoing the analysis.The authors describe benchmarking for agentic tasks as an open research question.Source: §4 · p. 9

• The generated script imports CMBAgent, initializes cosmocnc number_counts, varies bias_sz, updates parameters, and plots the likelihood.Source: A Research software example: cosmocnc · pp. 12–13
• The example asks cmbagent to compute an unbinned log-likelihood for the SO_sim_0 catalogue using q_so_sim at 40 bias_sz values from 0.79 to 0.81.Source: A Research software example: cosmocnc · p. 12
• The requested output plots the exponentiated log-likelihood normalized to its maximum and saves the plot as a PDF along with the code.Source: A Research software example: cosmocnc · p. 12
• The workflow instructs users to set cosmocnc parameters explicitly and save the code in output/results/ as a Python file.Source: A Research software example: cosmocnc · p. 12