Source-linked AI summary
Reliable LLM-Generated Programs for High-Energy Physics Experiments through Graph-Grounded Software Knowledge
Yue Sun, Tong Liu, Yipu Liao, Jingde Chen, Ke Li
TL;DR
LLMs often lack the API, dependency, and workflow knowledge needed to generate reliable programs for complex particle-physics software. The paper grounds ROOT generation with hybrid software-graph retrieval, skill-selected examples, and execution-guided repair, finding higher initial and final success across two orchestration settings with small cost increases. The results support transferring relation-based grounding beyond ROOT where software dependencies are complex or documentation is sparse.
Problem
User requests rarely specify the APIs, dependencies, and usage conventions required for reliable programs in large, interconnected particle-physics software ecosystems.
Method
The system combines hybrid retrieval over a heterogeneous ROOT software graph, hierarchical tutorial-example selection, and execution-guided repair.
Results
Grounding improves executable performance and final quality across Claude Code and standalone orchestration, while increasing average generation cost per successful task by only 1.3% and 3.2%, respectively.
Takeaways & Limitations
Because the method captures software relations common to large codebases rather than ROOT-specific facts, it may transfer to other experiment frameworks and proprietary systems.
Abstract
from arXiv · showhide
Extracting physics information from modern particle-physics experiments requires multistage analyses implemented on top of large and highly interconnected software ecosystems. General-purpose large language models (LLMs) often produce unreliable programs for such tasks because a user request alone rarely specifies the required APIs, dependencies, and usage conventions. We organize these software relations before generation and retrieve task-relevant knowledge at inference time. Using the open-source ROOT framework as a representative and reproducible testbed, we evaluate a complete grounding system that combines hybrid retrieval over a heterogeneous software knowledge graph, skill-selected workflow examples, and execution-guided repair. On a benchmark of 275 ROOT tasks, grounding improves first-attempt execution from 58.5% to 76.0% under Claude Code orchestration and from 51.3% to 64.0% under standalone orchestration. Final success increases from 90.5% to 96.0% and from 78.9% to 90.9%, respectively, while the average generation cost per successful task increases by only 1.3% and 3.2%. The gains persist under a strong coding agent, indicating that explicit software knowledge remains valuable even when agentic scaffolding is already in place. Because the method captures software relations common to large codebases rather than facts specific to ROOT or a particular model, it should transfer to other experiment frameworks and proprietary software, especially where documentation is sparse or internal dependencies are complex.
I. INTRODUCTION
Modern particle-physics analyses depend on complex software ecosystems whose APIs, dependencies, and usage conventions are difficult to infer from user requests alone. The paper grounds ROOT program generation with software-graph structure, workflow examples, and execution feedback, and evaluates these components across two orchestration settings.
- Motivation: Complex detector and analysis pipelines require programs that combine extensive scientific software stacks correctly.Reliable physics results depend on correct software use as well as data and physical models.
- Motivation: General-purpose LLMs can generate convincing programs that call nonexistent methods, use obsolete overloads, omit dependencies, or combine APIs incorrectly.Programming ability alone does not ensure reliable scientific software.
- Grounding strategy: Structural grounding organizes repository relations, procedural grounding supplies complete workflow examples, and diagnostic grounding converts execution failures into targeted repair context.Together, these sources identify valid components, show how they fit together, and support correction after failed runs.
- Evaluation design: The ROOT workflow retrieves graph and example evidence before generation and uses execution evidence for bounded repair, comparing grounded and direct generation with the same model in two orchestrations.The workflow uses hybrid retrieval, graph expansion, tutorial selection, and Error-RAG routing.
- Research questions: The evaluation addresses effectiveness, repair burden, resource trade-offs, robustness across orchestration settings, and Error-RAG versus direct error feedback.These questions cover both initial generation and subsequent failure recovery.
- Contributions: The study evaluates a 275-task ROOT benchmark and characterizes executable success, final-program quality, repair dynamics, resource overhead, and failure modes.The authors describe this as the first systematic experimental-particle-physics study grounding generated analysis programs and repair in a preconstructed framework-dependency graph.
2 Repository-Structure
The structural-grounding component represents ROOT as a heterogeneous software graph and retrieves task-relevant repository context through anchor selection and constrained, type-aware expansion. The resulting graph context is combined with tutorial and task information before ROOT program generation and execution.
- Workflow: The grounded workflow assembles graph-node contents and tutorial contents before deepseek-v4-pro generates or repairs a ROOT program.The workflow then executes the program in a ROOT environment and routes failures through a bounded repair loop.
- Evaluated workflow: Figure 1 depicts structural and procedural grounding before round-0 generation, followed by execution and a repair loop limited to five rounds.Failed executions are routed by a three-action Error-RAG planner.
- Type-Aware Graph Expansion: Relation-specific expansion constrains traversals by anchor type, permitted relation sequences, and separate hop budgets for depends, contains, and extends.This avoids unrestricted scalar-n breadth-first search and preserves node-type and relation-sequence constraints.
- Type-Aware Graph Expansion: The evaluated implementation sets every relation budget to 1, while the longest admitted schema is the two-edge path depends ◦contains.Class, source-file, and header anchors use different schemas to recover related files, headers, classes, and dependencies.
C. Procedural grounding from skill-selected examples
Procedural grounding retrieves complete ROOT tutorial workflows through hierarchical selection and injects their contents during initial generation. The study evaluates this mechanism within controlled benchmark, orchestration, execution, and repair procedures.
- Procedural grounding: Hierarchical tutorial retrieval traverses Markdown indexes and tutorial files, pruning branches and selecting a small set of complete workflow files.The full contents of selected files are injected during round-0 generation, while examples are not retrieved during repair.
- Procedural grounding: The tutorial skill supplies task-level combinations of ROOT data objects, fitting classes, and plotting calls that complement the software graph’s structural information.The tutorial corpus is independent of the 275-task benchmark and available only to the grounded condition.
- Benchmark: The benchmark contains exactly 275 natural-language requests for ROOT C++ programs derived from tutorial implementations whose framework identifiers were suppressed.Requests specify objectives, inputs, processing steps, and expected outputs without exposing source implementations to generation.
- Benchmark: The tasks span seven mutually exclusive workflow groups, led by 69 statistical-modeling, 63 data-processing and storage, and 57 histogram, fitting, and unfolding tasks.The remaining groups include visualization and graphics, mathematics and simulation, machine learning, and two outside the listed groups.
- Experimental controls: Direct and grounded conditions use the same generation model, while orchestration settings differ between Claude Code and a standalone runner.Claude Code permits up to 12 agent turns in direct generation but uses one effective turn with precomputed grounded context; the standalone runner uses five repairs and a 180-s timeout in both conditions.
- Repair ablation: The Error-RAG ablation compares targeted diagnostic retrieval with raw execution feedback on 113 actionable round-0 failures using byte-identical initial programs and matched repair limits.Error-RAG retains planner-based validated graph retrieval, whereas direct feedback removes the planner and graph retrieval.
E. Verified implementation parameters
The study defines verified execution, quality, reference-score, repair, and statistical endpoints for comparing grounded and direct ROOT program generation on a 275-task benchmark.
- Verified implementation parameters: 275 ROOT requests were evaluated using process-level execution, quality-gated acceptance, final reference scores, repair measures, and Error-RAG ablation endpoints.Process success requires ROOT exit status zero before timeout; final success requires process success and a valid positive quality verdict in any round from 0 through 5.
- Verified implementation parameters: The quality gate evaluates process-successful runs for compilation or runtime failures, invalid numerical results, failed fitting or plotting, crashes, and incomplete computation.Warnings may pass when they clearly do not affect the requested result, while missing completion phrases alone do not constitute failure.
- Verified implementation parameters: Final reference scores sum functional correctness, requirement completeness, ROOT API appropriateness, and robustness into an unweighted 0–100 score.A separate reference-evaluator pass requires a valid verdict, process-level success, and a summed score of at least 80.
- Verified implementation parameters: Binary endpoints use exact McNemar tests, while rate and mean-score differences use percentile-bootstrap intervals and sign-flip permutation tests.Holm–Bonferroni-adjusted p values control the family-wise error rate across eight headline tests; resource uncertainty is bootstrapped over complete task records.
- Results: Grounding improved round-0 execution from 58.5% to 76.0% under Claude Code and from 51.3% to 64.0% under standalone orchestration.Quality-gated acceptance also increased from 54.2% to 68.7% and from 44.7% to 58.9%, respectively, with all four differences significant after Holm correction.
- Results: Final success increased from 90.5% to 96.0% and from 78.9% to 90.9%, while mean final-program scores rose by 3.77 and 11.60 points.The grounded advantage persisted beyond round 0, and mean repair burden fell from 1.120 to 0.684 and from 1.720 to 0.975 rounds, respectively.
- Results: Grounding rescued 21 Claude Code tasks and 41 standalone tasks that failed under direct generation, compared with six and eight regressions.Final-failure counts consequently fell from 26 to 11 and from 58 to 25, respectively.
C. RQ2 ablation: Error-RAG versus direct error feedback
The Error-RAG ablation shows that targeted diagnostic retrieval improves repair success over direct error feedback on identical round-0 failures, with consistent benefits across evaluation measures and orchestration settings.
- Repair success: 77.9% of Error-RAG tasks reached final quality-gated success, versus 59.3% with direct error feedback.The comparison covers 113 actionable round-0 failures; the rate difference has a 95% confidence interval of [+9.7, +27.4].
- Repair trajectory: Error-RAG retained higher cumulative success after every repair round, widening from 46.0% versus 38.1% after round 1 to 77.9% versus 59.3% after round 5.The comparison is between Error-RAG and direct error feedback on the selected round-0 failures.
- Independent quality check: 65.9% of tasks passed the independent reference evaluator with Error-RAG, versus 51.8% under direct error feedback.This complementary endpoint was evaluated on 85 tasks present in both conditions, with a 95% confidence interval of [+3.5, +24.7].
- Resource trade-off: Grounding increased token use by 65.9% with Claude Code and 111.5% standalone, while average cost per successful task rose only 1.3% and 3.2%.Mean generation time increased by 25.2 s for Claude Code and 39.3 s for the standalone runner; the latter interval excluded zero.
- Cross-orchestration consistency: All eight adjusted headline tests favored grounding across the two orchestration implementations, indicating consistent effectiveness within the evaluated runner settings.The paper reports that all eight adjusted p values remained below 0.05 across both orchestration implementations.
- Interpretation: The lower repair burden suggests that grounding produces usable programs earlier, reducing reliance on repeated revisions when failed runs require substantial diagnosis.The discussion connects fewer repair rounds with practical relevance for specialized scientific software.
(d) Standalone orchestration
In standalone orchestration, grounding improves initial and final program outcomes while reducing repair effort, with only modest resource costs and a stated need for later physics validation.
- Resource trade-offs: Grounding uses more tokens and takes longer on average in standalone orchestration, but cost per successful task remains close to direct generation.The higher success rate and fewer repairs offset much of the added generation resource use.
- Practical boundary: The resulting programs still require physics validation and collaboration review despite reducing routine implementation and debugging effort.The practical benefit is framed as faster analysis iteration rather than replacing scientific review.
- Transferability: ROOT’s containment, dependency, and inheritance relations are presented as generic software relations that may transfer to other scientific and proprietary frameworks.Such deployment depends on access to the relevant source repository or interface metadata.
- Future work: Task-dependent graph attention is identified as future work because the present system expands anchors through fixed, type-dependent relations.The proposed refinement would rank and prune dependencies according to each request.
Appendix A: Benchmark construction prompt and example
Appendix A describes how hidden ROOT implementations are converted into concise user-oriented benchmark requests and specifies the constraints imposed on those descriptions.
- Appendix A: Benchmark construction prompt and example: The benchmark-generation process converts ROOT tutorial implementations into natural-language requirements while withholding framework-specific implementation details from evaluated generators.The source implementation is used only during construction and removed from subsequent requests.
- 1. Benchmark-generation prompt: The prompt asks for one concise English paragraph beginning with “Please write {language} code to” and containing no titles, lists, Markdown, or extra explanations.The requested output is limited to 2–3 sentences.
- 1. Benchmark-generation prompt: Requirements must describe user goals, inputs, processing, and outputs in plain domain language rather than framework classes, APIs, methods, or library identifiers.Examples replace named API calls with functional descriptions such as fitting, plotting, saving, and reading data.
- 1. Benchmark-generation prompt: External input and dependency files must be identified separately from generated outputs and handled according to normalized names, URLs, and the available-file inventory.Missing non-URL files should be generated in the unified data directory before the main workflow runs.
- 1. Benchmark-generation prompt: Existing files must be referenced under the unified data directory instead of original framework-specific or relative paths.URL dependencies may retain their URL source without requiring local generation.
- 1. Benchmark-generation prompt: Internally generated data should remain an in-program dependency, while random numbers, toy data, simulations, and fits require a fixed random seed.These rules target reproducibility without inventing external file dependencies.
- 1. Benchmark-generation prompt: The prompt explicitly forbids naming framework-specific classes and methods, including in examples, and requires functional behavior instead of underlying API calls.The examples contrast plain-language requests with prohibited ROOT identifiers.
2. Example benchmark record
The example benchmark record separates metadata, the user-facing request, and the hidden reference implementation, while the appendix also documents controlled retrieval-query generation.
- 2. Example benchmark record: Each benchmark JSON object contains an identifier, anonymized sample name, natural-language request, and hidden reference implementation.The answer field is used only by the independent evaluator.
- 2. Example benchmark record: The rendered example abbreviates the long answer value, but the stored benchmark retains the complete implementation.Escaped newlines are expanded for readability and an omission marker indicates the displayed truncation.
- Appendix B: Prompt templates used in the experiment: The appendix reproduces controlled prompt templates for retrieval, tutorial selection, code generation, execution-guided repair, and independent reference scoring.Runtime substitutions and retrieved artifacts are represented by placeholders.
- Appendix B: Prompt templates used in the experiment: Query rewriting produces a semantic query for dense retrieval and a keyword-rich query for BM25 retrieval and reranking.The keyword query may include likely ROOT APIs, classes, functions, and task terms.
- Appendix B: Prompt templates used in the experiment: The rewriting prompt preserves user intent, forbids invented names or tasks, prefers ROOT C++ APIs unless Python is requested, and returns JSON only.The prompt also requests English because ROOT API and source descriptions are mainly in English.
b. CodeGraph-node relevance selection
CodeGraph-node selection ranks candidate software entities using compact metadata, first across anchor nodes and then across one-hop expansions, before loading full node contents.
- b. CodeGraph-node relevance selection: The selector is applied first to bounded anchor candidates and then to bounded one-hop expanded candidates.Full node contents are loaded only after selection.
- b. CodeGraph-node relevance selection: Node selection uses only each candidate’s short description, name, and type, returning JSON with selected IDs and a brief reason.IDs must come from the candidates, remain ranked by usefulness, and not exceed the requested maximum.
- b. CodeGraph-node relevance selection: Candidate payloads include identifiers, names, types, descriptions, source labels, and rank scores for anchor or expanded nodes.The full payload is supplied to the selector only as structured candidate metadata.
- b. CodeGraph-node relevance selection: Tutorial-index traversal judges directories and source files by names, paths, and descriptions treated as untrusted reference data rather than instructions.Generic terms alone are insufficient, and returning no entries is valid.
- b. CodeGraph-node relevance selection: Traversal responses contain selected IDs, per-candidate relevance decisions, covered concepts, reasons, and an overall reason.The selection prompt again restricts outputs to candidate IDs.
- b. CodeGraph-node relevance selection: Final source-file selection ranks files by direct relevance to the complete task and seeks collective coverage of requested APIs and workflows.For C++ generation it prefers C++ tutorials and rejects launchers, menu files, and indirect examples.
- b. CodeGraph-node relevance selection: The final-selection payload supplies the original and rewritten questions, retrieval query, selected CodeGraph nodes, index path, selection limit, and source-file candidates.Only candidate source files are eligible at this stage.
3. Code-generation prompts
The prompts constrain ROOT macro generation, protect against untrusted retrieved content, and define execution-log quality adjudication. Initial generation inserts selected CodeGraph and tutorial context, while the quality gate evaluates whether execution completed correctly.
- a. Code-generation system message: The system asks a ROOT-experienced model to generate C++ code from input information and task requirements.
- a. Code-generation system message: Retrieved CodeGraph descriptions, comments, compiler output, and tutorials are treated as untrusted reference data rather than instructions.
- a. Code-generation system message: Generated code must avoid unauthorized shell, network, file, environment-variable, credential, home-directory, and unrelated-path access.
- a. Code-generation system message: ROOT invokes a macro function whose name exactly matches the filename without its extension.
- a. Code-generation system message: Generated files must contain complete code, required includes, a filename-matching main function, and runtime-bounded event or fit-test data.
- b. Initial generation user message: Initial generation inserts selected CodeGraph nodes and tutorial files before the original request and requires a named ROOT macro function returning only C++ code.
- a. ROOT execution-log quality gate: A clean-log fast path precedes model adjudication, while the quality gate returns passed=true only when logs show no correctness-threatening execution problem.
- a. ROOT execution-log quality gate: The quality gate emits strict JSON describing pass status, severity, reason, affected stream, evidence, and impact from bounded stdout and stderr payloads.
b. Error-RAG repair planner
The repair planner selects among direct repair, targeted retrieval, and environment-blocked routes, then passes bounded execution evidence and retrieved context to a constrained repair-generation call.
- Error-RAG repair planner: The planner chooses one of three repair routes and may request at most three validated CodeGraph nodes.
- Error-RAG repair planner: Its output is JSON containing an action, evidence-based diagnosis, repair instruction, and direct retrieval targets with names, types, path hints, and reasons.
- Error-RAG repair planner: Direct repair is used when logs and failed code suffice; retrieval is reserved for missing declarations, ownership rules, signatures, implementation details, or examples.
- Error-RAG repair planner: The planner limits final node names, prefers declaring headers and small working examples, and uses environment_blocked only when regeneration cannot fix the failure.
- Error-RAG repair planner: The planner receives the original task, failed code, execution status, timeout, logs, and a node limit as structured payload fields.
- Error-RAG repair planner: Repair generation retains the tutorial slot but evaluated repair calls leave it empty while reusing the generation system message.
- Error-RAG repair planner: Prepared stdout and stderr are included with explicit input-limit metadata in the repair payload.
- Error-RAG repair planner: The repair call preserves the original task and macro function name, supplies failed code and ROOT logs, and requests only complete corrected C++ code.
5. Independent reference-answer scoring prompt
The independent scorer evaluates only the final candidate against the task and a hidden known-correct reference, using structured multidimensional scores and execution evidence. The appendix audits representative rescue, regression, and environment-blocked transitions.
- 5. Independent reference-answer scoring prompt: The scorer receives only the final code attempt, task, execution evidence, and one hidden reference implementation.
- 5. Independent reference-answer scoring prompt: It treats task, code, reference, and logs as untrusted data and assesses only the supplied final candidate, whether initial or repaired.
- 5. Independent reference-answer scoring prompt: Candidates are judged by behavior against the task and reference, without penalizing different APIs, organization, names, or algorithms when requirements remain satisfied.
- 5. Independent reference-answer scoring prompt: The compact JSON verdict scores functional correctness, requirement completeness, ROOT API appropriateness, and robustness, alongside confidence and bounded evidence lists.
- 5. Independent reference-answer scoring prompt: The scorer payload records the scored round, return code, timeout, quality-gate result, candidate and reference code, logs, and truncation metadata.
- 5. Independent reference-answer scoring prompt: Schema validation failures trigger a retry that requests one corrected JSON object only.
- Appendix C: Artifact-level case audit: Case audits select the lowest task identifier within each relevant final-transition class.
- Appendix C: Artifact-level case audit: The audit documents a rescue from a hallucinated TGraphPolargram method, a regression from an invalid RuleFit option, and a standalone environment-blocked outcome.