Source-linked AI summary
CoLLM: AI engineering toolbox for end-to-end deep learning in collider analyses
W. Esmail, A. Hammad, M. Nojiri
TL;DR
Collider analyses increasingly require complex event selections and deep-learning methods, creating a substantial coding and expertise burden and exposing manual implementations to transcription errors. CoLLM addresses this with physics-aware LLM code generation, validation and correction, automated classifiers, and a graphical interface, producing executable and physically meaningful analyses across benchmark processes while remaining limited to LHCO data and vulnerable to ambiguity in plain-language specifications.
Problem
Collider analyses require substantial physics, programming, and deep-learning expertise, while manual code translation is time-consuming and vulnerable to silent transcription errors.
Method
CoLLM uses physics-aware LLMs to generate executable analysis code, validates and repairs it, and passes its outputs to automated MLP, GNN, and Transformer pipelines.
Results
Across five benchmark processes, CoLLM produced consistent, executable analysis code with physically meaningful outputs and high reproducibility from deterministic decoding and PyFixer correction.
Takeaways & Limitations
CoLLM connects plain-language collider-analysis specifications to automated deep-learning pipelines while reducing coding and technical effort.
Takeaways & Limitations
The current implementation supports only the LHCO data format, not ROOT structures commonly used in experimental analyses.
Abstract
from arXiv · showhide
Recent improvements in large language models have opened new opportunities for accelerating and automating scientific workflows. In parallel, modern collider analyses are becoming increasingly complex and demand substantial programming and deep learning expertise. \coll alleviates this workload by using pretrained large language models to generate physically consistent analysis code for event selection. Additionally, it automates subsequent deep learning analyses. To further reduce reliance on programming or deep learning experience, \coll provides a graphical user interface that allows users to perform end-to-end analyses through an interactive interface. The main motivation behind \coll is to lower the coding burden and simplify the technical complexity of collider analyses, which increasingly depend on sophisticated event selections and advanced deep learning methods.
1 Introduction
Collider analyses require substantial physics, programming, and deep learning expertise, while manual translation of analysis logic into code creates a time-consuming bottleneck vulnerable to silent transcription errors. CoLLM addresses this challenge by combining domain-aware LLM code generation with validation, error correction, and automated deep learning.
- LHC analyses combine object reconstruction, event selection, kinematic computation, and increasingly deep-learning classification, demanding expertise across physics, programming, and machine learning.
- Manual translation of physicists’ analysis logic into executable code can span thousands of lines across parsing, selection, histogram, and deep-learning components.
- Transcription errors, including incorrect particle identifiers or inconsistent kinematic cuts, can propagate silently and remain difficult to detect.
- CoLLM follows a disciplined “vibe engineering” approach in which LLM-generated code remains subject to physics constraints, automatic validation, and predictable execution.
- CoLLM translates plain-language specifications into executable Python analysis code whose output can feed user-specified deep-learning models.
- Its framework combines a physics-aware code-generation engine with automated deep-learning classifiers and iterative error correction.
2 The CoLLM Framework
CoLLM is a structured, modular framework that converts natural-language collider-analysis specifications into validated event-selection code and feeds the resulting features into automated deep-learning pipelines. Its design combines physics-aware generation, validation feedback, multiple model families, and user-facing interfaces while preserving extensibility.
- CoLLM integrates an LLM module for validated event-selection and preprocessing code with an automated deep-learning pipeline for classification and model optimization.
- The sequential design trains downstream models on physically motivated, consistently defined features and reduces analysis time and systematic ambiguity.
- User specifications are organized into selection cuts, validation plots, and output structure to define event criteria, diagnostics, and downstream observables.
- The closed-loop generator uses static checks and runtime tests to refine only affected code segments after failures.
- MLPs handle fixed-length kinematic inputs, GNNs model variable-multiplicity object relations, and Transformers capture global particle-set structure.
- CoLLM supports configurable automated training with user-controlled datasets, validation, optimization, learning-rate schedules, and numerical precision.
- The package includes terminal, graphical, MLP, GNN, and Transformer modes within a modular source structure.
- The GUI module supports interactive selection configuration, model definition, real-time monitoring, and direct training-job configuration.
3 Installation and Quick Start
CoLLM provides installation and quick-start paths for both interactive and scripted collider analyses. Its GUI and TUI share the same backend, while YAML configurations control reproducible code generation and automated training across supported model families.
- Installation: CoLLM is an open-source Python package designed for laptops through high-performance computing clusters on Linux and macOS with Python 3.9 or later.
- Installation: Users clone the public repository and are advised to use an isolated environment, with Python 3.11 recommended for current PyTorch and Hugging Face compatibility.
- Installation: The framework checks and installs missing dependencies automatically during first execution, including PyTorch, Transformers, Accelerate, Hugging Face Hub, LangChain, and Streamlit.
- Installation: Hardware support includes CUDA with 4-bit quantization on NVIDIA GPUs and MPS acceleration on Apple Silicon.
- User interfaces: The GUI supports real-time browser interaction, while the TUI enables scripted batch workflows; both use the same backend for reproducibility.
- User interfaces: YAML configuration files control TUI analysis parameters and can be version controlled with outputs.
- Training: Training configurations specify datasets, model architectures, optimization, early stopping, scheduling, precision, and evaluation metrics.
- Example validation: The generated top-pair analysis produces validation histograms whose reconstructed W and top masses and lepton-MET transverse-mass edge are physically meaningful.
4 Automated analysis pipeline
CoLLM’s internal design combines explicit collider-physics knowledge, deterministic code generation, and execution feedback to support robust analysis workflows. The framework couples code generation with automatic error correction and configurable deep-learning classifiers.
- The internal design combines domain knowledge, deterministic generation, and execution feedback to improve collider-analysis robustness.
4.1 LangChain orchestrator and generation pipeline
CoLLM uses LangChain to coordinate constrained, multistep generation and validation of executable collider-analysis workflows. Physics-aware prompts, deterministic decoding, and automatic error correction target reproducibility and common LHCO and physics-convention errors.
- Generated workflows: CoLLM generates complete executable workflows that read LHCO events, reconstruct objects, apply selections, compute observables, and produce structured outputs.The generated workflows are intended to resemble those written by experienced phenomenologists.
- Orchestration: LangChain coordinates multiple LLMs, prompt construction, sequential planning, code generation, validation, constraint enforcement, and error-guided correction.The orchestration layer decomposes code generation into multistep operations rather than a single prompt response.
- Reproducibility: Deterministic decoding with T = 0 and greedy token selection makes generation a function of the input prompt and model parameters, supporting reproducibility.Sampling is disabled, so identical inputs should produce similar analysis code across runs.
- Physics constraints: A physics-aware system prompt encodes LHCO formats, kinematic conventions, particle codes, and analysis requirements to constrain generated code.The prompt explicitly addresses errors such as MET misidentification, lepton-charge mistakes, and incorrect four-momentum summation.
- Error correction: PyFixer iteratively analyzes execution failures and guides corrective code modifications because generated analysis code can still fail at runtime.The repair mechanism addresses syntax errors, missing imports, and incorrect assumptions about data structures.
4.2 Automated deep learning analysis
CoLLM automates collider-classification workflows across MLP, GNN, and Transformer architectures, matching models to different event representations and user-selected configurations. The pipeline supports feature-based, relational, and particle-set modeling, while exposing computational and modeling trade-offs.
- Pipeline: CoLLM automates construction, training, and evaluation of MLP, GNN, and Transformer classifiers from CSV features produced by generated selection code.Users configure architecture and structural hyperparameters through graphical or terminal interfaces.
- Architecture choices: MLPs provide a fast baseline for fixed-length high-level kinematic features, whereas GNNs model particle relations and Transformers model particle sets with self-attention.The architectures correspond to tabular features, relational event structure, and particle-cloud representations.
- Graph networks: Fully connected graphs maximize information flow but introduce O(n^2) edges for n particles; CoLLM currently uses constant unit edge features as a simple relational baseline.More advanced graph structures are left for future versions.
- Graph networks: GNN message passing aggregates neighboring-node information, and permutation-invariant aggregation makes predictions independent of particle ordering.Global readout combines node representations into an event-level vector for classification.
- Training: Training supports regularization through dropout, batch normalization, and weight decay, with model parameters optimized by binary cross-entropy.AdamW is the default optimizer for Transformers, while Adam is used for MLP and GNN models.
5 Validation and Performance
CoLLM was validated on five simulated collider benchmarks spanning varied final states and selection requirements. Repeated generation exposed mostly consistent code behavior, while remaining differences traced to prompt ambiguity and motivated physics-level validation and manual review.
- Benchmark analyses: Five benchmark analyses tested CoLLM across complementary collider processes, final-state objects, selection criteria, and validation plots.The study used MadGraph5_aMC@NLO, Pythia8, and Delphes3-generated samples, each containing approximately O(104) events.
- Benchmark analyses: The benchmarks covered lepton, jet, photon, and tau selection; b-jet requirements; missing transverse energy; mass reconstruction; angular variables; isolation; and CSV feature export.These capabilities included derived variables such as HT and pT /mγγ ratios, plus cutflow reporting for later deep learning analyses.
- Reproducibility: Deterministic repeated generation under identical inputs and configuration was used to assess reproducibility of the generated analysis code.The repetitions used Llama-3.3-70B-Instruct, the same LHCO file, and up to five code-fixing trials.
- Generated-code behavior: Nine of ten sequential-cut analyses followed Case B, while Case A appeared once; both behaviors were judged correct for the ambiguous prompt.The numerical difference began at the W hadronic selection stage, and cutflow definitions also differed between Case C and Cases A/B.
- Generated-code behavior: Ambiguity in “the two leading jets” produced two different but prompt-consistent implementations of the W hadronic selection.Case A computed the invariant mass after basic η and pT jet cuts, whereas Case B applied the mass calculation to the two leading jets without requiring those η conditions.
- Validation and limitations: The validation histograms provided a physics-level sanity check, but subtle differences across approximately 300-line generated scripts made manual analysis inefficient.The paper recommends using an external LLM to assist with reviewing and validating generated analyses.
6 Discussion and conclusion
CoLLM connects plain-language collider-analysis specifications to automated deep learning pipelines, producing executable, physically meaningful analyses with reproducibility across repeated runs. Its current scope remains limited by LHCO-only code generation, ambiguous prompts, basic graph construction, and user-defined hyperparameters.
- CoLLM connects plain-language specifications to automated deep learning pipelines for collider event data, integrating physics-aware LLM code generation with MLP, GNN, and Transformer architectures.
- Five benchmark processes showed consistent, executable analysis code with physically meaningful outputs, while deterministic decoding and PyFixer supported reproducibility across repeated runs.
- Code generation is restricted to the LHCO format and does not yet support ROOT structures used in experimental analyses.
- Physically ambiguous prompts can produce different implementations and numerical results, although structured input reduces rather than eliminates this ambiguity.
- The current GNN uses fully connected graphs with trivial edge features and does not exploit internal particle correlations, limiting its treatment of complex final states.
- User-defined hyperparameters are not automatically optimized, so classifier performance may be suboptimal for non-standard analyses.
- CoLLM is designed to reduce collider-analysis technical overhead while preserving reproducibility and physical reliability.
A Using an external LLM for code validation
The external-LLM validation workflow addresses subtle differences between generated codes by comparing implementations and helping users identify discrepancies. It uses an LLM to explain differing outputs and guide prompt refinement.
- An external LLM is used to compare subtly different generated Python codes because manual analysis is inefficient.
- The workflow asks the external model to explain differing event counts after W hadronic and delta R selections.
- The reported output may vary with personal settings, so the example illustrates a validation procedure rather than a fixed response.
2. Different jet collections are used in the two codes
The two generated codes differ because they use different jet collections when computing the dijet invariant mass and angular separation. External LLM inspection helps expose these subtle semantic differences and summarize the resulting cutflow logic.
- Different jet collections are used in the two codes: The dominant discrepancy is that the two implementations use different jet collections for subsequent kinematic calculations.
- Different jet collections are used in the two codes: One code explicitly filters jets, then computes mjj and ∆Rjj from the two leading jets in that selected list.
- Different jet collections are used in the two codes: The other code requires at least two jets passing the kinematic criteria but computes mjj and ∆Rjj from the full pT-sorted jet collection without reapplying the selection.
- Cutflow and selection definitions: The cutflow reports cumulative event counts from the initial LHCO sample through lepton, jet, MET, b-jet, W, and angular-separation selections.
- Cutflow and selection definitions: The W hadronic step retains events with 60 GeV < mjj < 100 GeV, using the two highest-pT jets in the entire event without extra pT or η cuts.
- Cutflow and selection definitions: The delta R step requires ∆Rjj < 3.0 for the same two jets, with ∆ϕ wrapped into [−π, π].
- External validation: External LLM inspection can list parameter definitions and functions, helping users extract code semantics, refine prompts, and understand workflow interpretation.