Source-linked AI summary
The Belle II Core Software
T. Kuhr, C. Pulvermacher, M. Ritter, T. Hauth, N. Braun
TL;DR
Belle II needs a reliable, extensible software foundation for processing large simulation and experimental data sets. The paper describes core framework components, interfaces, parallel processing, data handling, and quality infrastructure; the resulting framework supported first public physics results two weeks after collision data taking began. It must nevertheless continue adapting to evolving technology and preserving expertise for new developers.
Problem
Belle II generates very large data samples requiring software that supports sophisticated simulation, reconstruction, visualization, and analysis while avoiding production and processing bottlenecks.
Method
The paper presents basf2 and its supporting externals and tools, including Python and Jupyter interfaces, parallel event processing, reproducible random-number handling, and geometry-testing modules.
Results
First public physics results were presented two weeks after collision data taking began, illustrating that the framework fulfilled Belle II requirements for data taking, simulation, reconstruction, and analysis.
Takeaways & Limitations
The resulting framework is described as reliable, easy to use, extensible, and suitable for Belle II data taking, simulation, reconstruction, and analysis.
Takeaways & Limitations
The software must continue accommodating evolving technology and requirements while preserving expertise for new developers.
Abstract
from arXiv · showhide
Modern high-energy physics (HEP) enterprises, such as the Belle II experiment at the KEK laboratory in Japan, create huge amounts of data. Sophisticated algorithms for simulation, reconstruction, visualization, and analysis are required to fully exploit the potential of these data. We describe the core components of the Belle II software that provide the foundation for the development of complex algorithms and their efficient application on large data sets.
1.1 Code Structure
The Belle II software separates experiment-specific code, third-party dependencies, and installation tools into a package-based structure. Its tooling supports versioned installations, development environments, and precompiled or source-built releases, although build-product relocatability remains unresolved.
- The core software is organized into basf2, externals, and tools for experiment code, third-party dependencies, and installation/configuration scripts.
- Belle II-specific code is partitioned into about 40 packages, each managed by one or two librarians.
- Packages use C++ sources and predefined subdirectories for modules, tools, data objects, scripts, data, tests, validation, and examples.
- Relocateability of compiled external software remains an issue despite efforts to make installations independent of their absolute path.
- Installation tools provide precompiled binaries when available and compile requested basf2 or external versions from source otherwise.
- The tooling supports version-specific development and analysis environments, including sparse checkouts and release-linked configuration.
1.2 Basf2 Development Infrastructure and Procedures
Belle II maintains software quality through version control, review, automated testing, validation, and release procedures. These practices aim to provide early feedback, reduce manual burdens, and preserve collaborative code quality.
- Git, Bitbucket pull requests, and Jira support review and discussion of code changes before merging them into the main development branch.
- Access controls allow package librarians and authorized authors to commit directly, while feature and bugfix merges require librarian approval.
- Unit and steering-file tests run through Bamboo, and branches can merge to master only when all tests succeed.
- Buildbot performs compiler checks and daily static-analysis, documentation, geometry, and memory checks while tracking execution time, output size, and memory usage.
- A validation framework compares generated plots with references and previous runs so quality shifters can identify regressions.
- Major releases receive extensive validation, while minor, patch, and light releases reduce validation effort or accelerate analysis-tool distribution.
1.3 Modules, Parameters, and Paths
basf2 processes event sequences through dynamically loaded modules configured and ordered with Python. Module lifecycle methods, runtime parameters, branching paths, and managed logging define the framework’s execution model.
- Events represent collision or cosmic-ray measurements within runs and experiments identified by unsigned experiment, run, and event numbers.
- basf2 executes dynamically loaded C++ or Python modules over collections of events, with selection, configuration, and ordering defined through Python.
- Module lifecycle methods initialize state, handle run boundaries, process events, collect end-of-run information, and terminate after all events.
- Module constructors define descriptions, runtime parameters, and parallel-processing capability flags.
- The Path class stores module execution order, while integer module results support conditional branching and process() starts event processing.
- Framework-managed logs support FATAL, ERROR, WARNING, INFO, and DEBUG levels with configurable global, package, and module verbosity.
1.4 Data Store and I/O
The Data Store provides flexible, named event data and relationships between objects, while ROOT modules persist and retrieve that data. Index files enable selected-event access and augmentation, but sparse reading is not optimal.
- The Data Store exposes mutable named objects and arrays with permanent or event-level durability for module data exchange.
- Weighted many-to-many relations connect objects across arrays, and separate relation storage permits partial event-data removal without corrupting data integrity.
- StoreObjPtr and StoreArray provide type-safe access to Data Store objects while accommodating event-by-event content changes.
- Relations can be filtered, updated, rebuilt, or copied alongside whole or partial Data Store arrays.
- ROOT persistence requires valid dictionaries and writes permanent and event-durable Data Store content into separate TTrees.
- Index files can select events through EventMetaData and access parent files, but ROOT’s TTree structure makes sparse event reading suboptimal.
1.5 Event Data Model
The event data model is flexible and dynamically defined through objects created by processing modules, while EventMetaData is mandatory. Reconstruction outputs are organized into analysis-focused mDST data, with cDST reserved for detailed detector, reconstruction, and calibration studies.
- The Data Store makes no assumptions about the event data model, which is defined dynamically as modules create objects.
- EventMetaData is the only mandatory Data Store component and uniquely identifies events using event, run, and experiment numbers plus a production identifier.
- Raw detector data are converted into detector-specific digit objects, while simulation digitizers create them from Geant4-generated SimHits.
- Reconstruction primarily produces detector-specific objects, while RecoTrack coordinates pattern recognition and track fitting across multiple detectors through GENFIT.
- mDST contains the reconstruction subset needed for general physics analyses, whereas cDST supports detailed detector, reconstruction, and calibration studies.mDST excludes raw data information to limit event size; cDST provides a dedicated format for specialized studies.
2 Central Services
basf2 exposes event-processing workflows through Python steering files and interactive use, using Python’s accessible syntax and ecosystem to support higher-level analysis tools. Its Jupyter integration enables interactive path execution, monitoring, and visualization.
- basf2 provides a Python interface for arranging modules into paths and starting event processing through steering files or interactive use.
- Python’s accessible syntax helps new users deploy the framework, while its third-party packages support higher-level analysis tasks and documentation generation.
- Boost.Python exposes basf2 features, and the basf2 executable adds framework-specific options for versions, modules, and file names.
- Users can write modules in Python for easier customization and rapid prototyping, while most framework modules remain implemented in C++ for compiled-code performance.
- Jupyter integration simplifies creating and processing basf2 module paths through interactive notebooks and the basf2-agnostic hep-ipython-tools project.Calculations can be decoupled, started, and monitored while users continue working in the notebook.
2.2 Parallel Processing
basf2 parallel processing uses forked processes that independently handle complete events while sharing unchanged memory through copy-on-write. It scales near linearly through the core count, but serialization can limit throughput for lightly computational workloads.
- Modern multicore CPUs motivate parallel execution, while shared-memory size and access can bottleneck independent single-core applications.Typical HEP sites have maintained approximately 2–3 GB of memory per core for many years.
- basf2 forks processes so each handles a complete event in an independent address space, avoiding thread-safe data-access requirements.
- Copy-on-write lets parallel jobs share unchanged detector geometry created before forking, reducing memory consumption compared with fully separate processes.
- Scaling remains equal or close to theoretical linear behavior through the number of cores for both smaller e+e− and larger B ¯B reconstruction events.A minor efficiency loss at the core count is attributed to shared resources such as level-3 caches.
- The framework splits paths around non-parallel input and output modules, executing only the middle section in multiple processes and transferring events through transmitter and receiver modules.
- The scheme works best when middle-section computation dominates input, output, and serialization costs; lightweight high-throughput jobs incur a sizable serialization penalty.Up to 20 concurrent processes were verified without input or output becoming a bottleneck for typical Belle II reconstruction jobs and event sizes.
2.3 Random Numbers
Belle II uses a fast, statistically tested xorshift1024* generator throughout its framework and seeds each event from identifying information to preserve independence and reproducibility.
- Large simulated samples require reliable pseudorandom-number generation because Belle II uses them for precise signal and background expectations.A PRNG is deterministic, so knowing its internal state allows subsequent numbers to be reproduced.
- Belle II chose xorshift1024*, a 64-bit PRNG with a 128-byte internal state that passes well-known statistical tests.Its simple implementation operates at high speed and is used from event generation through analysis.
- Each event receives an independent generator state derived from a common seed string and information uniquely identifying that event.This event-specific seeding prevents shared generator state across events.
- A 1024-bit SHAKE256 hash of the seed information is used as the generator seed state to minimize collisions between events.The common seed string may have arbitrary length.
- Passing each event’s small generator state with its data enables reproducibility regardless of the number of worker processes.The state can accompany events during parallel processing.
2.4 Conditions Data
Belle II stores evolving calibration and configuration data in a conditions database, while client interfaces transparently select valid payloads and support local or remote operation.
- Conditions are time-evolving settings or calibrations stored centrally as payloads with intervals of validity and global tags.Published global tags are immutable, while new tags remain modifiable until publication.
- The REST-based Conditions Database identifies payloads by name and revision, verifies integrity with checksums, and serves payloads valid for a run and global tag.Communication uses standard HTTP with XML or JSON.
- A standardized REST API decouples client implementations from database details and supports clients in different programming languages.
- Local and central metadata or payload sources can be combined, allowing operation without internet access or when the central database is unavailable.Local copies are used when available, with server downloads as fallback.
- DBObjPtr and DBArray provide user access to single-object and array payloads, while the framework transparently updates them for the current run.Users can poll for changes or register callbacks, and IntraRunDependency supports updates within a run.
- DBImportObj and DBImportArray let users create, populate, and commit new payloads with user-supplied intervals of validity.A local file-based database supports preparation and validation before uploading payloads to the Conditions Database.
- Figure 4 shows that users mainly interact with DBObjPtr and DBArray or configure sources, while communication with the Conditions Database is handled transparently.
- The b2conditionsdb tool supports listing, creating, modifying, inspecting, downloading, and uploading global-tag configurations.
2.5 Geometry and Magnetic Field
Belle II centralizes geometry and magnetic-field descriptions through conditions payloads while providing XML-based editing, automated overlap checks, and material-budget scans for validation.
- Geometry description: Belle II uses one detailed Geant4-based geometry description for simulation and reconstruction, assembled from sub-detector components.A central service configures each detector-specific top-level component.
- Geometry description: Geometry parameters are supplied through conditions payloads and can also be loaded from joined XML files with unit conversion and material definitions.
- Geometry description: Direct XML geometry creation lets developers edit and test geometry locally before creating and uploading conditions payloads.
- Geometry testing: Automated overlap checks test each volume against its siblings and parent using randomly generated surface points.Checks run nightly and use more samples before major releases or after substantial geometry changes.
- Geometry testing: A material-budget scanner tracks neutral particles through the detector and outputs ROOT histograms by coordinate, material, or detector component.Component-level distributions help visualize changes after updates to volume or material descriptions.
- Magnetic field: The magnetic field is loaded from conditions payloads generated from XML, and analysis jobs can obtain field values without instantiating Geant4 geometry.
- Magnetic field: Field handling supports multiple regional definitions, including constant, 2D radial-symmetric, full 3D, and accelerator-magnet implementations.Overlapping definitions may be summed or resolved by an exclusive declaration.
3 Conclusions
After a decade of development focused on software quality, Belle II reports a reliable, extensible framework supporting its core workflows and early physics output, while continued evolution requires preserving expertise.
- Ten years of software-quality-focused development produced a reliable framework that is easy to use and extend with new or improved algorithms.The framework fulfills requirements for data taking, simulation, reconstruction, and analysis.
- First physics results were presented publicly two weeks after Belle II collision data taking began in Spring 2018.
- The mature and robust core software must continue adapting to evolving technology and requirements, making preservation of expertise important.