Source-linked AI summary
BVR Sim: An Open and High-Throughput Environment for Heterogeneous Air-Combat Reinforcement Learning
Haocheng Sun, Mulai Tan
TL;DR
BVR air combat challenges reinforcement learning with partial observability, long horizons, heterogeneous platforms, and coupled sensing, energy, coordination, and weapon decisions. BVR Sim addresses this with an open environment using a shared tactical interface over aircraft-specific controllers, and reports practical high-throughput execution, cross-aircraft transfer, and compatibility with standard MARL pipelines.
Problem
BVR air combat combines long-horizon, partially observed, coupled decisions involving energy, sensing, coordination, and limited weapons, challenging multi-agent reinforcement learning.
Method
BVR Sim provides an open end-to-end heterogeneous BVR environment with shared tactical actions, aircraft-specific inner-loop controllers, entity observations, dual backends, and MARL adapters.
Results
BVR Sim remains practical through 10-vs-10 scenarios, transfers an F-16-trained policy to four unseen aircraft at a 45.50% mean win rate with controller adaptation, and supports MAPPO and HAPPO end to end.
Takeaways & Limitations
The released environment and workflow provide a foundation for heterogeneous BVR reinforcement-learning studies, reproducible evaluation, and standard CTDE experimentation.
Takeaways & Limitations
BVR Sim is intended for research-oriented tactical learning rather than certification-grade or operational combat modeling, with simplified or incomplete fidelity in several systems and empirical settings.
Abstract
from arXiv · showhide
Beyond-visual-range (BVR) air combat is a challenging reinforcement-learning domain characterized by partial observability, long-horizon decision making, energy management, and limited weapons. We present BVR Sim, an open-source Gymnasium-style environment designed for heterogeneous air-combat reinforcement learning. BVR Sim supports multiple JSBSim aircraft models, including the F-15, F-16, F/A-18, and F-22, with configurable weapons, sensors, controllers, and opponents. A unified tactical action interface specifies desired heading, altitude, speed, and weapon release above aircraft-specific inner-loop controllers, enabling policies to operate across heterogeneous platforms. The environment provides interchangeable Python and accelerated C++ backends, entity-oriented observations, compositional rewards, scripted opponents, replay and visualization, and adapters for multi-agent learning frameworks. At a 0.4-s decision interval, the C++ backend achieves 104 simulated seconds per wall-clock second in 1-vs-1 and remains practical through 10-vs-10 scenarios. A policy trained only on the F-16 transfers without retraining to four unseen aircraft, reaching a 45.5% mean win rate with aircraft-specific controller adaptation. MAPPO and HAPPO experiments further verify end-to-end compatibility with standard multi-agent reinforcement-learning pipelines.
1 Introduction
BVR Sim addresses the coupled, partially observed challenges of BVR air-combat reinforcement learning with an open environment for heterogeneous platforms. Its shared tactical interface separates policy decisions from aircraft-specific stabilization while supporting end-to-end MARL workflows.
- Motivation: BVR air combat combines continuous dynamics with delayed, irreversible decisions under partial information, creating long-horizon coordination and credit-assignment challenges.These include energy preservation, radar-track assessment, finite missile allocation, weapon support, and threat evasion.
- Interface: A shared tactical interface specifies heading, altitude, speed, and fire decisions, while model-specific inner-loop controllers translate them into aircraft actuation.This preserves consistent tactical action semantics across aircraft with substantially different low-level dynamics.
- Environment: BVR Sim is an open end-to-end BVR MARL environment integrating scenario generation, dynamics, sensing, weapon employment, learning interfaces, baselines, and visualization.It is designed as a complete environment rather than a standalone missile model or flight-control module.
- Heterogeneity: Per-unit configuration supports heterogeneous aircraft, dynamics, controllers, sensors, weapons, initial states, and opponent policies within one engagement.Rendering and telemetry are decoupled from Python and accelerated C++ headless backends.
2 Related Work
Prior open air-combat environments established accessible reinforcement-learning benchmarks, while general MARL benchmarks standardized interfaces without modeling BVR’s coupled flight and weapon constraints. BVR Sim combines JSBSim-based dynamics with tactical abstractions, native execution, and adapters for established MARL learners.
- Aircraft combat environments: BVR Gym, LAG, B-ACE, and WUKONG demonstrate the value of open air-combat reinforcement-learning environments but leave a design space between fidelity, throughput, and learning ergonomics.Their capabilities differ in simulator fidelity, task scope, hierarchy, and openness.
- Aircraft combat environments: Unlike LAG’s hierarchical low-level policy option, BVR Sim embeds model-specific flight controllers so tactical policies can train without separately learning basic flight control.Both environments use JSBSim aircraft dynamics, but BVR Sim places the conventional controller inside the simulator.
- MARL benchmarks: General MARL benchmarks standardize evaluation but do not encode BVR’s coupled flight, sensing, energy, and weapon constraints.BVR Sim exports standard spaces and supports established runners such as MAPPO and HAPPO through adapters.
- System design: BVR Sim closes the loop from scenario instantiation through tactical actions, inner-loop control, dynamics, sensing, weapons, observations, rewards, termination, and optional telemetry.Its configurable environment interval is paired with smaller JSBSim integration substeps for numerical flight integration.
4 Heterogeneous Engagement Modeling
BVR Sim models engagements through per-unit aircraft and weapon specifications, allowing mixed aircraft, asymmetric teams, controlled initial states, and multiple learning-control configurations. Its capability comparisons use explicit criteria for BVR tasks, mixed aircraft, and entity-table observations.
- Scenario generation: Each aircraft receives a per-unit specification covering identity, team, initial state, flight-dynamics type, aircraft specification, pylons, and optional scripted opponents.The current C++ mapping includes F-15, F-16, F/A-18, F-4N, AJ/JA-37, and F-22 families.
- Scenario generation: Aircraft models select corresponding controller parameters, while pylon maps independently determine weapon quantity and type; aircraft specifications can also be randomized at reset.This makes heterogeneity affect both flight behavior and weapon loadouts.
- Scenario generation: Opposing formations are generated from separation, spread, and engagement-axis settings, while explicit initial states support controlled experiments and wrappers support one-sided training or self-play.Team sizes are not restricted to symmetric 1-vs-1 encounters.
- Capability criteria: A mixed-aircraft engagement requires distinct flight-dynamics models and controller parameters to coexist within one episode.Different policies, teams, or initial conditions on one common aircraft model do not satisfy this criterion.
- Capability comparison: The native BVR Sim interface uses MultiDiscrete(15, 15, 9, 2), and its entity-table representation assigns fixed-width rows to aircraft or missiles.The comparison defines BVR-class engagement, mixed aircraft, and entity-table criteria explicitly rather than treating any missile or heterogeneous policy setting as sufficient.
5 Learning Interface
BVR Sim exposes a tactical learning interface over aircraft-specific control, with discrete heading, altitude, speed, and fire decisions and fixed-width entity observations. Its reward, termination, and framework APIs support diagnosis and standard MARL integration.
- High-level action space: The default action selects desired heading, altitude, and speed changes plus a binary fire decision, which are consumed by a model-specific flight controller.The action is represented as a discrete tactical abstraction rather than direct actuator commands.
- High-level action space: The first three action indices are normalized to [−1, 1] and scaled to maximum command changes of 45°, 80 m, and 80 m/s.These correspond respectively to heading, altitude, and speed changes.
- High-level action space: Simulator logic selects an eligible tracked target and available weapon when fire is requested, while lower-level fields remain available for learned target and weapon allocation.This keeps the default policy focused on tactical behavior while preserving an extension path for allocation tasks.
- Entity-oriented observation: Each entity row contains 40 features, with rows for self, aircraft, and bounded missile slots and zero padding for absent objects.For Ne allocated entities, the flattened observation dimension is 40Ne.
- Entity-oriented observation: The fixed-width entity schema preserves object boundaries and semantic roles while remaining compatible with a standard Box space and supporting encoders, attention, and masking.Compact and legacy text/vector observations remain available for compatibility.
- Rewards and integration: Rewards are independently weighted and compositional, episodes terminate on engagement conditions, destruction, or a configured horizon, and wrappers adapt the environment to MARL frameworks.Reward breakdowns support diagnosis, while Gymnasium-style APIs and HARL/MARLBenchmark adapters expose standard learning interfaces.
6 Simulation Components
BVR Sim combines model-based aircraft and missile simulation with heterogeneous sensing, control, and weapon configurations. Its matched missile case study provides a qualitative check of guidance behavior rather than statistical or operational validation.
- Aircraft dynamics and control: JSBSim integrates model-specific aircraft dynamics, while tactical commands are converted into normalized control-surface and throttle inputs.The standard flight controller maps heading, altitude, and speed commands to aileron, elevator, rudder, and throttle commands.
- Missiles and stores: The pylon manager tracks mounted, remaining, and committed weapons, and the weapon factory supports AIM-120 and alternative parameterized missile models.The generic missile model includes time-varying mass, thrust, gravity, atmospheric density, Mach-dependent drag and load limits, and command-rate limits.
- Missiles and stores: Missile translational dynamics are integrated with fourth-order Runge–Kutta, retaining major energy effects relevant to long-range intercept feasibility.Variants can use the built-in AIM-120 implementation or data-driven parameter tables through the generic model.
- Missiles and stores: An archived matched case found that changing proportional navigation to a multi-stage guidance law shortened time to hit by 8 s and retained more terminal energy.The study used an AIM-120C5 launched 16.21 NM from a straight-flying F-16 target under high-off-boresight geometry; it was a qualitative sanity check.
- Sensing and opponents: Aircraft maintain enemy, friendly, lock, and missile-threat state, while radar parameters and scripted opponent behaviors contribute to heterogeneous sensing and evaluation.Opponent behaviors include straight-line, random, aggressive, tactical, standoff, and mutual-assured-destruction-style policies.
7 Engineering for Reproducible Research
BVR Sim separates transparent inspection from accelerated execution while keeping a common task abstraction. Configuration, adapters, telemetry, and optional visualization support reproducible experimentation without coupling graphics to headless learning.
- Dual backends: The Python and C++ backends share a Python-level environment family but occupy different points in the transparency–throughput trade-off.The Python backend supports inspection and debugging, while the C++ backend moves core simulation computations into native code through pybind11.
- Configuration and extension: JSON or JSONC configurations vary force composition, stores, geometry, observations, opponents, episode length, and reward weights without changing the training loop.Componentized observation and reward interfaces and custom plugins make ablations explicit in configuration and easier to reproduce.
- Learning-framework adapters: Three adapters expose BVR Sim to MARLBenchmark, HARL, and UHRL while retaining the native multidiscrete tactical action and backend selection.MARLBenchmark and UHRL support denotes implemented interfaces rather than validation of every algorithm in those frameworks.
- Telemetry and visualization: Telemetry exports ACMI files and supports Web, OpenGL, Windows DX11, and experimental UnrealCV visualization paths decoupled from headless learning.Rendering is optional, so high-throughput training does not require graphics.
8 Performance Evaluation
The performance evaluation measures matched headless transitions across engagement sizes and compares Python with C++ execution. Native execution is consistently faster, although the largest-scale result has substantial variance.
- Protocol: The benchmark uses 0.4-s decision intervals, matched F-16 scenarios with six AIM-120C7 missiles per aircraft, and team sizes from 1-vs-1 through 10-vs-10.Each reported point aggregates three repeats of 500 timed transitions after 20 warm-up transitions, excluding construction, warm-up, and reset time.
- Protocol: Steps/s measures complete environment transitions per wall-clock second rather than per-aircraft physics updates.The benchmark script and raw per-repeat measurements are distributed with the repository.
- Results: The C++ backend is faster in every tested configuration, with mean speedups ranging from 2.72× to 6.55×.The comparison exercises complete tactical transitions rather than an empty straight-flight loop.
- Results: The 10-vs-10 C++ result is 55.43 ± 38.71 steps/s, so its speedup should not be interpreted as a stable asymptotic factor.The largest-scale measurements exhibit substantial run-to-run variance, partly because missile launches increase simulated objects and sensing interactions.
9 Multi-Agent Learning Integration
Archived MAPPO and HAPPO runs test BVR Sim as an end-to-end MARL environment on the same 2-vs-2 task. Both traces improve over long training, but their single-seed difference is not treated as an algorithm comparison.
- Experimental setup: MAPPO and HAPPO train two aircraft with centralized training and decentralized execution for approximately 22.5 million environment steps.Both archived HARL runs use the same MultipleCombat-2v2/ShootMissile task.
- Training traces: Figure 3 plots recorded mean episode reward with a trailing smoothing window of approximately one million environment steps.Faint curves show unsmoothed records rather than cross-seed uncertainty.
- Results: Both traces show sustained reward improvement over approximately 22.5 million environment steps, supporting compatibility with distinct on-policy MARL implementations.The extracted records, source hashes, and plotting script are included in the artifact.
- Interpretation: Because the runs use different single seeds, their numerical difference is not interpreted as evidence of relative performance, convergence, or learned coordination.Benchmark-quality comparison requires multiple seeds and frozen-policy evaluation.
10 Cross-Aircraft Policy Transfer
The cross-aircraft experiment tests whether a frozen tactical policy transfers across heterogeneous aircraft when paired with aircraft-specific controllers. PPO-F16 transfers to unseen models, while controller mismatch substantially reduces performance.
- Evaluation setup: 960 episodes per cell evaluate each frozen policy on each of five aircraft in the 1-vs-1 transfer study.PPO-F16 trains only on F-16, whereas PPO-All trains across all five models.
- Results: 45.50% mean win rate: PPO-F16 transfers without retraining to four unseen aircraft with aircraft-specific controller adaptation.Without adaptation, the mean win rate is 13.21%.
- Results: 22.00% versus 1.88%: F/A-18 shows one of the largest changes when replacing default F-16 controller parameters with aircraft-specific adaptation.For F-22, the corresponding comparison is 80.11% versus 0.00%.
- Results: 55.29% overall mean win rate: PPO-All improves from 46.96% to 55.29% with aircraft-specific controller adaptation.This result concerns training over all five aircraft models.
- Baseline: The maintained PPO helper models MultiDiscrete([15,15,9,2]) with four categorical branches and is distributed as an executable integration baseline.Framework-specific tuning and standardized multi-method comparisons remain future work.
11 Discussion and Limitations
BVR Sim is intended for research-oriented tactical learning, with simplified simulation fidelity, constrained learning scope, and limited empirical evidence. The paper identifies extensions toward broader platforms, variable-size policies, offline learning, and richer front ends.
- Scope: BVR Sim targets research-oriented tactical learning rather than certification-grade or operational combat modeling.The scope boundary is explicit in the paper’s stated purpose.
- Simulation fidelity: Simplified public radar and missile parameters, incomplete electronic warfare and environmental effects, and JSBSim model limitations constrain simulation fidelity.The listed omissions include communications, terrain masking, weather, and detailed countermeasures.
- Learning scope: The default tactical interface delegates target and weapon selection to simulator logic, so learned allocation requires retained lower-level fields or an extension.This limits what can be learned directly through the default interface.
- Empirical scope: One run per algorithm without frozen evaluation, aggregate-only transfer evidence, and a single module-level missile case do not support algorithmic ranking.The paper calls for multi-seed frozen MARL evaluation, self-play, explicit coordination analysis, and heterogeneous 5-vs-5 learning.
- Future directions: The per-unit specification can extend to additional platforms, while entity observations support transformer policies and variable-size masking.The discussion also identifies offline RL, imitation learning, and richer digital-twin front ends as directions.
12 Code and Data Availability
The paper releases the simulator, configurations, documentation, scripts, evidence, metadata, and provenance artifacts to support reproducible research. The conclusion positions these materials as a foundation for future evaluation and coordination studies.
- Release: The simulator, configurations, and documentation are available on GitHub under the GPLv3 license.The exact code release corresponding to the preprint is the annotated tag arxiv-v1.
- Reproducibility: The supplementary artifact contains paper-side scripts, configurations, aggregate evidence, and a SHA-256 manifest.Raw benchmark samples include full source identifiers and scenario metadata.
- Artifact contents: Controller-transfer and missile-guidance tables include provenance notes, while extracted HARL records and source hashes support Figure 3.Two legacy physics studies are identified because their per-episode or trajectory records were not recovered.
- Future use: The released configuration, structured observations, baselines, telemetry, and reproducible artifacts provide a foundation for frozen multi-seed evaluation, explicit coordination studies, and self-play.The conclusion frames these materials as support for future research rather than evidence of a superior learning algorithm.