PaperlayerView exact revision on arXiv

Source-linked AI summary

BC-Bench: Evaluating Agentic Engineering in a Domain-Specific Language for ERP

Haoran Sun, Klaus Marius Hansen

arXiv:2608.20851v1cs.SEcs.AI

TL;DR

Agentic engineering has been evaluated extensively for general-purpose programming but remains underexplored in ERP DSLs such as AL, where data, tooling, and execution constraints differ. The paper introduces BC-Bench, a reproducible benchmark of real-world AL tasks, and finds that model differences exceed harness differences while general-purpose gains do not consistently transfer. BC-Bench therefore provides a practical testbed for domain-specific evaluation and agent-tooling research.

  • Problem

    Agentic coding-agent evaluation remains limited for DSLs such as AL, whose scarce resources, specialized tooling, and environment-coupled workflows are not captured by general-purpose benchmarks.

  • Method

    BC-Bench adapts the SWE-Bench collection process to create 101 manually curated tasks from two Microsoft-maintained AL repositories with release-anchored, reproducible environments.

  • Results

    Between-model differences are larger than differences between the two evaluated harnesses, and general-purpose benchmark improvements do not reliably transfer to AL.

  • Takeaways & Limitations

    BC-Bench serves as a practical testbed for evaluating AL coding agents and guiding tooling, skills, and agent-configuration investments.

  • Takeaways & Limitations

    The 101-task dataset covers only single bug fixes from Microsoft-owned repositories and does not represent the full diversity of AL development.

Abstract

from arXiv · show

Agentic engineering systems have shown strong performance on general-purpose benchmarks, yet their effectiveness in enterprise resource planning (ERP) domain-specific languages (DSLs) remains underexplored. We introduce BC-Bench, a benchmark designed to evaluate agentic engineering on real-world tasks in AL, the DSL for Microsoft Dynamics 365 Business Central. BC-Bench comprises 101 manually curated tasks extracted from two Microsoft-owned production repositories, reflecting authentic ERP development workflows. Adapting the SWE-Bench methodology, we address the unique constraints of the AL ecosystem---including limited public resources and complex environment provisioning. Beyond generating functional code, BC-Bench evaluates test generation and supports multimodal problem statements where visual context is commonly present. We evaluate multiple frontier models across two agent harnesses, utilizing multi-run metrics to account for nondeterminism. In the Bug Fixing category, under our evaluated settings, between-model differences in resolution rate are larger than differences between the two evaluated agent harnesses, and improvements reported on general-purpose benchmarks do not consistently transfer to AL. These results highlight the need for domain-specific evaluation.

1. Background

AL is an ERP-focused DSL with environment-coupled execution and domain-specific conventions, creating challenges for evaluating coding agents. BC-Bench addresses these challenges with a manually curated, reproducible benchmark of real-world AL tasks.

  • AL and Business Central: AL implements Business Central business logic through typed objects such as pages, codeunits, and database-backed tables.AL code compiles to .NET assemblies deployed into a live Business Central environment.
  • AL and Business Central: AL tests require a live Business Central environment, while triggers, validation, permissions, and domain-specific libraries distinguish AL from general-purpose languages.These characteristics tightly couple code and testing to the target ERP environment.
  • Evaluation challenges: Public resources are scarce for ERP DSL benchmark construction, requiring substantial manual effort to curate realistic and diverse datasets.The paper contrasts large public repositories for Python and C# with far fewer MIT-licensed repositories for ABAP and AL, while noting that counts vary over time.
  • Evaluation challenges: DSL ecosystems also depend on specialized tooling and infrastructure, making existing benchmarking frameworks difficult to adapt.Business Central additionally has a large, extensible partner ecosystem with thousands of applications.
  • BC-Bench: BC-Bench contains 101 manually curated real-world tasks from the private NAV and public BCApps production repositories.The benchmark is designed for coding-agent evaluation in Business Central and AL.
  • BC-Bench: BC-Bench supports reproducible experimentation with agent configurations and runs directly in GitHub Actions without additional implementation.Its contributions include a benchmark, a production-repository dataset, and an AL-specific evaluation harness.

2. Related Work

Existing coding-agent benchmarks robustly evaluate general-purpose programming, but they do not capture the data, tooling, and execution constraints of DSLs. AL-specific efforts remain limited and generally do not target repository-level engineering workflows.

  • General-purpose benchmarks: SWE-Bench established real-world issue evaluation for Python, followed by multimodal, live-environment, realism, and terminal-oriented extensions.The cited benchmarks broaden languages, visual inputs, environment setup, task realism, or interactive evaluation.
  • DSL evaluation: General-purpose benchmarks do not adequately represent DSL constraints, including scarce data, limited tooling, and domain-specific execution workflows.Hardware-description-language benchmarks illustrate domain-specific specifications and verification workflows where language models still face performance gaps.
  • ERP benchmarks: ERP benchmarking in SAP and ABAP faces similar limitations because data availability and infrastructure are specialized.These constraints parallel the evaluation challenges identified for AL.
  • AL benchmarks: AL benchmarking remains limited because proprietary codebases, domain-specific abstractions, and tightly coupled business logic are poorly represented by existing benchmarks.CentralGauge evaluates AL proficiency across language aspects such as data modeling and event patterns, but not repository-level engineering tasks or real-world problems.

3. Methods

BC-Bench adapts SWE-Bench to evaluate AL engineering agents through curated production tasks, two task categories, reproducible environments, and multi-run metrics. Its design addresses AL-specific constraints while supporting practical agent configurations and tooling.

  • Execution environment: The benchmark tracks harness versions and standardized release-based environments, while default agents access AL code as plain text and selected experiments expose AL MCP tooling.GitHub Actions and containerized release versions support reproducibility, and AL MCP can expose development-environment capabilities such as compilation.
  • Dataset construction: BC-Bench uses a five-stage pipeline to curate tasks from two Microsoft-maintained AL repositories and validate their reproducibility.The process includes repository selection, attribute and manual filtering, bug-PR extraction, and execution filtering.
  • Dataset and categories: The final dataset covers Bug Fixing and Test Generation across broad functional areas, with most tasks drawn from BaseApp’s more than 2 million lines of code.Inventory, Finance, and Sales are among the concentrated core business domains, while specialized applications provide additional coverage.
  • Evaluation categories: Bug Fixing asks agents to generate a functional patch from a problem statement and codebase snapshot, including linked images when available.This setup follows SWE-Bench while accommodating multimodal problem statements.
  • Evaluation categories: Test Generation asks agents to create a test that fails on the base commit and passes after the gold patch is applied.The gold patch is provided as unstaged changes alongside the problem statement and codebase.
  • Evaluation metrics: Evaluations use five independent runs per task, reporting mean resolution rate, 95% confidence intervals, pass^5, and duration to capture stochasticity and practical latency.Resolution requires successful patch validation for Bug Fixing or the specified fail/pass behavior for Test Generation; duration excludes post-submission validation.

4. Results

Across BC-Bench results, model choice shows stronger effects than harness choice in matched Bug Fixing comparisons, while general-purpose benchmark gains do not consistently transfer to AL. Performance also varies with benchmark version, task complexity, functional area, and failure mode.

  • Evaluation setup: Five-run evaluations require caution because benchmark versions differ across configurations; within-version comparisons support stronger conclusions.Cross-version rankings are not definitive.
  • Model vs Agent Harness Effect: 5.3 percentage points and 11.9 percentage points: claude-opus-4.6 improves mean resolution rate and pass^5 over claudeopus-4.5 under matched Bug Fixing conditions.The comparison uses GitHub Copilot, benchmark version 0.2.0, and Bug Fixing.
  • Model vs Agent Harness Effect: Model comparisons are statistically significant, whereas matched Claude Code versus GitHub Copilot comparisons are not significant at the 0.05 threshold.These tests use paired differences across the same 101 tasks with five runs per task.
  • Model vs Agent Harness Effect: Model choice appears more important than harness choice in the evaluated Bug Fixing settings, but this conclusion rests on only four matched comparisons.The authors caution that the result may not generalize to additional harnesses or categories.
  • Cross-Benchmark Comparison: GPT-5.3-codex shows no statistically significant Test Generation advantage over GPT-5.2-codex despite improving on SWE-Bench Pro.This comparison fixes GitHub Copilot and the benchmark version.
  • Task Characteristics: More than 20 percentage points and more than 25 percentage points: accuracy drops for multi-file patches and patches exceeding 10 LoC, respectively.These patterns indicate difficulty with more complex changes.
  • Task Characteristics: Shopify and Warehouse tasks have lower resolution rates, while GPT-5.2 performs better on Manufacturing tasks; sparse areas make these results diagnostic.The authors suggest domain familiarity may affect resolution rate alongside task complexity.
  • Task Characteristics: Tasks with more images have higher resolution rates, but image count correlates with problem-statement length, making textual richness a competing explanation.The passage frames the visual-content interpretation as a hypothesis rather than a demonstrated cause.

5. Discussion

BC-Bench is presented as a practical testbed for real-world AL engineering, while the discussion identifies performance patterns, failure modes, and benchmark limitations that motivate further experimentation.

  • BC-Bench is intended as a practical testbed for guiding tools, skills, and agent configurations, not only as a leaderboard.Business Central engineering teams use it to prioritize tooling investments for coding agents.
  • Between-model differences exceed differences between the two evaluated agent harnesses, with no statistically significant matched-harness effect.GPT-5.3-codex also does not outperform its predecessor on BC-Bench despite gains on SWE-Bench Pro.
  • Unsuccessful trials are dominated by localization errors and semantically wrong solutions rather than build failures.The analysis therefore points toward repository navigation, context retrieval, and test feedback as targets for controlled improvements.
  • Limitations: The dataset’s 101 tasks from Microsoft-owned repositories do not represent the full diversity of AL development and cover only single bug fixes.Feature development, upgrades, localization, performance tuning, and partner-specific applications remain unexplored.
  • Limitations: By default, agents interact with code as plain text without AL-specific tooling such as compilation, unlike engineers working in Visual Studio Code.
  • Limitations: Passing tests does not necessarily guarantee solution correctness, and functional correctness alone does not characterize a high-quality fix.Future evaluation could include code review, security risks, duration, and token consumption.
  • Future Work: Future work proposes controlled experiments with AL-specific tooling, including compilation, symbol search, and go-to definition.BC-Bench enables empirical evaluation of whether these interventions yield meaningful gains.
  • Future Work: The static single-bug dataset creates contamination risk after release, motivating continuously updated or dynamically generated benchmarks.

A. Version Anchoring and Base-Commit Alignment

BC-Bench anchors tasks to stable Business Central minor releases rather than ephemeral CI builds, then aligns base commits to maintain compatibility while monitoring hotfix drift.

  • Business Central major versions introduce platform changes biannually, while minor releases provide incremental updates such as bug fixes and stability improvements.
  • BC-Bench anchors each task to a specific minor release to improve long-term reproducibility when benchmarks are re-executed.
  • Release anchoring can diverge from original development history, so base-commit alignment selects an earlier compatible commit within the release.
  • Hotfix backports may introduce inconsistencies, prompting weekly reruns of execution filtering to keep anchored tasks valid.

B. Version History

BC-Bench’s version history records successive updates to prompts, supported models and tools, environment behavior, repository setup, and dataset handling.

  • Version 0.1.0 curated 101 tasks for Bug Fixing and Test Generation, with all tasks passing the five dataset-construction stages.
  • Version 0.2.0 clarified problem statements, fixed unreachable links, and refined prompts so agents ignore localization changes.
  • Versions 0.2.1 and 0.2.2 added support for gpt-5.3-codex and claude-sonnet-4.6, respectively.
  • Version 0.3.0 made the Test Generation prompt explicitly require creating a new test case.
  • Version 0.3.2 updated altool to fix bugs and add tools, while version 0.4.0 improved settings, sparse checkouts, PNG conversion, timeouts, and altool integration.

C. Manual Labeling of Wrong Solution Failures

The manual labeling process audits unique Wrong Solution tasks by inspecting problem statements, gold patches, generated variants, and failing tests.

  • The appendix documents the manual labeling protocol and task-level audit table behind Table 13.
  • The 22 unique Wrong Solution tasks were labeled once each because repeated runs often produced related or identical patches.The audit used specified GitHub Copilot, model, benchmark-version, and GitHub Actions outputs.
  • Table 14 reports task-level manual labels and exact generated-diff variants for the Wrong Solution failure mode.
Loading 2608.20851v1…