Source-linked AI summary
Towards Standardized Evaluation in Automated Domain Modeling: Introducing a Benchmark
Vasiliy Seibert
TL;DR
Automated domain modeling lacks a standardized benchmark for comparing approaches. This paper introduces a 53-record benchmark with a generation task and deterministic metric, finding that LLM-driven candidates outperform a rule-based baseline across both corpora and all element types.
Problem
Automated domain modeling lacks a public benchmark with a defined task, dataset, and deterministic metric for comparative evaluation.
Method
The benchmark pairs 53 natural-language descriptions with reference PlantUML diagrams, evaluates generated diagrams using per-element similarity scores, and includes rule-based and LLM-driven candidates.
Results
LLM-driven candidates outperform the rule-based baseline on every element across both corpora, with absolute leads of 0.15–0.47.
Takeaways & Limitations
The benchmark provides a common basis for evaluating automated domain-modeling strategies across datasets differing in scale and composition.
Takeaways & Limitations
Results on the 8-record corpus show substantially greater cross-run variability than on the 45-record corpus.
Abstract
from arXiv · showhide
Domain modeling plays an essential role in domain-driven design, capturing essential entities and their relationships within a specific domain. Despite advancements in automated domain modeling, the absence of standardized benchmarks has hindered the comparative assessment of existing approaches. This paper introduces a benchmark designed to address this gap. The benchmark combines the 45-record Golden UML Modelset (Verbruggen et al., 2025) on Zenodo, as distributed by the Text2UML project of Calamo, Mecella, and Snoeck (Calamo et al., 2025), with the 8-record reference archive of Chen et al. (Chen et al., 2023a,b), enabling the evaluation of automated domain modeling approaches across different levels of complexity and scale. Given a natural language description, the task is to generate a corresponding domain model. For each description, a reference domain model is provided as ground truth. A metric is used to compare the generated domain model with the corresponding ground-truth model. To demonstrate the utility of the benchmark, we evaluate multiple automated domain modeling approaches, including heuristic rule-based methods and LLM-driven strategies. In accordance with the FAIR4RS recommendations (Chue Hong et al., 2022), the benchmark is provided as a research artifact to encourage reuse and support future research on automated domain modeling.
1 Introduction
The introduction presents domain modeling as a precise, communicative foundation for software development and motivates standardized evaluation of automated modeling. The paper addresses this gap with a benchmark defining a generation task, a 53-model dataset, evaluation metrics, and a FAIR4RS-aligned research artifact.
- Motivation: Domain models capture relevant entities and relationships in UML class-diagram syntax, supporting precise development foundations and clearer stakeholder communication.UML class diagrams are familiar to software developers and help reduce misunderstandings in requirements discussions.
- Motivation: Automated domain modeling can improve quality and reduce cost, evolving from heuristic rule-based extraction toward approaches relying entirely on Large Language Models.Traditional methods analyze natural-language requirement descriptions with tools such as Stanford CoreNLP and rules for extracting entities and relationships.
- Research gap: Performance measurement for automated domain modeling has been neglected, motivating a benchmark with a defined task, public dataset, and deterministic metric.The introduction frames such a benchmark as a basis for healthy competition and responds to repeated calls for better assessment of generative AI in modeling tasks.
- Contribution: The proposed benchmark converts natural-language domain descriptions into PlantUML class diagrams using 53 domain models, corresponding descriptions, and metrics drawn from the Seibert study.The metric comprises five metrics from four source papers.
- Contribution: The benchmark is released as a research artifact in accordance with FAIR4RS recommendations to encourage reuse.The artifact is available through Zenodo.
2 Related Work
Prior automated domain-modeling studies typically evaluate on private, case-specific datasets, while grading candidate models uses diverse approaches without a community-standard metric. Recent work includes LLM and prompting comparisons with human expert assessment, alongside automated grading tools in education.
- Evaluation datasets: Automated domain-modeling approaches generally use private, case-specific, or industrial datasets rather than a common public reference set.Heuristic rule-based methods report individual evaluations, and no stable public artifact is deposited; adoption of a common reference set is recent.
- Evaluation datasets: Chen et al. compare LLMs and prompting strategies using their own dataset and human expert comparisons to grade candidate models.Their approach is described alongside prior heuristic rule-based methods, which each report separate evaluations.
- Grading metrics: Candidate domain-model grading methods are diverse, and no metric has become a community standard.Education-focused work includes automatic grading from solution diagrams, heuristic and machine-learning grade prediction, and comparisons of LLM-generated scores with teaching-related assessments.
3 Approach
The benchmark evaluates automated domain modeling through data, task, and metric axes. It pairs natural-language descriptions with reference PlantUML class diagrams, evaluates generated diagrams using per-element similarity scores, and reuses rule-based and LLM-driven candidate strategies.
- Task axis: Each candidate strategy converts the natural-language text into a generated PlantUML class diagram for comparison with the reference.The task axis runs every candidate against the description and produces a candidate diagram.
- Data axis: The benchmark comprises 53 records, each pairing natural-language text with a reference PlantUML class diagram from two corpora.The records include a 45-record community-curated corpus and an 8-record human-expert reference archive derived from Chen et al.’s ground truth.
- Metric axis: The deterministic comparator accepts reference and generated PlantUML strings and returns class, attribute, and association similarity scores.The metric interface returns a 3-tuple with each score bounded to [0, 1].
- Candidate strategies: The benchmark reuses three open-source strategies: one deterministic rule-based pipeline and two LLM-driven zero-shot strategies.The rule-based pipeline uses spaCy dependency parsing and heuristic extraction of classes, attributes, and relationships without an LLM call.
4 Results
Across the 53-record DS-1 and DS-2 corpora, both LLM-driven strategies outperform the rule-based baseline on class, attribute, and relationship scores. The LLM strategies achieve comparable absolute performance across datasets, but DS-2 shows substantially greater cross-run variability.
- Datasets and evaluation: 53 records span DS-1’s 45-record corpus and DS-2’s 8-record reference archive, with human-expert reference diagrams for DS-2.The benchmark results report cross-run stability for three candidate strategies across both corpora.
- DS-1 results: On DS-1, both LLM-driven candidates outperform the rule-based baseline on every element.The Chen et al. candidate with GLM-5.1 leads class at 0.746 and attribute at 0.780, while the Calamo et al. candidate with GLM-5.1 leads relationship at 0.701.
- DS-2 results: On DS-2, both LLM-driven candidates again outperform the rule-based baseline on class, attribute, and relationship.The Calamo et al. candidate with GLM-5.1 achieves the best class score of 0.655 and relationship score of 0.689, while the Chen et al. candidate with GLM-5.1 achieves the best attribute score of 0.650.
- Cross-run stability: DS-2 cross-run standard deviations are 4–10× higher than on DS-1, despite comparable absolute LLM performance.The DS-2 LLM leads are comparable to DS-1 in absolute terms, with differences of 0.15–0.47; variability reaches 0.104 for the Calamo et al. candidate with Kimi.