PaperlayerView exact revision on arXiv

Source-linked AI summary

UML Class Diagram Evaluation and Repair Strategies based on LLMs

Jie Liang, Peng Liang, Chong Wang

arXiv:2608.28800v1cs.SE

TL;DR

This paper addresses limited evidence on how well LLMs generate UML class diagrams and how their structural defects can be repaired. It combines SDMetrics with expert review to evaluate diagrams across multiple dimensions and tests three repair strategies. Compared with expert-crafted benchmarks, generated diagrams contain recurring modeling defects, while the proposed repairs improve performance across evaluated issues, including 85% average correction for core-class identification and 100% repair for design-rule violations.

  • Problem

    Research on LLM application to UML class diagram modeling remains limited, particularly regarding multidimensional quality evaluation and systematic structural-defect repair.

  • Method

    The study compares GPT-4, o1-preview, and DeepSeek-R1 with expert-crafted gold standards using SDMetrics and expert review across size, coupling, inheritance, and design-rule dimensions, then applies three targeted repair strategies.

  • Results

    LLM-generated diagrams show recurring defects relative to expert benchmarks, while proposed remediation improves all evaluated dimensions; design-rule repair reaches 100% and average core-class correction reaches 85%.

  • Takeaways & Limitations

    The case studies provide preliminary evidence that multidimensional evaluation and targeted remediation can improve the quality of LLM-generated UML class diagrams.

  • Takeaways & Limitations

    The three relatively small or medium-sized cases contain 13–28 classes and may not capture the complexity of large-scale systems; industrial requirements and multi-team collaboration were not represented.

Abstract

from arXiv · show

UML class diagrams are a crucial tool for defining the structure of software systems, but designing accurate and comprehensive class diagrams is a challenging task. Traditionally, creating UML models relies on the expertise and experience of professionals. However, with the development of AI technologies, particularly LLMs, new opportunities for software modeling have emerged. Despite this, there has been limited research on the application of LLMs in software modeling, especially in UML class diagram modeling. This study conducts experiments on several typical software system cases. Combining SDMetrics with expert manual review, this paper comprehensively evaluates the practical performance of mainstream LLMs in UML class diagram modeling from multiple dimensions, including size and completeness, relationship correctness, inheritance hierarchy, and design rule compliance. Focusing on typical defects in LLM-generated UML class diagrams, this study reveals that LLMs exhibit uncertainties analogous to human memory. Accordingly, three targeted repair strategies are proposed, including memory reinforcement, external knowledge injection, and detection-guided automated targeted repair. Experimental results obtained from the case studies indicate that (1) compared to expert-crafted class diagrams, LLM-generated UML class diagrams exhibit several issues, such as incomplete identification of key classes, confusion or omissions in relationships, insufficient or absent inheritance relationships, unused classes, and circular dependencies, and (2) after applying the repair methods, all the LLMs show varying degrees of improvement in addressing these issues. The average repair rate for key class identification reaches 85%, the coupling relationship repair rate is 46%, the inheritance relationship repair rate is 69%, while repair rates for unused classes and circular dependencies both reach 100%.

1. Introduction

LLMs create new opportunities for UML class diagram modeling, but research has lacked systematic evaluation and repair strategies. This study evaluates generated diagrams across multiple dimensions and proposes targeted methods to address recurring defects.

  • UML class diagrams are important for representing software architecture, but producing high-quality models requires domain insight and modeling expertise.Traditional expert-led design also faces efficiency bottlenecks during rapid agile iterations.
  • Research on LLM-based UML class diagram generation remains limited, with few systematic evaluation frameworks or quality-enhancement strategies for structural defects.Existing work has largely focused on preliminary validation of individual models.
  • The study evaluates GPT-4, o1-preview, and DeepSeek-R1 against expert-crafted gold standards across size, coupling, inheritance, and design-rule compliance.The framework combines automated assessment with human review.
  • LLM-generated diagrams show recurring errors in key-class identification, relationships, inheritance structures, unused classes, and cyclic dependencies, while repair strategies substantially improve quality.The reported findings compare generated diagrams with expert-crafted benchmarks.
  • The paper proposes external knowledge injection, memory reinforcement, and automated targeted repair to address defects in generated class diagrams.Targeted repair is guided by automated defect detection, while memory reinforcement is motivated by similarities between LLM uncertainty and human forgetting.

2. Related Work

Prior studies established the potential of LLMs for UML modeling but generally examined limited dimensions and lacked comprehensive class-diagram evaluation and structural repair strategies. This paper addresses both gaps.

  • Prior research investigated LLM assistance for multiple UML diagram types and examined element identification and output-format effects.These studies provided an empirical foundation for LLM use in UML modeling.
  • Existing work has not established a comprehensive framework for evaluating class diagram quality across structural dimensions.The literature has predominantly pursued single-dimensional investigations.
  • Quality-improvement research has explored task decomposition and generation strategies, including multi-stage and multi-agent approaches.These approaches target completeness and semantic consistency during modeling.
  • Repair-oriented studies have focused mainly on error induction and surface-level correction, leaving systematic repair of structural defects largely unexplored.The paper positions its repair strategies as a response to this practical gap.

3. Research Design

The study evaluates and repairs LLM-generated UML class diagrams through a sequential experimental process, with the overall workflow presented as a research-process overview.

  • The research process consists of data collection, class-diagram generation, quality evaluation, and defect repair.The experiments are designed to assess diagram quality and remediate identified defects.

3.1. Data Collection

The study selected three cross-domain projects after screening case studies and evaluating their complexity, diagram quality, and requirements readability. The resulting cases range from 13 to 28 classes and have high quality scores.

  • The cases were screened using criteria covering diagram complexity, diagram quality, and requirements readability.Complexity requirements included at least 10 entity classes and 2–3 attributes per class.
  • Three projects were selected: Parking Lot, Online Movie, and Stack Overflow, representing IoT, e-commerce, and knowledge-service contexts.Each project included natural-language requirements and an expert-crafted class diagram.
  • 13 to 28 classes characterize the selected projects, while class diagram quality scores remain at least 4.6.The projects show gradual variation in complexity across the cases.

3.2. Class Diagram Generation

The study generated UML class diagrams from standardized natural-language requirements using three LLMs, then standardized their outputs alongside expert-crafted diagrams for evaluation. A one-shot design preserved each model’s original output characteristics without repeated generation or manual selection.

  • Generation procedure: Class diagrams were generated in two stages: LLMs first received natural-language case requirements, then outputs were standardized for SDMetrics evaluation.The standardized formats included both LLM-generated and expert-crafted diagrams.
  • Models: Three selected models were GPT-4.0, o1-preview, and DeepSeek-R1.The models included two OpenAI models and DeepSeek’s inference model.
  • Experimental control: The experiment used standardized project-requirement prompts under consistent conditions in a one-shot generation paradigm.Each model was instructed to produce a UML class diagram conforming to the specified requirements.
  • Experimental control: Each test case produced one independent generation per LLM without repeated generation, manual filtering, or superior-output selection.This design preserved the models’ original output characteristics and inherent stochasticity under standard inference conditions.
  • Data preparation: Outputs were formatted consistently in Visual Paradigm, exported as XMI, and imported into SDMetrics for quality assessment.The same processing was applied to expert-crafted and LLM-generated diagrams.

3.3. Class Diagram Evaluation

Class diagrams were evaluated across size, relationships, complexity, and design-rule adherence using automated metrics and manual review. The analysis covered class structure, coupling, inheritance depth, and rule violations affecting model quality and maintainability.

  • Evaluation framework: Evaluation combined automated testing tools with manual review across size, relationships, complexity, and design-rule adherence.These four dimensions define the study’s overall class-diagram evaluation framework.
  • Size and completeness: Class counts indicate system size and internal complexity, with too many or too few classes signaling potential decomposition problems.Appropriateness is assessed against functional requirements and expected system size.
  • Size and completeness: Attribute and method counts are examined as indicators of internal complexity and possible Single Responsibility Principle violations.Classes exceeding 10 attributes or 20 methods receive closer examination of responsibility coherence.
  • Relationships: Coupling is quantified by summing EC_Attr, EC_Par, IC_Attr, IC_Par, Dep_Out, and Dep_In for each class.Classes with substantially higher composite values are flagged for manual relationship review.
  • Inheritance hierarchy: Inheritance complexity is assessed with DIT and CLD, which measure distance from a class to the hierarchy root and to a descendant leaf.A hierarchy exceeding three to five levels is subjected to further examination, depending on system context.
  • Design-rule compliance: Rule-based analysis identifies incomplete, incorrect, redundant, or inconsistent elements, including unused classes, circular dependencies, and attribute overrides.These violations are examined for their underlying causes and effects on model quality and system reliability.

3.4. Class Diagram Repair

The paper proposes targeted repairs for deficiencies in automatically generated class diagrams using external knowledge injection, memory reinforcement, and detection-guided remediation. These methods address class identification, relationships, inheritance, and rule violations through structured prompts and analysis.

  • Repair strategy: Targeted repair methods are proposed to improve the accuracy and completeness of automatically generated class diagrams.The methods respond to deficiencies identified through metric-based and qualitative analyses.
  • External knowledge injection: External knowledge injection supplies UML concepts, principles, design patterns, and examples so models can apply domain-specific modeling standards.The knowledge is organized in a structured repository, transformed for model input, and used during diagram analysis and repair.
  • External knowledge injection: The external-knowledge workflow categorizes and identifies required entity classes before generating a PlantUML class diagram.Its prompts cover entity categorization, entity identification, and diagram generation.
  • Detection-guided repair: Detection-guided repair parses diagrams with SDMetrics, checks elements and relationships against built-in rules, and prioritizes detected violations for remediation.A targeted strategy is then formulated and applied to the generated diagram.
  • Memory reinforcement: Memory reinforcement uses repeated accurate information and CoT-based prompts to build a reviewed repository of UML knowledge for reducing modeling uncertainty.The approach draws on reinforcement in human memory and includes professional review of retrieved information.
  • Memory reinforcement: Coupling and inheritance repairs use prompts covering relationship definitions, representations, semantics, class responsibilities, inheritance concepts, and DIT/CLD hazards.Redesign prompts then adjust connections, arrow directions, or inheritance relationships in unresolved diagrams.

4. Results

The results section examines LLM performance against expert-crafted diagrams and evaluates whether the proposed repair strategies mitigate common modeling defects. The questions span four evaluation dimensions and defects involving classes, relationships, inheritance, and design rules.

  • Research questions: RQ1 compares LLM-generated and expert-created class diagrams using SDMetrics and manual inspection across four dimensions.The dimensions are size and completeness, relationship correctness, inheritance hierarchy, and design-rule compliance.
  • Research questions: The experiments therefore frame evaluation as both a comparison with expert modeling and an assessment of repair effectiveness.This sentence synthesizes the two explicitly stated research questions.
  • Research questions: RQ2 assesses whether memory reinforcement, external knowledge injection, and detection-driven targeted repair improve LLM-generated UML class diagrams.The targeted defects include missing critical classes, incorrect or omitted relationships, inadequate or absent inheritance, unused classes, and circular dependencies.

4.1. Answer to RQ1

Across four dimensions, the evaluation finds that LLM-generated UML class diagrams underperform expert-crafted diagrams, with weaknesses varying by model and task. Common problems include incomplete class coverage, high-risk relationship errors, unstable inheritance structures, and design-rule violations.

  • Evaluation overview: The evaluation combines SDMetrics with human assessment across size and completeness, relationship correctness, inheritance hierarchy, and design-rule compliance.
  • Size and completeness: o1-preview covered 93.6% of 47 core classes, compared with 59.6% for GPT-4 and 76.6% for DeepSeek-R1.
  • Size and completeness: Management-class coverage was 0% for GPT-4 and DeepSeek-R1, while o1-preview covered 10% overall; implicit-class coverage reached 71.4% for DeepSeek-R1.
  • Relationship correctness: All evaluated LLMs produced high-risk relationship errors, especially misclassified strong associations, misclassified combinations, and omitted essential associations.
  • Inheritance hierarchy: GPT-4 often omitted inheritance, whereas DeepSeek-R1 could introduce unwarranted multilevel hierarchies, showing inconsistent inheritance modeling.
  • Design-rule compliance: DeepSeek-R1 matched expert-crafted diagrams with zero detected design-rule violations, while GPT-4 recorded the highest weighted total score of 13.

4.2. Answer to RQ2

The proposed repair strategies improve LLM-generated UML class diagrams across core-class identification, coupling relationships, inheritance structures, and design-rule compliance, although effectiveness varies by model and defect type.

  • Repair strategies based on domain-knowledge injection, memory reinforcement, and rule-based targeted repair mitigate missing classes, incorrect relationships, insufficient inheritance, unused classes, and circular dependencies.
  • Core Class Repair: 85% average core-class coverage was achieved after repair, improving from 71% across the evaluated systems.
  • Coupling Relationship Repair: 46% was the average repair rate for coupling relationships, with o1-preview and GPT-4 showing substantial gains while DeepSeek-R1 showed more modest improvement.In the Parking Lot case, o1-preview achieved 78.57%, whereas DeepSeek-R1 achieved 26.67% in Online Movie.
  • Inheritance Relationship Repair: 69% was the average inheritance-group achievement rate across evaluated model–project combinations, with repair outcomes varying substantially by case and model.o1-preview outperformed the other models in reproducing target inheritance structures in the Parking Lot and Online Movie cases.
  • Inheritance Relationship Repair: Inheritance repair reduced DIT and CLD from 2 to 1 in the Stack Overflow case, while incomplete target structures could leave design deficiencies.The reported limitation is that incomplete inheritance repair may leave duplicated payment-related behaviors and increase technical debt and maintenance costs.
  • Design Rule Violation Repair: All three evaluated design-rule repair tasks achieved a 100% repair rate, resolving unused classes, circular dependencies, and attribute-overriding violations.These repairs improved structural clarity, maintainability, and conformance to the evaluated design rules, but did not establish runtime-performance gains.

5. Discussion

LLM-generated UML class diagrams show systematic deficiencies across class coverage, relationships, inheritance, and design-rule compliance, but targeted repair strategies improve quality with model-dependent effectiveness. The results support combining LLM generation with expert engineering judgment and tailoring repair strategies to individual models.

  • Class identification and coverage: LLMs consistently miss implicit and management-oriented classes because they rely mainly on explicit textual cues rather than broader domain and business-logic reasoning.o1-preview reaches 93.6% core-class coverage, but maximum implicit-class coverage is 71.4%, and management-class coverage is 10% only for o1-preview.
  • Relationship modeling: LLMs frequently misclassify lifecycle-dependent relationships, omit required associations, and coordinate relationships less reliably than human experts.The most frequent errors include association-to-composition or aggregation confusion, composition-to-association confusion, and omitted essential associations.
  • Inheritance complexity control: LLM inheritance designs are polarized, ranging from omitted valid generalizations in business scenarios to overly complex hierarchies in knowledge-oriented scenarios.These extremes reflect failure to identify valid generalizations or over-interpretation of semantic relatedness.
  • Design-rule compliance: Design-rule compliance varies substantially across models, with unused classes, circular dependencies, and attribute overriding appearing in generated diagrams.GPT-4 has a weighted cumulative violation score of 13, o1-preview has one redundant-unused-class violation, and DeepSeek-R1 has zero violations.
  • Repair strategies: 85% average core-class coverage follows external knowledge injection, increasing from 71% and reducing omissions caused by insufficient class-identification knowledge.Generic rules still provide limited improvement for implicit and management-related classes, which require scenario-specific engineering experience.
  • Repair strategies: 91.67% is o1-preview’s inheritance-defect repair rate, compared with 66.67% for DeepSeek-R1 and 50% for GPT-4.Memory reinforcement guides activation, risk bounding, diagnosis, and reconstruction to address both missing and excessive inheritance.
  • Repair strategies: Targeted repair improves reliability and controllability by using automated tools for objective error localization and restricting LLMs to localized corrections.The repairs consolidated unused classes, eliminated circular dependencies, and restored inheritance semantic consistency across the case studies.
  • Practical implications: Human–AI collaboration combines LLM generative efficiency with expert engineering judgment, while model-specific tailoring aligns model choice and repair strategy with project requirements.o1-preview is favored for core-class identification, DeepSeek-R1 for strict design-rule conformance, and GPT-4 shows repair potential in selected scenarios.

6. Threats to Validity

The study identifies threats to validity arising from stochastic generation, incomplete evaluation constructs, uncontrolled model parameters, human judgment, limited case and model coverage, and nonindustrial settings. These constraints bound how broadly and confidently the findings can be generalized.

  • Construct validity: LLM stochasticity means identical requirements and prompts may produce different diagrams, although selecting the best of three candidates reduces rather than eliminates this variance.The evaluation also leaves some engineering properties, including extensibility and testability, unquantified.
  • Construct validity: Relationship-error risk estimates rely on weights derived from the research team’s engineering experience and impact analysis rather than industry-wide standards.This weighting choice may bias the estimated risk values.
  • Internal validity: Generation parameters such as temperature and maximum output length were not fully standardized across models, potentially confounding comparisons.Core-class identification and inheritance reasonableness also relied partly on human reviewers.
  • External validity: The three cases contain 13–28 classes and may not capture large-scale system complexity; industrial factors such as evolving requirements and multi-team collaboration were also excluded.The experiments covered three representative LLMs, limiting cross-model coverage.
  • Conclusion validity: Conclusion validity is limited by a dataset of three cases and one best-performing class diagram per LLM per case, despite multidimensional evaluation and repeated trials.The study therefore supports conclusions within its tested scope more directly than broad generalization.

7. Conclusions and Future Work

The study presents an exploratory, multidimensional evaluation of LLM-generated UML class diagrams and tests targeted strategies for repairing their defects. It finds recurring quality problems relative to expert benchmarks, while the proposed remediation strategies improve evaluated dimensions in the tested scenarios and motivate broader future validation.

  • Conclusions: The framework evaluates UML class diagrams across size and completeness, relationship correctness, inheritance hierarchy, and design-rule compliance using SDMetrics and expert manual review.It compares three cross-domain cases and three representative LLMs.
  • Conclusions: The proposed strategies—external knowledge injection, memory reinforcement, and detection-guided targeted repair—achieve measurable improvements across all evaluated dimensions.These strategies are assessed empirically rather than presented only as conceptual proposals.
  • Conclusions: 100% of design-rule violations were repaired, while average correction of core-class identification reached 85% in the tested scenarios.The findings provide preliminary evidence of effectiveness and practical potential within the study scope.
  • Future Work: Future work will integrate domain ontologies and engineering-experience repositories, expand case complexity and model coverage, and evaluate additional UML diagram types.The authors also plan an integrated platform spanning requirements analysis, diagram generation, defect detection, repair, and result export.

CRediT authorship contribution statement

The contribution statement assigns work across conceptualization, investigation, data curation, formal analysis, methodology, supervision, writing, and review.

  • Authorship Contributions: Jie Liang contributed conceptualization, investigation, data curation, formal analysis, and original-draft writing; Peng Liang additionally contributed methodology and supervision; Chong Wang contributed methodology and review and editing.The statement attributes original-draft preparation to Jie Liang and Peng Liang.
Loading 2608.28800v1…