Model Generation with LLMs: From Requirements to UML Sequence Diagrams

• Graphical models can facilitate stakeholder communication and guide system design, but complementing natural-language requirements with models requires significant manual effort.Source: §I · p. 1
• Rule-based NLP approaches to generating UML diagrams require substantial construction and maintenance effort and can be difficult to adapt across contexts.The paper notes that pre-trained LLMs can be used out of the box, potentially reducing tool-building effort.Source: §I · p. 1
• The study targets sequence diagrams, a less-studied behavioral UML model, while prior LLM model-generation studies considered goal models or class diagrams.Source: §I · p. 1
• Three researchers examined ChatGPT diagrams for 28 requirements documents in different formats, including variants simulating requirement smells and evolution.They recorded observations in structured evaluation logs and used thematic analysis to identify issue categories.Source: §I · p. 1
• Generated diagrams score well for understandability, standard compliance, and terminological alignment, but often have completeness and correctness issues, especially with ambiguous or inconsistent requirements.The analysis identified 23 main issue categories.Source: §I · p. 2

• Sequence diagrams represent interactions between system components through function calls and messages.Source: A. Sequence Diagrams · p. 2
• Components are shown with vertical lifelines, while horizontal arrows distinguish function calls with solid lines from messages with dashed lines.Source: A. Sequence Diagrams · p. 2
• Alternative choices and optional steps can be enclosed in boxes, and horizontal lines can separate conceptual blocks.Source: A. Sequence Diagrams · p. 2
• The study generates abstract diagrams with free-form labels rather than pseudo-code, aiming to support stakeholder interaction without requiring programming experience.Source: A. Sequence Diagrams · p. 2

• Model generation maps input requirements into graphical abstractions and can support elicitation, analysis, design, or product-line engineering.Source: B. Related Work · p. 2
• Earlier sequence-diagram generation work used rule-based approaches, including tools that transformed scenario descriptions or use-case specifications into diagrams.One approach handled free-text use cases, while another used constrained natural language and an intermediate representation.Source: B. Related Work · pp. 2–3
• LLM applications in requirements engineering include goal-model and class-diagram generation, while prompt-pattern catalogs address tasks such as classification and traceability.Source: B. Related Work · p. 3
• Prior LLM model-generation studies did not target sequence diagrams and mainly used toy requirements, whereas this study targets industrial specifications across domains and formats.Source: B. Related Work · p. 3

• The study evaluates diagram quality and catalogs issues that emerge when ChatGPT generates sequence diagrams from natural-language requirements.The questions address quality degree and issues potentially associated with input characteristics or ChatGPT limitations.Source: §III · p. 3
• The evaluation focuses on sequence diagrams as complements to requirements for stakeholder communication, rather than as inputs for code generation.Source: §III · p. 3
• The study uses GPT-3.5 through the ChatGPT web application, chosen for its free access and intuitive interface for analysts without API coding skills.Source: §III · p. 3

A. Data Collection

• The collection covered 28 industrial requirements documents from the Lockheed Martin challenges, PURE, and a user-stories dataset.The documents spanned 18 domains and included “shall” requirements, user stories, and use case specifications.Source: A. Data Collection · p. 3
• Variants were devised incrementally by researcher intuition rather than systematically enumerating possible changes, which the authors considered infeasible and creativity-constraining.Evaluation stopped when no additional issues emerged, and the authors acknowledged potential bias from evaluating “shall” requirements first.Source: A. Data Collection · p. 4
• Researchers created 87 variants across 28 subsets, introducing requirement changes and smells such as ambiguity, inconsistency, and incompleteness.Only functional requirements were selected because sequence diagrams represent behavior rather than quality aspects.Source: A. Data Collection · pp. 3–4
• Each variant was prompted once in a separate ChatGPT session using a visualization-generator prompt, with diagrams rendered through PlantText and no prompting iterations.PlantText syntax served as the reference for checking adherence, although the authors note it may not fully comply with UML.Source: A. Data Collection · p. 4
• Two researchers independently assessed completeness, correctness, standard adherence, understandability, and terminological alignment using a five-point ordinal scale.They recorded scores, textual justifications, observed issues, and conversation links in evaluation logs; no manually defined ground truth was used because multiple diagrams can satisfy the same requirements.Source: A. Data Collection · pp. 4–5

B. Data Analysis

• Cross-evaluation of 30 models yielded square-weighted Cohen’s κ = 0.67, indicating substantial agreement between the two evaluators.The researchers also used Wilcoxon signed-rank tests against 3 and Cohen’s d, restricting this assessment to unmodified, smell-free cases.Source: B. Data Analysis · p. 5
• Thematic coding reduced 135 low-level codes to 62, then grouped them into 23 mid-level codes under the evaluation criteria and general issues.A third researcher coded the logs; another researcher revised the codes, and a third inspected the analysis.Source: B. Data Analysis · p. 5
• The analysis did not distinguish requirements formats or systematically trace modifications to issues, leaving those relationships outside the study’s scope.The authors focused instead on common issues that could inspire future research questions.Source: B. Data Analysis · p. 5

A. RQ1: Quality of the Diagrams

• Completeness, understandability, standard adherence, and terminological alignment scored significantly above 3, with medium-to-large effect sizes.These results concern diagrams generated from well-formed requirements.Source: A. RQ1: Quality of the Diagrams · p. 5
• Correctness scores did not significantly differ from 3, indicating that correctness was not adequately fulfilled on average.The reported p-value was greater than 0.05.Source: A. RQ1: Quality of the Diagrams · p. 5
• Completeness scores skewed high but remained imperfect, with several scores of 3 and 4; analysts should check both completeness and correctness.Source: A. RQ1: Quality of the Diagrams · p. 5

Completeness

• Ambiguous requirements can be abstracted into incomplete diagrams, concealing difficult conditions such as unavailable onboard track data behind “Movement Authority Rejected.”Generated diagrams also omit relevant requirements, components, conditions, calls, or messages; a train-radio example skips destination acquisition.Source: B. RQ2: Emerging Issues · p. 6

Correctness and inconsistency

• Conflicting requirements may be hidden by a function call, while the resulting diagram can omit required behavior or represent it incorrectly.In a triple-redundancy system, conflicting selected-value requirements are obscured by “Determine selected value”; another example follows only one conflicting requirement.Source: B. RQ2: Emerging Issues · pp. 6–7

Correctness and structure

• Diagrams can mishandle numeric constraints, interaction order, conceptual structure, or component roles, including ignoring a 10-statistic condition and treating data as an actor.A failure-management example checks only one branch where three are specified; diagrams may also overlook states or cyclic control loops.Source: B. RQ2: Emerging Issues · pp. 6–7

Requirements quality and understandability

• Ambiguous wording can prompt incorrect assignments, while conflicts and unclear or incomplete requirements can make generated behavior difficult to interpret.A passive braking-curve requirement leads ChatGPT to assign calculation to infrastructure rather than the train system.Source: B. RQ2: Emerging Issues · pp. 7–8

Terminology and traceability

• Terminology and traceability can be imperfect: relevant terms may be missing, and tracing notes may omit requirements even when their actions appear in the diagram.Introduced terms can differ from the requirements; such substitutions may be acceptable when they preserve intended meaning.Source: B. RQ2: Emerging Issues · pp. 7–8

Understandability and general issues

• Other issues include misplaced clarifying notes, redundancy or excessive detail, and memory or context effects; output variability may be useful when correctness is ensured.Earlier sessions can influence later outputs or cause requirement modifications to be ignored; missing domain context can also lower quality.Source: B. RQ2: Emerging Issues · p. 8

Requirements Quality and Diagram Quality

• Correctness, followed by completeness, are the main diagram-quality issues; their association with requirements defects needs confirmation through more rigorous experiments.If confirmed, diagram generation could help identify requirements-quality issues.Source: §V · p. 8

Incremental/Interactive Diagram Definition

• Multiple prompt iterations can support self-correction, while analysts may decompose requirements into clear steps before sequential representation.The required prompting strategies may vary across RE contexts, and analysts retain a primary role.Source: §V · p. 9

The Role of Domain/Contextual Knowledge

• Definitions and added context can yield richer, more accurate representations, while probing prompts can reveal when domain knowledge is needed.Within token limits, users can supply context; API users can also fine-tune with context-specific data.Source: §V · p. 9

Improving Understandability

• Clear diagrams may still be incorrect, so explanatory notes and natural-language rationales can help users assess representations and document them for stakeholders.ChatGPT may generate notes and traceability information spontaneously, though these can be imperfect.Source: §V · p. 9

Empirical Research on LLMs

• The study uses an exploratory design positioned between judgment and sample studies because ChatGPT’s behavior cannot be fully controlled across realistic requirements cases.The authors consider this design appropriate for investigating such situations.Source: §V · p. 9

Construct Validity

• Scoring-scale interpretation can be subjective, though shared criterion definitions and substantial inter-rater agreement mitigate this construct-validity threat.The study used five criteria, and the authors note that future work could add more.Source: §VI · p. 9

Internal Validity

• Without a gold standard, assessors judged diagram faithfulness, leaving subjectivity that triangulation and an independent thematic analyst only partially mitigate.The authors also note that memory from prior ChatGPT sessions could have affected results, while treating this as realistic behavior.Source: §VI · p. 9

External Validity

• The exploration is nonsystematic and incomplete, and selected documents could bias evaluation despite coverage of different requirements types.Independent analyses considered different model variants and documents, which the authors say mitigates incomplete exploration.Source: §VI · p. 9

Findings

• Generated diagrams showed terminological consistency, understandability, and standard adherence, but completeness and correctness remained challenging, particularly with ambiguous or inconsistent requirements.The study also observed occasional hallucinations and limitations in contextual understanding and domain-specific knowledge.Source: §VII · p. 10

Implications

• Providing additional context and domain knowledge can be expected to improve ChatGPT’s model-generation capability.Source: §VII · p. 10
• Incremental prompting with a human in the loop to decompose, verify, and refine requirements may enhance the correctness of produced models.Source: §VII · p. 10