PaperlayerView exact revision on arXiv

Source-linked AI summary

Is there a future for models in the LLM era?

Marco Calamo, Massimo Mecella, Monique Snoeck

arXiv:2609.35281v1cs.SE

TL;DR

MDE’s conceptual modeling is complex, while LLMs remain limited on full, complex applications, motivating investigation of how the approaches can be combined. The paper introduces ARTHUR, an LLM-powered multi-agent framework that refactors MDE-generated legacy code using requirements and conceptual models. The authors report that rigorous conceptual models improve reliability, cost-effectiveness, and speed compared with relying on legacy code alone, while outcomes depend on the LLM and cost expectations.

  • Problem

    The paper asks whether LLM agents can improve MDE-generated code, addressing the challenge of combining complex conceptual modeling with LLMs’ limitations on full applications.

    Source: §1 · pp. 1–2

  • Method

    ARTHUR uses LLM agents to refactor legacy Java code with textual requirements and conceptual models, and evaluates outputs using Model-Based Testing.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    The authors report that rigorous conceptual models improve reliability, cost-effectiveness, and speed compared with relying on legacy code alone, while the preferred cost-result balance depends on user expectations and LLM choice.

    Source: §5.1 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    The authors conclude that conceptual models can guide LLM agents in refactoring legacy code while maintaining almost complete alignment with original requirements.

    Source: §6 · p. 10

    Iterative refinement may over-optimize code for specific test cases rather than address broader architectural flaws.

    Source: §5.4 · p. 10

Abstract

from arXiv · show

In an era where software development is deeply tied with Large Language Models, does Model Driven Engineering (MDE) still make sense? This raises the question of the extent to which MDE can be successfully combined with an LLM approach to address the downsides of each approach separately. In this paper, we try to answer that question by investigating a central Research Question: Can Agents powered by Large Language Models improve code generated from UML diagrams and text specifications by Model Driven Engineering (MDE) tools? To investigate this problem, we developed a novel LLM-powered Multi-Agentic approach called ARTHUR (Architecture Refactoring Through Hybrid UML Reasoning), a framework that aims at combining the reliability of MDE and the ease of use of LLMs. ARTHUR is designed to refactor legacy Java code produced by traditional, rule-based, MDE code generators from UML diagrams. To asses the answer to our question, we have refactored the legacy code of several projects from our custom dataset crafted for this purpose. \name{} which made it possible to add support for modern frameworks like Spring Boot, while ensuring compliance with Model-Based Testing techniques to verify that the code still corresponds to the initial model's specifications. We then measured the results obtained in terms of time and cost, passing test rate, and \texttt{compile@k}, \texttt{pass@k} and \texttt{pass$^k$} metrics. We also observed the effect of generating code directly from the conceptual model without the refactoring. Our preliminary test results show that MDE could not be more far from retirement, after all.

1 Introduction

MDE can produce successful software but requires difficult conceptual modeling, while LLMs are efficient at code writing yet remain weak on complex applications. This paper asks whether LLM agents can improve MDE-generated code and introduces ARTHUR to explore that combination.

1 Introduction

  • MDE adoption declined despite its success when used correctly, with the authors identifying the intrinsic complexity of crafting conceptual models from client requirements as a key factor.The paper also notes a shift toward script-like programming as a possible contributor to declining adoption.Source: §1 · p. 1
  • LLMs make code writing more efficient and faster, but their problem-solving capabilities remain lackluster for full, complex applications.Source: §1 · p. 1
  • A March 2026 Google Trends review found increasing interest in MDE and MDA for the first time in a decade, possibly as a way to guide LLM generation of complex software.Source: §1 · p. 1, Figure 2 · p. 2
  • The paper asks whether LLM-powered agents can improve code generated from UML diagrams and text specifications by MDE tools.Source: §1 · pp. 1–2
  • ARTHUR combines MDE and LLMs to refactor legacy code using textual requirements, UML diagrams, and existing generated Java code, with Model-Driven Testing used to check conceptual-model compliance.The authors point to Spring Boot and React as target technology stacks.Source: §1 · p. 2

2 Backgroound and Related Works

The background describes UML-based modeling and code-generation tools alongside LLM-era code evaluation metrics. The related-work review finds gaps in rigorous testing, customizable stacks, and evidence about the impact of adding models.

2.1 Background

  • UML is presented as a standard modeling language, while tools such as EMF and Merlin translate conceptual architectures into code, including functional Java applications.Source: §2.1 · p. 2
  • The background describes pass@k as the frequency of passing functional tests within k attempts and compile@k as a measure of syntactical correctness.Source: §2.1 · p. 2

2.2 Related Works

  • The authors searched Scopus and Google Scholar and classified selected studies by target, UML use, language, LLM, technology stack, evaluation, and code availability.Source: §2.2 · p. 2
  • The review found that two of nine selected articles used LLMs to improve MDE-generated code for single tasks, with improvements assessed mainly through test-passing metrics.One approach used a Designer agent to create UML activity diagrams and improve a Coder agent’s pass@1 results; another found that adding requirements improved pass@1.Source: §2.2 · p. 2
  • Prior work combining LLMs and modeling often targets complex applications, but reported studies include single-case evaluations or lack rigorous validation and metrics.The review identifies a broader gap in rigorous testing, customizable technology stacks, and measuring the impact of adding models.Source: §2.2 · p. 3
  • The paper refines its questions to cover support for different languages and technology stacks, Model-Based Testing with compile@k and pass@k, and the added value of model input.Source: §2.2 · pp. 3–4

3 Methodology

The methodology builds a dataset of model-based legacy applications, introduces ARTHUR’s multi-agent refactoring workflow, and evaluates generated code with model-based tests and LLM code-generation metrics. Experiments compare refactoring legacy code with generating from conceptual models and include pilot and adjusted runs.

3.1 Dataset and MDE Code Generator

  • Eight vetted Golden UML cases pair natural-language requirements with conceptual models and generated Java, providing legacy-code and model inputs for refactoring.Merlin’s template-generated applications encode classes, associations, and state transitions but not business logic; TesCaV tests class, transition, and data criteria, while the refactored UI is not tested.Source: §3.1 · pp. 4–5

3.2 ARTHUR Architecture

  • ARTHUR divides refactoring among backend, frontend, and test agents that use tools to inspect and modify code, compile and debug, and run adapted TesCaV tests.Each agent can plan tasks and spawn sub-agents; the test agent reports passed, failed, and skipped tests.Source: §3.2 · p. 5

3.3 Testing Metrics

  • The evaluation tracks generation time, cost, test-passing rate, compile@k, pass@k, and pass^k across repeated generations.The metrics estimate compilation, test compliance, and repeated-generation reliability; pass@k is also calculated per individual TesCaV test for finer granularity.Source: §3.3 · p. 6

3.4 Experimental setup

  • The experiments evaluate refactored applications against TesCaV and compare legacy-code inputs with runs using conceptual models instead.The setup plans eight applications across two input types, for 16 frontend and backend generations, each with a full test run.Source: §3.4 · p. 6

3.4.1 The Pilot Run.

  • The initial pilot generated Java/Spring Boot backends and JavaScript/React frontends, with nearly perfect TesCaV results but time and cost growing more than linearly with case complexity.Generation time and cost, and TesCaV passing rates, are reported in Tables 2 and 3, respectively; Figure 5 breaks pilot cost and execution time down by case complexity.Source: §3.4.1 · p. 6, Figure 5 · p. 7

3.4.2 Adjusted Architecture and Experimental Setup.

  • The adjusted setup tested cheaper backend and frontend models, rejected Codestral’s incomplete results, and selected Kimi K2.5 before comparing legacy-code and conceptual-model inputs.The test agent remained powered by Claude 4.6; the conceptual-model run omitted legacy code as input.Source: §3.4.2 · pp. 6–7

4 Results

Across eight projects, the results compare optimized runs using legacy code or conceptual models and assess functional reliability and test-failure patterns. Conceptual-model input produced stronger test results than legacy-code input in the optimized comparison.

4 Results

  • The evaluation covered eight projects and examined execution time, financial cost, and functional reliability across three differently configured runs.Run 1 used text and Merlin code; Run 2 used an optimized architecture with text and Merlin code; Run 3 used an optimized architecture with text and the conceptual model.Source: §4 · p. 7

4.1 Optimized Runs: Cost-Benefit Analysis

  • Run 3 took 375.38 minutes and cost $36.81, compared with Run 2’s 546.82 minutes and $62.42.The authors report less-than-linear cost growth for Runs 2 and 3, compared with quadratic growth in Run 1.Source: §4.1 · pp. 7–8, Table 4 · p. 8
  • Run 3 achieved a 96.33% overall pass rate, versus 65.75% for Run 2, with 100.00% pass rates in three projects.Run 2’s rates fell to 20.09% for hfte and 41.90% for kinepolis.Source: §4.1 · pp. 7–8, Table 5 · p. 8

4.2 Reliability Metrics For The Three Runs

  • All runs maintained 100.00% Compile@k, while Run 2’s project-level Pass^k consistency was 0.74% and Run 3 matched Run 1’s project-level consistency scores.At the individual-test level, Run 3 reached Pass@5 of 100.00% and Pass^k consistency of 82.94%.Source: §4.2 · p. 8

4.3 Test Failures Broken Down By Category

  • Association-end multiplicity and finite-state transition-pair and loop tests had the highest error percentages across runs.The authors aggregated results across runs because the failure pattern was similar.Source: §4.3 · p. 8

5 Discussion

The discussion finds that ARTHUR can refactor legacy code, but outcomes and practical convenience depend on the LLM and user priorities. It also argues that conceptual-model input can outperform legacy-code input, while noting limits to the evaluation’s generalizability.

5.1 RQ1: Results in terms of time and money spent

  • Flagship models handled the refactoring almost perfectly but required substantial time and money, while smaller models did not fully handle the task’s complexity.The authors describe a hybrid using Claude 4.6 Sonnet for testing and Kimi K2.5 for code generation as their preferred cost-results balance, despite lower pass@k scores.Source: §5.1 · p. 8

5.2 RQ2: Metric Results compared with existing literature

  • ARTHUR matched iEcoreGen’s perfect compile@k scores but had shorter pass@1 results across all three runs; comparisons are not strict because datasets differ.The authors note that TesCaV averages over 100 tests per case versus 30 for iEcoreGen, and ARTHUR’s eight cases versus iEcoreGen’s 20, which contribute to lower pass results.Source: §5.2 · pp. 9–10

5.3 RQ3: Adding Conceptual Models

  • Conceptual-model input led to about 10% more tests passed per project on average, and the authors report similar or better results than legacy-code input in some cases.They suggest that tested attributes, object states, events, and methods are explicit in the model but diluted in the larger legacy code base.Source: §5.3 · p. 10

5.4 Limitations and threats to validity

  • The evaluation covers only eight projects, limiting how readily its findings generalize to larger industrial-scale applications.The authors also say high reliability depends on input quality, particularly a well-done conceptual model.Source: §5.4 · p. 10

6 Conclusion and Future Work

The conclusion presents ARTHUR as a multi-agent framework combining MDE and LLMs to modernize architectures while maintaining alignment with requirements. The authors identify larger-scale evaluation and support for additional languages and frameworks as future work.

6 Conclusion and Future Work

  • ARTHUR uses conceptual models and legacy code to refactor outdated architectures into modern technology stacks while maintaining almost complete alignment with original requirements.The conclusion reports that conceptual models improved reliability, cost-effectiveness, and speed compared with relying on legacy code alone.Source: §6 · p. 10
  • Future work will evaluate ARTHUR on a bigger dataset, including more complex real-world MDE projects.The authors say this would help strengthen confidence in scalability and robustness.Source: §6 · p. 10
  • The authors also plan to extend ARTHUR to Python/Flask or other languages and frameworks.They describe the architecture as prompt-driven and aim to demonstrate its language-agnostic potential.Source: §6 · p. 10
Loading 2609.35281v1…