PaperlayerView exact revision on arXiv

Source-linked AI summary

SENAI: Towards Software Engineering Native Generative Artificial Intelligence

Mootez Saad, José Antonio Hernández López, Boqi Chen, Neil Ernst, Dániel Varró, Tushar Sharma

arXiv:2503.15282v1cs.SE

TL;DR

Code-generation models are commonly trained on code and judged mainly by functional correctness, while software engineering also demands design principles and practices that support maintainability. The paper proposes incorporating higher-level design knowledge into LLM training and extending evaluation beyond correctness; it concludes that SE grounding holds promise for software quality and technical-debt reduction.

  • Problem

    Code-model development emphasizes functional correctness, although software must also satisfy constraints and practices that support integration, maintainability, and adaptability.

    Source: §1 · p. 1

  • Method

    The paper proposes adding higher-level design and architectural insights to training and using evaluation strategies that examine internalized software-engineering concepts beyond correctness.

    Source: §1 · p. 2, §3.1.2 · pp. 3–4

  • Results

    The conclusion says SE grounding holds promise for enhancing software quality and reducing potential technical debt in real-world applications.

    Source: §4 · p. 5

  • Takeaways & Limitations

    The paper presents grounding code generation in SE principles as essential for moving beyond functional correctness.

    Source: §4 · p. 5

    UML diagrams and architectural decision records are less widely available than raw code, potentially limiting training-data volume and diversity.

    Source: §3.2.3 · p. 4

Abstract

from arXiv · show

Large Language Models have significantly advanced the field of code generation, demonstrating the ability to produce functionally correct code snippets. However, advancements in generative AI for code overlook foundational Software Engineering (SE) principles such as modularity, and single responsibility, and concepts such as cohesion and coupling which are critical for creating maintainable, scalable, and robust software systems. These concepts are missing in pipelines that start with pre-training and end with the evaluation using benchmarks. This vision paper argues for the integration of SE knowledge into LLMs to enhance their capability to understand, analyze, and generate code and other SE artifacts following established SE knowledge. The aim is to propose a new direction where LLMs can move beyond mere functional accuracy to perform generative tasks that require adherence to SE principles and best practices. In addition, given the interactive nature of these conversational models, we propose using Bloom's Taxonomy as a framework to assess the extent to which they internalize SE knowledge. The proposed evaluation framework offers a sound and more comprehensive evaluation technique compared to existing approaches such as linear probing. Software engineering native generative models will not only overcome the shortcomings present in current models but also pave the way for the next generation of generative models capable of handling real-world software engineering.

1 Introduction

Code-generation models often prioritize functional correctness without being trained on abstract software-engineering principles, even though software must meet broader constraints to remain maintainable and adaptable. The paper proposes integrating higher-level design knowledge into training and broadening evaluation beyond correctness.

  • LLMs already generate code and provide programming assistance, including code completions and functions or classes from textual prompts.Source: §1 · p. 1
  • Models trained mainly on source code and evaluated chiefly for functional correctness may miss software-engineering principles needed for code to fit larger systems.Source: §1 · p. 1
  • Software must integrate with existing systems, remain understandable and maintainable, and adapt as requirements change, bugs are fixed, and features are added.Source: §1 · p. 1
  • Prior work reports struggles with design patterns, architectural tactics, inheritance, and encapsulation; the authors suggest poorly structured designs could increase technical debt and maintenance costs.Source: §1 · p. 2
  • The paper proposes integrating higher-level design and architectural insights into training and extending evaluation beyond software correctness.Source: §1 · p. 2

2 The Status Quo

The section surveys conventional code-model pretraining, input representations, and evaluation, including the benchmarks and metrics used to assess code-generation systems.

2 The Status Quo

  • The status-quo overview covers common pretraining practices and systematic evaluation of code language models.Source: §2 · p. 2, §2.2 · p. 2

2.1 Pre-training of LLMs (of Code)

  • Code language models learn code patterns and semantics from source-code datasets, often using next-token prediction to generate code.Source: §2.1.1 · p. 2

2.1.1 Pre-training objectives:

  • CodeT5 combines masked code-span and identifier prediction with code-to-text and text-to-code generation objectives.Fill-in-the-middle training instead predicts a missing code segment between a prefix and suffix, supporting bidirectional context.Source: §2.1.1 · p. 2

2.1.2 Data representation:

  • Representations range from paired natural-language and code inputs to data-flow graphs; most models still operate at function or file level, while newer approaches add repository context.Source: §2.1.2 · p. 2

2.2 Benchmarks and Evaluation

  • Token-based metrics compare generated code with references, whereas execution-based metrics test functional correctness by running code against expected outputs.Source: §2.2.1 · p. 2

2.2.2 Benchmarks:

  • HumanEval, MBPP, and BigCodeBench evaluate code-generation tasks, with BigCodeBench using more complex instructions and diverse function calls than HumanEval-like tasks.Source: §2.2.2 · pp. 2–3

3 Vision: Software Engineering Grounded Language Models

The paper envisions grounding language models in software engineering (SE) principles, since current models and evaluations focus largely on syntactically and functionally correct code. It proposes changes to evaluation and training so models can better reflect SE knowledge and practices.

3.1.1 Benchmarks and metrics.

  • Current benchmarks can reward functionally correct code despite poor design; proposed tasks assess design practices through refactoring and quality metrics such as LCOM and CBO.Source: §3.1.1 · p. 3

3.1 Evaluation Strategy

  • Evaluation should gauge how much SE knowledge models grasp, since linear classifier probes may miss complex, abstract concepts.Source: §3.1.2 · p. 3, §3.1 · p. 3

3.1.2 Model understanding.

  • Bloom’s Taxonomy can structure questions assessing whether models internalize SE concepts, including cohesion and coupling.Source: Table 1 · p. 3, §3.1.2 · p. 4

3.2 Enriching LLMs with se knowledge

  • Training should extend beyond syntactically correct, functionally accurate code to include SE knowledge that supports maintainability, scalability, and sound design.Source: §3.2 · p. 4

3.2.2 Beyond Code and Its Abstractions:

  • Training on UML diagrams alongside code could connect high-level designs with implementations, while architectural decision records could link code to design rationales.Source: §3.2.1 · p. 4, §3.2.2 · p. 4

3.2.1 SE Guided Training:

  • SE-guided training could predict class and object relationships and use reinforcement-learning feedback grounded in SE knowledge to encourage sound architecture.Source: §3.2.1 · p. 4

3.2.3 Potential Challenges:

  • UML diagrams and architectural decision records are less available than raw code, potentially limiting the volume and diversity of training data.Source: §3.2.3 · p. 4

4 Conclusions

The paper calls for grounding language models in SE principles to move code generation beyond functional correctness. It presents this integration as a way to steer models toward maintainable, scalable code aligned with best practices.

4 Conclusions

  • Integrating high-level design insights into language models could steer code toward maintainability, scalability, and best practices, while improving benchmarks and evaluation practices.Source: §4 · p. 5

Related papers

Loading 2503.15282v1…