PaperlayerView exact revision on arXiv

Source-linked AI summary

From Metrics to Improvement: A Lifecycle-Aware LLM Feedback Framework for Research Software Quality

Nafis Tanveer Islam, Nafiseh Soveizi, Yutong Li, Zhiming Zhao

arXiv:2608.23118v1cs.SE

TL;DR

The paper addresses limited support for translating research-software quality diagnostics into actionable improvements, especially for Tier 1 notebook-centric workflows. It proposes a lifecycle-aware Quality Model coupled with iterative LLM refinement and finds improvements in selected attributes, alongside multi-objective trade-offs and model-reliability concerns.

  • Problem

    Existing quality tools identify research-software issues but provide limited support for interpreting diagnostics and translating them into actionable improvements.

  • Method

    The framework combines lifecycle-aware quantitative assessment with an iterative LLM-based refinement loop that reuses structured diagnostics for repeated reassessment.

  • Results

    The framework improves specific attributes, particularly code duplication and structural quality, but effects vary across metrics and depend on LLM capability.

  • Takeaways & Limitations

    Metric-driven feedback can guide research-code improvement, but quality optimization must consider interacting attributes rather than a single metric.

  • Takeaways & Limitations

    Some local LLMs produced invalid or incomplete code while receiving favorable static-analysis scores, limiting their use in several prompting comparisons.

Abstract

from arXiv · show

Research software is increasingly central to scientific workflows, yet it is often developed by researchers with limited software engineering expertise. This can lead to quality issues that hinder maintainability, reproducibility, reuse, and sustainability. Existing static analysis tools can identify such issues, but their outputs often require expert interpretation and provide limited support for translating quality assessments into actionable improvements. To address this gap, we propose a lifecycle-aware framework that integrates quantitative software quality assessment with Large Language Model (LLM)-based code refinement. The framework comprises two stages. First, a lifecycle-aware Quality Model is developed from established software quality standards and practitioner requirements. The model defines five quality dimensions and 25 candidate metrics, of which 14 are operationalized using existing analysis tools and custom measurements. Second, the resulting quality diagnostics are used as structured feedback within an iterative LLM-based refinement process, enabling generated improvements to be repeatedly reassessed against the Quality Model. We evaluate the framework on notebook-centric research software using multiple LLMs and compare iterative structured feedback with single-step feedback and unstructured prompting. The results show improvements in specific quality attributes, particularly code duplication and structural quality, while also revealing trade-offs among maintainability, code size, documentation, and complexity. These findings demonstrate the potential of metric-driven LLM feedback for research software quality improvement while highlighting its inherently multi-objective nature \footnote{The source code and experimental data are publicly available at https://github.com/QCDIS/Software_Quality_Control_LLM . }

1 Introduction

Research software, especially Tier 1 notebooks, is essential but often suffers quality problems that hinder reproducibility and reuse. The paper proposes a lifecycle-aware Quality Model linked to iterative LLM refinement to turn quantitative assessment into actionable improvement.

  • Tier 1 research software comprises exploratory scripts or notebooks whose quality affects the scalability, reusability, and sustainability of broader scientific software ecosystems.
  • Poor-quality research software can undermine reproducibility, reuse, transparency, and maintainability through issues including duplication, weak documentation, and inconsistent notebook execution order.
  • The paper addresses how quality in Tier 1 artifacts can be measured and improved by integrating metric-based evaluation with LLM-driven refinement.
  • The proposed framework combines a lifecycle-aware Quality Model with an LLM feedback loop tailored to notebook-centric workflows, bridging quantitative evaluation and automated improvement.
  • The paper defines lifecycle-aware metrics, integrates static-analysis metrics with iterative LLM refinement, and evaluates the approach quantitatively and through a qualitative case study.

2 Related Work

Prior work can detect notebook and software-quality problems or support LLM-based code improvement, but these strands remain insufficiently integrated for lifecycle-aware Tier 1 research software. The paper addresses gaps in integration, lifecycle context, and traceability.

  • Static and Rule-Based Tools: Existing notebook tools often address isolated quality aspects, require researchers to interpret diagnostics and apply fixes, and lack continuous lifecycle-aware improvement support.
  • General software-engineering LLM approaches do not explicitly account for exploratory, evolving, and reproducibility-related characteristics of Tier 1 notebook-centric development.
  • Research-software quality tools detect issues in notebooks, while LLM approaches increasingly assess and improve code, but the two capabilities are rarely integrated for Tier 1 workflows.
  • Research Gaps: The literature identifies limited lifecycle-aware assessment and improvement, despite quality priorities changing across stages such as implementation, publication, deployment, and community feedback.
  • Research Gaps: Existing approaches also provide limited traceability between research-software requirements, measurable diagnostics, broader quality dimensions, lifecycle context, and generated improvements.
  • The proposed methodology responds by combining standards and practitioner perspectives, operationalizing requirements in a Quality Model, and using diagnostics as structured LLM feedback.

3 Methodology

The methodology derives framework requirements from standards and practitioner perspectives, then implements a two-stage lifecycle-aware assessment and LLM-refinement process. A survey of 20 participants identifies practical concerns, lifecycle engagement, and differing quality priorities.

  • The methodology combines established software-quality standards with practitioner perspectives to derive requirements for notebook-centric Tier 1 research software.
  • Framework Design: Stage 1 develops a lifecycle-aware Quality Model for structured quantitative assessment, while Stage 2 uses its metrics as feedback in iterative LLM-driven refinement.
  • Figure 1 presents the workflow from requirement analysis through Quality Model design to LLM-based quality improvement.
  • Practitioner Perspectives: The survey involved 20 participants from scientific, research-software, data, and AI backgrounds with varied programming experience and notebook familiarity.
  • Community Diversity: Reported reuse concerns included insufficient documentation, dependency problems, poor modularization, and missing structured testing, motivating requirements for the framework.
  • Lifecycle Diversity: Participants encountered diverse lifecycle stages, and 40% reported not explicitly thinking about their work in lifecycle terms, supporting lightweight lifecycle-aware assessment.

3.2 Framework Requirements

The framework requirements address lifecycle context, diverse practitioner needs, accessible assessment, and actionable improvement for notebook-centric research software. They are organized into a two-stage architecture linking quantitative assessment with iterative quality refinement.

  • Framework Requirements: Researchers face diverse quality challenges, lifecycle engagement, and priorities, requiring a context-aware and accessible framework.The framework must also transform quality assessments into actionable improvement guidance.
  • Framework Requirements: The validated requirements comprise five functional and six non-functional requirements.Functional requirements cover lifecycle-aware assessment, notebook integration, unified reporting, standards-based feedback, and multiple reporting granularities.
  • Framework Requirements: Non-functional requirements target lightweight, efficient, portable, extensible, privacy-preserving, and robust operation.These requirements are intended for practical research environments.
  • Framework Design: Stage 1 provides consistent lifecycle-aware quality assessment, while Stage 2 uses its results for actionable, iterative quality improvement.The two stages remain connected through a continuous feedback loop involving the researcher.
  • Research Questions: The framework is structured around research questions concerning model design, metric-driven LLM feedback, and practical quality improvement.These questions correspond to assessment, refinement, and evaluation of Tier 1 research software.

4 Stage One: Lifecycle-aware Quality Model

Stage 1 develops a lifecycle-aware Quality Model by translating requirements into quality dimensions and candidate metrics, evaluating their feasibility, and implementing a measurable subset. The resulting diagnostics unify heterogeneous measurements and support subsequent LLM-based improvement.

  • Quality Model Development: Stage 1 translates validated requirements into a measurable model for notebook-centric research software through three sequential activities.The activities are metric definition, feasibility evaluation, and implementation with lifecycle integration.
  • Quality Dimensions: The model defines five dimensions: Maintainability, Security, FAIRness, Functional Suitability, and Sustainability.Together they address structure, correctness, security, reuse, and long-term sustainability.
  • Quality Dimensions: The conceptual model comprises 25 candidate metrics, with Maintainability containing the largest metric set.The other dimensions cover secure reuse, FAIR practices, functional correctness and verification, and dependency and sustainability concerns.
  • Metric Operational Feasibility: Candidate metrics are assessed for tool availability and maintenance, notebook compatibility, and computation reliability.The evaluation uses tools including Pylint, Radon, JSCPD, Bandit, Gitleaks, and HowFairIs, alongside custom scripts.
  • Metric Operational Feasibility: More than half of the candidate metrics are measurable automatically or semi-automatically with existing tooling.Manual or non-operationalizable metrics generally require human judgment, project-specific knowledge, testing infrastructure, reproducibility environments, or defect-tracking data.
  • Executable Quality Model: Implemented metrics are integrated with lifecycle information and a common diagnostic representation.Each diagnostic records the metric name, value, status, interpretation, quality dimension, and lifecycle stage, producing structured guidance for LLM refinement.

5 Stage Two: Using LLM to Enhance Software Quality

Stage 2 converts Stage 1 diagnostics into structured guidance for LLM-based code refinement while retaining researchers as the final decision-makers. The approach aims to suggest higher-quality code across concerns such as maintainability, scalability, and security.

  • LLM-based Quality Improvement: Stage 2 uses diagnostic reports to guide LLM-based refinement of research software quality.Researchers ultimately decide whether to use the resulting software.
  • LLM-based Quality Improvement: The approach supports adaptation of software quality measures such as maintainability, scalability, and security across research domains.Its purpose is to help suggest higher-quality code rather than replace researcher judgment.

5.1 Methodology

The methodology feeds lifecycle-aware diagnostic evidence into an iterative LLM refinement process. It packages project files, metric results, lifecycle context, code excerpts, metadata, and quality guidance so the model can revise affected code units.

  • Data Ingestion: The refinement workflow begins by ingesting project files and attaching Stage 1 diagnostics to each code unit.Each file is paired with metric results, lifecycle stage, and diagnostic messages.
  • Prompting Design: Prompting transforms file-level diagnostics and lifecycle tags into compact, constrained evidence bundles.Bundles combine metric violations, code excerpts, contextual metadata, and applicable quality guidelines.
  • LLM Inference & Program Repair: The LLM produces revised code units that target documentation, duplication, modularity, and other identified quality issues.Guidance is adapted to the current lifecycle stage and its relevant quality concerns.

5.2 System Design

The system closes an iterative loop between lifecycle-aware quantitative assessment and LLM-guided code refinement. A coordinating Quality Control Engine applies preferences and thresholds while diagnostics guide targeted revisions and reassessment.

  • The framework integrates a Quality Model with an AI-guided refinement process in a closed, iterative improvement loop.
  • The Quality Control Engine coordinates quantitative assessment and LLM refinement by managing information flow, preferences, thresholds, and verify–rescore cycles.
  • Three modules cooperate: Quality Control Engine, Quantitative Quality Engine, and LLM-Guided Refinement Engine.
  • The Evaluation Module obtains diagnostics from six tools, applies metric weights, and computes an aggregated quality score.
  • Lifecycle-aware orchestration selects phase-relevant metrics, while tool integration executes analyses and returns normalized diagnostic reports.
  • The refinement engine converts diagnostics, code excerpts, lifecycle context, and guidelines into evidence packages that guide revised code generation and reassessment.

5.3 Experimental Analysis

The experiments use a cost-conscious repository sample to evaluate LLM-based feedback strategies with both paid OpenAI models and local HuggingFace models.

  • The study randomly selected 10 of 277 extracted repositories for feedback experiments because LLM experimentation costs increase linearly with token count.
  • Experiments primarily used OpenAI’s paid LLMs and subsequently evaluated four local HuggingFace LLMs.

5.4 Dataset Preparation

The study prepares notebook-centric research software from environmental and Earth science projects, characterizes baseline quality, and evaluates LLM refinements across multiple metrics and models. Results show structural improvements alongside trade-offs and reliability problems for some local models.

  • Dataset Preparation: The dataset comprises publicly accessible environmental and Earth science Jupyter Notebook projects located using the keywords “ocean” and “forest.”
  • Baseline Analysis: Baseline analysis covers 277 collected repositories before the feedback-based experiments.
  • Baseline Analysis: Most projects contain 744–1,610 lines of code, with some significant outliers.
  • Baseline Analysis: Maintainability scores are concentrated toward the higher end, with mean 77.04 and median 76.55, although some outliers fall below 20.
  • Baseline Analysis: Cyclomatic complexity is generally low, with median 2.31 and mean 3.12, but a few repositories exceed 10.
  • Baseline Analysis: Most repositories score between 5 and 8 on Pylint, with median 5.65 and mean 5.45, indicating room for improvement.
  • Baseline Analysis: A substantial documentation gap remains: 904 of 2,796 repositories have no comments at all.
  • Model Reliability: Smaller local models frequently generated syntactically or structurally invalid code, including incorrect indentation, semicolons, Markdown delimiters, or fully commented-out code.

6 Discussion

The discussion frames quality assessment and improvement as lifecycle-aware, multidimensional, and iterative rather than reducible to one metric. The framework improves selected attributes, but outcomes depend on metric interactions and model reliability.

  • RQ1. Designing a Lifecycle-Aware Quality Model: A practical Quality Model for Tier 1 research software should be multidimensional, operationalizable, and sensitive to lifecycle context.
  • RQ2. Using Quality Metrics for LLM-Based Refinement: Iterative structured feedback can guide improvements in modularization, documentation, and separation of responsibilities more effectively than single-step or unstructured feedback for some models.
  • RQ2. Using Quality Metrics for LLM-Based Refinement: Quality metrics are interconnected, so refinement should update the overall quality profile rather than optimize individual metrics independently.
  • RQ3. Effectiveness of the Feedback-Driven Framework: The framework improves specific attributes, particularly code duplication and code smells, but additional iterations can worsen maintainability, code size, comment density, or other measures.
  • RQ3. Effectiveness of the Feedback-Driven Framework: Effectiveness depends on LLM capability because some local models produced invalid or incomplete code despite favorable static-analysis scores.
  • Limitations of the System: Only 6 of the Quality Model’s 25 metrics were operationalized automatically or semi-automatically in the current implementation.
  • Limitations of the System: The Decision Module reports metric-level improvements and regressions but does not explicitly optimize among competing quality objectives.

7 Conclusion and Future Work

The framework combines a quantitative Quality Model with structured, lifecycle-aware feedback for iterative LLM refinement of Tier-1 research software. Evaluation found improvements in selected quality attributes, but also trade-offs across metrics, underscoring the need to consider overall quality profiles and functional validation.

  • The framework combines a quantitative Quality Model with lifecycle-aware diagnostics to guide iterative LLM-based code refinement.The diagnostics are transformed into structured feedback for repeated refinement.
  • Evaluation found improvements in several quality attributes, particularly code duplication and structural quality, compared with original code and alternative prompting strategies.A qualitative case study also illustrated movement toward more modular and maintainable implementations.
  • Optimizing one quality attribute can negatively affect others, including code size, complexity, and documentation-related measures.The findings characterize software quality improvement as inherently multi-objective.
  • Overall quality profiles should be considered rather than optimizing individual metrics in isolation.
  • The absence of test suites limited systematic validation of the functional correctness of generated code.Future work targets functional validation, broader metric coverage, and more advanced multi-objective optimization.
Loading 2608.23118v1…