Software Engineering LLMs: Why We Need Them and How Software Engineering Helps Build Them

I. INTRODUCTION

• Refactoring requires improving software structure while preserving behavior, and includes repository-level, multi-file operations across 13 categories.The paper identifies compatibility, dependency management, removal of unnecessary code, and adaptation to changing requirements as engineering work beyond code generation.Source: §I · p. 1
• GPT-5.2 scored 62% and GLM-5.1 scored 51% overall on RefactorBench, yet both failed every Extract Class task.GPT-5.2 also failed every Move Class and Introduce Parameter Object task, illustrating uneven performance across established engineering operations.Source: §I · pp. 1–2, Figure 1 · p. 1
• Capability-focused training targets a foundational engineering capability in the expectation that it can strengthen related tasks and workflows.Refactoring was selected because it exercises code localization, dependency reasoning, coordinated modification, and behavior preservation.Source: §I · p. 2
• RefactorForge uses refactoring preconditions, program analysis, IDE tools, and structural and regression tests to generate and verify training trajectories.The study starts with seven bounded, tool-verifiable refactoring types and treats the work as an industrial case study, not a new training algorithm.Source: §I · p. 2
• 18.00 percentage points on RefactorBench and 7.06 points on SWE-Bench ProMax: fine-tuning on verified RefactorForge trajectories improved the targeted capability and a related issue-resolution task.The gains extended to SWE-Bench Verified (+4.97 points) and SWE-Bench Pro (+1.23 points); average scores were higher under all four evaluated harnesses, while greenfield repository generation improved under two harnesses and regressed under two.Source: §I · p. 2
• The authors argue that SE should shape what coding LLMs learn and contribute methods for their training, development, maintenance, and capability definition.Their industrial case study also surfaced data-construction losses, verification limits on capability coverage, and the need to assess performance across workloads and agent harnesses.Source: §I · p. 2

A. Industrial Background

• The team maintains a 92B-parameter coding LLM serving more than 100,000 developers across heterogeneous repositories, workflows, and systems.Its capabilities must support diverse engineering workloads through different coding agents.Source: A. Industrial Background · p. 3
• Full post-training runs follow scheduled cycles because they require substantial computing resources and finite data budgets, so teams assess candidate datasets through incremental fine-tuning between cycles.Evaluation checks targeted capability improvement, transfer without unacceptable regressions, and persistence across agent harnesses.Source: A. Industrial Background · p. 3

B. Motivating Study: Tool-Guided Refactoring Across Repositories

• Three of five tool-guided refactoring runs exhausted their budgets before completion despite consuming nearly 99 hours, 2.18 billion tokens, and US$1,500.The unfinished runs improved Code Health by 1.12% to 7.22%; completed Aiomonitor improved by 5.86%, while Dateutil gained 1.76 points but remained below other repositories’ initial scores.Source: B. Motivating Study: Tool-Guided… · p. 3
• The modest, uneven refactoring gains motivated training the coding LLM’s underlying refactoring capability rather than relying on open-ended agent budgets or repository-specific playbooks.The motivating runs used a common procedure across five repositories, retaining changes only when project tests passed and Code Health improved.Source: B. Motivating Study: Tool-Guided… · p. 3

III. REFACTORFORGE: FROM SE KNOWLEDGE TO VERIFIED TRAINING DATA

• RefactorForge constructs executable tasks from controlled repository transformations, has a teacher agent attempt them without reference patches or hidden tests, and retains only trajectories that pass specified checks.Each retained trajectory records the task instruction, tool calls, repository observations, and code edits for supervised fine-tuning.Source: §III · p. 3
• The pipeline selects repository-level operations with checkable eligibility conditions and structural objectives, using fail-to-pass tests and pass-to-pass regression tests to verify outcomes.Figure 2 summarizes the twelve-step pipeline across four phases.Source: §III · p. 3, A. Selecting Refactoring Types · p. 4, Figure 2 · p. 4

A. Selecting Refactoring Types

• RefactorForge supports seven types across Rename, Move, and Remove, using program analysis and tools such as PyCharm or Ruff to identify candidates and apply transformations.The selected families draw on evidence about common rename and move practices, persistent unused methods, and coding models’ reluctance to remove code.Source: A. Selecting Refactoring Types · p. 4
• 96.4% of generated tasks are renames, and the pilot targets correct execution of specified transformations rather than judging whether they improve software design.Coverage is limited to operations established tools can apply and the pipeline can verify; the approach requires repository-level changes coordinating definitions and references across files.Source: A. Selecting Refactoring Types · p. 4

B. Preparing Reproducible Repository Environments (Phase I)

• The pipeline derives installation recipes from package metadata, applies deterministic repairs first, then uses bounded LLM assistance for unresolved failures and validates and caches successful repairs.Projects have limited build-and-test attempts and repair requests; those with unresolved failures are rejected.Source: B. Preparing Reproducible Repository… · p. 5
• Reproducible-build and regression-testing requirements restrict the training pool because repositories without reproducible environments are excluded.The phase selects versioned PyPI projects and records their repository commit to fix the project version.Source: B. Preparing Reproducible Repository… · p. 5
• RefactorForge accepts an environment only after installation and test collection succeed, at least one test passes, and no tests fail or error; it freezes and replays the recipe to verify reproducibility.Passing tests and baseline execution time are recorded, and the passing tests become the later pass-to-pass regression suite.Source: B. Preparing Reproducible Repository… · p. 5
• 306 of 729 assessed projects met final acceptance criteria (42.0%), after 614 built successfully and 476 reached test execution.Environment construction cost $11.38 in LLM expenditure, or 3.72 cents per accepted recipe.Source: B. Preparing Reproducible Repository… · p. 5

C. Constructing Reference Refactorings (Phase II)

• RefactorForge defines eligibility conditions and structural procedures for each refactoring, using program analysis and PyCharm or Ruff to apply supported transformations.LLM assistance is bounded to parameters such as alternative identifiers, not code changes; this avoids dependence on mined developer patches but limits coverage to transformations supported by the selected tools and checks.Source: C. Constructing Reference Refactorings… · p. 6
• Program analysis selects eligible symbols and records their locations, names, and usage metadata; Remove Unused Function candidates must be private, undecorated top-level functions without resolved usages.Unused-import removal relies on Ruff diagnostics restricted to from ... import ... statements.Source: C. Constructing Reference Refactorings… · p. 6
• For Move Function, RefactorForge selects a plausible destination using resolved usage counts, but the heuristic does not guarantee an architecturally optimal module.The destination must have at least two usages and more than the source module; ties are resolved by file-path order, and conflicting definitions or imports are rejected.Source: C. Constructing Reference Refactorings… · p. 6
• PyCharm APIs construct supported rename and move patches, while Ruff removes unused imports; the original repository is restored after each candidate.The approach uses refactoring semantics rather than unconstrained LLM-generated changes, addressing the need for independently checkable reference transformations.Source: C. Constructing Reference Refactorings… · p. 6
• At the 95th percentile, reference patches changed six files, spanned 17 diff hunks, and involved 56 lines of patch churn.The median patch changed two files, spanned four hunks, and involved eight lines of churn.Source: C. Constructing Reference Refactorings… · p. 6
• Reference patches are accepted only when operation-specific structural checks pass and all project tests that passed before the transformation still pass.For Move Function, checks verify the definition's destination and removal from its source, while guarding unrelated definitions and duplicate imports.Source: C. Constructing Reference Refactorings… · p. 6

D. Constructing Verifiable Training Tasks (Phase III)

• Refactoring tasks require both structural checks and behavioral validation because passing project tests alone does not establish that the requested transformation occurred.RefactorForge derives structural F2P tests from the reference transformation and reuses previously passing project tests for regression checking.Source: D. Constructing Verifiable Training… · p. 6
• Task instructions name the required symbols but omit file paths and patch contents, and unsupported details or raw patch syntax trigger rejection and revision.GLM 5.3 generates the brief developer-facing instruction from the refactoring type, symbol metadata, and validated reference patch.Source: D. Constructing Verifiable Training… · p. 7
• Structural F2P tests are retained only when they fail on the original repository and pass after the reference patch is applied.The tests check operation-specific changes to names, definitions, references, and imports; validated tasks are packaged with the pinned project, Docker environment, setup, instruction, and F2P and P2P tests.Source: D. Constructing Verifiable Training… · p. 7
• 5,704 executable tasks were produced across seven refactoring types.Source: D. Constructing Verifiable Training… · p. 7

E. Selecting Verified Training Trajectories (Phase IV)

• A teacher trajectory is retained only if its final solution satisfies both structural and behavioral requirements.The approach applies established post-training data collection practice, using correctness criteria defined in preceding phases rather than adding a new SE-specific mechanism.Source: E. Selecting Verified Training… · p. 7
• 4,889 of 5,704 tasks (85.71%) yielded successful teacher trajectories after GLM 5.3 solutions passed both F2P and P2P tests.The teacher received the task instruction and original repository, but neither the reference solution nor hidden grading tests.Source: E. Selecting Verified Training… · p. 7
• After filtering malformed or incomplete conversations, repetition, garbled text, restricted terms, non-English characters, and invalid multi-turn structure, 4,572 trajectories remained.These filters assess the trajectory rather than the final code.Source: E. Selecting Verified Training… · p. 7

IV. INDUSTRIAL CASE STUDY AND FINDINGS

• The intervention is assessed for targeted capability improvement, transfer to related tasks with acceptable effects on other capabilities, and persistence across agent harnesses.Source: §IV · p. 7

A. Study Design

• Baseline-92B was incrementally fine-tuned on 4,572 verified trajectories totaling 72.51 million tokens, or 0.46% of the preceding code-related allocation, to produce Refact-92B.The trajectories covered seven refactoring types across Rename, Move, and Remove families; evaluation reports Avg. @3 and percentage-point differences, with paired McNemar tests on each model's highest-scoring run.Source: A. Study Design · p. 7
• Targeted refactoring was evaluated on RefactorBench's 100-task Python Lazy set and SWE-Bench ProMax's 170 refactoring tasks across seven languages, using OpenCode.Source: A. Study Design · p. 8
• Transfer was evaluated on SWE-Bench Verified and SWE-Bench Pro for issue resolution, and NL2Repo-Bench for generating complete installable Python repositories from requirements.Source: A. Study Design · p. 8
• Code-smell comparisons were restricted to Python SWE-Bench Pro tasks both models resolved and patch pairs PyExamine could analyze.Source: A. Study Design · p. 8
• Cross-harness persistence was assessed on SWE issue-resolution and greenfield benchmarks using OpenCode, Claude Code, OpenHands, and Mini-SWE-Agent.Source: A. Study Design · p. 8

Refactoring performance

• 18.00 pp on RefactorBench and 7.06 pp on SWE-Bench ProMax: Refact-92B outperformed Baseline-92B on both benchmarks.Success rates rose from 43.00% to 61.00% on RefactorBench and from 51.18% to 58.24% on ProMax.Source: B. Findings · p. 8

Refactoring categories

• Refact-92B outperformed Baseline-92B in seven of thirteen refactoring categories and completed tasks in eleven, despite training on only seven refactoring types.Renames made up 96.4% of training tasks; Introduce Parameter Object success rose from 0% to 75.0%, and categories below 50% success fell from seven to five.Source: B. Findings · p. 8, Figure 3 · p. 8

Transfer to related SE tasks

• 4.97 pp on SWE-Bench Verified and 1.2 pp on SWE-Bench Pro: Refact-92B outperformed Baseline-92B under OpenCode.These benchmarks test related repository issue-resolution tasks beyond the refactorings used for training.Source: B. Findings · p. 8

Greenfield repository generation

• Refact-92B improved over Baseline-92B on NL2Repo-Bench under Claude Code and OpenHands, but declined under Mini-SWE-Agent and OpenCode.The mixed results contrast with the consistent improvements on related SWE issue-resolution tasks.Source: B. Findings · pp. 8–9

Code quality

• Refact-92B introduced 6.9% fewer detected code smells than Baseline-92B across the 111 analyzed patch pairs both models resolved.PyExamine detected 54 introduced smells in Refact-92B patches and 58 in Baseline-92B patches.Source: B. Findings · p. 9

Performance across harnesses

• Refact-92B outperformed Baseline-92B in all eight benchmark-harness combinations across four agent harnesses.Improvements ranged from 1.23 to 6.57 pp on SWE-Bench Pro and 1.91 to 7.10 pp on SWE-Bench Verified; the largest gains on both benchmarks occurred outside OpenCode.Source: B. Findings · p. 9, Figure 4 · p. 9

• The section reports lessons learned from developing RefactorForge to construct capability-focused training data.Source: §V · p. 9
• The lessons address evaluation of capability-focused training data's effects on a coding LLM.Source: §V · p. 9
• The coding LLM is designed to support diverse workloads at industrial scale.Source: §V · p. 9

L1. Account for losses at each stage of training-data

• Only 306 of 729 assessed projects met environment criteria, yielding 5,704 executable tasks and 4,572 trajectories after quality filtering.The teacher solved 4,889 tasks; each stage required suitable environments, correct solutions, and training-quality trajectories.Source: L1. Account for losses at each stage… · p. 9
• Verification capacity shapes which capabilities can be trained: renames made up 96.4% of generated tasks, and behavior preservation requires checks distinct from transformation checks.RefactorForge selected seven refactoring types based on established practices; the passage notes separate checks in SWE Refactor Bench [40].Source: L1. Account for losses at each stage… · p. 9
• Neither model solved Extract Class tasks, which RefactorForge could not generate because choosing class members requires design judgment beyond its structural checks and existing tests.Expanding coverage therefore requires both more task generators and verification methods for more complex engineering objectives.Source: L1. Account for losses at each stage… · p. 9

L3. Evaluate capability improvements against regression

• Refact-92B improved refactoring and related issue-resolution performance but regressed on greenfield repository generation under two of four harnesses.Incremental fine-tuning exposed this trade-off before integration into the full post-training mixture, supporting regression gates for model updates.Source: L3. Evaluate capability improvements… · pp. 9–10
• Refact-92B improved SWE-Bench Pro under all four evaluated harnesses, with gains from 1.2 pp under OpenCode to 6.57 pp under Claude Code.The largest gains on both SWE issue-resolution benchmarks occurred outside OpenCode, the training-trajectory collection harness.Source: L3. Evaluate capability improvements… · p. 10

VI. SE MUST SHAPE CODING LLM DEVELOPMENT: A RESEARCH AGENDA

• Salsabil et al. [61] found newer single-prompt models outperformed 13 of 35 ICSE 2026 techniques, while domain knowledge and external feedback proved more resilient.The paper presents this as support for the enduring value of engineering expertise.Source: §VI · p. 10
• SE researchers should use established engineering knowledge to select training operations, construct executable tasks, and verify correct solutions.RefactorForge uses refactoring knowledge to determine what the model practices and what counts as a correct solution.Source: §VI · p. 10
• Capability specifications should connect engineering knowledge to operations, training tasks, correctness criteria, capability relationships, and evaluations of acquisition and transfer.SWEBOK provides a foundation; RefactorForge is an initial example for refactoring, while its verification challenges indicate work remains for more complex activities.Source: §VI · p. 10
• Model development should apply SE methods across brownfield post-training, including regression-aware prediction, testing, data validation, reproducibility, tracing, integration, and release decisions.Brownfield maintenance updates existing checkpoints under finite budgets while preserving established capabilities [9].Source: §VI · p. 10
• SE research should move beyond studying and adapting to coding LLMs toward actively contributing to their development.The paper frames engineering the models themselves as an expanded role for SE as coding LLMs increasingly engineer software.Source: §VI · p. 10

VII. CONCLUSION

• The industrial case study evaluates one 92B model and seven bounded refactoring types, predominantly renames, so its findings are an initial demonstration rather than evidence of generality.Constructing verified training data, fine-tuning, and evaluating improvements require substantial engineering effort, computational resources, and financial investment.Source: §VII · pp. 10–11