PaperlayerView exact revision on arXiv

Source-linked AI summary

CoSTAR: Data Synthesis-Driven Constraint-Aware COBOL Section Summarization for Legacy System Modernization

Hao Lin, He Jiang, Xiaochen Li, Weihong Sun, Yufu Wang, Zhilei Ren, Ang Jia

arXiv:2609.11332v1cs.SE

TL;DR

COBOL modernization faces scarce section-level summarization data and the need to preserve migration-relevant constraints amid aging systems and shrinking expertise. CoSTAR synthesizes execution-validated COBOL code-summary data and trains smaller models with constraint-aware context and rationales, improving summarization across base models and metrics.

  • Problem

    COBOL modernization is hindered by incomplete documentation, shrinking developer expertise, and scarce supervision for section-level summarization.

  • Method

    CoSTAR repurposes general-purpose programming tasks to generate and execute-validate COBOL code-summary instances, then augments sections with relevant declarations and trains models using constraint-aware rationales.

  • Results

    Across four 7B or 8B base LLMs, CoSTAR improves all five automatic metrics, including average relative gains of 53.84% on METEOR and 37.22% on chrF.

  • Takeaways & Limitations

    CoSTAR supports effective COBOL section summarization with small models for privacy-sensitive modernization settings.

  • Takeaways & Limitations

    Generalization beyond COBOL section-level summarization may require adjusted prompts and validation criteria, and compiler dialects, coding conventions, and application domains may limit broader applicability.

Abstract

from arXiv · show

COBOL remains critical to governments, financial institutions, and large enterprises; yet, aging technologies, shrinking expertise, and missing documentation make modernization of COBOL-based legacy systems increasingly urgent. Before migration, code summarization is a common practice to support legacy system understanding. However, COBOL code summarization, especially on section-level, faces two key challenges: data scarcity and migration constraint preservation. To address these challenges, we propose CoSTAR, an integrated framework that combines execution-validated data synthesis with constraint-aware model training. CoSTAR repurposes general-purpose programming tasks to synthesize execution-validated COBOL code-summary data through LLM-based generation to overcome data scarcity. Based on the synthesized data, CoSTAR augments target sections with relevant data declarations and natural-language explanations, and uses constraint-guided structured rationales to train smaller base LLMs. The trained LLMs preserve the migration constraints for COBOL section summarization. We evaluate CoSTAR on both public and confidential enterprise COBOL systems. CoSTAR effectively synthesizes 3,764 execution-validated training instances. Based on these instances, CoSTAR built on 7B/8B base LLMs can improve these LLMs with average relative gains of 25.38% on ROUGE-L, 53.84% on METEOR, and 37.22% on chrF. In real-world enterprise evaluation, CoSTAR built on only Qwen3-8B, outperforms the enterprise-deployed Qwen3-235B in accuracy, completeness, and conciseness. These results show that CoSTAR enables small, locally deployable LLMs to achieve performance competitive with substantially larger LLMs for privacy-sensitive COBOL legacy systems.

I. INTRODUCTION

COBOL modernization is urgent because critical systems face fragile infrastructure, shrinking expertise, and incomplete documentation. CoSTAR addresses scarce section-level data and migration-constraint preservation through validated synthesis and constraint-aware summarization.

  • Motivation: COBOL supports critical services, but aging infrastructure, cybersecurity vulnerabilities, shrinking expertise, and incomplete documentation make modernization increasingly urgent.These systems remain widely used across government, finance, and enterprise settings.
  • Challenges: Section-level COBOL summaries are scarce, while migration requires preserving data constraints that affect behavior beyond executable logic.Relevant constraints include definitions such as PIC and VALUE and their effects in comparisons, arithmetic, assignments, and state updates.
  • Approach: CoSTAR synthesizes execution-validated COBOL code-summary supervision from general-purpose programming tasks and trains models to incorporate migration-relevant constraints.The framework uses task descriptions to guide generation and executable tests to validate generated programs.
  • Approach: CoSTAR sequentially uses model-explain to recover identifier semantics and model-summary to generate constraint-aware summaries from expanded context.Large teacher and judge models are used offline, while the smaller task-specific models run locally at inference.
  • Results: 3,764 validated instances enabled average relative gains of 25.38% on ROUGE-L, 53.84% on METEOR, and 37.22% on chrF across 7B/8B models.On enterprise COBOL, CoSTAR-Qwen3-8B outperformed Qwen3-235B by 4.35%, 8.06%, and 4.21% in accuracy, completeness, and conciseness.

II. RELATED WORK

Prior legacy-code understanding and summarization work largely targets mainstream languages and conventional program context. CoSTAR focuses on COBOL section-level documentation by recovering cross-division data constraints and training smaller models with structured rationale supervision.

  • Code Summarization: Code summarization research has progressed from templates and retrieval to neural, structure-aware, pretrained-code, and LLM-based methods, but datasets remain concentrated on Java and Python.This concentration leaves legacy-language summarization comparatively underexplored.
  • Code Context: Prior context augmentation captures syntax, control flow, data flow, semantic facts, variable-related statements, or calling relationships, mainly in mainstream languages.These studies establish that target code alone may be insufficient for complete summaries.
  • Legacy-Code Documentation: CoSTAR retrieves relevant COBOL data declarations and explains identifiers to provide focused context across the data and procedure divisions.This directly addresses COBOL’s separation of executable logic from data definitions.
  • LLM-Based Methods: CoSTAR extends teacher-based summarization by using constraint-guided structured rationales rather than relying only on teacher-generated final summaries.The supplied passage introduces this distinction but does not provide further comparative results.
  • Legacy System Understanding: Legacy modernization requires recovering business functions, interactions, data and state changes, and business rules for feasibility assessment and target-system design.Traditional techniques recover architecture, dependency, flow, data-dictionary, and business-rule artifacts.
  • Legacy-Code Documentation: Existing legacy-code documentation has not specifically addressed COBOL section-level summarization with recovered data-division constraints informing final summaries.CoSTAR positions this granularity as a localized unit that organizes related paragraphs into more complete procedural logic.

A. Overview

CoSTAR repurposes general-purpose programming tasks to synthesize execution-validated COBOL section-summary data. It filters tasks for executable, bounded tests before generating COBOL implementations for validation.

  • A. Overview: CoSTAR represents each programming task with a specification, implementation, and executable tests for synthesizing COBOL section-summary supervision.The specification contains a natural-language description and input/output specification, while tests provide executable behavioral evidence.
  • A. Overview: The synthesis pipeline uses a pre-filter, COBOL code synthesizer, and compiler-and-test executor to construct validated data.The process passes eligible task specifications to synthesis while reserving executable tests for later validation.
  • A. Overview: Function-level CodeFlowBench tasks provide a granularity match for individual COBOL sections.Each subproblem supplies a task description, input/output specification, and executable tests.

3) COBOL Code Synthesizer:

The COBOL code synthesizer converts task specifications into complete COBOL programs and retains only implementations that compile and pass every executable test. Each accepted program yields a validated section-summary instance.

  • 3) COBOL Code Synthesizer:: Prompts specify COBOL source format, program structure, input/output conventions, and section organization required for compilation and validation.Executable tests are withheld from the Code LLM and used only by the compiler-and-test executor.
  • 3) COBOL Code Synthesizer:: The synthesizer generates complete COBOL programs so compilation, execution, and data-division declarations are available for validation and constraint-aware summarization.The source task description becomes the paired summary after validation.
  • 3) COBOL Code Synthesizer:: Compilation verifies compiler acceptance, while testing checks whether observable behavior satisfies the source task requirements.Passing finite tests provides executable evidence for tested behaviors but does not prove complete semantic correctness.
  • 3) COBOL Code Synthesizer:: 3,764 of 10,924 eligible subproblems produce programs that compile successfully and pass all executable tests.These accepted programs form the COBOL code-summary dataset for subsequent constraint-aware model training.

C. Constraint-Aware Model Training

CoSTAR trains constraint-aware summarizers by recovering relevant COBOL data definitions, explaining their semantics, and supervising structured reasoning. Judge-based refinement filters rationales before they become training data.

  • C. Constraint-Aware Model Training: Relevant identifier definitions preserve behavior-relevant constraints such as PIC formats, VALUE initialization, hierarchy, and shared-state effects.The extractor retrieves declarations referenced by the target section while retaining necessary parent groups and key defining information.
  • C. Constraint-Aware Model Training: Structured rationales progress through identifier explanation, constraint analysis, logic abstraction, and final summary generation.The final phase integrates the preceding phases so summaries incorporate constraints from the concrete COBOL implementation rather than directly reusing the source description.
  • C. Constraint-Aware Model Training: Judge evaluation accepts rationales only when Context Adherence and Logical Coherence score at least 4 and Constraint Coverage scores 5.Constraint Coverage receives the strictest threshold because omitted migration-relevant constraints or state effects weaken summarization supervision.
  • C. Constraint-Aware Model Training: A generate–judge–refine loop revises failed rationale candidates, with limits on refinements and independent generation rounds.Candidates that remain unqualified across all allowed rounds are excluded from the dataset.
  • C. Constraint-Aware Model Training: The training pipeline combines identifier-explanation supervision with structured-rationale supervision for constraint-aware summarization.The first dataset teaches identifier semantics; the second teaches constraint reasoning and final summary generation from expanded context.

3) Task-Specific Fine-Tuning:

CoSTAR transfers teacher-derived identifier semantics and constraint reasoning to two LoRA-trained task-specific models. At inference, they run sequentially using only the target section and relevant data division.

  • 3) Task-Specific Fine-Tuning:: LoRA fine-tuning produces model-explain for identifier explanations and model-summary for four-phase constraint-aware rationales.Phase 4 of model-summary supplies the final summary.
  • 3) Task-Specific Fine-Tuning:: Inference requires only the target section and its data division, without the source description, teacher LLM, or judge LLM.Teacher and judge models are used offline for supervision construction and quality control.
  • 3) Task-Specific Fine-Tuning:: Inference first extracts relevant definitions and generates identifier explanations, then aggregates them into expanded context for model-summary.The two models separate identifier-semantic recovery from constraint-aware summarization.
  • 3) Task-Specific Fine-Tuning:: Deployment uses two smaller locally deployable models while incorporating cross-division data constraints relevant to section behavior.This design is intended for privacy-sensitive enterprise COBOL environments.

IV. EVALUATION

CoSTAR is evaluated on open-source and enterprise COBOL systems through five research questions covering overall effectiveness, data scale, context, rationale supervision, and industrial applicability.

  • The evaluation examines CoSTAR’s overall performance, synthesized-data scale, expanded context, structured rationale supervision, and industrial applicability.
  • Stack-120 contains 120 manually selected COBOL sections with relevant data-division declarations from open-source programs.
  • Expert-reviewed reference summaries describe behavior, data items, state effects, constraints, boundary behavior, and migration risks.
  • Relevant declarations are extracted without inferring unavailable copybook definitions, using the same preprocessing across comparable settings.

2) Evaluation Metrics:

CoSTAR is evaluated with textual, semantic, and human-oriented metrics under controlled preprocessing, decoding, and fine-tuning conditions.

  • The evaluation combines textual similarity, semantic similarity, and human evaluation metrics for COBOL summaries.
  • ROUGE-L, METEOR, and chrF measure textual similarity, while BERTScore and SentenceBERT assess semantic similarity.
  • Large-scale baselines receive the same task instructions, input information, and output requirements as CoSTAR whenever supported.
  • Controlled experiments keep training and inference settings unchanged except for the factor examined by each research question.
  • Automatic and human evaluations use only CoSTAR’s final summary, excluding the intermediate structured rationale from metric computation.

B. RQ1: Overall Effectiveness

CoSTAR improves all evaluated base LLMs across automatic metrics and makes 7B/8B variants competitive with substantially larger general-purpose LLMs on Stack-120.

  • CoSTAR improves every base LLM across all five automatic metrics, averaging 25.38% on ROUGE-L, 53.84% on METEOR, and 37.22% on chrF.
  • DeepSeek-R1-Distill-Llama-8B obtains gains of 69.08% on ROUGE-L, 93.08% on METEOR, and 50.87% on chrF.
  • CoSTAR variants reach 88.50%, 100.60%, 95.36%, 100.31%, and 96.10% of the best large-scale baselines on ROUGE-L, METEOR, chrF, BERTScore, and SentenceBERT.
  • CoSTAR-Llama-3.1-8B-Instruct achieves the highest METEOR, exceeding the best large-scale baseline by 0.142 points.
  • RQ2: Synthesized Data Effectiveness: Using 3,764 synthesized instances, CoSTAR improves performance as training-data scale increases, with the full setting best on every metric.

D. RQ3: Expanded Context

Expanded context improves COBOL section summarization by exposing data constraints outside local procedure code and combining declarations with identifier explanations.

  • Relevant data declarations improve all five metrics because they expose properties and constraints unavailable from local section code.
  • Identifier explanations improve over section code by 18.84% on ROUGE-L, 27.89% on METEOR, and 26.32% on chrF.
  • CoSTAR-Full improves over section code alone by 27.20% on ROUGE-L, 34.69% on METEOR, 27.42% on chrF, 8.54% on BERTScore, and 14.23% on SentenceBERT.
  • Combining declarations and identifier explanations outperforms identifier explanations alone, showing that the two representations are complementary.

E. RQ4: Structured Rationale Supervision

Under matched base-model, data, context, and hyperparameter settings, CoSTAR's structured rationale supervision yields substantially larger gains than summary-only fine-tuning and also improves enterprise evaluation scores over a much larger deployed model.

  • RQ4: Structured Rationale Supervision: Summary-only SFT improves all five metrics, but its gains reach only 19.09% on ROUGE-L, 26.31% on METEOR, and 18.24% on chrF.Its gains on BERTScore and SentenceBERT are 0.33% and 2.56%, respectively.
  • RQ4: Structured Rationale Supervision: 69.08%, 93.08%, 50.87%, 2.82%, and 15.56% are CoSTAR's relative gains over the base LLM on ROUGE-L, METEOR, chrF, BERTScore, and SentenceBERT, respectively.Compared directly with Summary-only SFT, CoSTAR further improves these metrics by 41.98%, 52.87%, 27.60%, 2.48%, and 12.67%.
  • RQ4: Structured Rationale Supervision: Structured rationales organize identifier roles, data constraints, control and data flow, and modernization-relevant behavior before final summary generation.This supervision provides a task-specific learning signal for using expanded context.
  • RQ4: Structured Rationale Supervision: Under the same base LLM, training data, expanded context, and hyperparameters, CoSTAR further improves METEOR by 52.87% and SentenceBERT by 12.67% over Summary-only SFT.The matched settings isolate the difference in supervision as the source of these additional gains.
  • Enterprise Evaluation: CoSTAR-Qwen3-8B scores higher than the enterprise-deployed Qwen3-235B on accuracy, completeness, and conciseness in Industrial-200 human evaluation.The improvements are 4.35%, 8.06%, and 4.21%, respectively, with all differences statistically significant after Holm correction.

V. THREATS TO VALIDITY

The paper identifies threats involving supervision reliability, data leakage, generalizability, and evaluation validity, while using expert review and complementary metrics to mitigate some risks.

  • Internal Threats: Finite tests, automated verification, and LLM-based evaluation cannot completely eliminate semantic errors, noise, hallucinations, or prior pretraining exposure.The synthesized data and evaluation datasets are constructed independently, but base-model pretraining remains non-transparent.
  • External Threats: Findings are evaluated only for COBOL section-level summarization, so transfer to other languages, granularities, tasks, dialects, conventions, and domains may require adaptation.The authors identify broader COBOL environments and heterogeneous backbones as future work.
  • Construct Threats: Automatic metrics may miss COBOL-specific semantics, while expert ratings and teacher-generated references can introduce subjectivity or inaccuracies.Three experienced COBOL engineers rated Industrial-200 summaries with Fleiss’ kappa of 0.763, and Stack-120 references received independent expert review.
  • Conclusion: Across four 7B or 8B base LLMs, CoSTAR improves all five automatic metrics and outperforms Qwen3-235B in enterprise human evaluation.The conclusion reports average relative gains of 53.84% on METEOR and 37.22% on chrF.
Loading 2609.11332v1…