Source-linked AI summary
Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus Generation
Jean-Baptiste Espinasse, Djamel Eddine Khelladi, Mathieu Acher
TL;DR
Representative production COBOL corpora are rarely available, limiting rigorous benchmarking of modernization approaches. This paper generates COBOL from specifications and degrades it using production-derived patterns and complexity targets; across three programs, it reaches structural targets reliably, while functional preservation remains partial.
Problem
Representative COBOL corpora reflecting production code are rarely available, limiting rigorous benchmarking of modernization approaches.
Source: §1 · pp. 1–2Method
The pipeline generates programs from natural-language specifications, then iteratively degrades them toward complexity targets using patterns and metrics derived from production COBOL.
Source: §1 · p. 2Results
Takeaways & Limitations
A maintenance plan driving incremental legacy-style modifications is proposed as a potentially more realistic alternative, though at higher generation cost and left for future evaluation.
Source: §8 · p. 8The limited number of generated programs and a 14-program reference corpus from one industrial partner constrain generalization across enterprises and domains.
Source: §7 · p. 7
Abstract
from arXiv · showhide
COBOL remains widely deployed, yet representative corpora reflecting real production code are rarely available, limiting rigorous benchmarking of modernization approaches. We propose a systematic agentic AI pipeline for generating realistic COBOL programs, combining specification-driven generation with iterative degradation guided by patterns and complexity targets extracted from real production code. Here, realism is understood as structural fidelity to production code as captured by our metrics. We evaluate whether degradation reaches target complexity levels while preserving business behavior, and examine the limits of the approach, across three programs from distinct business domains. Results show the pipeline reliably produces syntactically valid programs and moves them toward realistic structural complexity. However, preserving business behavior is not always achieved by construction, and targeting structural metrics independently of business logic risks producing programs whose complexity does not reflect a plausible maintenance history. We discuss these limitations and outline a more realistic alternative as a direction for future work, generating legacy programs from scratch along a simulated development history.
1 Introduction
Representative production COBOL corpora are scarce, limiting rigorous benchmarking of modernization approaches. This paper proposes a specification-driven generation and iterative degradation pipeline, then evaluates its structural targets, behavior preservation, and practical limits.
1 Introduction
- Representative production COBOL corpora are rarely openly available, limiting rigorous benchmarking of modernization approaches.Existing public collections are heterogeneous and often lack enterprise patterns such as copybooks, file handling, and business-oriented control flows.Source: §1 · pp. 1–2
- The pipeline generates a valid, functionally correct program from a specification, then degrades it toward calibrated complexity using production-derived patterns and targets.Realism is defined as structural fidelity to production code as captured by the paper’s metrics.Source: §1 · p. 2
- Across three business domains, the pipeline reliably produces syntactically valid programs and moves them toward realistic structural complexity, while revealing limits in behavior preservation and scalability.The evaluation uses target intervals derived from an industrial partner’s production programs and includes expert review.Source: §1 · p. 2
2 Related Work
COBOL corpus-building work includes translation, augmentation, and full synthetic generation, each with different constraints on source data or structural realism. This paper combines specification-driven generation with production-pattern degradation toward calibrated complexity.
Program translation
Program augmentation
- Augmentation transforms existing corpora for diversity or realism, but prior methods target syntactic coverage, functionality, or plausible student errors rather than production COBOL realism.The paper’s degradation phase instead injects plausible legacy patterns, analogous to generating plausible mistakes [17].Source: §2 · p. 2
Full synthetic generation
Positioning
- Unlike prior augmentation work, the degradation phase targets a calibrated, corpus-derived complexity profile rather than syntactic coverage or edit diversity alone.The approach combines full synthetic generation with augmentation based on patterns extracted from real production code.Source: §2 · p. 2
3 Corpus Analysis
The corpus analysis characterizes 14 industrial-partner payroll programs and uses structural metrics spanning scale, complexity, maintenance drift, and modularization. Metric distributions are uneven, while correlations among several metrics motivate deriving complexity targets jointly.
3.1 Corpus Description
- The reference corpus comprises 14 batch payroll programs spanning a processing pipeline, with file-oriented design, extensive copybooks, and large global data structures.The programs were incrementally developed and maintained over years.Source: §3.1 · p. 2
3.2 Structural Complexity Metrics
- The metric set covers scale, code complexity, maintenance drift, and modularization, using indicators such as SLOC, NPAR, DIVR, CC, NGO, VUR, DPR, and APL.DIVR captures data-to-logic balance; CC and NGO jointly characterize control-flow complexity; VUR and DPR capture unused variables and paragraphs.Source: §3.2 · pp. 2–3
3.3 Descriptive Statistics
- SLOC, CC, and NGO have right-skewed distributions, with the same program several times above the corpus median; nearly all programs have 0% DPR, except one at 1.36%.The 14-program statistics are descriptive characteristics, not generalizable estimates of legacy COBOL systems.Source: §3.3 · p. 3
3.4 Inter-Metric Correlation
- SLOC, NPAR, CC, NGO, and VUR are strongly correlated (ρ > 0.90), while DPR is largely independent (|ρ| < 0.25), motivating correlation-based complexity targets.DR is strongly anti-correlated with the size and control-flow metrics (ρ < −0.90), and APL has moderate correlation (ρ ≈ 0.6–0.7).Source: §3.4 · p. 3
4 Approach
Spec2COBOLRot separates clean, specification-driven COBOL generation from iterative degradation toward production-derived structural complexity. It uses expert-validated specifications and a library of production-code patterns, with complexity intervals guiding degradation.
4 Approach
- Separating clean generation from degradation makes the initial program available for expert review and as an input/output behavior reference.The degraded program can be compared against that clean version to assess whether it preserves behavior.Source: §4 · p. 3
4.1 Inputs: Specifications and Asset Library
- Specifications capture program functionality, while the asset library supplies validated production patterns: idioms, behavior-preserving smells, and modernization-relevant mismatches.Specifications include a functional objective, data inputs and outputs, and business rules.Source: §4.1 · p. 4
4.2 Phase 1: Specification driven COBOL Generation
- Phase 1 generates a COBOL program and synthetic data from a natural-language specification, then retries compilation and execution failures with error feedback.The loop is bounded by k iterations; if the program still fails, Phase 2 does not launch.Source: §4.2 · pp. 4–5
4.3 Phase 2: Iterative Degradation
- Phase 2 injects library patterns and recomputes structural metrics until all fall within target intervals or the iteration budget is exhausted.Intervals are derived from SLOC-correlated metrics using regression, widened by one residual standard deviation; weakly correlated metrics use the corpus-wide range. Targets set without a production corpus risk being less representative.Source: §4.3 · p. 5
5 Experimental Setup
The evaluation uses three expert-validated specifications from different business domains and an asset library extracted from industrial-partner production programs. Complexity targets span four manually selected SLOC levels, with metric intervals derived from their relationship to SLOC.
5 Experimental Setup
- Three specifications cover banking, payroll, and insurance, each based on a real-world use case and validated by a COBOL expert.Source: §5 · p. 5
- The production-derived asset library contains 26 assets: 8 idioms, 15 smells, and 3 mismatches.Source: §5 · p. 5
- Four manually chosen SLOC intervals span approximately 2.5k to 25k lines; metric intervals are regression-derived when correlation with SLOC is strong.APL uses the corpus-wide observed range because its correlation is weak (R2 = 0.36), while DPR is excluded due to near-zero variance.Source: §5 · pp. 5–6
6 Evaluation
The evaluation examines validity, realism degradation, practical scale limits, and expert perceptions. Degradation moves generated programs toward production-derived structural targets, but functional preservation and realistic legacy patterns remain challenges.
6 Evaluation
- The evaluation asks whether the pipeline produces valid, structurally complete programs, increases complexity while preserving behavior, scales, and appears realistic to experts.The study frames these as four research questions.Source: §6 · p. 6
- Table 5 derives target intervals across metrics and complexity levels by linear regression on SLOC.Intervals for adjacent levels may partially overlap because of residual variability around the SLOC-metric trend.Source: Table 5 · p. 6
6.1 RQ1 - Syntactic Validity
- Every generated program across TRNRECBT, PAYCOMBT, and CLMPREMT compiled, ran without runtime errors, and produced required files.Qualitative inspection found key business rules and data formats mostly reflected, but validation was not systematic or exhaustive.Source: §6.1 · p. 6
6.2 RQ2 - Realism Degradation
- After degradation, profiles enter the target band on nearly every axis across all three programs, while functional equivalence is preserved in 5/9 L1 runs.Figure 3 compares pre- and post-degradation structural profiles against L1 and L2 target bands; the functional checks compare output files on the same synthetic dataset.Source: §6.2 · p. 6, Figure 3 · p. 7
6.3 RQ3 - Practical Scale Limits
- For L3 and L4, degraded programs compiled and often retained consistent data, but threshold-driven duplication of similar paragraphs reduced realism.The authors characterize this as a qualitative realism limit rather than a technical limit on compilability or functional equivalence.Source: §6.3 · p. 6
6.4 RQ4 - Expert Review
- Experts recognized a PAYCOMBT excerpt as a legacy pattern, but judged the programs too clean and the injected patterns insufficiently diverse.Observed gaps included absent COPY statements, systematic explicit END-IF closures, recurrent STRING usage, and rarely introduced REDEFINES clauses.Source: (c) TRNRECBT (L1) · p. 7, Figure 4 · p. 7
7 Threats to Validity
The validity discussion limits what can be inferred from the study’s metrics, reproducibility, and corpus scope. It also identifies an execution-environment boundary for some COBOL constructs.
Construct and external validity
- Syntactic metrics do not directly measure business-domain comprehension, and expert review of realistic specifications remains subjective.The reference corpus contains 14 programs from one industrial partner and one business domain, limiting generalization.Source: §7 · p. 7
Internal validity
- Non-deterministic LLM calls limit exact reproducibility, although the authors want variability to cover a wider range of implementations.Running the same specification may produce different programs.Source: §7 · p. 7
External validity
- Programs using CICS or embedded SQL require a supporting execution environment that the current toolchain does not provide.The authors distinguish this execution constraint from a limitation of the methodology.Source: §7 · p. 8
8 Discussion
The discussion finds that degradation moves generated programs toward reference structural complexity, but functional correctness is only partly demonstrated. It identifies risks in targeting complexity independently of business logic and proposes a maintenance-history-based alternative, while noting omitted comment metrics.
8 Discussion
- Functional correctness holds only in part under input/output comparison on a synthetic dataset, with test-data coverage limiting the result.The authors also identify runtime performance as a possible extension of the evaluation.Source: §8 · p. 8
- Targeting absolute structural metrics independently of business complexity may produce artificial programs, despite their potential value as difficult modernization benchmarks.The authors favor traceable production-derived patterns over a generic legacy-code prompt and leave quantitative comparison for future work.Source: §8 · p. 8
- A maintenance plan of decades of patches could guide incremental code changes by a legacy-developer agent, plausibly yielding more realistic programs at higher generation cost.The proposed alternative retains separate generation and degradation stages; its evaluation is future work.Source: §8 · p. 8
- The evaluation omits comment density and comment-code alignment because assessing semantic consistency requires analysis beyond static structural parsing.The authors identify outdated documentation as a reason comment-code alignment could help assess documentation reliability.Source: §8 · p. 8
9 Conclusion
Across three programs from distinct business domains, the pipeline moved generated COBOL toward target complexity while producing syntactically valid programs, though behavior preservation was partial. Expert review found that some legacy patterns remained missing despite matching metrics; future work plans to scale the pipeline and broaden its reference corpus.
9 Conclusion
- Across three programs from distinct business domains, degradation reliably produced syntactically valid programs and moved them toward target complexity, with business-behavior preservation only in part.Experts appreciated the programs but noted that some legacy patterns were missing despite metric matches, which the authors attribute to pattern coverage.Source: §9 · p. 8
- Future work plans to scale the pipeline and extend its reference corpus to other business domains.Source: §9 · p. 8