PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural-Language to SysMLv2 Translation via Conformance-Driven Iterative Refinement

Chance LaVoie, Eladio Andujar Lugo, Taylan G. Topcu, Levent Burak Kara

arXiv:2607.14162v1cs.SEcs.AI

TL;DR

Natural-language–to–SysMLv2 generation must produce models accepted by industrial tools, not merely grammatically valid text. The paper embeds a production conformance checker in a generate–check–repair loop and reports 100.00% acceptance across 604 cases, versus 51.16% single-shot acceptance.

  • Problem

    Single-shot LLM generation does not reliably produce production-conformant SysMLv2 models, while grammar validity does not guarantee operational usability in industrial workflows.

  • Method

    The framework iteratively generates SysMLv2, checks candidates with a production conformance oracle, and repairs them using deterministic diagnostics until zero errors are reported.

  • Results

    100.00% of 604 SysMBench prompt–model cases achieved production conformance after refinement, compared with 51.16% for single-shot generation.

  • Takeaways & Limitations

    Production conformance can serve as the generation stopping condition for producing SysMLv2 artifacts suitable for loading, visualization, and engineering use.

  • Takeaways & Limitations

    The study evaluates syntactic and production-level conformance, not semantic correctness, architectural adequacy, or requirements fidelity.

Abstract

from arXiv · show

Model-Based Systems Engineering (MBSE) relies on formal system models as primary technical artifacts for representing requirements, structure, and behavior across the system lifecycle. With the standardization of SysMLv2 as a textual language, interest is increasing in translating natural-language descriptions directly into executable models. For practical deployment, generated models must be accepted by industrial modeling environments, not merely satisfy grammar constraints. We present a conformance-checker-driven framework for reliable natural-language-to-SysMLv2 translation that enforces production-level acceptance as the termination condition. The system embeds a SysMLv2 conformance checker within a generate-check-repair loop. Each model is evaluated using the checker, and deterministic diagnostics are incorporated into revisions until zero conformance errors are achieved. Using the production checker as the oracle ensures the framework targets deployability rather than grammar plausibility. We evaluate the approach on the full SysMBench prompt set of 151 prompts across four large language model backends, yielding 604 prompt-model cases. Single-shot generation achieves 51.16% production-conformance acceptance, while our approach achieves 100.00% conformance. By elevating production conformance from a post-processing check to a control mechanism within generation, the framework converts probabilistic outputs into production-accepted SysMLv2 artifacts suitable for loading, visualization, and engineering use.

1 Introduction

The paper targets reliable natural-language–to–SysMLv2 generation by replacing grammar plausibility with production-tool conformance as the acceptance criterion. A checker-guided repair loop raises acceptance from 51.16% single-shot generation to 100.00% across 604 cases.

  • Limits of Grammar-Level Parsing: Grammar parsing alone cannot enforce model-wide constraints such as name resolution, type consistency, ownership, multiplicity, and cross-reference integrity.Parser-valid models may therefore remain unusable for visualization, simulation, or downstream analysis.
  • Current State of MBSE Automation: Production conformance is defined as acceptance by the production checker with zero reported errors, indicating tool-level usability beyond grammar validity.This criterion targets models that can be loaded and used in industrial modeling environments.
  • Digital Engineering and Trustworthy AI for Systems Engineering: The work addresses trust and reliability concerns by verifying generated artifacts against the same production-level checks used by industrial modeling environments.Without verification, AI-generated models risk ad hoc integration into digital ecosystems and increased synchronization burdens.
  • Proposed Advance: The framework generates a SysMLv2 candidate, checks it with SysIDE, feeds diagnostics into repair prompts, and stops only when zero errors remain.Diagnostics include unresolved references, typing errors, and ownership violations.
  • Proposed Advance: 100.00% of 604 prompt–model cases reached zero-error production conformance, compared with 51.16% for single-shot generation.The evaluation used all 151 SysMBench prompts across four model backends, and most failures resolved within a few repair cycles.
  • Proposed Advance: The study contributes a conformance-checker-in-the-loop architecture, a benchmark-scale evaluation, and refinement traces for future reliability and repair-dynamics research.The benchmark covers the full SysMBench prompt set and four model backends.

2 Related Work

Related work establishes structured natural-language–to–SysML generation and deterministic feedback, but grammar-focused methods do not ensure production-level model consistency. This paper extends verifier-guided refinement to SysMLv2 by using an industrial modeling tool as the oracle.

  • 2.1 Natural-Language–to–SysML Model Generation: SysTemp uses template-first multi-agent generation, while Cibrián et al. combine retrieval-augmented context with ANTLR grammar validation.These approaches target structured synthesis and syntactic correctness in natural-language–to–SysML pipelines.
  • 2.1 Natural-Language–to–SysML Model Generation: Studies also report structural inconsistencies and hallucinations in generated SysML behavioral models and examine LLM-assisted interaction with MBSE artifacts.These contributions reflect growing interest in integrating generative AI with systems engineering models.
  • 2.1 Natural-Language–to–SysML Model Generation: Earlier approaches used restricted language subsets, rule-based extraction, or diagram heuristics, often requiring controlled vocabularies or domain-specific grammars.More recent work explores semantic alignment and integration across SysMLv2 artifacts.
  • 2.2 Grammar-Constrained and Structured Synthesis: Grammar and template constraints improve structural correctness but do not necessarily enforce model-wide consistency or tool-specific constraints.Production environments additionally check reference resolution, typing consistency, and cross-model integrity.
  • 2.3 Verifier- and Tool-Guided Refinement: Verifier-guided refinement generates candidates, evaluates them with deterministic backends, and revises them until the backend reports success.Prior examples use formal verifiers, compilers, or test suites as correction oracles.
  • 2.3 Verifier- and Tool-Guided Refinement: This work instantiates verifier-guided refinement for SysMLv2 MBSE by aligning termination with the production modeling tool used in industrial environments.The approach examines this pattern at benchmark scale rather than relying only on parser conformity.

3 Methodology

The study compares single-shot and iterative conformance-guided SysMLv2 generation using paired prompt–model cases, with a production checker controlling repair until zero errors. It evaluates 604 cases across 151 SysMBench prompts and four model configurations, while measuring acceptance and repair convergence.

  • 3.1 Study Objective and Paired Design: The paired design compares initial single-shot and iteratively repaired outcomes for the same prompt and model, isolating the effect of conformance-driven refinement.Only the presence or absence of iterative feedback differs between conditions.
  • 3.2 Conformance-checker-in-the-Loop Generation Procedure: The workflow generates a complete SysMLv2 candidate, checks it with SysIDE, feeds deterministic diagnostics into revision prompts, and stops only at zero checker errors.Diagnostics include unresolved references, typing inconsistencies, ownership violations, and other model-wide constraint failures.
  • 3.3 Dataset and Outcome Extraction: The evaluation covers 604 prompt–model cases from 151 SysMBench prompts run with four model configurations.The curated prompt set was selected to stress generation across diverse SysMLv2 modeling patterns.
  • 3.3 Dataset and Outcome Extraction: The study reports single-shot and final pipeline conformance rates, iterations-to-success, runtime, token usage, and error counts from saved run records.Grammar-versus-production analysis also compares ANTLR parsing with SysIDE acceptance on the same initial artifacts.
  • 3.4.2 Convergence Metrics: Cumulative acceptance A_k tracks cases reaching production conformance by repair cycle k, while residual failure R_k = 1 − A_k and threshold times summarize convergence.The analysis reports T90, T95, and T99 and interprets contraction descriptively rather than as a formal convergence guarantee.
  • 3.4.3 Statistical Reliability Analysis: Reliability bounds are scoped to SysMBench-style prompt distributions under the evaluated configuration rather than universal arbitrary natural-language inputs.The all-success analysis uses a one-sided 95% Clopper–Pearson lower confidence bound.

4 Results

Across 604 SysMBench prompt–model trials, conformance-guided refinement raised production acceptance from 51.16% single-shot to 100.00%, with rapid convergence across four backends. Results also quantify the grammar–production gap, reliability boundary, repair behavior, and released trajectory corpus.

  • 4.1 Primary Outcome: Production Conformance: 100.00% of cases reached production conformance with refinement, compared with 51.16% for single-shot generation across 604 trials.The final pipeline achieved 604/604 zero-error cases, versus 309/604 initially.
  • 4.2 Per-Model Reliability: All four models reached 151/151 eventual conformance, although Anthropic Sonnet 4.6 led single-shot performance at 82.78%.OpenAI Codex 5.2, DeepSeek Reasoner, and Mistral Large had single-shot rates of 41.72%, 41.06%, and 39.07%, respectively.
  • 4.3 Convergence Behavior: Acceptance increased from 51.16% at k = 0 to 84.44% at k = 1, 94.37% at k = 2, and 99.67% at k = 4, reaching 100.00% by k = 5–7.First conformance required a mean of 1.727 total attempts and a median of 1; the maximum was 8 attempts.
  • 4.3 Convergence Behavior: Convergence was rapid across backends despite different starting points, with thresholds T90 = 2, T95 = 3, and T99 = 4 repair cycles.Early residual-failure ratios clustered around approximately 0.32–0.36, indicating a front-loaded reduction in failures.
  • 4.4 Statistical Reliability: With no observed failures in 604 cases, the evaluated four-backend mixture had a 95%-confidence convergence probability of at least 99.51%.The bound applies to SysMBench-style prompts under the evaluated controller, checker, and backend configuration, not arbitrary natural-language inputs.
  • 4.5 Grammar Validity vs. Production Conformance: Grammar parsing accepted 61.09% of initial candidates, versus 51.16% production conformance, including 60 grammar-valid but production-invalid cases.No production-conformant model failed grammar parsing, making grammar validity necessary but insufficient under the evaluated configuration.
  • 4.6 Repair Analysis: Persistent repair failures mainly occurred when models ignored the checker-highlighted region or introduced another invalid construction.Iterations-to-success showed no meaningful relationship with SysMBench difficulty or generated-output length.
  • 4.7 Open-Source Dataset Release: The released corpus contains 1,043 iteration artifacts from 604 cases, including 604 positive and 439 negative production-conformance artifacts.Full trajectories preserve intermediate models and diagnostics for studying repair dynamics and cost–reliability tradeoffs.

5 Discussion

The framework makes production conformance the generation stopping condition, using deterministic checker feedback to improve deployability. Results indicate rapid convergence within the evaluated scope, while structural acceptance remains distinct from semantic correctness.

  • Production Conformance: 100.00% production-conformance acceptance was achieved across 604 cases, up from 51.16% with single-shot generation.The benchmark comprised the full SysMBench prompt set across four model backends.
  • Reliability Mechanism: Deterministic diagnostics shift reliability enforcement from first-pass model output to iterative evaluation and targeted revision.The checker reports concrete modeling errors at each iteration, which guide subsequent corrections.
  • Deployment Implications: Conformance guidance supports loading, rendering, visualization, simulation, and verification workflows by enforcing usability during generation.This reduces the need for engineers to correct structural syntax errors before using generated models.
  • Convergence Dynamics: Most recoverable failures resolve within a few repair cycles, with T90 = 2, T95 = 3, and T99 = 4.The observed pattern is consistent with correction of shallow but compound structural inconsistencies rather than unstable or oscillatory refinement.
  • Scope Limits: The study guarantees structural and operational compatibility with a modeling tool, not semantic correctness, behavioral adequacy, or trace completeness.Its empirical scope is also limited to SysMBench-style prompts, one production conformance backend, and predominantly structural model content.
  • Scope Limits: Production conformance provides a stable baseline for adding semantic validation, cross-view consistency analysis, simulation checks, and requirement-trace evaluation.The paper presents these higher-level assurances as mechanisms that can build on structural acceptance.

6 Conclusion

The paper addresses unreliable single-shot natural-language–to–SysMLv2 generation by placing production conformance inside an iterative generate–check–repair loop. Across the evaluated benchmark, refinement raised acceptance to 100.00%, while the conclusions remain limited to syntactic and operational acceptance.

  • 6 Conclusion: The method generates a candidate model, checks it with a production SysMLv2 conformance checker, and revises it until zero errors are reported.Production acceptance, rather than grammar parsing alone, is the stopping condition.
  • 6 Conclusion: 100.00% production-conformance acceptance was reached after refinement, compared with 51.16% for single-shot generation across 604 cases.These cases covered 151 SysMBench prompts and four model backends.
  • 6 Conclusion: Deterministic checker feedback forms a reliability layer between LLM-generated text and MBSE tools without retraining model weights or depending on a specific language model.The released trajectory-level data supports further research on the refinement process.
  • 6 Conclusion: The findings establish syntactic and operational acceptance under one production checker, but do not guarantee semantic correctness or architectural adequacy.Future work includes cross-conformance replication, semantic alignment with requirements, and behavioral evaluation.

7 Limitations and Future Work

The framework reaches production-conformance acceptance across all benchmark cases, but its scope and generality remain bounded by several unresolved limitations. Future work targets semantic validation, cross-tool robustness, adaptive repair, and cost characterization.

  • Production conformance does not guarantee semantic correctness, architectural adequacy, or requirements fidelity.Models can satisfy name-resolution, typing, and ownership constraints while misrepresenting intended system behavior.
  • The study relies on a single production checker, SysIDE, so other toolchains may impose different constraints or diagnostics.Cross-tool replication is needed to assess tooling effects on repair trajectories and convergence behavior.
  • The empirical evidence is limited to the SysMBench prompt distribution and does not cover noisy, incomplete, or inconsistent industrial requirements.Longer and less structured inputs remain important for evaluating deployment readiness.
  • Although all 604 cases converged empirically, the study provides no formal guarantee of convergence or characterization of failure modes.Early contraction-like behavior suggests stable dynamics under deterministic diagnostics, but convergence conditions remain open.
  • Conformance-guided refinement adds model calls and runtime overhead, but the cost–reliability tradeoff is not quantified in detail.Iteration counts, token usage, and latency under varying prompt complexity remain to be studied.
  • Future work will add semantic and simulation-based checks, evaluate cross-conformance, and explore adaptive repair prioritizing high-impact diagnostics.

A Auxiliary Demonstration: Grammar Parsability vs. Production Conformance

The auxiliary demonstration separates grammar parsability from production acceptance using intentionally mismatched SysMLv2 examples. All examples parsed with ANTLR, yet none satisfied SysIDE conformance.

  • The demonstration treats parser acceptance and production conformance as distinct outcomes rather than a second primary experiment.
  • All 10/10 intentionally distinct examples passed ANTLR parsing, while 0/10 passed production conformance.The cases were designed to remain grammar-parseable while violating model-wide constraints checked by production tooling.
  • Most production failures were unresolved-reference errors, occurring in 9/10 cases, while one case had an invocation-typing failure.
  • The example Wheel model parses structurally but fails production conformance because no Feature named ‘height’ is found.

B Extended Analysis: Difficulty and Output-Length Signals

The extended difficulty analysis tests whether repair effort tracks benchmark difficulty based on hand-authored model size. Across difficulty levels, pooled repair iterations show no strong linear relationship.

  • SysMBench difficulty is assigned from hand-authored ground-truth line counts, with buckets spanning fewer than 30 lines to 120 or more lines.The benchmark contains 151 prompts distributed across the five difficulty buckets.
  • 1.766, 1.668, 1.938, 1.667, and 1.583 pooled mean iterations-to-success occur at difficulty levels 1 through 5, respectively.The pooled linear fit is weak, with R2 ≈0.183.
  • The results indicate that the ground-truth line-count difficulty label is not strongly correlated with repair effort as measured by iterations-to-success.

B.2 Generated Output Length Versus Iterations-to-Converge

The output-length analysis finds that generated SysML size explains almost none of the variation in repair effort. Model-level averages reinforce that longer outputs do not necessarily require more repair iterations.

  • R2 = 0.0011 and slope = 0.00062 show that generated SysML line count explains almost none of the variation in iterations-to-converge.The pooled linear fit in Figure 9 is essentially flat.
  • Anthropic generated about 103 lines on average yet required 1.23 iterations-to-success, fewer than OpenAI at 1.73, DeepSeek at 1.97, and Mistral at 1.98.This contrast indicates that line count alone is a poor standalone proxy for repair effort.
  • OpenAI, DeepSeek, and Mistral produced approximately 49, 44, and 50 lines on average, respectively, whereas Anthropic produced roughly twice as many.
  • Raw line count may miss structural complexity because shorter models can contain more varied declarations, references, and nested relationships.Uniform repetition can produce many lines without proportionally increasing conformance sensitivity.
  • Persistent diagnostics are defined as identical SysIDE error families and stderr messages appearing in consecutive iterations, enabling analysis of revision behavior.

C.1 Error Transition Outcome Explanations

The analysis tracks persistent identical conformance errors across consecutive repair iterations by comparing SysIDE diagnostics and highlighted code regions. It distinguishes failures left unchanged from attempted repairs that remain unsuccessful.

  • Transition classification: 1,502 exact-error transitions were examined across consecutive iterations, with 1,262 resolved by the next iteration and 240 remaining persistent.The persistent cases are the transitions retained for outcome analysis.
  • Transition classification: Persistent cases are classified as unaddressed when the identical error and highlighted snippet recur, or addressed when the error remains but SysIDE highlights a different snippet.The classification uses direct comparison of SysIDE stderr blocks between iterations.
  • Transition classification: The analysis compares exact diagnostic messages and highlighted code regions rather than relying on manual inspection of model edits.An example involves a parsing error highlighting `attribute state : VehicleState;`.
  • Classification caveat: The addressed-but-not-fixed category can contain limited noise when multiple identical errors occur in different code regions and only one occurrence is repaired.The authors expect this edge case to be uncommon because pattern-level repairs may affect repeated structures.

C.2 Persistent Error Analysis

Among persistent identical errors, most remain tied to the same highlighted code region, while a smaller share reflects attempted but unsuccessful edits. These outcomes separate repair-loop inefficiency from limitations in the model’s correction capability.

  • Persistent outcomes: 64.6% of persistent transitions were unaddressed and not fixed, compared with 35.4% addressed but not fixed.Figure 10 reports the split as 155/240 versus 85/240 persistent transitions.
  • Persistent outcomes: The larger unaddressed share suggests that the model often leaves the checker-highlighted offending region unchanged after receiving feedback.The authors propose stronger prompt grounding or repair policies requiring revision of the highlighted region.
  • Persistent outcomes: The addressed-but-not-fixed share indicates attempted edits that fail to produce a syntactically valid correction, pointing to repair-capability limitations.Suggested future directions include constrained editing and fine-tuning on syntactically valid code datasets.
Loading 2607.14162v1…