PaperlayerView exact revision on arXiv

Source-linked AI summary

Generation and Verification: Human-Agent Collaboration for Safety-Critical Automotive Software Engineering

Elizaveta Zinovyeva, Aiham Taleb, Nikita Kozodoi, Daniel Schleicher, Amir Mahdi Namazi, Martin Kraus, Levent Kent, Brian Jensen

arXiv:2610.35982v1cs.SEcs.AI

TL;DR

Safety-critical automotive coding requires both developer-verifiable checks and domain knowledge that may be underrepresented in general-purpose models. The paper presents an MCP assistant and compares prompt engineering with fine-tuning on production-code context. In a two-week pilot, verify was the most-used tool, and prompt engineering had fewer C violations while C++ strategy differences were too small to call.

  • Problem

    Safety-critical automotive development requires reviewed, requirement-traced, standards-checked changes and domain-specific conventions that general-purpose models may not represent well.

    Source: §1 · p. 1

  • Method

    The assistant exposes generate, edit, and developer-invocable verify tools, and compares prompt engineering with fine-tuning using production code paired with requirements and UML behavior.

    Source: §1 · p. 2, §3.2 · p. 4, §6 · p. 8

  • Results

    Verify was the most-used pilot tool, prompt engineering yielded fewer C violations, and C++ strategy counts differed too little to favor either approach.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Developers valued checking their own code, fast responses that fit a review loop, and clarity about the scope of automated verdicts.

    Source: §6 · p. 8

    The evaluation reports no functional-correctness results, and its two-week pilot involved nine self-selected developers at one organization with self-reported evidence.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Research on coding agents mainly focuses on generation quality. Yet, safety-critical automotive software makes two additional demands. It prioritizes verification, since every change is reviewed, traced to a requirement, and checked against industry standards such as MISRA and AUTOSAR before it can be committed. It is also a narrow domain, written against layered architectures, embedded resource constraints, and organization-specific interface conventions that are scarce in the data general-purpose LLMs are trained on. We present a coding assistant for automotive C/C++ that addresses both demands. First, it exposes three tools to the user over the Model Context Protocol (generate, edit, and verify), allowing developers to run verify on any code on demand rather than relying on the agent to check its own output. Second, rather than using a general-purpose model, we customize the underlying LLM for the automotive domain through prompt engineering and fine-tuning. We construct a dataset from automotive production code of AUMOVIO, where each function is paired with the requirement texts and Unified Modeling Language (UML) behavior it must satisfy. In a two-week pilot on active production projects, developers report verify as the most-used tool and confirm using it on their own code and not only on generated output. Comparing model customization strategies, 75% prefer the faster prompt-engineered configuration. MISRA- and AUTOSAR-inspired pattern checkers indicate that prompt engineering achieves a lower number of violations on C, while fine-tuning shows better conformance with architecture patterns for C++.

1 Introduction

Automotive coding assistants must support both independent verification and domain-specific conventions that general-purpose models may not know. The paper presents an MCP assistant with developer-invocable verification and compares prompt engineering with fine-tuning using production-code context.

1 Introduction

  • Automotive development requires changes to be reviewed, traced to requirements, and checked against coding standards before commit.The domain also involves real-time determinism, layered architectures, embedded resource constraints, and organization-specific interface conventions.Source: §1 · pp. 1–2
  • The assistant exposes verification as a developer-invocable tool alongside generation and editing, so developers can check code independently of generation.Its three tools are generate, edit, and verify; the same automotive rule checks can be run on demand and independently of generation.Source: §1 · p. 2
  • The paper compares prompt engineering and fine-tuning using production functions paired with requirement texts and UML behavior.Both strategies are scored with custom rule sets inspired by automotive standards; the authors treat checker output as pattern conformance, not full compliance.Source: §1 · p. 2
  • The two-week pilot found verify was the most-used tool, while generated C scored better than shipped production code under the checkers.The authors attribute the C comparison to what the checkers reward rather than a genuine quality advantage, and report a pilot with nine developers.Source: §1 · p. 2

2 Related Work

Prior work studies code generation, domain adaptation, verification, and developer productivity, but the paper positions its contribution around developer-controlled verification in production automotive work. It compares customization strategies using production code and reports a pilot on active projects.

2 Related Work

  • Domain-specific code-generation studies report that general-purpose models underuse specialized libraries, while prompting and fine-tuning can inject domain knowledge.Prior work also finds domain-adapted CodeGen4Libs outperforms non-domain-specific baselines.Source: §2 · p. 2
  • The preferred customization strategy remains unsettled: comparisons report inconsistent wins for prompting and fine-tuning across tasks.Shin et al. (2025) find prompting a stronger general-purpose model does not consistently win; Freire et al. (2026) report LoRA fine-tuning beats few-shot prompting and retrieval for domain-specific text-to-code.Source: §2 · pp. 2–3
  • Safety-critical coding work combines model adaptation with domain-aware validation, often through generation-loop checks and self-repair.Prior studies report modest self-repair gains and difficulty correcting code without external feedback.Source: §2 · p. 3
  • Productivity and adoption depend on oversight and collaboration as well as generation quality, while trust-focused work emphasizes calibrated review signals.The paper distinguishes its approach by making verification a developer-invocable peer of generation and evaluating it on production code and an active automotive project.Source: §2 · p. 3

3 Methodology

The methodology builds a production-code dataset with requirements and behavioral context, then compares prompt engineering and fine-tuning using shared domain information. It integrates custom automotive rule sets into generation and verification tools exposed through MCP.

3 Methodology

  • The assistant compares prompt engineering, which supplies domain knowledge at inference, with fine-tuning, which incorporates it into model parameters.Qwen3-32B is evaluated in prompt-engineered, fine-tuned, and curated-data fine-tuned variants, alongside Claude Sonnet 4 and Amazon Nova Pro prompt-engineered baselines.Source: §3 · p. 3

3.1 Dataset Preparation

  • The dataset pipeline extracts production C/C++ functions and augments them with API context, requirements, and UML artifacts; over 6,000 functions are produced, with 822 held out.Functions are filtered for sufficiently complete documentation, validated against custom rules, and stratified by complexity; requirement identifiers are resolved to full text and UML diagrams provide behavioral specifications.Source: §3.1 · pp. 3–4

3.2 Model Customization

  • Prompt-engineered models receive the assembled task context and full rule sets at inference, while fine-tuned models learn from production implementations using the same context.Fine-tuning uses Qwen3-32B with rank-8 LoRA adapters; one variant uses the full corpus and another a curated corpus with near-duplicates removed.Source: §3.2 · p. 4

3.3 Automotive Rule Sets Integration

  • Custom rule sets cover coding patterns and architectural patterns, and serve both as generation context and evaluation rubrics.The sets contain 57 C and 107 C++ coding rules, plus 37 C and 30 C++ architecture rules, reviewed with automotive engineers.Source: §3.3 · p. 5

3.4 Deployment

  • The MCP assistant provides generate, edit, and verify tools, with verify directly available to developers to preserve independent checking.The tools respectively produce implementations, make contextual modifications, and check code against the rule sets; the architecture separates IDE orchestration from specialized models.Source: §3.4 · p. 5

4 Evaluation Setup

The evaluation combines pattern-rule scoring against production code, generation latency, and a developer pilot. Its automated checkers deliberately cover selected patterns rather than semantic correctness.

4 Evaluation Setup

  • The study prioritizes standards-conformance checks over reference similarity because closeness to a reference implementation is a weak proxy for adherence developers must certify.The evaluation considers reference-based similarity, static and dynamic analysis, and conformance with MISRA, CERT C, and AUTOSAR.Source: §4 · p. 5
  • The evaluation compares generated functions with a human production-code baseline on 822 held-out functions, measures latency, and surveys engineers in a pilot.The held-out set comprises 574 C and 248 C++ functions.Source: §4 · p. 5
  • Automated scores come from regular-expression pattern checkers that exclude rules needing semantic, dataflow, or inter-procedural reasoning.The checkers may produce false positives and false negatives, so production use requires established automotive linters.Source: §4 · p. 6

5 Results

Pattern-conformance results differ by language, while the pilot shows developers value verification and prefer the faster prompt-engineered configuration. Developers also report less than consistent conformance, despite favorable automated scores.

5.1 Pattern Conformance Evaluation

  • For Qwen3 32B, prompt engineering yields the fewest C violations (1.22 coding, 0.54 architecture); curated fine-tuning yields the fewest C++ violations (1.25 coding, 0.02 architecture).The C++ architecture differences across Qwen variants are at most 0.08 violations per function and are not treated as meaningful; curated fine-tuning also has the highest C violations among those variants (1.67 each).Source: §5.1 · p. 6
  • Observed generation latency ranges from about 15 seconds for Nova Pro (PE) to about 110 seconds for Qwen3 32B (PE), but this is not a controlled comparison of strategies.The models use different serving infrastructure; the observed range nonetheless motivates considering faster models for interactive editing and slower ones for long-form generation.Source: §5.1 · p. 6

5.2 Comparison with the Human Baseline

  • Human production code is best on C++ coding patterns (0.84), and the checkers reward recognizable patterns rather than establishing functional correctness or standards compliance.Function-local checks can penalize code relying on caller or module-level invariants, while redundant branches consume code space and complicate timing analysis.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 Pilot Study with Developers

  • Six of eight respondents prefer PE, whose pilot configurations were faster than the fine-tuned models; the authors note that mode preference and response time are confounded.Developer feedback led the authors to read latency as important in tight read-edit-review loops.Source: §5.3 · p. 8
  • Six of nine developers report conformance always or most of the time, but only one reports always, despite automated scores at or better than human code on the rule sets.Two respondents report sometimes and one never; developers do not treat favorable automated numbers as trust.Source: §5.3 · p. 8
  • Verify receives the largest share of most-used-tool selections (4 of 10) and is used on existing human-written code as well as generated output.The item allowed multiple selections; generate and edit each received three selections.Source: §5.3 · p. 8

6 Conclusion

The paper presents an automotive coding assistant that treats verification as a separate tool and adapts models using production code paired with requirements. In the pilot, developers used verification on their own code, while customization results varied by language and developers preferred the faster configuration.

6 Conclusion

  • The assistant addresses safety-critical workflows by making verification a peer of generation and adapting models with production code paired to requirements.It was deployed on active automotive C/C++ projects, evaluated for pattern conformance, and piloted in production.Source: §6 · p. 8
  • Verify was the most-used pilot tool, and developers ran it on existing code as well as generated output.With the base model held fixed, prompt engineering had fewer C violations, while C++ strategy differences were too small to call a winner.Source: §6 · p. 8
  • The results suggest safety-critical coding assistants should be judged by developers’ ability to check code, receive timely answers, and understand where automated verdicts apply.The pilot also found that developers did not fully trust the output despite its scores.Source: §6 · p. 8

Limitations

The evaluation does not test functional correctness, and its model comparisons and human evidence have specific methodological and scope limitations.

Limitations

  • The evaluation reports no compilation or unit-test rates and does not check whether functions satisfy prompted requirements or UML behavior.Pattern-checker results therefore do not establish functional correctness.Source: Limitations · p. 9
  • The model comparison uses LoRA rather than full fine-tuning, includes no ablations or significance tests, and measures latency on different serving stacks.The small C++ margins should not be read as ranking the strategies; latency describes these deployments, not the strategies generally.Source: Limitations · p. 9
  • The two-week pilot involved nine self-selected developers and relied on self-reports, with no telemetry or record of code commits or prior edits.The authors characterize these findings as directional signals rather than effect estimates; all data and proprietary code, rule sets, and checkers come from one organization.Source: Limitations · p. 9

Ethical Considerations

The assistant is intended to support, not automate, compliance work: qualified engineers must review its output, and deployment risks include false confidence, shared blind spots, and non-conforming reuse.

Ethical Considerations

  • Every assistant-produced change needs review by qualified engineers because defective automotive code can harm people.The system is not intended to automate compliance work.Source: Ethical Considerations · p. 9
  • Passing pattern checks can still leave functional or timing requirements unmet, while added defensive patterns can raise scores without making code safer.Extra branches may also affect code size and worst-case execution time.Source: Ethical Considerations · p. 9
  • A generator and checker from the same model family may share blind spots, so the checker is kept independent; approval gates and logging also help.Logged information includes prompts, context, model IDs, and analyzer reports.Source: Ethical Considerations · p. 9
  • The study did not red-team insecure outputs, checker evasion, or vulnerabilities outside its rules; production readiness would require that work.The study uses AUMOVIO proprietary code under its data governance, and pilot participation was voluntary with only anonymized survey responses collected.Source: Ethical Considerations · p. 9

A Evaluation Metrics Overview

The evaluation treats metric selection as a trade-off because no single measure captures automotive code quality. It uses standards conformance and support metrics while declining reference similarity as a primary measure.

A Evaluation Metrics Overview

  • No single metric captures automotive code quality, and candidate metrics differ in tooling, implementation complexity, reliability, and cost.Table 2 compares metric families on those dimensions.Source: A Evaluation Metrics Overview · p. 12
  • Standards conformance was selected because it combines high tool support with high reliability.The selected standards include MISRA and AUTOSAR.Source: A Evaluation Metrics Overview · p. 12
  • Reference-based similarity was not adopted as a primary metric because its reliability is only moderate and its relevance to certification is weak.Latency is used as a support metric.Source: A Evaluation Metrics Overview · p. 12

B Dataset Preprocessing Pipeline

The preprocessing pipeline extracts automotive C/C++ functions and assembles implementation context, then applies quality checks and complexity-stratified sampling to form the dataset.

B Dataset Preprocessing Pipeline

  • The pipeline traverses automotive C/C++ repositories to extract individual functions with surrounding context for the preprocessing workflow.Context includes documentation, requirement texts, UML behavior, and API information.Source: B Dataset Preprocessing Pipeline · p. 12
  • Requirement identifiers are expanded into full texts, referenced PlantUML diagrams provide behavioral specifications, and related headers provide API and data-structure context.Header linking also gathers dependencies so generated code can use existing APIs rather than reimplement them.Source: B Dataset Preprocessing Pipeline · p. 12
  • Quality assurance corrects signatures against headers, retains functions with behavior-specifying documentation, and checks initial compliance against AUTOSAR and MISRA-C patterns.The pipeline extracts Doxygen-style and inline documentation and links it to function implementations.Source: B Dataset Preprocessing Pipeline · pp. 12–13
  • Stratified sampling by function length and complexity preserves distribution balance across training and evaluation sets, yielding approximately 6,000 contextualized function implementations.The resulting split maintains complexity-distribution balance.Source: B Dataset Preprocessing Pipeline · p. 13

C Model Architecture and Training Configuration

The fine-tuning setup combines a conventional parameter-efficient recipe with distributed-training choices shaped by long prompts. Training examples provide requirements and behavioral specifications alongside function signatures.

C Model Architecture and Training Configuration

  • Qwen3-32B was selected for code-generation performance at a scale fine-tunable on available hardware; rank-8 LoRA (α = 16) and 4-bit quantization are reproducibility choices, not contributions.Adapters are applied to attention projections and feed-forward components.Source: C Model Architecture and Training… · p. 13
  • Under quantization, the setup defaults to ZeRO-2 because quantized parameters interact poorly with ZeRO-3 parameter gathering; long prompts, rather than model size, drove most memory pressure.The prompts carry resolved requirements, UML fragments, and header context.Source: C Model Architecture and Training… · p. 13
  • The distributed SageMaker training setup reached approximately 85% GPU utilization with stable convergence, keeping each fine-tuning cycle within a development sprint.Source: C Model Architecture and Training… · p. 13

D Conversation Format Example

  • Training examples place resolved requirements, safety level, UML behavior, and the function signature in one conversational prompt, supplying obligations rather than leaving them to be inferred from the signature.Source: D Conversation Format Example · p. 13

Related papers

Loading 2610.35982v1…