Source-linked AI summary
Context operations to architecture modelling output from large language models and evaluation criteria for their use in systems engineering design
Vinicius Kaster Marini, Petter Krus
TL;DR
Systems engineering needs ways to use LLMs despite probabilistic outputs and incomplete or inaccurate design artefacts. The paper formalizes context operations for architecture modelling and evaluates generated models against design intent, finding that context placement and µ-Templates influence convergence and compliance while some outputs remain incomplete.
Problem
LLM outputs for engineering design can be probabilistic, incomplete, inaccurate, and insufficiently compliant with design intent, motivating careful evaluation of generated work products.
Method
The paper formalizes context operations for LLM-based engineering design and evaluates modelling-as-code outputs through assertions against architecture intent.
Results
µ-Templates have a convergence effect toward desired model format, structure, and compliance, while context primacy and recency affect smaller models under substantial workloads.
Takeaways & Limitations
Context handling and evaluation assertions support convergence criteria for modelling loops and the potential combination of deterministic and probabilistic systems-design approaches.
Takeaways & Limitations
Generated responses can omit components or place subcomponents incorrectly, although code syntax and modelling conventions may still be satisfied.
Abstract
from arXiv · showhide
The development of generative artificial intelligence resources enables opportunities of speeding up systems and engineering design work. This contribution introduces a framework of formal operations for assembling context in LLM-based engineering design. This framework involves the assembly of modular context units, including policy prompts, reference units with persistence, and user questions with prompt vectoring. This approach enables the systematic structuring of interactions with generative models. A formal method for evaluating modelling-as-code LLM outputs is also presented, which enables the evaluation of compliance to intent from LLM answers and thereby asses the support from LLMs for systems architecture modelling.
1 Introduction
LLMs offer potential support for early systems engineering and design, but their probabilistic outputs require careful review against design intent. This contribution positions a structured approach to context operations for architecture modelling.
- Motivation: Early design is a target context for LLM support because conventional design automation approaches have fallen short.The paper frames LLMs as conversational tools pretrained on large internet-scale corpora.
- Motivation: Probabilistic LLM outputs can lack matter-of-factness and therefore require careful review against engineering design work products.The paper identifies this as a central challenge for engineering and design use.
- Research question: The motivating question is how to improve GenAI accuracy relative to design intent in system architecture modelling.The paper addresses this question through systematic context input for architecture modelling.
- Related applications: LLMs are being explored across systems modelling, system tools, safety, reliability, dependency analysis, and human-machine design assistance.Examples include model generation, HARA and FMEA support, STPA, assurance cases, dependency exploration, and sequenced prompt chains.
- Contribution: The contribution develops a formal structured approach to LLM operation and demonstrates how it works for systems engineering and design.It aims to support human-machine collaboration rather than remove the designer from the process.
2 Overview of LLM-based engineering approaches
LLM-based engineering approaches organize intent, context, methods, tools, and outputs to support different design work products. Their effectiveness depends on how context and prompting are structured around the intended model or representation.
- Approach dimensions: Engineering use cases vary by intent, context, meta-model, method, tool or environment, and output type.Intent concerns requirements, architecture, and X-ability; context distinguishes product and mission design; meta-models include modelling languages, procedure models, boilerplates, and concept maps.
- Methods: Instructions to chatbox, code to chatbox, and custom applications are the main methods for conveying context to LLMs.Custom applications embed prompts in purpose-specific workflows and perform actions through API calls.
- Context support: Zero-shot, one-shot, few-shot, and retrieval-augmented generation provide progressively richer contextual support for query intent.Exemplars convey relevant grammar, semantics, and situations, while RAG draws on domain-specific information.
- Prompting: Instruction prompts provide general guidance, while formulae, rules, and templates impose more structured control over generated outputs.These mechanisms remain probabilistic but target compliance with specified relationships and structures.
- Outputs: Chatbox workflows commonly produce models, sentences, tables, or spreadsheets, with models requiring an underlying grammar.MBSE and custom application environments are among the tools used to interface with LLM outputs.
3 Degraded attention and mitigating factors
LLM attention degrades when context is too long, distant, ambiguous, complex, or structurally incoherent. The paper identifies placement, relevance, role-based positioning, tagging, and workload reduction as mitigation strategies.
- Degraded attention: Long context workloads can exceed model capacity and reduce attention to information in the middle or far from the main query.Tokens near the context ends are more likely to receive attention, producing the lost-in-the-middle effect.
- Degraded attention: Ambiguous, overlapping, or conflicting context units can cause confusion and misinterpretation, especially when their relation to the query is unclear.The paper links this ambiguity to reduced recall probability P[tki ∈AM].
- Degraded attention: Complex evidence degrades context processing because multi-hop connections across context positions are challenging for LLMs to maintain.The difficulty concerns reasoning about specific evidence relative to the main query.
- Degraded attention: Context units ordered incoherently relative to work-product intent can be misinterpreted by the sequence-dependent attention mechanism.The paper represents context as a concatenated workload CQ(i) = {tk1..tkn}.
- Mitigation: Mitigation strategies include placing relevant or duplicated information near context ends, positioning units by role, tagging important units, and reducing workload size.These strategies exploit primacy and recency while reducing competition for attention capacity.
4 Context assembly levels in LLM use cases
Systems-engineering LLM use cases assemble supportive context through different formats, adapters, interpreters, and composition strategies. The literature connects context assembly with work products that better comply with design intent, while noting attention-related constraints.
- LLM use in safety engineering has produced incomplete and inaccurate specifications, motivating attention to context workload and work-product quality.
- Context assembly supports systems design through conveyor formats, grammar formats, application adapters, and downstream interpreters.These elements make context machine-readable and enable outputs to become actionable in modelling environments.
- Unit composition: Use cases combine pretraining knowledge, accumulated question-answer turns, formal grammar exemplars, and alternating context workloads with deterministic model-building code.
- Unit order: Context units are ordered to address primacy, recency, memory, and attention degradation during LLM processing.
- Modelling-as-code: Modelling-as-code sequences directives with template code and supports modular intake of separate context units around the main query.
- Context memory: Persistent context units can be reused through retrieval databases or structured storage, reducing workload per query and mitigating attention degradation.
5 Model generation parameters
The paper formalizes an LLM function class in which users select a model and configure parameters governing context intake, context capacity, answer length, and sampling behavior.
- The LLMX function class represents selectable language-model operators, with X replaceable by a particular LLM.
- Equation 1 frames model selection as choosing one model and version from several available model lineups.
- LLMX parameters include model size, temperature, context length, answer length, context workload length, and tuning parameters such as topp and topk.
- The main operational parameters are context capacity CM(i)len, answer capacity AM(i)len, and user-determined workload length CQ(i)len.
- Parameter settings and syntax vary across API libraries, model providers, and specific models, particularly for context and answer-length limits.
6 Context operations
The paper models an LLM call as an ordered assembly of policy, reference, and question units into one context workload. Generic assembly places upstream context before the question and downstream references after it, while role-focused assembly assigns policy, references, and exemplars distinct positions.
- 6.1 Generic context formulation: A complete LLM call assembles context units into a single workload CQ(i) that supports the model’s processing of the intended request.
- 6.1 Generic context formulation: Generic assembly places upstream context before the core question and downstream context after it, with multiple units permitted in each group.
- 6.1 Generic context formulation: The resulting answer AM(i) is generated by LLMX from the assembled context workload CQ(i).
- 6.2 Role-focused units onto single-call: Role-focused assembly places policy rules first, imported references next, and the core question with exemplars afterward to exploit primacy and recency.
- 6.2.1 Policy units: Policy units can include recurring global directives, boundary prompts, and additional single-call policy context units.
7 Case study
The case study evaluates how cumulative context treatments and different LLM models support modelling-as-code generation for a hybrid SAR UAV architecture. It validates generated outputs against plantUML syntax and architecture requirements, using manual correction and metrics to assess compliance.
- Case study setup: The case study assembles context units to generate a component architecture model for the hybrid SAR UAV Suthern-Cross.The modelling task starts from required functionality and uses modelling-as-code to generate a model from an LLM answer.
- Context treatments: Four cumulative treatments progress from the query alone to global and boundary prompts, prompt vectors, and reference priors.T1 includes Qc; T2 adds Gsp and Bsp; T3 adds Ov and Qv; and T4 adds PP.
- Model treatments: Five LLM treatments compare model characteristics under the same context workload, spanning local, cloud-capable, and frontier models.The assortment is selected with regard to processing resources, parameter scale, context length, and answer-length capacity.
- Output validation: Generated model outputs are evaluated against plantUML syntax conventions and architecture requirements, with correctness requiring compliance with both.The verification process exports raw codeblocks, checks syntax, iterates corrections, and then verifies architectural requirements.
- Evaluation metrics: The evaluation metrics quantify answer characteristics together with syntax and architecture compliance across the generated model corpus.The framework accounts for LLM processing statistics and verification criteria applied to the raw model codeblock exported to a model file.
8 Results
The results show that context treatments, especially policy and µ-Template prompt vectors, improve compliance of generated architecture models, while model capacity affects structural detail and processing time. Additional references provide only marginal benefit because longer workloads can trigger lost-in-the-middle effects.
- 8.1 Modelling examples: The T1 Claude Sonnet 4.6 model produced 38 components and 46 flows but lacked component ports and specified flows, so it did not comply as a component architecture.The output possibly represented a heavy SAR UAV design, but deviated from the intended architecture type.
- 8.1 Modelling examples: The T2 Nemotron3-super model contained 28 elements and 32 flows, but its raw output required correction because of syntax errors and still included dangling ports.The corrected model represented ports as nested components and flows as declarations between port pairs.
- 8.1 Modelling examples: µ-Templates embedded in prompt vectors improved component, port, and flow representations, with T3 producing a valid component architecture after correction.T4 achieved the best results only by a marginal difference, while additional references were limited by the models’ ability to process and integrate them.
- 8.2 Modelling analysis: Higher-capacity models generated deeper and more structured outputs, with line counts ranging from 31 to 195 and flow counts from 21 to 109, while larger cloud models responded faster.Processing times ranged from up to 08:21 for local or smaller models to 0:30 for larger cloud models.
- 8.2 Modelling analysis: The largest quality gain occurred when µ-Templates were combined with policy and prompt-vector ancillaries in T3, while T4 was constrained by lost-in-the-middle attention.µ-Templates practically annulled component misses in T3 and T4, whereas component misses remained high on smaller models and increased under T2.
- 8.3 Discussion of results: The experiment evaluated 20 queries across context workloads and selected LLMs by generating system-architecture PlantUML code models against requirements and assertions.The quantitative analysis covered the generated models and their compliance with the defined requirements.
9 Conclusions
The contribution presents a modular context-operations framework and demonstrates it in an aircraft-design case study with architectural verification. The results support discussion of LLM modelling capability and motivate future expansion toward conversational and agentic modelling.
- Contribution: The framework was successfully operated in a heavy SAR UAV aircraft-design case study.The case study applied context operations through a chatbox application.
- Contribution: The case study moved beyond modelling examples toward modelling requests and verification through measured architectural properties.This approach evaluates generated modelling results against architectural characteristics.
- Results: Evaluating modelling quality from modular context workloads enabled discussion of LLM modelling capability under a single query.The reported results were described as consistent with current understanding of LLM operation in system design.
- Future work: Future work will extend modular workloads to conversation threading and agentic modelling, requiring expanded learning bases for model counting and validation rules.The proposed expansion includes fine-tuning model counting and validation rules.