DTScribe: Generating Models for Digital Twin Reporting using Local Large Language Models

1 Introduction

• DT reporting remains time-consuming and unstructured, making consensus harder for experts and comprehension harder for nonexperts.A common system view can facilitate interdisciplinary collaboration, improve cross-team communication, and reduce technical ambiguity.Source: §1 · p. 1

1.1 Challenges

• Modeling domain knowledge is slow and labor-intensive, while DT documentation may also be too sensitive for cloud processing.The authors identify limited SME and researcher experience with ontology engineering, alongside privacy and sovereignty requirements, as barriers.Source: §1.1 · p. 2
• DT reporting also needs to serve nontechnical stakeholders and keep pace with evolving data and processes across the DT life cycle.The paper identifies a gap between static documentation and dynamic reporting, particularly when reports are not first-class CI/CD artifacts.Source: §1.1 · p. 2

1.2 Proposed Approach

• DTScribe uses LLMs to extract DTDF characteristics from documents and generate an OML description, targeting local, privacy-preserving deployment.The generated ontology is designed to work with DTInsight and fit into its continuous reporting pipeline.Source: §1.2 · pp. 2–3

1.3 Research Questions

• The paper asks how to build a local-LLM architecture for characteristic extraction and valid OML generation, how to evaluate extraction, and whether refinement improves validity.The third research question compares iterative reasoner-guided repair and LLM-as-a-judge with zero-shot extraction.Source: §1.3 · p. 3

1.4 Contributions and Structure

• The contributions include automated document-to-OML generation, iterative refinement loops, DTInsight integration, and an empirical evaluation of model and deployment trade-offs.The paper also reports recommendations on model size, judge use, clustering, and OML refinement.Source: §1.4 · p. 3

2.1 DT Reporting with DTInsight

• DT reporting is systematic documentation and communication of a twin’s architecture, capabilities, operational state, and life cycle to stakeholders.The DTDF provides a domain-agnostic, twenty-one-dimension framework, while standards such as ISO 23247 and AAS address different concerns.Source: §2.1 · p. 4
• DTInsight uses ontology-based descriptions to generate a reporting page and DT constellation that stakeholders can explore and update through a continuous pipeline.Its underlying DTDF ontology remains largely manual and effort-intensive to create and maintain.Source: §2.1 · pp. 4–5

2.2 Ontologies and OML

• OML separates a vocabulary defining concepts and constraints from a description that instantiates them; DTScribe generates the DTDF OML description using the existing vocabulary.The paper frames this as instance model generation, which assumes a pre-existing metamodel.Source: §2.2 · pp. 5–6

2.3 Large Language Models

• LLMs can omit context in long inputs, while retrieval-augmented generation retrieves relevant document chunks to ground responses in domain-specific information.The background describes RAG as useful for information extraction and aspect-based summarization.Source: §2.3 · p. 6
• Prompt chaining decomposes complex workflows into narrower prompts, though the background notes that larger context windows and built-in reasoning may make such techniques less necessary.The cited discussion specifically connects prompt chaining with improving output quality for smaller models with small context windows.Source: §2.3 · p. 6

3 Approach

• The approach ingests user documents, extracts DTDF characteristics, and generates an OML description for the DT constellation visualization.Two optional loops refine extracted characteristics and repair or validate the generated OML.Source: §3 · pp. 6–7

3.1 Component A: DT Documents

• Although the pipeline supports .docx and .txt, this article processes PDFs of academic papers describing a DT exemplar.The input documents are assumed to describe a DT, avoid explicit DTDF terminology, and contain no malicious prompt injections.Source: §3.1 · p. 7
• Preprocessing converts document text to Markdown, removes academic-paper references, preserves structure and boundaries during chunking, and embeds segments for clustered extraction.The authors report that references degraded retrieval quality in initial experiments.Source: §3.1 · p. 7
• The authors plan to extend the workflow with multimodal processing, such as converting figures to text to enrich retrieval context.This is described as future work.Source: §3.1 · p. 7

3.2 Component B: LLM

• The configurable LLM component handles both structured extraction of DTDF characteristics and generation of an OML description.The configuration allows model comparisons without changing pipeline logic.Source: §3.2 · p. 7
• The study compares six LLMs because no single model is optimal for both required tasks.The alternatives vary in architecture, scale, and deployment modality, with trade-offs in latency, cost, reliability, and hardware requirements.Source: §3.2 · pp. 7–8
• Model choices account for evidence-based extraction and strict OML validity, with final selections based on performance and output quality across both tasks.Source: §3.2 · p. 8

3.3.1 Clustered Extraction with RAG

• Clustered extraction degraded extraction accuracy, rejecting the hypothesis that grouping would improve retrieval precision and context management.The authors include this strategy to document the pipeline and the testing and rejection of the hypothesis.Source: §3.3.1 · p. 8
• The twenty-one characteristics are grouped by semantic similarity into six clusters, whose tailored retrieval context and prompts produce structured JSON descriptions for ontology generation.For example, the Orchestration cluster includes C7, C8, C12, and C18.Source: §3.3.1 · p. 8
• Extraction prompts divide the task into reasoning, evidence-grounded description, and validation, with missing evidence marked “Not in Document.”The prompts also specify technical correctness, completeness, and consistency with judge feedback when available.Source: §3.3.1 · p. 8

3.3.2 Component D: LLM-based Evaluation

• A secondary LLM evaluates descriptions against source evidence and iteratively refines low-quality items, preserving characteristics scoring at least 4.The evaluator uses factual consistency, specificity, and relevance criteria; the implementation uses cloud-hosted GLM-4.7, though users can substitute a local model or bypass evaluation.Source: §3.3.2 · pp. 9–10

3.4.1 Split Generation

• Component characteristics are multi-attribute and interdependent, so the LLM generates their OML instances; independent description characteristics are constructed programmatically.The split is intended to reduce context length, inference cost, and task complexity for smaller LLMs.Source: §3.4.1 · p. 11

3.4.2 Component F: LLM-based Ontology Repair

• The repair loop addresses common generation errors, including undefined references, type mismatches, and malformed OML syntax.The authors assume that independently represented programmatic descriptions and LLM-generated components do not create interaction errors during repair.Source: §3.4.2 · pp. 12–13
• A reasoner checks the integrated ontology for validity, and its error messages are injected beside problematic lines to guide iterative LLM repair.The loop repeats until validation succeeds or the maximum retry limit is reached.Source: §3.4.2 · p. 13

3.5 Approach Summary

• The architecture integrates local LLMs, symbolic verification, and iterative refinement to generate syntactically valid OML description models for DTInsight visualization.The system is designed to support local LLM deployment for secure, automated processing of digital twin documentation.Source: §3.5 · pp. 14–15
• The pipeline converts documentation to Markdown and chunks it for retrieval-augmented generation before extracting DTDF characteristics.The extraction stage tests six semantic clusters and includes optional LLM-as-a-judge refinement, but clustering degraded extraction accuracy.Source: §3.5 · p. 14
• Ontology generation uses LLMs for complex component instances and programmatic construction for simple descriptions, followed by reasoner-guided repair.Reasoner error reports are fed back into the LLM context to guide self-correction.Source: §3.5 · p. 14

4.3 Extraction Metrics

• The evaluation tracks extraction accuracy and extraction quality, alongside generation time.Accuracy is the primary benchmark against partial ground truth; quality is used as a feedback signal in the judge loop.Source: §4 · p. 15, §4.3 · p. 16

4.1 DT Exemplar Papers

• The benchmark uses five digital twin paper exemplars, with four recent conference papers and an incubator case study as a recurring example.None of the four recent exemplars explicitly uses the DTDF, making them suitable for evaluating extraction against the framework.Source: §4.1 · p. 15

4.2 Establishing Partial Ground Truth

• Three authors independently rated the presence of twenty-one characteristics across the five exemplars, yielding Fleiss’ κ = 0.625, reported as substantial agreement.Majority voting classified a characteristic as present when at least two raters agreed.Source: §4.2 · p. 16

4.3.1 Extraction Accuracy

• Extraction accuracy treats each characteristic as a binary presence label and compares the LLM-judged extraction vector with the human-derived ground-truth vector.It ignores the textual quality of the extracted description.Source: §4.3.1 · p. 16

4.3.2 Extraction Quality

• Extraction quality uses an LLM to assess description precision, relevance, and formulation without requiring a single reference summary.The authors motivate this reference-free approach by noting that stakeholder reporting needs vary from high-level summaries to technical specifications.Source: §4.3.2 · pp. 16–17

4.4 Ontology Generation Metrics

• Ontology metrics measure success at each retry step, cumulative success by a retry budget, and conditional success among runs not yet validated.The OWL reasoner determines validity by detecting syntactic and logical errors.Source: §4.4 · p. 17

5 Evaluation

• The evaluation examines local-LLM capability, metrics for extraction completeness and accuracy, and whether judge-guided repair improves OML structural validity over zero-shot extraction.The repair analysis includes iterative reasoner-guided repair, while the architectural questions also cover clustering and LLM judging.Source: §5 · pp. 17–18

5.1 Experimental Setup

• Six open-source LLM configurations were evaluated, with EmbeddingGemma for vector embeddings and GLM-4.7 as the judge.Extraction and ontology generation used default sampling parameters; the judge used temperature 0 and a fixed seed.Source: §5.1 · pp. 18–19

5.2 Experimental Process

• The ablation varied document clustering and judge feedback, and measured OML success before and after the refinement loop.The four configurations were Base, +Cluster, +Judge, and +Cluster+Judge; first-pass success isolated performance before OML retry logic.Source: §5.2 · p. 19
• Each configuration ran 25 times; the full pipeline ran 3,000 experiments over about five days using cloud infrastructure to parallelize computation.Base and +Cluster could also run fully locally with Ministral3 3B and 8B.Source: §5.2 · p. 19

6 Results

• The reporting page ingested by DTInsight used a +Judge configuration for P2 with gpt-oss-20b, whose extractions were around 99% accurate.Source: §6 · p. 19

6.1 RQ3.1: Local LLM Capability

• Baseline models, including Ministral3 3B, exceeded 70% extraction accuracy, though larger models generally did better and P5 was difficult across model sizes.The alternative view in Figure 4 suggests scaling improves accuracy in most cases, but shows dips for P4 and P5, likely due to missing source information.Source: Extraction Accuracy · p. 20, Figure 4 · p. 20
• On the first OML generation attempt, all but the largest model had success rates around or below 40%; the largest achieved around 60%.Across models, small and medium baselines could extract above 70% depending on the paper, yet consistently struggled to generate OML models on the first try.Source: Extraction Accuracy · pp. 20–21

6.2 RQ3.2: Impact of Clustering and LLM Judging

• Clustering’s accuracy degradation was most pronounced for larger models, which may produce unhelpful “Not in Document” outputs when context is fragmented.The authors hypothesize that smaller models may guess over gaps, whereas larger models adhere more strictly to the fragmented context.Source: §6.2 · p. 22
• The +Judge configuration improved extraction accuracy by an average of 2.11% but tripled processing time, while +Cluster reduced accuracy by 18.51% overall.Combining the methods raised processing time by a factor of 9 to over 45 minutes per run, the least efficient trade-off.Source: §6.2 · pp. 21–22, Table 3 · p. 21, Table 4 · p. 22

6.3 RQ3.3: Effectiveness of Iterative OML Repair

• Repair-loop benefits depend on model size and extraction configuration: small models gained little, while low-accuracy clustered inputs could make OML generation easier by yielding mostly “Not in Document” characteristics.The largest model showed diminishing returns, with a 19.0% improvement, while gpt-oss-20b and Qwen3-Next-80B-A3B improved by 33–39%.Source: §6.3 · p. 23
• With repair, 20B models reached OML generation success rates comparable to a 120B model, with improvements up to 39.6%.Medium-sized models generally required two to three retries to reach peak success; the largest model’s gains diminished beyond one or two retries.Source: §6.3 · p. 23, Table 6 · p. 23
• The authors recommend at least two rounds of reasoner-based refinement for medium-sized models and three to four generation attempts for optimal reliability.They recommend using judge loops only when computational budget permits and avoiding clustering for semantic extraction.Source: §6.3 · pp. 23–24

7.1 Addressing Pain Points

• Generated characteristics and editable OML drafts provide shared starting points for expert negotiation, clearer conceptualization, automated diagramming, and interactive, continuously updated documentation.Prior work cautions that time spent negotiating definitions may be necessary rather than wasteful.Source: §7.1 · p. 24

7.2 Expected Benefits

• Processing existing documentation in minutes can lower the entry barrier compared with manual reporting that takes hours, but exact usability gains remain future work.The authors also leave incremental documentation updates for future work.Source: §7.2 · p. 25

7.3 Lessons Learned

• Vision-LLM preprocessing was discarded for hallucinations or omissions, roughly one minute per page latency, instability, and an added failure point; preliminary hyperparameter tuning yielded no substantial benefit.Specialized multimodal parsing tools may offer a more robust alternative.Source: §7.3.1 · p. 25, §7.3.2 · p. 25

7.3.3 Local/Hosted Approach Tradeoff

• An 8 GB VRAM consumer GPU supported local extraction and generation, but retries and large contexts caused slowdowns and small models sometimes failed structured outputs; cloud services produced higher-quality outputs but bring privacy, reproducibility, and availability costs.The authors describe deployment choice as dependent on operational constraints.Source: §7.3.3 · pp. 25–26

7.3.4 LLM-as-a-Judge

• The judge loop raised extraction accuracy on average but did not consistently improve OML generation; scores may favor polished writing and remain misaligned with expert judgment.The authors identify complex, multi-criteria scoring and domain-specific terminology as challenges for judge accuracy.Source: §7.3.4 · p. 26
• For around 2% marginal average accuracy gain, adding LLM-as-a-judge raises processing time from 317.5 seconds (Base) to 905.1 seconds (+Judge), discouraging latency-sensitive use.The authors note that judge scores can be biased toward polished writing and remain misaligned with human expert judgment, so they should be treated as automated proxies for iterative refinement.Source: §7.3.4 · p. 26

7.4.2 Construct Validity

• Evaluation metrics are constrained: partial coauthor-annotated ground truth (κ = 0.625) leaves shared-bias risk, while reasoner validity can reward oversimplified OML and generation success depends on extraction accuracy.The authors note that standard recall and accuracy metrics were unavailable because robust mappings of DTs to DTDF descriptions were lacking.Source: §7.4.2 · p. 27

7.4.4 External Validity

• Generalizability is limited by the small set of papers, models, and configurations and homogeneous academic inputs; complex, distributed company documents may make observed extraction quality an upper bound [8].The authors nevertheless report confidence in their recommendations based on the experiments.Source: §7.4.4 · pp. 27–28, §7.4.3 · p. 27

8.1 LLMs for DTs

• To the authors’ knowledge, DTScribe is the first work targeting DT reporting; related DT approaches instead address integration, simulation, 3D models, or natural-language interaction.Its focus is extracting and presenting concrete DT characteristics from text reports.Source: §8.1 · p. 28

8.2.1 Guiding LLMs

• Prior work guides generation with metamodels or repairs SysML errors; DTScribe uses OML’s semantic reasoner to assess multiple rounds of LLM-based repair.The paper contrasts this with approaches that generate valid JSON for transformation or use ontology-backed verification as future work.Source: §8.2.1 · p. 28
• Chen et al. synthesize candidate Mermaid graphs, with five candidates improving Llama3.1-8B quality; DTScribe instead automates DT characteristic extraction and ontology-based visualization.The cited work is described as especially efficient with smaller LLMs.Source: §8.2.1 · pp. 28–29

8.2.2 Requirements and Architecture Models

• Requirements and architecture studies generate problem diagrams, architecture diagrams, or state machines, while this work models DT characteristics as a conceptual architecture and plain-text report.The conceptual architecture is represented as a DT constellation.Source: §8.2.2 · p. 29

8.2.3 Ontologies and Knowledge Graphs

• LLM-generated ontology research has focused mainly on OWL rather than OML; two unpublished NASA JPL efforts involve OML but do not target DT reporting.DTScribe generates OML description models from documents and iteratively refines them using reasoner feedback.Source: §8.2.3 · p. 29
• Other ontology and knowledge-graph systems use hand-built ontologies, RAG, competency questions, or user interaction; DTScribe uses DTDF vocabulary to extract characteristics and automate reporting from documents.The user may indicate source documents and correct the output model, but is otherwise not in the loop.Source: §8.2.3 · p. 29

9 Conclusion

• DTScribe extracts textual and structural DTDF characteristics, generates OML description models, and uses DTInsight to produce interactive visualizations and reporting pages.Source: §9 · p. 30
• The evaluation supports automation with local LLMs; the authors recommend medium-sized models, two or three refinement loops, and avoiding LLM-as-judge and clustering approaches.Source: §9 · p. 30

Future Work

• Future work proposes adding multimodal documents, including video, audio, meeting notes, and images, and investigating fine-tuned LLMs for extraction and OML generation.Source: Future Work · p. 30
• The authors propose exploring SHERPA or Graph of Thoughts prompting and automatically searching existing knowledge to build the ontology.Source: Future Work · p. 30
• Current reports are general rather than stakeholder-customized; future work proposes persona-based metrics and reports tailored to users’ expertise.Source: Future Work · p. 30
• Future work proposes user collaboration to fill missing characteristic details and update the DT model, alongside wider-scale validation and alignment of DTDF with emerging standards.Source: Future Work · p. 30

• Table 7 presents DTDF characteristic-extraction prompts across clusters, with a continuation on the next page.Source: Table 7 · pp. 37–38
• Figures 6–10 report cumulative OML generation success across four repair attempts for P1–P5, with n = 25 for each case.Source: Figure 6 · p. 39, Figure 7 · p. 39, Figure 8 · p. 40
• Table 8 lists RAG querying keywords for each characteristic cluster.Source: Table 8 · p. 42