Quality Metrics for LLM-Generated Asset Administration Shells: A Perturbation-Based Evaluation Approach

1 Introduction

• Modeling company-foreign or legacy products requires manual extraction from technical documents, a time-consuming, error-prone task requiring domain expertise.AAS provide standardized digital representations of hardware and software assets and enable interoperable digital twins in Industry 4.0.Source: §1 · pp. 1–2
• LLM-based AAS generation tools remain unreliable, while non-unique AAS content and scarce labeled datasets complicate the choice of suitable quality metrics.Property names, units, and structures may vary, and generation can combine multiple information-extraction steps in one prompt-response cycle.Source: §1 · pp. 2–3
• The study uses controlled perturbations to assess which metrics reflect AAS quality degradation and how quality varies across perturbations, models, manufacturers, and product segments.The research questions also examine how perturbation types and model characteristics affect generation quality.Source: §1 · p. 3
• The paper contributes AAS-specific full-reference and no-reference metrics, a perturbation-based evaluation method, and empirical analyses across generation conditions and product groups.The authors state that these contributions support tool selection and benchmarking, automated quality assurance, and integration of AI assistants into industrial processes.Source: §1 · p. 3

2 Background and Related Work

• Traditional software quality models do not readily extend to AAS submodels, so this work empirically evaluates metrics tailored to AI-generated artifacts rather than proposing a broad quality model.BLEU and ROUGE rely on n-gram overlap and are not well suited to structured outputs with short fields and semantically equivalent representations.Source: §2 · pp. 3–4
• AAS generation combines property recognition, value and unit normalization, and template mapping without labeled data for each step, motivating end-to-end evaluation with adapted metrics.Adaptations include soft matching to accommodate permissible variations and outputs without a unique correct representation.Source: §2 · p. 4
• RAG evaluation offers related alternatives, including downstream-task measures and LLM judgments, with some methods assessed against human judgments and controlled retrieval quality.Sander and Dietz use follow-up-question support; RAGAS uses prompting and embeddings; Saad-Falcon et al. apply prediction-powered inference.Source: §2 · p. 4
• The paper introduces a framework for systematically benchmarking AAS quality metrics without high-quality labeled reference data, drawing on IE and RAG evaluation.Source: §2 · p. 4

3 Study Design

• The pipeline combines datasheet content with a reference-derived property dictionary, generates structured property records with an LLM, and evaluates the resulting AAS.The dictionary specifies expected property names, value types, and units; perturbations are introduced during generation, followed by metric-based assessment.Source: §3 · pp. 4–5, Figure 2 · p. 5

3.1 AAS Quality Metrics

• The 18 evaluated metrics vary by reference type, matching strategy, and information-extraction measure, while full-reference and no-reference metrics assess complementary aspects of generated AAS.Property matching uses normalized Levenshtein distance or embedding cosine similarity followed by one-to-one Hungarian assignment; value comparisons use numeric tolerances or string-similarity thresholds.Source: §3.1 · p. 6

3.2 Perturbation-Based Evaluation Approach

• Metric sensitivity is measured through rank correlation between perturbation intensity and metric scores, using prompt degradation, temperature variation, and reduced model size.Prompt degradation varies in intensity from 0.0 to 1.0 and includes semantic, grammatical, contextual, and lexical changes.Source: §3.2 · p. 7

3.3 Dataset and Preprocessing

• The dataset contains 200 products from four manufacturers, selected for diversity and filtered for usable technical submodels and at least 10 technical properties per product.Most products fall in ECLASS segments 27, Electrical engineering, and 51, Fluid Power; manufacturer-specific property dictionaries specify names, definitions, value types, and units.Source: §3.3 · pp. 7–8, Table 3 · p. 8

3.4 Models and Experimental Procedure

• Experiments evaluate GPT-4o-mini, Qwen3 variants, and DeepSeek-R1 variants, with generated outputs structured as records containing property names, values, units, and text references.The study runs the evaluated metrics on generated AAS and applies perturbations to assess metric sensitivity.Source: §3.4 · pp. 8–9

4 Results

• 6,400 submodels covered 32 experimental conditions and 200 product datasheets, from which 231,276 properties were extracted.Source: §4 · p. 9
• 83.5% of prompted properties were extracted on average, versus 96.8% for gpt-4o-mini in the unperturbed baseline; Fig. 3 ratios above 100% indicate hallucinations or duplicates.Ratios below 100% indicate incomplete extractions.Source: Figure 3 · p. 9, §4 · p. 10

4.1 Metric Performance

• Full-reference recall with cosine-matched values and exact-matched names led (ρ = 0.65); no-reference F1 had the strongest monotonic relationship, while soft matching showed no clear advantage over exact matching in no-reference metrics.Source: Figure 4 · p. 10, §4.1 · pp. 10–11

4.2 Impact of Perturbations

• Across perturbations, performance fell by 35–40 percentage points, with a sharp decline below approximately 8 billion parameters and more gradual declines for larger models.Source: Figure 5 · p. 11, §4.2 · p. 11
• Temperature increases and prompt degradation produced more linear declines, while Qwen3:32b performed best in the baseline scenario.Source: §4.2 · p. 11

4.3 Manufacturer and Product Segment Analysis

• Manufacturer differences were significant (p < 0.001; η2 ≈ 0.13–0.17), but product-specific characteristics had stronger influence; Fluid Power tended to score higher than Electrical Engineering, with substantial within-segment variation.No-reference metrics also showed a ceiling effect, with many near-perfect scores.Source: §4.3 · p. 12, Figure 6 · p. 12

• Exact matching of property names outperformed more complex similarity-based approaches, whereas soft value matching accommodated spelling variants and semantically equivalent phrasing.Source: §5 · pp. 12–13
• No-reference metrics support scalable quality assurance when reference AAS are unavailable, and the identified metrics can support benchmarking and acceptance decisions.Source: §5 · p. 13
• Single-metric optimization can cause metric overfitting; no-reference metrics are indirect and may show ceiling effects, so robust evaluation may require complementary metrics and human validation.Source: §5 · p. 13
• Controlled perturbations enable reproducible assessment of metric sensitivity without labeled ground truth and may extend to other structured information extraction tasks.Source: §5 · p. 13
• Quality varies with model characteristics and data properties: architecture and training can outweigh parameter count, while product-segment differences suggest domain-specific influences.Source: §5 · p. 13

Construct Validity

• Quality is measured mainly through correctness and completeness of extracted technical-data property names and values, not contextual appropriateness, modeling-convention compliance, or downstream usefulness.No-reference metrics assess plausibility indirectly and do not detect semantic errors.Source: §6 · p. 13

Internal Validity

• The approach assumes stronger perturbations lower AAS quality, but that relationship may not be strictly monotonic across all models and products.Threshold optimization against perturbation intensity may favor metrics aligned with the perturbation design.Source: §6 · p. 14

Conclusion Validity

• Statistical conclusions may vary with product, model, and perturbation effects; some manufacturer and product-segment analyses use smaller effective samples.Spearman’s rank correlation captures monotonic relationships but overlooks non-monotonic effects.Source: §6 · p. 14

External Validity

• Generalizability is bounded by 200 products from four manufacturers, technical-data submodels, selected ECLASS segments, and a selected set of LLMs.Results may not transfer directly to other industrial domains, AAS submodels, documentation styles, model families, or future model generations.Source: §6 · p. 14

Conclusion

• Exact name matching and similarity-based value matching provide the most reliable indicators of quality degradation across perturbation types.The findings support soft matching for benchmarking when property values have non-unique spelling and phrasing.Source: §7 · p. 14
• The perturbation-based methodology offers a general way to compare metrics when labeled ground truth is scarce or unavailable, with potential transfer to other structured information-extraction tasks and LLM systems.Source: §7 · p. 14

Future Work

• Future work will incorporate expert assessments to align automated metrics with human judgment and calibrate them in realistic evaluation settings.Source: §7 · p. 14
• Planned extensions include semantics-aware and LLM-based metrics, additional AAS submodels and industrial domains, and adaptive pipelines for continuous quality monitoring and improvement.Source: §7 · p. 15