PaperlayerView exact revision on arXiv

Source-linked AI summary

An Evaluation of Requirements Modeling for Cyber-Physical Systems via LLMs

Dongming Jin, Shengxin Zhao, Zhi Jin, Xiaohong Chen, Chunhui Wang, Zheng Fang, Hongbin Xiao

arXiv:2408.02450v1cs.SE

TL;DR

CPS requirements modeling from natural-language documents is difficult and labor-intensive, while systematic evidence on LLMs’ ability to perform it is needed. The paper defines entity-recognition and interaction-extraction tasks, builds CPSBENCH, and evaluates seven LLMs. The models show limited effectiveness, with around 60% recall, and the paper analyzes their limitations and hallucinations.

  • Problem

    CPS requirements are difficult to model accurately because complex domain documents require labor-intensive, error-prone identification of problem-diagram elements, and systematic evaluation of LLM capabilities for this task is needed.

    Source: §I · pp. 1–2

  • Method

    The paper formalizes CPS requirements modeling as entity recognition and interaction extraction, constructs CPSBENCH from real-world requirements documents and tutorial cases, and evaluates seven LLMs using few-shot reasoning.

    Source: §I · p. 2

  • Results

    LLMs achieve only around 60% recall; GPT-3.5 performs best in entity recognition and GPT-4 in interaction recognition, with stronger understanding of general than specialized concepts.

    Source: §I · p. 2, §VI · p. 8

  • Takeaways & Limitations

    The paper establishes a taxonomy of hallucinations in CPS requirements modeling and discusses future directions for improving LLM-based modeling.

    Source: §IX · p. 11

    The evaluation selects the latest versions of models with around 7 billion parameters, so its findings concern that model-size and selection setting.

    Source: A. Threats to validity · p. 10

Abstract

from arXiv · show

Cyber-physical systems (CPSs) integrate cyber and physical components and enable them to interact with each other to meet user needs. The needs for CPSs span rich application domains such as healthcare and medicine, smart home, smart building, etc. This indicates that CPSs are all about solving real-world problems. With the increasing abundance of sensing devices and effectors, the problems wanted to solve with CPSs are becoming more and more complex. It is also becoming increasingly difficult to extract and express CPS requirements accurately. Problem frame approach aims to shape real-world problems by capturing the characteristics and interconnections of components, where the problem diagram is central to expressing the requirements. CPSs requirements are generally presented in domain-specific documents that are normally expressed in natural language. There is currently no effective way to extract problem diagrams from natural language documents. CPSs requirements extraction and modeling are generally done manually, which is time-consuming, labor-intensive, and error-prone. Large language models (LLMs) have shown excellent performance in natural language understanding. It can be interesting to explore the abilities of LLMs to understand domain-specific documents and identify modeling elements, which this paper is working on. To achieve this goal, we first formulate two tasks (i.e., entity recognition and interaction extraction) and propose a benchmark called CPSBench. Based on this benchmark, extensive experiments are conducted to evaluate the abilities and limitations of seven advanced LLMs. We find some interesting insights. Finally, we establish a taxonomy of LLMs hallucinations in CPSs requirements modeling using problem diagrams. These results will inspire research on the use of LLMs for automated CPSs requirements modeling.

I. INTRODUCTION

The paper examines whether LLMs can model CPS requirements from natural-language documents, a task made difficult by domain complexity and the effort required to identify problem-diagram elements. It formalizes the task, introduces CPSBENCH, evaluates seven LLMs, and analyzes their strengths, limitations, and hallucinations.

I. INTRODUCTION

  • CPS complexity makes accurate requirements difficult, while problem-diagram modeling requires labor-intensive, error-prone identification of elements scattered across natural-language documents.The problem-frame approach emphasizes components and their interconnections, including interactions between software and its environment.Source: §I · p. 1
  • The paper formalizes requirements modeling as entity recognition and interaction extraction, reflecting analysts’ process of identifying elements before determining their interactions and constraints.The tasks are motivated by the manual modeling process illustrated in Figure 1.Source: §I · p. 2
  • CPSBENCH combines real-world requirements documents with tutorial cases to support evaluation of CPS requirements modeling.The benchmark contains 12 enterprise-level documents and 30 tutorial cases, with requirements, entity, interaction, and problem-diagram components per sample.Source: §I · p. 2
  • Evaluated on CPS requirements modeling with problem diagrams using CPSBENCH, current LLMs achieved only around 60% recall, failing to recognize almost half of modeling elements, and understood general requirements concepts better than specialized concepts.Performance improved with more shots in the prompt.Source: §I · p. 2
  • The paper establishes a taxonomy of LLM hallucinations in CPS requirements modeling and discusses directions for future improvement.It also reports an extensive evaluation of seven LLMs’ strengths and limitations.Source: §I · p. 2

II. TASK DEFINITION

The task-definition section frames requirements modeling as a pipeline inspired by manual analysis: identify model elements in natural-language requirements, then determine their interactions. It defines entity recognition in terms of typed entities and introduces six entity categories for problem diagrams.

A. Overview

  • Requirements modeling is decomposed into entity recognition and interaction extraction, which form a pipeline for constructing models from natural-language requirements.The decomposition is inspired by the manual modeling process.Source: A. Overview · pp. 2–3

B. Entity Recognition

  • Entity recognition identifies entities in a requirements description and assigns each a type; the formulation denotes entities as value-type pairs.The task definition identifies six entity types for CPS problem diagrams.Source: B. Entity Recognition · p. 3
  • The six entity types are Machine Domain, Physical Device, Environment Entity, Design Domain, Requirements Domain, and Shared Phenomena.They cover the software system, real-world devices, external objects, existing third-party systems, system purpose, and shared events, states, or values.Source: B. Entity Recognition · p. 3
  • Shared Phenomena denotes shared events, states, and values between connected entities, such as a notification or button click.This category is among the six entity types used in the task definition.Source: B. Entity Recognition · p. 3

C. Interaction Extraction

Interaction extraction finds typed relations among entities already recognized in a requirements description. CPSBENCH draws on real-world requirements documents and tutorial cases, with preprocessing to focus and clarify the functional requirements.

C. Interaction Extraction

  • Interaction extraction identifies relations among recognized entities, representing each interaction by its head entity, relation type, and tail entity.The paper distinguishes Interface, Requirements Reference, and Requirements Constraint interactions.Source: C. Interaction Extraction · p. 3

III. BENCHMARK CONSTRUCTION

  • CPSBENCH is constructed from requirements documents and cases, with the benchmark process described as having three steps.The available passage identifies the construction process but does not specify those steps.Source: §III · p. 3

A. Data Collection

  • The collected materials span embedded, control, and real-time systems, drawing on public datasets, private industry documents, and requirements-engineering books.The selected materials include 10 PURE documents, one Lockheed Martin autopilot document, the private SSCS document, and 30 book cases.Source: A. Data Collection · p. 3
  • Preprocessing removes non-functional requirements and short sentences, corrects spelling, rewrites incomplete sentences, standardizes phrases, and splits documents into sentences.The authors state that they focus on functional requirements rather than non-functional requirements.Source: A. Data Collection · p. 4

B. Data Annotation

Annotators labeled entities and their types, then judged and typed interactions between entity pairs; the resulting labels serve as CPSBENCH ground truth.

B. Data Annotation

  • Ground truth was produced by annotating entities and types sentence by sentence, then judging and typing interactions for each entity pair.A separate group reviewed each group's results; disagreements were discussed and resolved by team vote.Source: B. Data Annotation · p. 4
  • The annotation effort covered 2,633 requirement sentences, 5,795 entities, and 3,092 interactions, taking over 500 person-hours.Six annotators with requirements-modeling experience or coursework participated in the inspection team.Source: B. Data Annotation · p. 4

C. Quality Evaluation

The benchmark annotations were trained and checked for agreement, and the paper describes benchmark samples and an in-context-learning evaluation approach. The approach retrieves examples for prompts and parses model responses into entities and interactions.

C. Quality Evaluation

  • Entity-labeling agreement reached Cohen’s Kappa 0.74, while average interaction-labeling agreement reached Cohen’s Kappa 0.78.Annotators refined guidelines through disagreement discussion until agreement exceeded 0.6.Source: C. Quality Evaluation · p. 5

D. CPSBENCH benchmark

  • Each CPSBENCH sample contains requirements, an entity dictionary with entity types, an interaction dictionary, and a problem diagram.The diagram is constructed from the sample's entities and interactions.Source: D. CPSBENCH benchmark · p. 5

IV. APPROACH

  • The evaluation approach follows in-context learning and is organized around prompt construction, response generation, and answer parsing.The section identifies prompt construction and few-shot retrieval, response generation, and answer parsing as the described stages.Source: §IV · p. 5, A. Overview · p. 5

A. Overview

  • For each requirements text, the method retrieves k examples to construct a prompt, then parses the response into entities and interactions for a requirements model.The prompt includes a task description, schema definition, few-shot examples, and test requirements text.Source: A. Overview · p. 5

B. Prompt Construction

Prompts combine task instructions, a modeling schema, few-shot examples, and the test requirements; retrieval selects semantically similar training examples to supply those shots.

B. Prompt Construction

  • Each prompt combines task description, schema definition, few-shot examples, and test requirements text.The task description outlines the task, the schema specifies target types, examples provide references, and the requirements text is the input to analyze.Source: B. Prompt Construction · p. 5
  • The task description instructs the LLM about the task and output format.It frames the model as an expert in CPS requirements modeling and information extraction, then specifies entity or interaction recognition.Source: B. Prompt Construction · p. 5
  • The schema defines the target entity or interaction types based on prior work.Source: B. Prompt Construction · p. 5
  • Few-shot examples provide task tutorials and references while regulating output format for parsing.The generated output mimics the examples' format, which supports extracting the final requirements models.Source: B. Prompt Construction · p. 5
  • The requirements text supplies the CPS requirements to analyze against the defined schema.Source: B. Prompt Construction · p. 5

C. Few-shot Retrieval

  • Semantic similarity retrieval selects k training examples whose requirements resemble each test input.The approach embeds the training examples and test requirements text, then ranks examples by cosine similarity.Source: C. Few-shot Retrieval · pp. 5–6
  • The retrieval step sorts examples by descending similarity and uses the top k as shots.Source: C. Few-shot Retrieval · p. 6

D. Response Generation

The study uses few-shot prompts to evaluate LLMs on CPS requirements modeling, while parsing generated answers for metric computation. It examines recognition performance, shot counts, and hallucination types.

D. Response Generation

  • Greedy sampling with temperature 0 generates responses consistently, and regular expressions extract content within braces for metric computation.The response is first generated from the constructed prompt, then formatted as an answer for easier computation.Source: D. Response Generation · p. 6, E. Answer Parsing · p. 6

V. STUDY DESIGN

A. Research Questions

  • RQ1 uses 10-fold cross-validation to assess entity recognition and interaction extraction with multiple metrics.Source: A. Research Questions · p. 6
  • RQ2 varies the number of retrieved shots from one to three to estimate its effect on modeling performance.The range reflects response-speed limitations.Source: A. Research Questions · p. 6
  • RQ3 manually compares GPT-4 predictions with ground truth to summarize hallucination types.Source: A. Research Questions · p. 6

B. Studied LLMs

The evaluation covers seven closed- and open-source LLMs, assesses recognition using micro precision, recall, and F1, and uses sentence-level cross-validation with shared inference settings.

B. Studied LLMs

  • Seven evaluated LLMs include closed-source GPT-4 and GPT-3.5 and open-source Qwen2, Llama3, Gemma, GLM4, and Mistral.The study used official interfaces or implementations to reproduce them.Source: B. Studied LLMs · p. 6

C. Evaluation Metrics

  • Micro precision, recall, and F1 assess recognition by aggregating true positives, false positives, and false negatives.The counts are aggregated across entities or interactions before computing precision and recall.Source: C. Evaluation Metrics · pp. 6–7

D. Experiment Settings

  • Sentence-level 10-fold cross-validation keeps each requirements text in either training or test data.Each fold retains one of ten randomly partitioned parts for testing and uses the other nine for training.Source: D. Experiment Settings · p. 7
  • All LLMs use greedy sampling at temperature 0 and the same inference library to support evaluation fairness.Closed-source models are accessed through APIs, while open-source models are instantiated with replication packages and pretrained weights.Source: D. Experiment Settings · p. 7
  • Shot retrieval uses SimCSE embeddings and FAISS to select the three most similar training samples.Source: D. Experiment Settings · p. 7

VI. RESULT AND ANALYSIS

Across CPSBench, LLMs show limited requirements-modeling performance and vary in how well they recognize different concepts. More few-shot examples generally improve performance, with diminishing gains, while the hallucination analysis identifies distinct entity and interaction error types.

RQ1

  • For entity identification, GPT-3.5 with one-shot reasoning achieved average F1 45 and outperformed other LLMs by 4%–18.4%; for interaction identification, GPT-4 scored 57, ahead by 5%–19%.Source: §VI · p. 8
  • LLMs understood general requirements concepts better than specialized ones, including weaker recognition of design and requirements domains, references, and constraints.The authors attribute this pattern to limited training material on specialized concepts.Source: §VI · p. 8, Figure 6 · p. 8

RQ2

  • F1 generally rose with more shots except for Qwen2, but gains diminished; GPT-4’s entity F1 rose 12% then stayed at 48, while interaction F1 rose 16% then 1%.Figures 7 and 8 visualize the entity-recognition and interaction-extraction results across one to three shots.Source: §VI · pp. 8–9, Figure 7 · p. 8, Figure 8 · p. 8

RQ3

  • A PhD candidate and a master’s student familiar with CPS requirements modeling reviewed GPT-4 predictions against requirements and ground truth to summarize hallucinations.The reported taxonomy and cases are summarized in Table IV.Source: §VI · p. 9, Table IV · p. 9
  • Entity-recognition hallucinations include type, boundary, complete, and omission errors; interaction-extraction hallucinations include type, complete, and omission errors.Boundary errors include predictions that contain, are contained by, or overlap the gold entity.Source: §VI · pp. 9–10, Table IV · p. 9

A. Threats to validity

The authors discuss threats to validity, propose three future research directions, review related requirements-modeling work, and conclude with an empirical evaluation and CPSBENCH-based study of LLMs.

A. Threats to validity

  • Manual annotations are somewhat subjective; model performance may vary with decoding or inference setup, and manual hallucination review may be incomplete.The authors describe annotation guidance, independent annotation, cross-validation, consistency scoring, a shared inference framework, and team review as mitigations.Source: A. Threats to validity · p. 10

B. Future Directions

  • Future work could develop CPS-specific LLMs, inject modeling knowledge through retrieval or fine-tuning, and combine models as a mixture of experts.The proposed directions address domain knowledge, specialized-concept identification, and integration of model strengths.Source: B. Future Directions · p. 10

VIII. RELATED WORKS

  • Prior natural-language modeling approaches include rules, text processing, deep learning, and LLMs; the reviewed LLM work targets single concerns, while CPS problem diagrams require multiple concerns.The authors also note that earlier evaluations used limited cases and models, and describe their study as evaluating advanced LLMs across multiple CPS systems.Source: §VIII · pp. 10–11

IX. CONCLUSION

  • The paper formulates CPS requirements modeling as two tasks, constructs CPSBENCH, empirically evaluates advanced LLMs to gain insights into their strengths and limitations, and establishes a hallucination taxonomy with future directions.Source: §IX · p. 11

Related papers

Loading 2408.02450v1…