Source-linked AI summary
Mutation Testing of Simulink Cyber-Physical System Models: Challenges and Solutions in Practice
Murat Kavak, Onur Kilincceker, Serge Demeyer, Kevin Vandenbroucke, Halim Abdurrahman Ceylan
TL;DR
Practical evidence on incorporating mutation testing into Simulink and Stateflow design models remains limited. This paper studies a DUCO pilot with MUT4SLX and identifies equivalent mutants, requirement traceability, and Stateflow mutation testing as the main challenges.
Problem
Experience reports on applying mutation testing to Simulink and Stateflow models in practice remain lacking despite the technical feasibility of existing tools.
Method
The authors analyse lessons from applying MUT4SLX to three DUCO production models and develop practical solutions to encountered challenges.
Results
The main challenges are equivalent mutants, requirement traceability, and mutation testing for Stateflow, ranked by severity.
Takeaways & Limitations
Model-based mutation testing practice must address equivalent-mutant handling, requirement traceability, and Stateflow-specific testing challenges.
Takeaways & Limitations
Direct comparison of Simulink and Stateflow model files is not meaningful, so binary comparison requires generated code and additional optimization steps.
Abstract
from arXiv · showhide
Several experience reports illustrate that mutation testing is capable of supporting a shift-left testing strategy, especially in industries where late bug discovery incurs very high costs and risks. In the context of cyber-physical systems, a shift left implies that engineers need to test the design models used to simulate, prototype, and analyse the feasibility of the system under design. In this paper, we analyse the challenges we encountered and the lessons we learned when incorporating mutation testing in the context of DUCO, a company producing ventilation systems for buildings. The engineers within DUCO have years of experience with model-based engineering centered around Simulink and StateFlow, including unit tests for their models. During a pilot project with our tool prototype MUT4SLX we learned that equivalent mutants, requirement traceability, and mutation testing for Stateflow represent particular challenges not yet reported in the academic literature.
1 Introduction
The introduction motivates model-based shift-left testing for complex cyber-physical systems and positions mutation testing as a way to assess test-suite weaknesses. It frames the paper as an experience report on incorporating mutation testing into DUCO’s Simulink and Stateflow development context.
- Motivation: Cyber-physical-system testing is difficult because software, physical components, and processes are complex, heterogeneous, and interdependent.These systems operate under strict safety, timing, and reliability constraints, increasing the need for early testing.
- Motivation: Model-based shift-left testing moves testing earlier by using executable system and environment models to simulate dependencies and interactions.It manages complexity through higher-level modeling and supports cyber-physical-system safety and reliability.
- Mutation testing: Mutation testing injects faults into code or models and measures their detection to diagnose weaknesses in test suites.The resulting mutant score reflects how many injected faults the tests detect, and the practice is recommended in several functional-safety standards.
- Study context: The paper addresses the lack of practical experience reports by analysing challenges and lessons from incorporating mutation testing at DUCO.DUCO had adopted model-based engineering practices by 2019, with average projects incorporating 20% Simulink and 80% Stateflow.
2 Related Work
Prior work established mutation testing for Simulink and Stateflow through mutation operators, fault-injection tools, test-generation approaches, and emerging LLM-based mutant generation. This work instead examines practical challenges and potential solutions when applying MUT4SLX to Simulink cyber-physical-system models in a DUCO pilot project.
- Simulink mutation testing: Mutation testing for Simulink models began with three signal mutation operators and later supported bounded model-checking-based test-data generation.The initial operators were add, multiply, and assign operations on input-carried signals.
- Simulink mutation testing: Research proposed four mutation categories for simple Simulink designer faults and later expanded them to eight categories for model-clone-detector testing.Table 1 organizes these classes by underlying change logic and related work.
- Mutation-testing tools: SIMULTATE automated Simulink mutation testing through Python, while FIM extended its inherited operators with ten signal fault-injection operators and five block mutation operators.SIMULTATE initially proposed four mutation operators and supported five fault models; FIM emphasized safety analysis and property-based mutation testing.
- Mutation-testing tools: MUT4SLX provides automated mutant generation and execution, with 28 Simulink and 18 Stateflow parameter-based operators, requirement traceability, and CI/CD compatibility.Its operators were modeled after realistic faults mined from an industrial bug database, and DUCO selected it because it was tailored to the company’s requirements and development pipeline.
- Study focus: Unlike prior work, this study focuses on challenges and potential solutions for applying mutation testing to Simulink cyber-physical-system models with MUT4SLX in practice through a DUCO pilot project.Related approaches also include six timed Stateflow operators and LLM-based Simulink-Stateflow mutant-generation pipelines.
3 Driving the Pilot Project
The pilot evaluated MUT4SLX on three undisclosed DUCO production models within the 2021–2024 EFFECTS project. It examined practical challenges and possible solutions for mutation testing of Simulink and Stateflow models.
- Pilot setup: The pilot used three DUCO production models, M1, M2, and M3, with MUT4SLX applied using the complete operator set without filtering.The models’ names and internal structures cannot be disclosed.
- Pilot setup: DUCO reported a 7.5% improvement in its test suites over the first three consecutive trials of MUT4SLX.This report covered a considerably larger internal model set than the three models analysed in detail.
- Pilot results: 459 mutants were generated across the three models, with 250 killed and 209 surviving, producing mutation scores of 82.0% for M1, 41.0% for M2, and 95.7% for M3.M1 achieved 100% decision and condition coverage, yet 16 of its 89 mutants survived.
- Pilot results: M2 required 3.6 hours to simulate its 324 mutants, demonstrating execution cost as a practical concern.The passage identifies simulation cost as a further concern but does not provide comparable timings for the other models.
- Test-suite analysis: For M1, individual test cases killed 44.9%–68.5% of mutants while the complete suite reached 82.0%, whereas M2’s suite was largely redundant.M2’s strongest single test case reached 40.1%, compared with 41.0% for the full suite; 16 of 21 test cases scored 37.7%–38.9%.
- Research questions: The study asked which challenges arise when applying mutation testing to Simulink and Stateflow models and what possible solutions address them.The authors prioritized severe challenges and explored solutions based on DUCO engineers’ feedback and relevant literature.
4 Results and Discussion
The pilot with MUT4SLX exposed equivalent mutants as a major challenge in Simulink and Stateflow, particularly for common block and operator mutations. The adopted solutions used semantic analysis and model-based equivalence testing, while requirement traceability and continuous testing remained important practical considerations.
- Equivalent Mutants: Most equivalent mutants arose from Constant Replacement, Relational Operator Replacement, and multiplication-parameter changes in Product and Gain blocks.MUT4SLX could not detect equivalent mutants automatically, so they required manual analysis.
- Equivalent Mutants: Gain and Product block mutations can be equivalent because multiplication-mode changes produce identical results for scalar constants or certain scalar and non-scalar input combinations.The implemented solution identifies Gain constant types and Product input combinations to assess equivalence semantically.
- Equivalent Mutants: Comparing original and mutated .slx files is ineffective because they are compressed XML packages without compiler optimization differences.Generating optimized C code and compiling it was considered as an alternative, but the adopted approach instead used formal-verification-inspired testing.
- Equivalent Mutants: The equivalence strategy generates a broad input space with Simulink Design Verifier, then compares original and surviving-mutant outputs under those inputs.For large models, the resulting simulations create significant time costs, motivating filtering the surviving-mutant pool for scalability.
- Requirement Traceability: MUT4SLX’s requirement-traceability prototype currently supports Signal Builder and the Simulink Requirements Toolbox, limiting coverage across organizational toolchains.Real-world adoption requires integrations tailored to each organization’s specific testing tools and requirement-document types.
- Continuous Testing: A Jenkins CI/CD pipeline automates tests for existing and newly implemented features as mutation operators and tool capabilities evolve.Artificial models were extended with new features, while tests were continuously expanded or updated when operators were introduced or refactored.
5 Lessons Learned and Practical Insights
The pilot identified four practical lessons: equivalent mutants require operator-specific filtering, code-level detection methods need model-level adaptation, traceability is an integration problem, and mutation operators should reflect company-specific bug data.
- Lesson 1: Equivalent mutants concentrated in three operators, making an inexpensive dimensionality-tracing filter more effective than general-purpose detection.The operators were Constant Replacement, Relational Operator Replacement, and Multiplication-parameter mutation in Gain/Product blocks; the filter operated at block level and eliminated many equivalent mutants before simulation.
- Lesson 2: Code-level equivalent-mutant detection techniques required substantial adaptation for Simulink and Stateflow models.TCE was not directly applicable because .slx files are compressed XML packets, while generating and comparing optimized C binaries was much more expensive than C code comparison.
- Lesson 3: Requirement traceability proved costly and difficult to generalize because industrial requirements and test tools use diverse formats.A prototype integrating Signal Builder with .slreqx was costly even for that single combination, so .slreqx support was positioned as a proof-of-concept rather than a general solution.
- Lesson 4: Mutation operators should be selected and customized according to company-specific requirements and the software team’s bug database.This customization makes operators more targeted, relevant, and impactful.
6 Conclusion
The paper reports practical experience incorporating mutation testing for Simulink and Stateflow models at DUCO, addressing challenges and lessons absent from earlier proof-of-concept work.
- 6 Conclusion: The study examines how mutation testing can be incorporated into DUCO’s shift-left testing of Simulink and Stateflow models.DUCO engineers have experience with model-based engineering and unit testing, while prior tools mainly demonstrated technical feasibility without practical experience reports.
7 Mandatory Data Availability Statement
The study’s supporting data and experimental findings cannot be shared because third-party restrictions and the industrial partner’s data sensitivity limit availability. Authors may provide further supporting resources upon request.
- Mandatory Data Availability Statement: Third-party restrictions and the high sensitivity of the industrial partner’s data prevent sharing the study’s supporting data and experiment results.Readers may contact the authors for further supporting resources.