PaperlayerView exact revision on arXiv

Source-linked AI summary

Stress Testing Control Loops in Cyber-Physical Systems

Claudio Mandrioli, Seung Yeob Shin, Martina Maggio, Domenico Bianculli, Lionel Briand

arXiv:2302.13913v4cs.SE

TL;DR

Control-based CPSs rely on control-design assumptions for guarantees, but those assumptions may not hold after implementation, making it important to identify affected operating scenarios. The paper develops frequency-based stress tests for linearised-model assumptions and evaluates them across three case studies, generating tests spanning actuator saturation from 0% to 95%.

  • Problem

    The paper addresses how to test control-based CPSs when implemented systems may violate control-design assumptions that underpin guarantees.

    Source: §1 · pp. 2, 4

  • Method

    It characterises control-loop inputs using control theory and frequency-domain features, then uses a test parametrisation and stress metric to generate tests of linearised-model assumptions.

    Source: §1 · p. 4, §4.3 · p. 23, §5 · p. 24

  • Results

    Across three case studies, the approach generated tests with actuator saturation times ranging from 0% to 95%.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The tests let engineers examine system behavior at design-scope boundaries where a priori control-theoretical guarantees become uncertain.

    Source: §1 · p. 4

    The frequency-amplitude characterisation cannot identify a boundary between linear and nonlinear behavior for arbitrary input sequences.

    Source: §4.3 · p. 23

Abstract

from arXiv · show

Cyber-Physical Systems (CPSs) are often safety-critical and deployed in uncertain environments. Identifying scenarios where CPSs do not comply with requirements is fundamental but difficult due to the multidisciplinary nature of CPSs. We investigate the testing of control-based CPSs, where control and software engineers develop the software collaboratively. Control engineers make design assumptions during system development to leverage control theory and obtain guarantees on CPS behaviour. In the implemented system, however, such assumptions are not always satisfied, and their falsification can lead to loss of guarantees. We define stress testing of control-based CPSs as generating tests to falsify such design assumptions. We highlight different types of assumptions, focusing on the use of linearised physics models. To generate stress tests falsifying such assumptions, we leverage control theory to qualitatively characterise the input space of a control-based CPS. We propose a novel test parametrisation for control-based CPSs and use it with the input space characterisation to develop a stress testing approach. We evaluate our approach on three case study systems, including a drone, a continuous-current motor (in five configurations), and an aircraft.Our results show the effectiveness of the proposed testing approach in falsifying the design assumptions and highlighting the causes of assumption violations.

1 INTRODUCTION

Control-based CPS testing must account for design assumptions that may fail in implemented systems, limiting the scenarios where control-theoretic guarantees apply. The paper defines stress testing around falsifying those assumptions and proposes a frequency-based approach, evaluated on three case studies.

1 INTRODUCTION

  • CPS testing is challenging because requirements concern physical behavior while design choices span hardware, software, and their interaction.The control layer uses sensors and actuators to affect the physical component, where CPS outputs capture requirements.Source: §1 · p. 2
  • A CPS's design scope includes scenarios where assumptions hold or their falsification does not affect behavior; other scenarios can reduce the set covered by a priori guarantees.For the drone, motor saturation that limits flight capability is outside the design scope.Source: §1 · p. 3
  • The paper shifts testing from requirement violations to falsifying control-design assumptions, since their validity determines when control-theoretic guarantees can be relied on.When assumptions are falsified, testing is needed to examine system behavior empirically.Source: §1 · p. 4
  • The approach targets assumptions behind linearised physics models, characterising control-layer inputs by frequency and using shape, amplitude, and time scaling to define test cases.It also introduces a metric and uses the parametrisation to develop metamorphic relations and a stress-testing approach.Source: §1 · p. 4
  • Across three case studies, the approach generated tests with actuator saturation times from 0% to 95%, exposing varied conditions for assessing assumption violations.The cases were drone altitude control, DC servo position control, and aircraft altitude control.Source: §1 · p. 4

2 CONTEXT AND BACKGROUND

The background describes how control-based CPSs are developed across physical, control, and software engineering, then introduces control-loop and frequency-domain concepts. It explains why linearised models support control design and why their assumptions restrict the range of conditions for which guarantees apply.

2.1 Development of Control-Based CPS

  • Control-based CPS development brings together physical-component, control, and software engineers, whose design and implementation choices shape system behavior.Testing the control layer must account for its interaction with the physical component, not software in isolation.Source: §2.1 · p. 6

2.2 Control Engineering Primer

  • A control loop repeatedly uses sensor measurements and references to compute actuator commands, which affect the physical system and its subsequent measurements.The drone example repeats this cycle every millisecond.Source: §2.2 · p. 7
  • The control-design process models the physics, linearises nonlinear models around an operating point, and uses control theory to design algorithms with formal performance guarantees under assumptions.The linearised model is intended for small variations around the selected operating point.Source: §2.2 · pp. 7–8
  • The DFT represents a signal through frequency components: constant and sinusoidal signals map to one component, while a step contains multiple components.This provides a frequency-domain counterpart to representing signals as values over time.Source: §2.2 · p. 8, Figure 3 · p. 9
  • Control systems track slower reference changes and filter faster ones; closed-loop bandwidth marks the threshold, while higher bandwidth trades speed against actuation and noise concerns.High-frequency responses can raise actuation and measurement-noise concerns.Source: §2.2 · p. 10

3 CONTROL ENGINEERING PERSPECTIVE ON CPS STRESS TESTING

Control-theory guarantees depend on assumptions that implementation and operation can violate, so this section focuses stress testing on linearised-model assumptions for individual control loops. It motivates the scope, prioritizes assumptions, and outlines an approach to generate tests that expose their falsification.

3.1 Problem Motivation

  • Implementation choices can falsify design assumptions: motor saturation, for example, breaks the proportional thrust model when commands exceed motor capacity.Reference updates that change too much or too fast can trigger saturation; the resulting behaviour may impair flight performance and requirements fulfilment.Source: §3.1 · p. 11

3.2 Design Assumptions in Control Algorithms

  • The identified assumptions span ignored loop interactions and mode changes, physical-model fidelity, validity near the linearisation point, finite precision, and discrete-time execution effects.The assumptions arise across control problem definition, control design, and algorithm implementation.Source: §3.2 · pp. 12–13
  • Linearised-model validity is prioritized before loop and mode interactions, while physical-model validity is application-specific and numerical and timing assumptions are outside this work’s scope.The work focuses on the assumption that operation stays sufficiently near the linearisation point.Source: §3.2 · pp. 13–14

3.3 Problem Statement

  • The problem targets individual control loops, using reference-value sequences as inputs and physical-quantity tracking traces as outputs to test when linearity violations remove control-theoretical guarantees.The loop is assumed to have an estimated closed-loop bandwidth and a valid reference range.Source: §3.3 · p. 14
  • The approach characterizes a control loop’s input space with control-domain knowledge, then uses frequency-domain test parametrisation, a stress metric, and metamorphic relations to generate tests.The parametrisation contrasts with traditional time-domain approaches.Source: §3.3 · p. 14

4 CONTROL LOOP INPUT SPACE CHARACTERISATION

This section characterizes control-loop inputs by frequency, amplitude, and component positioning to distinguish tracking, filtering, and nonlinear behavior relative to the system’s design scope. It defines measures and expected properties to guide stress-test generation and analysis, noting that behavior depends on the other test components as well as the frequency–amplitude point.

4.1 Qualitative Input Space Characterisation

  • The frequency–amplitude plane separates linear tracking and filtering from nonlinear behavior, with the design scope comprising both linear behavior and nonlinearity that does not impair performance.Its boundaries are qualitative simplifications; the drone example shows that saturation can occur while the reference is still tracked, whereas stronger saturation can produce unpredictable output.Source: Figure 5 · p. 15, §4.1 · pp. 15, 17

4.2.1 Mapping of Tests to Frequency-Amplitude Points.

  • A test input maps to frequency–amplitude coordinates using its DFT components above a relative threshold ρ of the largest component, capturing signal speed and size.The threshold selects relevant components from the discrete spectrum; faster square waves map to higher frequencies and larger-amplitude waves map to higher amplitudes.Source: §4.2.1 · p. 19

4.2.2 Degree of Non-Linearity Definition.

  • The degree of non-linearity measures the largest output-spectrum amplitude at frequencies outside the relevant input components, normalized by the maximum input amplitude.Repeating the input adds intermediate DFT samples without changing its frequency content, improving detection of new output frequencies at the cost of longer tests.Source: §4.2.2 · pp. 20–21
  • The drone examples yield degrees of non-linearity of 0.05 and 0.06 without saturation, 0.01 with saturation but little behavioral impact, and 0.16 when output misbehaves.The low value during limited saturation distinguishes its non-impairing effect from the misbehaving case; numerical noise can affect values when new-frequency amplitudes are small.Source: §4.2.2 · p. 21

4.2.3 Mapping of Behaviour to Frequency-Amplitude Points.

  • For tests exhibiting linear behaviour, the metric takes values in [−∞, 1]: 1 describes complete filtering, 0 describes perfect tracking, and negative values correspond to input amplification.Source: §4.2.3 · p. 22

4.2.4 Expected Properties of the Characterisation.

  • The characterization expects non-linearity to increase with input amplitude and frequency, filtering to increase with frequency, and closed-loop bandwidth to remain test-independent under linear behavior.The examples support the latter two properties, with bandwidth around 0.9 Hz for the linear tests; the saturated higher-amplitude test shows about 0.6 Hz, while the non-linearity trend has an exception when saturation occurs without impairing behavior.Source: §4.2.4 · pp. 22–23

4.3 Benefits and Limitations of the Frequency-Amplitude Characterisation

  • 𝔡nl quantifies new output frequency components, identifying tests that depart from the linear-model design scope but remain near its validity limits.The approach targets tests where 𝔡nl is non-zero but not too large, avoiding tests fully within scope or far outside it.Source: §4.3 · p. 23
  • A frequency-amplitude point cannot define a universal boundary between linear and nonlinear behavior because its effect depends on other components in the same input.Tests with a main component near 1 Hz and amplitude 0.2 m can yield different 𝔡nl values and behaviors when paired with different components.Source: §4.3 · p. 23
  • The proposed test parametrisation separately defines frequency content, amplitude content, and the relative positioning of frequency-amplitude components.This separation lets the approach use the input-space characterisation to control expected test behavior.Source: §4.3 · pp. 23–24

5 TESTING APPROACH

This section presents a control-layer test parametrisation that uses metamorphic relations and control-theoretic metrics to sample shaped inputs by frequency and amplitude within an optimistic bound, then evaluates output traces to locate stress cases and check the relations.

5.3 Approach Steps

  • The approach bounds amplitude with sinusoidal tests, samples frequency-amplitude coordinates using other shapes, then evaluates executions with 𝔡nl, 𝔡f, and the relations to identify tests outside design scope.The preliminary bound is optimistic because sinusoidal inputs represent a single frequency-amplitude component and do not account for interactions among components.Source: §5 · p. 24, §5.3 · pp. 27–28, §5.3.2 · p. 29

5.1 Test Case Parametrisation

  • Tests combine a periodic shape, amplitude gain, and time scaling; the shape sets the relative frequency-amplitude pattern, while the gains move components along the amplitude and frequency axes.A test is fully defined by ⟨ϕ_i, α_i, γ_i⟩, and periodic inputs enable use of the degree-of-nonlinearity metric 𝔡nl.Source: §5.1 · pp. 24–25

5.2 Definition of Metamorphic Relations

  • Three metamorphic relations capture expected behavior: increased input amplitude and frequency should raise 𝔡nl, frequency should increase filtering for linear tests, and estimated bandwidth should agree across shapes.Linear tests are identified using a 𝔡nl threshold; filtering is assessed with 𝔡f for corresponding frequency components, and bandwidth uses the 0.5 output-to-input ratio.Source: §5.2 · pp. 25–26

5.3.1 Step 0: Setting of Required Input.

  • Engineers provide a maximum amplitude, a frequency range around expected closed-loop bandwidth, and amplitude resolution; a rule of thumb is to cover a factor of 10 around expected bandwidth.The amplitude resolution represents the smallest input variation expected to affect behavior; broader ranges or finer resolution increase the test count.Source: §5.3.1 · p. 28

5.3.2 Step 1: Optimistic Bounding of Amplitude Values.

  • Sinusoidal tests and binary searches estimate a frequency-dependent amplitude bound near the non-linearity threshold, with additional frequencies sampled until neighboring bounds differ by less than the amplitude resolution.The bound excludes trivially large inputs from later sampling. If the assumed relation between amplitude and non-linearity fails, the bound may be overestimated; if no tested amplitude causes non-linearity, the estimate may approach A_max.Source: §5.3.2 · pp. 29–31

5.3.3 Step 2: Generation of Test-Set.

  • The test set uses square, sawtooth, triangular, and trapezoidal waves, with sampled amplitude and time-scaling parameters that move each shape’s main component within the bounded input space.The shape set can be expanded when a CPS use case supplies other relevant patterns.Source: §5.3.3 · p. 31
  • Frequency samples uniformly cover [f_min, f_max], with resolution based on the average frequency gap from Step 1’s amplitude bound; application-specific sampling may bias coverage toward relevant ranges.Amplitude sampling can likewise be adapted when particular regions are of interest.Source: §5.3.3 · p. 31

5.3.2 Step 1: Optimistic Bounding of Amplitude Values.

  • Step 1’s optimistic amplitude bound limits sampling at each frequency, avoiding amplitudes that already produce very high non-linearity under sinusoidal input.Within that range, amplitudes are sampled from a beta distribution skewed toward higher values, with the sample count set by the desired amplitude resolution.Source: §5.3.3 · p. 31

5.3.4 Step 3: Execution of Tests and Properties Evaluation.

  • Tests at transitions between low- and high-nonlinearity regions identify stress cases that can reveal phenomena falsifying design assumptions and pushing the CPS outside its design scope.The method displays each test’s largest-amplitude frequency–amplitude component and colours it by measured degree of non-linearity.Source: §5.3.4 · p. 33
  • MR1 expects non-linearity to increase with frequency and amplitude, while MR2 and MR3 are checked on in-scope, linear-behaving tests using filtering degree versus frequency.MR2 expects filtering to increase with frequency irrespective of shape; MR3 expects different input shapes to yield the same closed-loop bandwidth, where filtering crosses 0.5.Source: §5.3.4 · pp. 33–34

6 EMPIRICAL EVALUATION

The empirical evaluation examines whether frequency–amplitude input characterization can generate tests that trigger nonlinear behavior and whether metamorphic relations can identify tests near design-scope bounds. It applies the approach to seven SUTs, considering nonlinear effects and design-scope violations under relevant inputs.

6.1 Systems Under Test

  • The test systems span PID-controlled drone and aircraft altitude loops and a state-feedback DC servo, chosen to cover common control algorithms and nonlinear phenomena.The drone and servo models were built from implementation and development documentation; the aircraft model was publicly available. The servo provides application-relevant variants with individually injected nonlinearities.Source: §6.1 · pp. 34–35
  • The evaluation uses drone and DC servo models with detailed nonlinear-phenomenon knowledge, but aircraft RQ1 analysis is limited to actuator saturation because its other nonlinearities are unknown.Across seven SUTs, the study includes drone and aircraft altitude control plus five DC servo versions. The servo versions share saturation, quantisation, and PWM, with Coulomb friction, quadratic friction, dead-zone, or backlash injected individually in four versions.Source: §6.1 · p. 36, §6.3 · p. 39

6.2 Testing Settings

  • Frequency ranges were selected around expected bandwidths, while maximum reference amplitudes were set to practical operating ranges for each system.Ranges were 0.1–2 Hz for the drone, 0.005–3 Hz for the servo, and 0.005–0.5 Hz for the aircraft; maximum amplitudes were 6 m, 2π rad, and 750 m, respectively.Source: §6.2 · p. 37
  • The drone’s dnl exceeded the threshold at two periods and rose through five; tests used 5 periods for the drone, 7 for the DC servo, and 10 for the lightweight aircraft.The campaign executed 30,187 tests; Step 1 reduced the tests needed to cover the input space for the drone and lightweight aircraft.Source: §6.2 · pp. 38–39

6.3 RQ1 - Generation of Stress Tests

  • The study quantifies saturation by the percentage of test time it occurs and injected nonlinearities by average deviation from linear-model behavior; PWM and quantisation are present throughout DC servo tests.For the drone, motor saturation includes constrained power and inability to generate negative force; aircraft saturation is measured at the elevator’s maximum or minimum angle. The models’ quantified phenomena are plotted against each test’s main frequency–amplitude component.Source: §6.3 · pp. 39–40
  • Tests exposed actuator saturation across a wide range of degrees and covered transition regions; higher input frequency and amplitude, especially together, tended to trigger saturation.For the DC servo, tests above 90% actuator-saturation time were missed, but would require inputs beyond the practical one-rotation range. The authors report coverage across the SUTs, including the drone and aircraft.Source: Figure 12 · p. 41, §6.3 · p. 41, Figure 13 · p. 42
  • DC servo tests exposed sensor saturation from 0% to 90%; it appeared with large amplitudes and low frequencies, with a sharp transition into sustained saturation.High-frequency filtering reduces the input signal in the output, limiting sensor saturation. Once saturation occurs, the control algorithm cannot restore an unsaturated state; higher percentages would require amplitudes beyond the sensor range.Source: §6.3 · pp. 41, 43, Figure 13 · p. 42
  • Generated tests trigger nonlinearities within valid input bounds; saturation and quadratic friction vary with frequency and amplitude, while Coulomb friction, dead-zone, and backlash occur readily.Quadratic friction is most exposed near the DC servo’s 0.2–0.3 Hz closed-loop bandwidth; backlash is less exposed at low frequencies because inputs change direction less often.Source: §6.3 · p. 43

6.4 RQ2 - Relevance of Metamorphic Relations

  • MR1 fails when high-frequency filtering masks nonlinear effects or specific phenomena cause unpredictable behavior, so testing must cover the full input range rather than only large amplitudes and frequencies.Examples include low-frequency sensor saturation in the DC servo and high-frequency tests in the drone and lightweight aircraft where actuator saturation does not produce high 𝔡nl.Source: §6.4 · p. 48
  • MR2 usually shows increasing 𝔡f with frequency, but exceptions expose untracked low-frequency components that 𝔡nl can miss.Large-amplitude DC servo tests can saturate sensors and actuators, while lightweight-aircraft trapezoidal tests show small deviations during flight transitions; both patterns warrant further investigation.Source: §6.4 · p. 49
  • MR3’s 0.5-threshold crossing is generally similar across shapes, but drone ground impacts and lightweight-aircraft amplitude-dependent tracking exceptions reveal scope limitations.The threshold occurs around 0.6 Hz for the drone, just below 1 Hz for most DC servo versions (below 0.5 Hz for DC3), and around 0.15 Hz for the lightweight aircraft.Source: §6.4 · pp. 49–50
  • MR1 generally holds, while MR2 and MR3 hold in most tests and can complement 𝔡nl by detecting cases outside the CPS design scope.MR1 can fail because high-frequency filtering masks nonlinear effects or specific nonlinear phenomena cause unpredictable behavior; MR2 and MR3 expose some misbehavior that 𝔡nl misses.Source: §6.4 · p. 50

6.5 Threats to Validity

  • The findings may generalize beyond the case studies, though model developers’ understanding and manual plot inspection are limitations; automated MR evaluation remains future work.The drone and DC servo models follow real-system documentation, and the independently developed lightweight aircraft showed similar observations; access to physical systems also helped validate model choices.Source: §6.5 · p. 50

6.6 Data Availability

  • The authors provide SUT models, testing code, reproduction instructions, and output traces for recreating the article’s figures.Repositories are provided for the Crazyflie, DC servo, and lightweight aircraft.Source: §6.6 · p. 51

7 APPROACH APPLICABILITY AND LIMITATIONS

The approach identifies when nonlinear phenomena affect control-system behaviour, but practical use depends on SUT knowledge and can require many tests. Across the case studies, test shapes and system-specific design bounds influence what is exposed; design-scope information can inform verification and development.

7 APPROACH APPLICABILITY AND LIMITATIONS

  • The approach generates tests that falsify linearisation assumptions, while the metrics and metamorphic relations identify when nonlinearities affect reference tracking and push a system outside its design scope.The reported results support stress testing individual control loops across the studied systems.Source: §7 · p. 51

7.1 Identification of Non-Linear Phenomena in Practice

  • Detailed SUT knowledge may be needed to identify nonlinear phenomena, and d_nl can miss cases with weak new frequency components; d_f and MR2/MR3 can help detect these cases.The complementary relations identify low-frequency filtering or inconsistent closed-loop bandwidth where d_nl alone may be small.Source: §7.1 · p. 51

7.2 Campaign Duration

  • Step 1 reduces test counts for the drone and lightweight aircraft, but not the DC servo; many tests may still burden campaigns that cover multiple control loops.The campaigns took hours for the drone and lightweight aircraft and no more than a day for the DC servo versions, while minimizing test counts remains future work.Source: §7.2 · pp. 51–52

7.3 On the use of Different Shapes

  • Ramp and step inputs more readily expose nonlinear behaviour, plausibly because they include many frequency-amplitude components; trapezoidal and triangular inputs share a similar component pattern.The paper identifies the selection of useful frequency-amplitude patterns for test shapes as an area for further research.Source: §7.3 · p. 52

7.4 General Applicability of the Approach

  • Across the case studies, stress tests expose nonlinear phenomena, but whether those phenomena affect design scope depends on the system; MR2 and MR3 are consistent, while MR1 varies with system bounds.The tests triggered input and friction nonlinearities without those necessarily impairing design scope, whereas the reported sensor-saturation cases were limited to some DC servo tests; the study focuses on valid inputs.Source: §7.4 · pp. 52–53

7.5 Practical Implications of the Design Scope

  • Design-scope information can guide verification and runtime checks, and can help developers identify when system components may need redesign.Runtime checks could reshape reference signals or warn users; an unexpectedly narrow scope may indicate a need to redesign actuators, sensors, or filters.Source: §7.5 · p. 53

8 RELATED WORK

Prior work addresses CPS testing through model testing, falsification, runtime verification, and implementation-focused studies. This paper distinguishes its contribution by using a frequency-amplitude input description and control-design knowledge to test the validity of control assumptions in implemented systems.

8 RELATED WORK

  • Earlier CPS model-testing work generated traces using search algorithms [2, 28, 35–37], classification trees, and system-identification refinements [39]; Matinnejad et al. explored derivatives linked to input frequency and amplitude.The paper says its explicit frequency-amplitude description enables qualitative characterisation and metamorphic relations informed by control theory.Source: §8 · p. 54
  • CPS falsification work targets requirements, including robustness-guided falsification, invalidating temporal-logic formulas, and rapid-exploration methods; this article instead targets control-design assumptions.The cited reinforcement-learning work notes the value of access to internal system dynamics.Source: §8 · p. 54
  • Runtime-verification research includes frequency-domain signal specifications, hybrid signal logic [41], and dynamics-based monitoring, while implementation-focused studies include software-in-the-loop quadcopter simulations.The paper observes that control-theoretical knowledge has been used in prior testing to reduce tests or define oracles, but not to explicitly leverage the SUT's control design in defining the testing problem [39].Source: §8 · pp. 54–55
  • The paper positions frequency-amplitude input-space characterisation and control-informed metamorphic relations as extensions beyond requirement-driven testing and time-domain descriptions.It presents these as part of a control-engineering perspective on CPS testing.Source: §8 · p. 55

9 CONCLUSION

The paper defines stress testing for control-based CPSs as testing whether implemented systems violate control-design assumptions. It uses frequency-amplitude test generation, metrics, and metamorphic relations to identify tests that approach or exceed the design scope; future work includes broader assumptions and signals.

9 CONCLUSION

  • The approach generates tests around linearity assumptions using a qualitative frequency-amplitude characterisation, novel test parametrisation, a stress-test metric, and metamorphic relations for expected behaviour.Evaluation across six case studies found that the approach falsified linearity assumptions and that d_nl, d_f, and the relations helped identify stress tests, particularly those outside design scope.Source: §9 · p. 55
  • Future work includes testing assumptions about control-mode changes and interactions between control loops, as well as combined falsification of different assumptions.The paper also identifies testing arbitrary signals and reducing test counts as future directions.Source: §9 · pp. 55–56
Loading 2302.13913v4…