Source-linked AI summary
Stateful Testing for Deep Learning Compilers
Jiatao Ge, Ruixiang Qian, Yibo Hu, Yanzhou Mu, Yinglong Zou, Jiawei Liu, Zhenyu Chen, Chunrong Fang
TL;DR
Retained compiler state can make a later invocation incorrect even when it is correct in isolation, a gap that single-invocation tests do not address. Rondo explores model invocation sequences and compares uncompiled execution with compiled execution under reset and retained state. Across four frameworks, it detects 117 bugs, including 35 stateful bugs, and achieves the highest behavior and compiler Python source-line coverage among evaluated methods.
Problem
Single-invocation tests do not capture compiler state established by earlier invocations, although that state can make a later invocation incorrect.
Source: §1 · p. 2Method
Rondo represents tests as models paired with invocation sequences, guides sequence search with compilation-behavior feedback, and compares uncompiled, compiled-reset, and compiled-retained executions.
Source: §1 · p. 2, §4.2 · p. 9Results
Rondo detects 117 bugs, including 35 stateful bugs, and achieves the highest behavior coverage and compiler Python source-line coverage among evaluated methods.
Source: §1 · p. 3Takeaways & Limitations
Developers confirmed 35 of 39 submitted bug reports, including 16 stateful reports, and fixed 18 bugs, including 11 stateful bugs.
Source: §1 · p. 3The selected programs, frameworks, and hardware may limit generalization.
Source: §7 · p. 23
Abstract
from arXiv · showhide
Modern deep learning compilers compile models lazily and cache artifacts across invocations. Retained compiler state can make a later invocation incorrect even when it is correct in isolation. An analysis of 26 publicly reported issues in PyTorch, JAX, and TensorFlow shows that detecting such bugs requires sequences that first establish compiler state and then expose a failure. We present Rondo, a stateful testing method that represents each test case as a model paired with an invocation sequence. Rondo mutates invocation sequences and uses a Markov chain Monte Carlo sampler guided by compilation behavior to favor rarely observed behaviors. Its three-way differential oracle compares execution without compilation, compiled execution with compiler state reset, and compiled execution with state retained. It confirms stateful bugs by replaying the same invocation with state retained and reset while preserving inputs and program state, then shortens sequences while preserving the original bug. We evaluate Rondo on four deep learning frameworks using 24 common model architectures. Rondo detects 117 bugs, including 35 stateful bugs. We submit 39 bug reports, including 18 for stateful bugs; developers confirm 35 reports, including 16 for stateful bugs, and fix 18 bugs, including 11 stateful bugs. Rondo achieves the highest behavior coverage and compiler Python source-line coverage among the compared methods, with 99.26% alarm precision. It shortens 44 of 117 invocation sequences, reducing the lengths of these 44 sequences by 83.34% on average.
1 Introduction
Retained compiler state can make a later invocation incorrect even when it is correct in isolation, while ordinary single-invocation tests do not represent the state established earlier. Rondo represents tests as models paired with invocation sequences, guides sequence generation using compiler behavior, and checks results across compilation and state conditions.
Problem
- Stateful compiler bugs arise when compiler information retained from earlier invocations makes a later invocation incorrect, even though that invocation is correct in isolation.Compiler state includes retained artifacts that can affect later invocations.Source: §1 · p. 2
- Single-invocation test cases omit the earlier invocations needed to establish compiler state and expose these bugs later.The analysis of 26 reported issues shows that detecting them requires invocation sequences.Source: §1 · p. 2
Method
- Rondo represents each test as a model paired with an invocation sequence and guides generation toward rarely observed compiler behaviors using compilation feedback.It generates sequences through mutation or SMT-based synthesis.Source: §1 · pp. 2–3
- Rondo compares uncompiled execution with compiled execution both after a state reset and with earlier compiler state retained, then shortens sequences while preserving the bug.Confirmation holds the invocation’s inputs and program state fixed when comparing retained and reset compiler state.Source: §1 · pp. 2–3
Evaluation
- Rondo detects 117 bugs, including 35 stateful bugs, and achieves the highest behavior coverage and compiler Python source-line coverage among evaluated methods.Developers confirmed 35 of 39 submitted reports, including 16 stateful reports, and fixed 18 bugs, including 11 stateful bugs; alarm precision was 99.26%.Source: §1 · p. 3
2 Key Observations
An analysis of public compiler issues identifies how stateful bugs are triggered, what they affect, and why tests must represent the compiler state established by earlier invocations. These findings motivate sequence-based testing that checks more than returned outputs.
2.1 Bug Collection and Analysis
- The analysis retained 26 bug cases after filtering public reports, and all 26 were successfully reproduced on their affected versions.The cases were collected from public issue trackers and examined for their invocation sequences, failure mechanisms, and effects.Source: §2.1 · pp. 3–4
2.2 Bug Triggers across Invocations
- Stateful bugs require sequences that establish compiler state before a later invocation exposes an incorrect result.In 17 of 26 cases, compiled code was reused after its required conditions no longer held; in 9, earlier values or structures did not match current program state.Source: §2.2 · p. 5
2.3 Impact of Stateful Bugs
- These bugs affect outputs, gradients, and observable program state, and may do so without raising an exception.Examples include stale outputs, missing gradients despite correct outputs, and incorrect random-generator results or state.Source: §2.3 · p. 6
2.4 Characterizing Compiler State
- Testing must characterize compiler state so it can establish and reproduce the state that exposes a later failure.The invocation that establishes relevant compiler state can complete correctly even when a later invocation fails after program state changes.Source: §2.4 · pp. 6–7
3 Related Work
Prior deep-learning compiler testing uses generation or mutation to diversify models, programs, and intermediate representations, with differential or metamorphic checks to identify bugs. Rondo instead targets invocation sequences that establish compiler state and confirms whether retained state is necessary for the failure.
Prior testing methods
- Existing generation-based approaches construct models, programs, or IRs under compilation constraints, while mutation-based approaches modify or combine existing tests.The surveyed methods exercise different compiler levels and use operator constraints, language rules, compiler information, or existing tests.Source: §3 · p. 7
Test oracles
- Prior oracles compare compiled and reference results, different compiler configurations, or programs linked by a declared result relation.Other checks include rules for eliminating undefined behavior and selecting runnable lowering paths for differential comparison.Source: §3 · pp. 7–8
Comparison with prior work
- Rondo searches and mutates invocation sequences to establish compiler state, then compares the same invocation with that state retained and reset while holding inputs and relevant program state fixed.It checks outputs, gradients, and observable program state, then shortens sequences while preserving the bug and its dependence on retained compiler state.Source: §3 · p. 8
4 Methodology
Rondo represents a test as a model and an ordered invocation sequence that can establish compiler state. It confirms a stateful bug by comparing the same invocation under retained and reset compiler state while keeping inputs and relevant program state unchanged.
4.1 Problem Formulation
- Because compiler state is not directly observable, Rondo represents it by the invocation prefix that establishes it from the fixed initial state.Replaying a legal prefix determines the reached configuration, and replaying the same sequence can reproduce the compiler state.Source: §4.1 · p. 8
- A Rondo test case pairs a model with an invocation sequence containing model invocations or declared state operations in order.The fixed scenario specifies the framework build, backend, device, compiler options, and initial conditions.Source: §4.1 · p. 8
- A stateful witness pairs a state-establishing prefix with an invocation that fails under retained compiler state but succeeds after reset.The comparison keeps the invocation’s inputs and relevant program state, including alias relations, unchanged.Source: §4.1 · p. 9
4.2 Method Overview
Rondo searches invocation sequences that establish compiler state, checks selected sequences for candidate bugs, and refines candidates into minimized witnesses. Compilation behavior guides exploration, while controlled comparisons determine whether a bug depends on retained compiler state.
4.2 Method Overview
- Rondo’s three modules generate legal test cases, compare execution modes and semantic properties, then confirm and minimize candidate bugs.Exploration uses mutation or SMT-based synthesis; refinement preserves the witness class, exposing invocation, and recorded symptoms.Source: §4.2 · p. 9
- The test oracle compares outputs, gradients, and observable program state across uncompiled, compiled-with-reset, and compiled-with-retained-state modes.It also applies independent semantic checks to identify candidate bugs.Source: §4.2 · p. 9
- Stateful candidate refinement replays the failing invocation with compiler state retained and reset, holding inputs and relevant program state unchanged.It then minimizes the sequence while preserving the bug class, exposing invocation, and recorded symptoms.Source: §4.2 · p. 9
4.3 Test-space Exploration
Rondo explores compiler states through legal invocation sequences generated by mutation or, when framework constraints are available, SMT-based synthesis. An adaptive MCMC sampler uses compilation-behavior feedback to favor rarely observed behaviors.
4.3.1 Generating Test Cases.
- Mutation varies invocation inputs, program state, and gradient behavior; 13 operators also revisit invocations or explore alternative continuations.The operators include input changes, state updates, mode switches, backward passes, repeated invocations, and truncation.Source: §4.3.1 · pp. 10–11
- SMT-based synthesis targets unvisited guard outcomes by solving for legal inputs, with mutation as fallback when constraints cannot be solved.A guard checks whether a compiled artifact can be used for the current invocation.Source: §4.3.1 · p. 11
4.3.2 Guiding Test Case Generation.
- The adaptive MCMC search scores sequences using observed behavior keys, giving greater weight to rare behaviors and accounting for sequence-generation probabilities when sampling.New behavior keys can also cause test cases to be saved as starting points for later searches.Source: §4.3.2 · p. 12
- A sequence with no new behavior may still be retained because added invocations can reveal new compiler behavior later.The example contrasts ordinary artifact reuse with reuse after a state update, which had not yet been observed.Source: §4.3.2 · p. 12
4.4 Test Oracle
Rondo’s oracle compares three execution modes and checks structured observations against numeric rules and declared semantic properties. These checks identify candidates, while controlled replay tests whether retained compiler state is necessary for a failure.
4.4 Test Oracle
- The oracle runs each sequence without compilation, compiled with state reset, and compiled with state retained, then compares observations and applies semantic checks.The three modes share the same initial program state but maintain their own program state as invocations proceed.Source: §4.4 · p. 12, §4.4.1 · p. 13
4.4.1 Invocation Modes.
- Compiled-with-reset execution clears or isolates compiler state at a reset boundary while preserving the program state reached by preceding invocations.Compiled-with-state execution retains compiler state throughout the sequence, allowing later invocations to reuse earlier compiled artifacts.Source: §4.4.1 · p. 13
4.4.2 Semantic Checking.
- Structured observations combine returned-value trees with declared observable program state, and comparison checks structure, shape, dtype, and representation-aware numeric values.Discrete values must match exactly; floating-point checks distinguish NaNs, infinities, signed zeros, zero versus nonzero, normal values, and subnormals.Source: §4.4.2 · p. 14
- Six independent semantic checks can detect incorrect results shared across all three modes, and violations become candidates for controlled replay.The checks cover precision, deterministic stability, scalar bit patterns, mathematical identities, implementation consensus, and result bounds.Source: §4.4.2 · p. 14
4.5 Stateful Candidate Refinement
Rondo confirms stateful failures by replaying the exposing invocation with compiler state retained and reset from aligned program state, then minimizes sequences while preserving reproduction conditions. Its refinement distinguishes witness classes and retains the original failure’s defining conditions.
4.5.1 Confirming Stateful Bugs.
- Stateful confirmation reruns the exposing invocation with compiler state retained and reset from the same reached program state, aligning inputs and relevant state.For random operations, alignment also includes relevant random-generator state; if validity and alignment conditions cannot be established, classification remains undetermined.Source: §4.5.1 · p. 15, §4.5 · p. 15
- In the example, retaining the compiled artifact after the prefix makes invocation B return 6, while resetting compiler state at w = 4 with x = 3 makes it return 12.Deleting the prefix would also change program state, so it cannot substitute for the aligned comparison.Source: §4.5.1 · p. 15, Figure 5 · p. 16
- Rondo distinguishes point-wise witnesses, reproducible by the final invocation alone, from history-exposed witnesses that require prefix-established program state but not retained compiler state.A bug is classified as stateful when at least one witness satisfies the stateful-witness definition.Source: §4.5.1 · pp. 15–16
4.5.2 Sequence Minimization.
- Rondo accepts a shortened sequence only when it preserves the witness class, exposing invocation, failing modes, and original symptom signatures.Mutations delete invocation blocks or individual invocations, or simplify retained inputs; after single-invocation deletions are exhausted, the sequence is 1-minimal under that deletion operation.Source: §4.5.2 · pp. 16–17
5 Experiment Design
The experiment evaluates bug detection, coverage, ablations, oracle effectiveness, and sequence refinement across four frameworks and 24 model architectures. Methods share evaluation settings, while coverage, oracle, and refinement measurements use distinct protocols.
5.1 Research Questions
- The five research questions assess bug detection, diversity and coverage, component ablations, oracle effectiveness, and sequence refinement.Baselines include NNSmith, OATest, AlignGuard, and RandomHistory; ablations remove feedback, SMT synthesis, the return operator, or state operations.Source: §5.1 · p. 17, §5.2 · p. 17
5.3 Measurements
- Coverage comparison measures input, operator-path, compiler-guard, and across-invocation behavior coverage, plus compiler Python source-line coverage.The four evaluation metrics use shared total sets, so absolute coverage depends on those selected sets and is primarily used for relative comparisons.Source: §5.3 · p. 18
- Oracle evaluation reports true and false positives across five cumulative checking levels, while refinement counts only strictly shorter sequences that preserve the original bug.Refinement statistics include oracle checks and time; cases that cannot shorten, exceed time, or fail to preserve the bug do not count as effective successes.Source: §5.3 · p. 18
5.4 Implementations
- Rondo is evaluated on four frameworks and 24 common model architectures, with three-run coverage comparisons after 600 test cases and five oracle configurations on 50,074 labeled cases.The frameworks are PyTorch, MindSpore, TensorFlow, and JAX; sequence-refinement statistics cover test cases reproducing all 117 bugs.Source: §5.4 · pp. 18–19
5.2 Baseline Methods
- Baseline and ablation methods use the same test programs, seeds, settings, metric definitions, and reference protocol.All methods are implemented with the same framework versions, environment, and configurations.Source: §5.4 · pp. 18–19
6 Results and Analysis
Rondo detects bugs across four deep learning frameworks and achieves the strongest reported behavior and compiler source-line coverage among evaluated methods. Its ablations, oracle checks, and refinement results characterize how the approach performs and where its gains arise.
6.1 RQ1: Bug Detection
- 117 bugs were detected across four frameworks, including 35 stateful bugs; developers confirmed 35 reports and fixed 18 bugs.Of 39 submitted reports, 18 concerned stateful bugs; developers confirmed 16 stateful reports and fixed 11 stateful bugs.Source: §6.1 · pp. 19–20, Table 3 · p. 19
6.2 RQ2: Diversity and Coverage
- 106.66% higher HBC than RandomHistory, the strongest baseline, accompanied Rondo’s highest coverage on all four behavior metrics and 14.94% compiler source-line coverage.Its compiler source-line coverage exceeded RandomHistory’s 14.17% by 0.77 percentage points, or 1,727 covered lines.Source: Table 4 · p. 20, Figure 6 · p. 20, §6.2 · p. 21
6.3 RQ3: Component Ablation
- Removing feedback caused the largest declines in Input, Path, and Guard, while removing SMT-based synthesis caused smaller declines in Input, Path, and HBC.Without SMT slightly increased Guard from 0.3837 to 0.3849 and added three compiler source lines; rounded source-line coverage was 14.94% in both configurations.Source: Table 5 · p. 21, §6.3 · pp. 21–22
- Removing state operations reduced HBC by 49.01%, the largest HBC decrease among the four ablations; removing the return operator reduced Guard by 40.42% and HBC by 34.06%.Removing state operations also reduced Input, Path, and Guard by 32.59%, 37.71%, and 45.50%, respectively.Source: §6.3 · pp. 21–22
6.4 RQ4: Oracle Effectiveness
- 539 true-positive test cases were detected by Full Oracle, up from 31 with crash-only checking; it produced four false positives among 543 alarms.Full Oracle achieved 99.26% alarm precision and a 0.74% false discovery rate; semantic checks added 52 true positives.Source: Table 6 · p. 22, §6.4 · p. 22
6.5 RQ5: Sequence Refinement
- 83.34% mean compression was achieved for the 44 of 117 test cases whose sequences were shortened while preserving the original bugs.Across all test cases, the mean compression ratio was 31.34%; refinement averaged 7.32 oracle checks and 184.55 seconds per test case.Source: §6.5 · pp. 22–23, Table 7 · p. 23
7 Threats to Validity
The evaluation identifies internal, external, and construct threats to validity and describes measures used to address them.
Internal Threats to Validity
- Errors in test runs, oracle checks, or compiler-state reset can affect bug detection and stateful-bug classification.The authors compare three execution modes, use independent semantic checks, align inputs and relevant program state when confirming stateful bugs, and recheck bug relations during minimization.Source: §7 · p. 23
External Threats to Validity
- Selected programs, frameworks, and hardware may limit how broadly the findings generalize.The evaluation covers four frameworks and 24 common architectures, including MLPs, CNNs, and Transformers.Source: §7 · p. 23
Construct Threats to Validity
- Behavior metrics and oracle alarms may not fully reflect compiler coverage or the number of underlying bugs.The study reports compiler Python source-line coverage alongside behavior coverage and distinguishes detected bugs from oracle true-positive test cases.Source: §7 · p. 23
8 Conclusion
Single-invocation tests do not represent compiler state that can affect later calls. Rondo uses invocation sequences to explore that state, and its evaluation detects bugs across four frameworks.
8 Conclusion
- Single-invocation tests do not represent compiler state that can affect later invocations.Rondo represents each test case as a model paired with an invocation sequence.Source: §8 · p. 23
- Rondo guides sequence generation with compilation-behavior feedback and checks execution without compilation, with state reset, and with state retained.It confirms stateful bugs by replaying an invocation with compiler state retained and reset while preserving inputs and relevant program state.Source: §8 · p. 23
- Across four deep learning frameworks, Rondo detected 117 bugs, including 35 stateful bugs; developers confirmed 35 reports and fixed 18 bugs.The study submitted 39 reports, including 18 for stateful bugs; developers confirmed 16 stateful reports and fixed 11 stateful bugs.Source: §8 · p. 23
Data-Availability Statement
The public artifact contains Rondo’s implementation, bug-discovery results, and experimental results; reproduction requires the framework versions and device support described in Section 5.4.
Data-Availability Statement
- Reproduction requires the framework versions and device support described in Section 5.4.The public artifact includes Rondo’s implementation, bug-discovery results, and experimental results.Source: Data-Availability Statement · p. 24