PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding Deep Learning Framework Testing in Data Flow

Jiatao Ge, Yanzhou Mu, Yinglong Zou, Yibo Hu, Zhenyu Chen, Chunrong Fang

arXiv:2610.15808v1cs.SEcs.LG

TL;DR

Data-flow bugs can affect gradient propagation and fusion behavior, while topology-oriented testing does not capture runtime tensor states and operator input requirements. FlowMuT uses a Tensor Flow Graph to guide constrained mutation testing; it submitted 55 bug reports for cases missed by baselines and improved data flow diversity by 43.4% over the strongest baseline.

  • Problem

    Data-flow bugs can affect gradient propagation or change the computation represented by a fusion, while topology-oriented rules omit additional data flow information.

    Source: §1 · p. 2, §3.3.2 · p. 8

  • Method

    FlowMuT uses a Tensor Flow Graph to represent runtime tensor states and operator input requirements, then selects constrained subgraph mutations with LinUCB and execution feedback.

    Source: §3.1 · p. 5, §7 · p. 18

  • Results

    FlowMuT submitted 55 bug reports for cases missed by baselines, improved data flow diversity by 43.4% over the strongest baseline, and achieved a 96.7% legal-mutant rate.

    Source: §7 · p. 18

  • Takeaways & Limitations

    Each accepted mutant refreshes the TFG and serves as the seed for the next iteration, with execution feedback guiding subsequent selection.

    Source: §3.1 · p. 5, §7 · p. 18

    The evaluation covers two frameworks and diverse application tasks but cannot represent every architecture or data flow pattern.

    Source: §6 · p. 18

Abstract

from arXiv · show

Data flow underlies deep learning (DL) framework execution, including automatic differentiation and operator-fusion optimization. Bugs involving data flow can cause incorrect results or unexpected behavior. Existing testing methods represent models primarily by operator topology, omitting runtime tensor values and properties and operator input requirements. They therefore cannot distinguish data flows in models with similar topologies, even when these differences affect framework behavior. We introduce the Tensor Flow Graph (TFG) to capture runtime tensor states and operator input requirements. Its nodes represent operators and their input requirements, while its edges record tensor values and properties. Building on the TFG, we propose FlowMuT, a mutation-based testing method grounded in data flow. Mutations are represented as constrained subgraph replacements, with mutation operators derived from prior work and an analysis of mutable data flow objects. In each round, FlowMuT builds a TFG for the seed model, pairs mutation operators with TFG sites, and filters illegal candidates using data flow constraints. It uses LinUCB to select a candidate based on coverage and impact, executes the mutant, updates selection feedback, and uses each accepted mutant as the next seed. We evaluate FlowMuT on PyTorch and MindSpore using representative models from diverse application scenarios. FlowMuT submitted 55 bug reports for cases missed by all baseline methods; developers confirmed 41 reports and fixed the bugs described in 7 reports. FlowMuT achieves 43.4% higher data flow diversity than the strongest baseline and a legal mutant rate of 96.7%, compared with 86.4% without constraints. These results show that testing from a data flow perspective is practical and effective.

1 Introduction

Topology-based model descriptions omit runtime tensor states and operator input requirements, leaving data-flow behavior unrepresented. FlowMuT addresses this with a Tensor Flow Graph and constrained, data-flow-guided mutation; evaluation reports more detected bugs, higher data-flow diversity, and more legal mutants than the baselines or unconstrained variant.

1 Introduction

  • Topology-only graphs omit runtime tensor values, properties, and operator input requirements, so they cannot distinguish executions with the same operators and connections but different data flows.Such differences can affect framework behavior, including layout and storage aliasing in the SDPA example.Source: §1 · p. 2
  • Mutation legality depends on simultaneously satisfying dtype and shape compatibility, operator input requirements, and preservation of intended tensor values.For the slice-and-expand example, sharing K and V heads is legal when their values match, but changes unequal values and must be rejected.Source: §1 · pp. 2–3
  • FlowMuT represents profiled executions with a TFG and uses constrained subgraph mutations, filtering, and LinUCB selection based on coverage and impact.Its 147 mutation operators combine adapted CG-derived operators with operators constructed from TFG objects and mutation primitives.Source: §1 · p. 3
  • FlowMuT detected 79 bugs versus 28 for the strongest baseline, and 55 cases found only by FlowMuT led to new bug reports.Developers confirmed 41 reports and fixed the bugs described in 7 reports.Source: §1 · p. 3
  • 43.4% higher data-flow diversity than the strongest baseline and a 96.7% legal mutant rate with constraints, compared with 86.4% without constraints.The evaluation covers PyTorch and MindSpore and 14 representative models from ten application scenarios.Source: §1 · p. 3

2 Background and Related Work

Data flow includes the tensors produced and consumed during execution and their runtime properties, which matter to both gradient computation and compiler transformations. Existing testing approaches include API-level, generation-based, and mutation-based methods, while FlowMuT uses a profiled whole-model representation to build, filter, and select mutations.

2.1 Data Flow in DL Frameworks

  • Data flow describes the tensors and runtime properties realized on a model’s operator routes, so the same operator topology can produce different data flows under different inputs or execution conditions.Operator topology records operators and their static connections.Source: §2.1 · p. 4
  • Runtime tensor states matter alongside operator dependencies: automatic differentiation uses saved intermediate tensors and gradient states, while compiler transformations use dependencies and tensor properties such as shape and layout.These graphs and representations serve execution or optimization tasks, rather than providing a common mutation-testing abstraction.Source: §2.1 · p. 4

2.2 DL Framework Testing

  • DL framework tests range from API-level calls to complete models; model-level test generation synthesizes models, while mutation-based methods transform executable seed models.Prior generation approaches include templates, grammars, and operator constraints; prior mutation approaches edit model structures, inputs, parameters, or weights and use feedback to guide testing.Source: §2.2 · pp. 4–5
  • FlowMuT mutates complete executable models using a profiled whole-model TFG, forming and filtering site-specific candidates from runtime tensor states and operator input requirements.It selects retained candidates using coverage and impact features, unlike prior mutation methods organized primarily around layers, parameters, weights, or structural graph units.Source: §2.2 · p. 5

3.1 Overview

FlowMuT repeatedly profiles a seed model, constructs and filters mutation candidates, selects one with LinUCB, and executes it. Successful mutants become the next seed, with the TFG refreshed after acceptance.

3.1 Overview

  • Each round profiles the seed model and builds its TFG, then generates candidates, selects one with LinUCB, and executes the resulting mutant.The workflow checks for framework bugs and updates the selection state from execution feedback.Source: §3.1 · pp. 5–6
  • Candidate generation instantiates mutation operators as constrained subgraph replacements and filters illegal candidates before LinUCB selects an operator and site.The workflow takes a seed model, profiling inputs, and a preconstructed operator set as inputs.Source: §3.1 · p. 5
  • After successful execution, FlowMuT accepts the mutant as the next seed and rebuilds its TFG from a newly profiled execution.Bug-reporting executions are added to the reported bug set rather than advancing the seed in that step.Source: §3.1 · p. 6

3.2 Tensor Flow Graph and Mutation Abstraction

The Tensor Flow Graph combines operator topology, runtime tensor information, and operator input requirements. FlowMuT represents mutations as constrained replacements of bounded subgraphs, checking compatibility with the surrounding model.

3.2.1 Tensor Flow Graph.

  • The TFG combines operator topology, runtime tensor values and properties, and operator input requirements in one representation.Its topology connects operator nodes by tensor edges; profiled edge annotations capture runtime states, while node annotations record requirements.Source: §3.2.1 · pp. 6–7, Figure 3 · p. 7
  • In the TFG, topology identifies mutation sites, runtime tensor annotations determine which operators can be instantiated, and input requirements check compatibility with the surrounding model.Runtime annotations include tensor properties collected by profiling, and input requirements include individual-input properties and relations among inputs.Source: §3.2.1 · p. 7

3.2.2 Mutation as Constrained Subgraph Replacement.

  • Each mutation is a constrained replacement of a bounded subgraph, with a pattern specifying where it applies, a replacement specifying the change, and constraints specifying when it is expected to remain legal.The abstraction unifies mutations over tensors, operators, and bounded computations.Source: §3.2.2 · p. 7
  • Input, internal, and output constraints check the data flow and topology around and within a replacement to prevent mutations incompatible with the surrounding model.The checks cover edges entering the matched subgraph, its internal structure, and edges leaving the replacement.Source: §3.2.2 · p. 7

3.3 Mutation Operator Pool

FlowMuT builds a unified mutation-operator pool from prior topology-based rules and additional data-flow mutations. Each operator is represented as a constrained subgraph replacement and checked for applicability and effect.

3.3.1 CG-Derived Mutation Operators.

  • The CG-derived pool retains 120 unique operators from 155 extracted rules across 19 source studies after mapping, filtering, and deduplication.Rules were retained when their changes mapped to TFG elements and the information needed to check them was available.Source: §3.3.1 · p. 8

3.3.2 TFG-Derived Mutation Operators.

  • 48 TFG-derived operators combine mutable objects with update, insertion, deletion, or rewiring primitives, subject to boundary and implementation checks.Examples include inserting a copy to break sharing or making an implicit dtype or layout conversion explicit.Source: §3.3.2 · pp. 8–9

3.3.3 Merging, Deduplication, and Validation.

  • FlowMuT combines CG- and TFG-derived operators into a validated pool of 147 after removing 21 overlaps.Operators are implemented and checked to perform their intended changes while rejecting violations of declared constraints.Source: §3.3 · p. 8, §3.3.3 · pp. 9–10

3.4 Test Case Generation

FlowMuT profiles each seed model into a TFG, constructs and filters mutation candidates, then selects and executes a candidate. Successful mutants become seeds for subsequent iterations.

3.4 Test Case Generation

  • Each iteration profiles the current seed, pairs mutation operators with TFG sites, filters candidates, selects and executes a mutant, and uses successful mutants as new seeds.Accepted mutants are reprofiled to refresh the TFG and candidate set.Source: §3.4 · p. 10, §3.4.2 · p. 11

3.4.1 Efficient Candidate Construction.

  • Candidate construction pairs every mutation operator with sites anchored at tensor edges, giving O(|E|) construction time, then retains only candidates satisfying input, internal, and output constraints.The fixed operator pool makes the work per tensor edge constant.Source: §3.4.1 · p. 10

3.4.2 Candidate Selection.

  • Coverage features describe each candidate’s site context, tensor state, and mutation operator, with contributions decreasing as elements are exercised repeatedly.Tensor-state features include rank, dtype, device, layout, gradient state, and numerical properties.Source: §3.4.2 · pp. 10–11
  • Impact features estimate downstream computation reach and gradient influence; without gradients, FlowMuT estimates influence by perturbing the anchor tensor and measuring output response.Gradient influence is normalized within the current candidate set, and tensor size is factored out.Source: §3.4.2 · pp. 10–11
  • LinUCB selects candidates using estimated reward and uncertainty, then updates its feedback from structured execution traces and novelty in events and transitions.Executions without new behavior or a complete run receive zero reward, and failure labels are excluded.Source: §3.4.2 · p. 11

4 Experiment Design

The experiment design evaluates FlowMuT’s bug detection, diversity and coverage, mutation legality, and selection across benchmark models and mutation-based baselines, using campaigns and repeated fixed-round comparisons.

4.1 Research Questions

  • The study asks whether FlowMuT detects framework bugs, improves test diversity and coverage, filters illegal mutations, and selects operators and sites effectively.These are framed as four research questions.Source: §4.1 · p. 11

4.2 Benchmarks

  • The benchmark uses 14 representative seed models spanning ten application tasks, evaluated across PyTorch and MindSpore execution modes.Table 2 summarizes the models by application task, input size, and parameter count.Source: §4.2 · p. 12, Table 2 · p. 12

4.3 Baseline Methods

  • LEMON, COMET, and DevMuT are the mutation-based baselines, adapted to the same framework versions as FlowMuT.The adaptations follow the baselines’ papers and released implementations.Source: §4.3 · p. 12

4.4 Measurements

  • Bug detection counts distinct reproducible bugs; a potential bug is an unexpected crash, uncaught exception, semantic violation, or cross-mode difference beyond expected numerical tolerance.A failure counts only when reproducible.Source: §4.4 · p. 12
  • Diversity is measured with LIC, LPC, LSC, and DFSD, while framework coverage includes Python statements, C/C++ source lines, and exercised public interfaces.DFSD signatures include operator and kernel paths, binned value profiles, and tensor properties; final diversity and coverage measures are averaged at round 250.Source: §4.4 · p. 13
  • Mutation legality is measured by the Legal-Mutant Rate, comparing the full constraint system and variants that omit constraint types on 1,000 candidates.Manual inspection labels 864 candidates legal and 136 illegal; the study also records retention and filter selectivity.Source: §4.4 · p. 13
  • Selection policies are compared over 250 mutation rounds, with diversity tracked at checkpoints and LinUCB context features tested by removal and partial-context variants.The policies include one-shot, random-search, fixedpriority, and UCB1; practical cost is average generation and detection time per test case.Source: §4.4 · p. 13

4.5 Implementations

  • Each configuration uses ten independent runs per model–framework pair, with 250 rounds per run; bug discovery uses 12-hour campaigns for each seed model and framework.The evaluation targets PyTorch 2.12.1 and MindSpore 2.7.1 in the specified eager/compiled and PyNative/Graph modes.Source: §4.5 · p. 13

5 Results and Analysis

FlowMuT detects more bugs than the baselines and improves test-case diversity, framework coverage, and mutation legality. Its selection results show contributions from both coverage and impact features.

5.1 RQ1: Bug Detection

  • 79 bugs were detected by FlowMuT, compared with 28 for DevMuT; 55 cases found only by FlowMuT led to new bug reports.The campaigns also recorded 12 bugs for LEMON and 23 for COMET.Source: §5.1 · p. 14

5.2 RQ2: Diversity and Coverage

  • 43.4% higher DFSD than the strongest baseline accompanies FlowMuT’s lead across test-case diversity and framework-coverage metrics.Its combined generation and detection time was 10.63 seconds per test case, lower than COMET but higher than LEMON and DevMuT.Source: Table 4 · p. 15, §5.2 · p. 15
  • The full operator pool achieves the highest test-case diversity and framework coverage, while the CG-derived pool exceeds the TFG-derived pool on every reported metric.Removing any listed TFG-derived object type or mutation primitive lowers LIC, DFSD, and all three framework-coverage measures, though some reduced pools improve LPC or LSC.Source: §5.2 · p. 15, Table 5 · p. 16

5.3 RQ3: Mutation Legality

  • 96.7% legal-mutant rate, up from 86.4% without constraints, is achieved while retaining 88.0% of candidates.The complete constraint system has 89.2% rejection precision, 78.7% recall, and a 1.5% false-rejection rate; removing input constraints causes the largest leakage.Source: §5.3 · p. 17, Table 6 · p. 17

5.4 RQ4: Mutation Selection

  • LinUCB leads in diversity and coverage, reaching 48.1% C++ line coverage and a mean of 486.5 distinct signatures.Against context-free UCB1, it increases DFSD and mean signature count by 10.6%; against random search, it improves both by 1.1%, while raising C++ line coverage from 33.3% to 48.1% and API coverage from 22.7% to 42.2%.Source: Table 7 · p. 17, §5.4 · p. 18
  • Removing downstream impact reduces DFSD and mean signature count by 27.5%, while coverage-only and impact-only variants both trail the complete context.Removing operator coverage reduces LPC by 76.6% and LSC by 27.9%; removing tensor-state coverage lowers C++ and API coverage by 10.4 and 10.9 percentage points, respectively.Source: §5.4 · p. 18

6 Threats to Validity

The evaluation is subject to implementation, execution, measurement, and scope limitations. It covers two frameworks and diverse tasks, but not every architecture or data-flow pattern.

Internal Validity

  • Implementation differences and stochastic execution may bias method comparisons, despite controlled configurations and repeated runs.Component interactions may also affect ablation results; the authors use matched variants and interpret components through primary metrics.Source: §6 · p. 18

Construct Validity

  • Data-flow signatures may not represent source coverage, and failure symptoms may differ across executions.The evaluation reports diversity, coverage, and detected bugs separately and records developer confirmation and fix status.Source: §6 · p. 18

External Validity

  • The evaluation covers two frameworks and diverse application tasks, but cannot represent every architecture or data-flow pattern.Additional frameworks, accelerators, and deployment environments remain future evaluation scope.Source: §6 · p. 18

7 Conclusion

FlowMuT grounds model-level deep-learning framework testing in data flow through the Tensor Flow Graph and iterative mutation. On PyTorch and MindSpore, it reports additional bug reports, higher diversity, and a higher legal-mutant rate.

7 Conclusion

  • The TFG records operator topology, runtime tensor values and properties, and operator input requirements.It supports mutation construction, legality checking, and LinUCB selection using coverage and impact features.Source: §7 · p. 18
  • Execution feedback guides subsequent selection, and each accepted mutant supplies the TFG for the next iteration.Source: §7 · p. 18
  • On PyTorch and MindSpore, FlowMuT submitted 55 bug reports for cases missed by the baselines, improved data flow diversity by 43.4% over the strongest baseline, and achieved a 96.7% legal-mutant rate.Source: §7 · p. 18
  • FlowMuT improved data-flow diversity by 43.4% over the strongest baseline and achieved a 96.7% legal-mutant rate.Source: §7 · p. 18

8 Data Availability

The implementation and experimental materials are available in the accompanying artifact.

8 Data Availability

  • The accompanying artifact provides the implementation and experimental results, including configurations, mutants, triggering test cases, and bug reports.Source: §8 · p. 18
  • Researchers are encouraged to contribute new mutation rules following this study.Source: §8 · p. 18

Related papers

Loading 2610.15808v1…