PaperlayerView exact revision on arXiv

Source-linked AI summary

Building real-time digital twin instances with Function+Data Flow: user evaluation and extension for iterative pipelines

Eduardo de Conto, Blaise Genest, Arvind Easwaran, Nicholas Ng, Shweta Menon

arXiv:2608.18480v1cs.SEcs.CL

TL;DR

AI-based digital-twin pipelines are difficult to specify, validate, and reuse, motivating dedicated tooling. The paper evaluates FDF and DesCartes Builder and proposes H-FDF for iterative pipelines; the study reports good usability and the extension supports complex dual-training workflows.

  • Problem

    AI-based digital-twin engineering needs better support because pipelines are difficult to specify, validate, and reuse across data-driven phases.

  • Method

    The paper combines a controlled empirical user study of FDF and DesCartes Builder with the formal H-FDF extension for iterative pipelines.

  • Results

    Median SUS 72.5 (mean: 68.9 ± 15.2) indicates good perceived usability for FDF and DesCartes Builder.

  • Takeaways & Limitations

    H-FDF supports modular iterative pipelines, and its dual-training case study shows flexible co-induction with explicit loop conditions.

  • Takeaways & Limitations

    Warning messages remain a key improvement area, with 57% of users rating them friendly and 24% providing diagnostics-related feedback.

Abstract

from arXiv · show

Digital twins (DTs) increasingly leverage artificial intelligence (AI) and machine learning (ML) pipelines, both to build real-time DTs from high-fidelity simulations and to instantiate them with historical data. However, engineering these pipelines remains largely ad-hoc: pipelines are hard to specify, validate, and reuse, with scarce dedicated tooling. Function+Data Flow (FDF) addresses this by defining a visual domain-specific language (DSL) that represents functions (ML models) explicitly, enabling their composition and reuse. We implemented FDF in DesCartes Builder, an integrated modeling environment supporting FDF-based DT synthesis and validation. In this paper, we report on an empirical user study evaluating whether FDF and DesCartes Builder can make AI-based DT development more accessible and reliable. Participants implemented a representative real-time DT prototype within DesCartes Builder, and we measured perceived usability and feature adequacy through quantitative and qualitative measures. Our results indicate that DesCartes Builder and FDF achieve a good level of usability across a broad range of potential users, and particularly for the intended audience of domain experts. The study additionally surfaces concrete strengths and areas for improvement of both the tool and the underlying FDF framework. Informed by these findings, we propose H-FDF, a Hierarchical extension of FDF supporting iterative and modular pipelines, enabling the formal specification of more complex DT pipelines such as dual training. Our findings suggest that integrated, model-driven platforms are a promising direction to transform AI-based DT engineering into a disciplined modeling practice.

1 Introduction

Digital-twin engineering phases Φ2 and Φ3 require specialized support for data-driven surrogate models, yet existing frameworks remain limited. This paper evaluates FDF and DesCartes Builder and proposes H-FDF for iterative pipelines.

  • Motivation: DT engineering is complex despite its use in safety-critical domains and rapid market growth.The process is structured into four phases, from high-fidelity modeling to physical-twin interaction.
  • Use case: Model order reduction and supervised learning transform deformation and strain data into a real-time prototype for material-strain prediction.PCA reduces mesh dimensionality before supervised learning predicts reduced plastic strain from reduced deformation.
  • Research gap: Phases Φ2 and Φ3 lack frameworks that account for the particularities of digital-twin engineering.These phases create real-time prototypes and instantiated twins from simulation or historical data.
  • Contributions: FDF and DesCartes Builder provide a visual, model-driven approach that treats learned functions explicitly for composition and reuse.The approach targets model-order reduction, data assimilation, or their combination across Φ2 and Φ3.
  • Evaluation: The empirical study measures usability and feature adequacy by having academic participants implement a representative material-strain prototype.The study uses a controlled workshop with PhD students and researchers.
  • Contributions: H-FDF extends FDF with internally iterative modules while keeping the overall pipeline acyclic.The extension is illustrated with a dual-training use case.

2 Related Work

Related tools support simulation, workflow construction, or ML operations, but commonly leave models implicit or provide limited validation and reuse. FDF instead makes functions explicit and H-FDF separates iterative control from data flow.

  • Simulation environments: Simulink supports multi-domain physics simulation but lacks native ML training and offers limited interface typing and model reuse.These limitations distinguish its focus from FDF’s targeted support for data-driven DT phases.
  • ML workflow tools: ML workflow tools support experiment tracking, model versioning, and monitoring but generally keep the generated model implicit.Scikit-learn and PyTorch commonly execute the ML tasks underlying Φ2 and Φ3.
  • ML pipeline DSLs: DSL-based approaches increase abstraction and may support hierarchical decomposition, yet typically lack robust validation while keeping ML models implicit.This limits their ability to formalize the complete pipeline semantics.
  • FDF comparison: FDF identifies misconnections through implicit typing, while DesCartes Builder supports pipeline design, execution, parametrization, and validation.H-FDF additionally supports hierarchical decomposition and iteration until a suitable model is attained.
  • Comparison with KNIME: KNIME uses explicit nominal typing, whereas FDF uses implicit typing to represent data and functions uniformly.H-FDF also explicitly distinguishes data flow from control flow rather than coupling loops implicitly.

3 FDF Overview

FDF is a higher-order visual data-flow DSL for composing ML functions and data across fast DT learning and instantiation. Its boxes, implicit typing, and reusable function outputs support pipeline construction and validation.

  • Core concepts: FDF targets Φ2 and Φ3 by treating ML functions as first-class citizens that can be passed, produced, and reused.Its primitive boxes correspond to model reuse, reduced-order modeling, and surrogate modeling.
  • Core concepts: An FDF pipeline is an acyclic directed graph with separate function ports and data ports.Function ports carry learned functions, while data ports carry data batches.
  • Primitive boxes: Processor, Coder, and Trainer boxes respectively process data and functions, perform unsupervised learning, and perform supervised learning.DataIO and FuncIO boxes represent data and function dependencies, with shapes and colors providing redundant visual encoding.
  • Minimal example: A minimal real-time-DTP pipeline loads X and y, reduces X with a Coder, encodes X through Processor, and trains predict with Trainer.The pipeline exports reduce and predict for later use with new data and physical-twin interaction.
  • Implicit typing: FDF implicitly infers data and function types and propagates them through the graph to support structural consistency.This avoids manual type declarations that may be infeasible while learned functions are determined dynamically.
  • Design rationale: The pipeline can be described using high-level concepts aligned with user intent rather than low-level algorithm details.This follows from treating functions uniformly and separating learned-function production from their application.
  • Comparison with KNIME: FDF represents PCA computation and application with generic Coder and Processor components rather than tightly coupled operation pairs.Replacing a KNIME regression model requires five actions, whereas FDF model swapping is handled as a Trainer parameter with one click.

4 The DesCartes Builder Tool

DesCartes Builder is an open-source visual environment that uses FDF to specify, execute, validate, and reuse ML-based digital-twin pipelines. Its material-strain example combines model-order reduction, surrogate learning, validation, and expert-review outputs.

  • DesCartes Builder systematizes ML-based digital-twin pipeline engineering through an open-source visual modeling environment for domain experts.Users can specify, execute, and validate synthesis workflows without deep programming expertise by leveraging FDF.
  • Tool architecture: The tool separates its graphical front end from a Python/Kedro execution back end through an abstract execution engine.The front end handles pipeline interaction, while the back end orchestrates execution and implements FDF boxes using Scikit-learn and PyTorch.
  • User interface: The interface supports drag-and-drop FDF modeling, box selection and parameter editing, pipeline execution, execution traces, and chart-based validation metrics.Function and data connections are visually differentiated, and users can inspect results through the output and chart-viewer components.
  • Material-strain application: The material-strain pipeline reduces displacement and strain dimensions with PCA, learns a surrogate model, and evaluates it with scoring and sensitivity-analysis processors.The pipeline uses standardized PCA retaining 99.9% of original variance before surrogate learning and validation.
  • Validation and exploitation: The resulting reduced-order surrogate achieves an R2 score of 0.8, with predictions aligning closely to ground truth but higher residuals in a localized sparse-data region.The authors suggest improving the design-of-experiments strategy to address the localized data scarcity and allow expert review through exported results.
  • Validation and exploitation: The more complex reduced-order pipeline outperforms a simplified alternative, whose R2 score is only 0.5 and whose unexplained variance is reported as 150% higher.The paper attributes the additional pipeline complexity to its accuracy.

5 Empirical Study on FDF and DesCartes Builder

The controlled user study assessed DesCartes Builder and FDF for DT modeling, finding good usability and generally positive feature adequacy while identifying learning, guidance, diagnostic, and scope-related limitations.

  • Study design: The controlled study evaluated whether FDF and DesCartes Builder support DT development for target users, including manufacturing domain experts.Participants implemented a material-strain real-time DTP, and the evaluation measured usability and feature adequacy.
  • Perceived usability: The SUS mean was 68.9 ± 15.2, with a median of 72.5 and 16% of participants below the unacceptable threshold of 50.The mean’s 95% CI was [62.6, 75.2], with scores ranging from 37.5 to 95.
  • Perceived usability: Among technical participants, the mean SUS score increased to 72.8 ± 12.1 after excluding the low-technical subgroup.The exploratory comparison had n = 15 and Cohen’s d = 0.67, while very low technical proficiency presented adoption barriers.
  • Feature adequacy: Core feature ratings were positive: 89% found the canvas intuitive, 81% found parameter setting straightforward, and 85% were satisfied with results visualization.The overall mean feature rating was 3.9 ± 0.6 on a 5-point scale.
  • Positive feedback: Participants valued plug-and-play visual blocks, no-code interaction, the graph interface, and an integrated platform for quickly changing models and running experiments.The FDF modeling paradigm was well received, although participants reported an initial learning curve and faster adaptation among users familiar with MDE tools or Simulink.
  • Areas for improvement: The main improvement areas were built-in guidance, documentation, diagnostics, runtime inspection, and warning-message usability.Only 57% considered warning messages friendly, while documentation and diagnostics were recurring qualitative concerns.
  • Validity and scope: The study’s two-hour session may have limited participants to simpler scenarios, excluding complex multi-domain pipelines and long-term maintainability.The workshop was supervised by the authors, creating a potential internal-validity threat, although anonymity and critical feedback were emphasized.

6 The H-FDF extension

H-FDF extends FDF with hierarchical modules that support reusable subpipelines and internally iterative processes while preserving global acyclicity. Its E-FDF foundation formalizes module interfaces, optional feedback, and explicit stopping conditions.

  • Motivation: FDF’s strictly acyclic architecture cannot directly represent iterative processes, and it lacks native subcomponents, making reuse and composition tedious.These limitations motivated the H-FDF extension.
  • Hierarchical extension: H-FDF introduces modules that call lower-level subpipelines, allowing local iteration while keeping the externally visible workflow globally acyclic.Higher-level modules call lower-level ones, preserving unidirectional composition.
  • E-FDF foundations: E-FDF adds a Module box with data and function inputs and outputs, extending the three standard FDF box classes.Module boxes encapsulate subpipelines invoked from higher-level modules.
  • E-FDF foundations: E-FDF supports function initialization in Trainer and Coder boxes, enabling iterative function learning rather than only standard random initialization.A supplied function initializes the model in these boxes.
  • Iteration semantics: Optional stop and feedback elements make an E-FDF pipeline inductive, repeating execution until a Boolean stopping condition is satisfied.Without these elements, the pipeline performs a single iteration.
  • Iteration semantics: Module execution initializes internal inputs, runs the acyclic graph in partial order, evaluates the stop condition, and either returns outputs or updates feedback inputs for another iteration.Feedback edges are excluded from the execution graph’s partial order, preserving acyclic analysis.

3 Y 4 Predict

The H-FDF examples show how hierarchical subpipelines compose model-learning components and how iterative feedback implements dual training. Explicit stop ports also make convergence or iteration limits part of the pipeline graph.

  • Hierarchical pipeline structure: An H-FDF pipeline is a sequence of E-FDF subpipelines connected by a hierarchy mapping that matches module ports with subpipeline boundary ports and types.The topmost pipeline has no iteration boundary elements, while the lowest-level subpipeline has no Module box and is non-inductive.
  • Hierarchical pipeline structure: In the example, top-level pipeline P1 uses Module M1 to call P2, which computes an Encode function, applies it to a dataset, and returns both.Data and function ports are matched according to their ordered interfaces.
  • Execution and typing: H-FDF executes each subpipeline as its own directed graph, while a Module executes its associated lower-level pipeline and returns its data and function outputs.Implicit typing propagates through module outputs and warns when feedback connects incompatible types.
  • Dual training: The dual-training case study uses an inductive subpipeline to cotrain a linear-regression operator and a Koopman operator, feeding updated models into the next iteration.The dataset inputs remain fixed from the higher-level caller, while learned functions receive feedback.
  • Dual training: Model combination iteratively refines two predictors by alternating residual updates until convergence, then adds both model outputs for the final prediction.Unlike one-shot residual learning, the two updates recur across iterations.
  • Dual training: A non-inductive subpipeline can encapsulate training and prediction, returning both the learned model and its prediction to the calling modules.This reusable subpipeline is called by both M2 and M3 in the dual-training design.
  • Stopping conditions: H-FDF represents stopping explicitly through a stop port, so inserting a new process requires adapting the stop-condition computation to include accuracy and loop count.This separates process execution from the conditions governing iteration.

7 Conclusion

The paper evaluates FDF and DesCartes Builder, reports positive usability findings, and proposes H-FDF for iterative pipelines. It also identifies future implementation and user-experience work.

  • 7 Conclusion: H-FDF extends FDF with a Module box that supports iteration until a stop condition while keeping the global pipeline acyclic.The extension is intended to make iterative pipelines easier to reason about.
  • 7 Conclusion: The user evaluation found FDF and DesCartes Builder intuitive and easy to use, with a median SUS score of 72.5.The mean SUS score was 68.9 ± 15.2.
  • 7 Conclusion: Qualitative feedback characterized DesCartes Builder as an integrated platform for rapid experimentation with a plug-and-play graph interface.Participants also requested tooltips and integrated documentation.
  • 7 Conclusion: A dual-training case study demonstrates H-FDF’s ability to combine models co-inductively, while comparison with KNIME suggests explicit loop conditions reduce iteration-strategy evolution effort.The paper formalizes H-FDF syntax and semantics before applying it to this case.
  • 7 Conclusion: Future work includes fully implementing H-FDF in DesCartes Builder and adding contextual documentation and real-time pipeline inspection.

A.1 Informed Consent

The study materials describe a two-hour evaluation of DesCartes Builder involving a tutorial, hands-on task, and experience survey. Participants were asked to provide informed consent covering the project’s ethical considerations.

  • A.1 Informed Consent: Participants signed an informed consent document covering the project’s basic ethical considerations.
  • A.1 Informed Consent: The study evaluated whether DesCartes Builder’s abstraction and tooling could facilitate machine-learning-based digital-twin development.
  • A.1 Informed Consent: The experiment required about two hours and included a short tutorial, a hands-on tool task, and an experience survey.
  • A.1 Informed Consent: Collected study data were to be anonymized and kept confidential.

A.2.1 Background

The questionnaire collected participants’ roles, expertise areas, familiarity with relevant concepts and tools, and agreement ratings based on the SUS questionnaire.

  • A.2.1 Background: Participants reported roles including researcher, student, engineer, industry practitioner, or other.
  • A.2.1 Background: Participants identified primary expertise areas such as artificial intelligence, systems engineering, software engineering, and digital twins.
  • A.2.1 Background: Familiarity was rated on a five-point scale ranging from 1, None, to 5, Expert.The assessed topics included machine learning, digital twins, reduced-order modeling, system modeling, and engineering simulation tools.
  • A.2.1 Background: Agreement with questionnaire statements was rated from 1, Strongly Disagree, to 5, Strongly Agree.The questionnaire was based on the SUS questionnaire.

A.2.3 Experience with DesCartes Builder

The questionnaire assessed how user-friendly participants found key DesCartes Builder tasks and invited suggestions for improving the experience.

  • A.2.3 Experience with DesCartes Builder: Participants rated interface user-friendliness for box parametrization, pipeline modeling, result visualization, and warning messages.Ratings ranged from 1, Very Demanding, to 5, Very Intuitive.
  • A.2.3 Experience with DesCartes Builder: The experience questionnaire covered four concrete interaction areas: box parametrization, pipeline modeling, result visualization, and warning messages.
  • A.2.3 Experience with DesCartes Builder: Participants were asked which features or improvements would most enhance their experience with DesCartes Builder.
  • A.2.3 Experience with DesCartes Builder: Additional comments were solicited about usability, ML integration, and digital-twin functionality.

Appendix B Sensitivity Analysis

The appendix presents a feature-based sensitivity analysis for finding brittle regions in ML surrogates without exhaustive physics-model evaluation. A scalable heuristic uses feature attribution, local refinement, and grid-based comparisons to identify input pairs with large output discrepancies.

  • Motivation: Low MSE on holdout data may miss regions where small input perturbations cause large output changes.The problem is linked to neural-network brittleness and inadequate design-of-experiments coverage of critical input regions.
  • Procedure: It then refines around influential samples, computes finer attributions, and evaluates a grid varying two features while fixing the others.The grid is used to evaluate the model at all queried points and inspect local output discrepancies among adjacent elements.
  • Sensitivity objective: The analysis seeks neighboring inputs whose surrogate outputs have maximized discrepancy.The output difference may be measured with the ℓ∞-norm, as commonly used in adversarial analysis.
  • Search strategy: A scalable heuristic uses feature-attribution scores to search high-sensitivity regions because exhaustive search is infeasible.The method produces candidate input pairs that expose brittle regions in the surrogate.
  • Procedure: The procedure first samples the full input space and computes attribution scores for a selected output target.Uniform random samples are drawn over the bounded input domain before influential samples are identified.
  • Implementation: The implementation is integrated into the DesCartes Builder backend and uses the Captum library.This connects the sensitivity-analysis procedure to the paper’s model-driven DT engineering environment.
Loading 2608.18480v1…