PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Bounds as a Silent-Fault Detector for AI-Ready Scientific Data

Kathryn Knight, Patrick Widener, Heather Bort

arXiv:2609.30507v1cs.CE

TL;DR

Scientific data reuse can involve artifacts whose categories or measures were created under conditions a downstream user cannot inspect. The paper proposes recording those conditions as domain bounds and checking reuse against them; in injected faults, the bounds detector caught 17 of 18 domain-bound faults that provenance and descriptive records missed. A measured sepsis reuse also produced death-rate predictions 9.2 percentage points too high against a 7.4% rate, despite no error or warning.

  • Problem

    Data artifacts may be reused without their production conditions being recorded, even though those conditions affect how their categories and measures should be interpreted.

    Source: §I · pp. 1–2, §VI · p. 10

  • Method

    The paper specifies authored domain-bound records and evaluates them against reuse requests, alongside provenance and descriptive records.

    Source: B. A taxonomy of injectable faults · p. 4, A. Setup · p. 7

  • Results

    17 of 18 domain-bound faults were caught by the bounds detector and none by the provenance or descriptive detectors; no detector flagged any of the 144 controls.

    Source: B. Detection · p. 8

  • Takeaways & Limitations

    Checking declared reuse conditions can detect faults that lineage and descriptive records accept, including cases where model outputs are wrong without an error or warning.

    Source: C. A measured silent fault · p. 9, §VI · p. 10

    The study’s fault-class assignment partly constructs the detector separation, and a consumer declaring nothing passes every check.

    Source: E. Discussion · p. 9

Abstract

from arXiv · show

We address an incipient type of data-based faults, driven by increasing amounts of data reuse in AI-enhanced computational science workflows. A dataset is created under specific conditions; its categories, measures, and labels depend on that context, but that context is usually unavailable to downstream users and is completely abandoned when datasets are used to train models. We propose the encoding and association of this context, called domain bounds, with the data to which it applies. By specifying the conditions for valid reuse, a domain bound flags apparently valid data from being reused in an incompatible context, even when every value falls within its expected range. We present cases from computational science and biomedicine where missing domain bounds allow silent reuse errors, and show why description and provenance approaches do not detect them. Our detector uses standard data container technologies, catches faults drawn from a taxonomy of published cases, irrespective of dataset size. Against injected faults it caught 17 of 18 domain-bound reuses that lineage and descriptive records accepted.

I. INTRODUCTION

Scientific data can be reused under conditions different from those that gave its categories, measures, and labels meaning, while pipelines still run without error. The paper frames those undocumented conditions as domain bounds and proposes recording and checking them to detect reuse faults.

I. INTRODUCTION

  • When reuse conditions differ from those at data creation, files can parse and workflows can finish while producing scientifically unwarranted results.AI training can make the training data opaque, while the model carries its context into later inferences.Source: §I · p. 1
  • Different approximation methods can yield different, valid band gaps for the same material, although a band_gap_ev column does not record which method produced its values [4].Merging those columns can create a well-formed, correctly typed table that is misleading.Source: §I · p. 1
  • Sepsis cohorts from the same data differed by onset interpretation, and that choice mattered more to model results than the model type.The cohorts contained 867 to 2,178 admissions; Sepsis-3 does not specify when sepsis begins.Source: §I · p. 2
  • The paper presents a fault taxonomy grounded in published cases and specifies authored domain bounds that compile into an RO-Crate profile extension and SHACL for validation.The proposed checks let a consuming workflow evaluate reuse without a person in the loop.Source: §I · p. 2
  • The paper proposes testing injected reuse faults with provenance, descriptive metadata, and domain bounds, and measuring consequences with two published sepsis cohorts.The cohorts come from the same population and use two definitions of the same condition.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

Earlier fault-tolerance approaches use references available within a computation, while semantic-compatibility systems check workflow composition or rely on constraints kept in their originating environment. This paper focuses on already-produced data whose context must travel with it.

A. Silent errors in HPC

  • Prior HPC fault-tolerance methods compare results with checksums, predictions, replicas, or checkpoints, requiring a reference for detection,,,,.The paper extends silent-error framing to cases where values are correct but may disagree across contexts or timeframes because a concept has multiple defensible forms.Source: A. Silent errors in HPC · pp. 2–3

B. Semantic compatibility in scientific workflows

  • Semantic-compatibility work checks types, domains, representations, or extents, including at workflow composition time [18], [19], [20], [21],.A dataset removed from its originating environment loses system-maintained constraints.Source: B. Semantic compatibility in… · p. 3
  • For already-produced artifacts reused outside their originating system, composition-time checks are too late and catalogs are unavailable, so the reference must travel with the artifact.The consumer may never access the system that produced the data.Source: B. Semantic compatibility in… · p. 3

C. Silent semantic failures in systems and AI pipelines

Systems that infer checkers from prior specifications do not address reuse faults where multiple interpretations are defensible and no test can distinguish them. The paper argues that a useful domain-bound approach needs a declared reference that travels with the artifact.

C. Silent semantic failures in systems and AI pipelines

  • Prior systems infer runtime or pipeline checkers from specifications that predate failures, such as tests, API contracts, or stated intent [23], [24],, [26].Domain-bound faults differ because cohorts or band gaps can satisfy distinct, defensible interpretations, so the condition must be declared rather than inferred.Source: C. Silent semantic failures in systems… · p. 3

D. Limits of provenance, description, and shift detection

  • Provenance can document activities without stating when an artifact stops being valid, while descriptive metadata and dataset-shift methods do not identify which reuse condition was violated.Static labels may not cover unlisted uses, and metadata labels may not travel with the dataset.Source: D. Limits of provenance, description,… · pp. 3–4
  • The paper concludes that a domain-bound approach must provide a reference, keep it with the artifact, and declare it because there is nothing to infer.Source: D. Limits of provenance, description,… · p. 4

III. USING DOMAIN BOUNDS TO CHARACTERIZE DATA-REUSE FAULTS

Domain bounds record the conditions under which an artifact’s categories were stabilized; reuse under incompatible conditions can activate a dormant fault and let an error propagate. Shared labels can enable clean integration despite weak scientific warrant, while tracing the resulting error may stop at the dataset.

III. USING DOMAIN BOUNDS TO CHARACTERIZE DATA-REUSE FAULTS

  • A domain bound records the conditions under which an artifact’s categories were stabilized, decisions made by particular people at a particular time.When those conditions go unrecorded, the artifact carries a domain-bound fault.Source: §III · p. 4
  • Incompatible reuse activates a dormant domain-bound fault, producing an error that can propagate while the pipeline continues normally and reports nothing.The error may manifest later as an erroneous result, such as a value or prediction.Source: §III · p. 4

A. Propagation and attribution

  • Shared labels can make datasets combine cleanly even when their records follow different rules or values use different methods, weakening the scientific warrant for integration.The paper calls this false interoperability; retrieval-augmented generation can also combine semantically similar records without checking whether their production conditions are compatible.Source: A. Propagation and attribution · p. 4
  • When poor transfer has multiple possible causes, diagnosis may trace the error to a dataset but not to the undocumented decisions and theories that shaped its categories,.Pooled data may reflect local coding practices, workflow conventions, measurement differences, or historical concept drift.Source: A. Propagation and attribution · p. 4

B. A taxonomy of injectable faults

The taxonomy organizes published reuse faults by the type of record positioned to detect them, distinguishing domain-bound cases from faults addressed by lineage or descriptive metadata and residual cases. Its classes are grounded in documented examples, so the taxonomy is necessarily incomplete.

  • The taxonomy is assembled from published cases where groups used the same data under different conditions and reported different results.Each class names a documented reuse fault that can be checked against its source.Source: B. A taxonomy of injectable faults · pp. 4–5
  • Its classes are grouped by the record positioned to detect them: domain-bound, lineage, descriptive-metadata, or residual.Residual classes are included because the present study's records detect none of them.Source: B. A taxonomy of injectable faults · p. 5
  • Band-gap pooling can combine computed values from different approximation methods with measured values, although these are distinct quantities and not interchangeable (D1 and D4).The files themselves do not reveal this mismatch.Source: B. A taxonomy of injectable faults · p. 5
  • Sepsis criteria can produce different cohort sizes: SEPSIS-1 identified 212 patients, SEPSIS-3 identified 272, and qSOFA identified 50 among 380 patients across 13 hospitals.Which definition is appropriate depends on the demands of the setting, including sensitivity, specificity, timeliness, and ease of implementation.Source: B. A taxonomy of injectable faults · p. 5

IV. MECHANISM

A domain bound is authored once as a record of the conditions under which dataset categories, measures, and labels were stabilized; packaging and validation artifacts are then derived from it. The author supplies interpretive context that cannot be inferred automatically, while revisions receive a new identifier.

A. Authoring

  • A DomainBoundSpec records the conditions under which a dataset’s categories, measures, and labels were stabilized for a particular purpose and time.The sepsis example includes fields such as population, setting, temporal scope, operational definition, measurement regime, and intended use.Source: A. Authoring · p. 6
  • Of the sepsis record’s 16 populated fields, four require author knowledge and 12 could in principle be derived from existing workflow information.Interpretive fields include intended use, non-interchangeability, portability limits, and assumptions.Source: A. Authoring · p. 6
  • Interpretive fields capture settled construction choices, so authors need not anticipate future uses; revisions preserve earlier versions by changing the record’s digest.Editing the bound changes its content digest, giving the revision a new identifier.Source: A. Authoring · p. 6

B. Exchange

The exchange design adds a DomainBoundedWR profile to the existing Workflow Run RO-Crate composition rather than replacing it. A crate consumer unfamiliar with the extension can still read the crate as an ordinary Workflow Run Crate.

  • DomainBoundedWR extends the existing Workflow Run RO-Crate composition, which includes process-run, workflow-run, and workflow descriptions.The extension names the profile chain on the root Dataset entity.Source: B. Exchange · p. 6
  • A data entity links to its DomainBoundSpec through db:domainBounds, and the bound is identified by its content digest.The digest is the identifier described for the authored record.Source: B. Exchange · p. 6
  • Because profile conformance is additive, consumers unfamiliar with DomainBoundedWR can treat the crate as an ordinary Workflow Run Crate and disregard the extra entity.Source: B. Exchange · p. 6

C. Validation

At reuse time, a workflow’s stated intent is checked against constraints compiled from a domain bound. The resulting SHACL validation can be automated, reports the failed condition, and does not require inspecting dataset contents.

  • A ReuseRequest states the proposed task, population, setting, operational definition, and transformations, and the bound is compiled into constraints targeting that request.Scope fields become single-member sh:in constraints, valid transforms become a multi-member sh:in constraint, and non-interchangeable categories generate negated constraints.Source: C. Validation · pp. 6–7
  • The 16 populated sepsis-bound fields compile to 17 SHACL Core constraints; four fields, including free-text portability limits and assumptions, generate none.Any conforming SHACL engine can check the resulting node shape against a ReuseRequest.Source: C. Validation · p. 7
  • A dimension left unasserted in a reuse request is treated as unconstrained, not as a violation.This permits legitimate partial requests when a consuming workflow does not know every production dimension.Source: C. Validation · p. 7
  • A violation identifies the condition the proposed reuse fails, and validation can run automatically at orchestration or job submission without examining the dataset.The cost of a check does not vary with dataset size.Source: C. Validation · p. 7

D. Generality and Limits

Domain bounds need not use SHACL or RO-Crate; the evaluation tests programmatic checking against faults that provenance and descriptive metadata may miss, along with the consequences of missed faults.

D. Generality and Limits

  • Domain bounds need not use SHACL or RO-Crate; any mechanism that serializes, checks, and carries the bound can serve.The paper presents SHACL and RO-Crate as convenient implementation choices.Source: D. Generality and Limits · p. 7
  • The mechanism trusts the authored bound: it does not verify that the bound accurately describes the dataset's conditions.An unrepresented hazard or an erroneous bound can therefore leave an incompatible reuse undetected.Source: D. Generality and Limits · p. 7

V. EVALUATION

  • The evaluation tests whether bounds can be checked programmatically, catch faults missed by other records, and whether undetected reuse has consequences.Source: §V · p. 7

A. Setup

  • Two authored records, for materials science and biomedicine, pair data-stabilization conditions with proposed reuse requests, without referring to the data itself.Source: A. Setup · p. 7
  • Three detectors receive the same artifact and reuse request, but each can consult only its designated record: lineage, descriptive metadata, or domain bounds.Source: A. Setup · p. 8
  • Fault classes were grouped by the record that would need the information to catch them; residual classes have no detector in this study.Source: A. Setup · p. 8
  • Across both records, 16 fault classes yield 32 injected faults and 144 benign controls, including requests that leave some conditions unstated.The controls test whether the bounds detector distinguishes faults from valid reuses rather than flagging every request.Source: A. Setup · p. 8

B. Detection

Domain-bound faults were caught by the bounds detector in 17 of 18 trials and by neither other detector; no detector flagged any of the 144 controls. The checker only catches violations of recorded conditions.

B. Detection

  • In the Sepsis-3-to-ICD-coded reuse, provenance and descriptive metadata pass while the bounds detector reports the violated label-definition condition.The example combines faults D1 and D4 in a single request.Source: Figure 3 · p. 8, B. Detection · p. 8
  • 17 of 18 domain-bound fault trials were caught by the bounds detector and none by the other detectors; no detector flagged any of 144 controls.Each fault group was caught by one detector and missed by the other two.Source: Table II · p. 8, B. Detection · p. 8
  • Complete lineage and descriptive metadata still accept domain-bound reuses because those records do not check the conditions under which the artifact was prepared.Source: B. Detection · p. 8
  • The missed fault involved an unrecorded onset-reconstruction condition, so no constraint existed for the checker to evaluate.The checker can only catch violations of recorded conditions.Source: B. Detection · pp. 8–9

C. A measured silent fault

Across two sepsis cohorts from the same population, applying Sepsis-3 changed patient composition and mortality rates; a model transferred to the wider cohort overpredicted deaths despite higher AUROC. Two residual fault classes remain: mis-authored bounds and dimensions left unconstrained.

C. A measured silent fault

  • The wider cohort’s 7.4% mortality contrasted with 18.9% in the Sepsis-3 cohort, which contained 19,051 admissions versus 110,204 in the primary cohort.The mean age also rose from 62.7 to 72.5 years after applying Sepsis-3.Source: C. A measured silent fault · p. 9
  • The model overpredicted deaths by 9.2 points against a 7.4% rate out of bounds, despite AUROC rising from 0.589 on held-out Sepsis-3 patients to 0.705 on the wider population.Neither run raised an error or warning.Source: C. A measured silent fault · p. 9
  • Because the model’s estimates average near the 18.9% training mortality, shifting them similarly leaves AUROC unchanged while miscalibrating predictions for the 7.4% population.A domain bound could flag the Sepsis-3 population mismatch before model fitting.Source: C. A measured silent fault · p. 9

D. Future work: Residual classes not detected

  • A mis-authored bound can pass validation because the consumer has no other account of the artifact; checking it requires an independent reference.Versioned vocabulary identifiers could be re-checked when the vocabulary changes.Source: D. Future work: Residual classes not… · p. 9
  • An unmodeled dimension cannot trigger a violation, and bounds surface recorded judgments without ensuring they are correct.A blank field may mean the dimension is irrelevant or was omitted; experts’ uncaptured assumptions make distinguishing these cases difficult.Source: D. Future work: Residual classes not… · p. 9

E. Discussion

The discussion places domain-bound checks at reuse time, before workflow execution, and describes their dataset-size-independent cost. It also bounds the evidence: the taxonomy is incomplete, the measured sepsis transfer has constraints, and the validator implements only part of SHACL Core.

E. Discussion

  • Checks run at reuse request time, before workflow execution, with cost independent of dataset size and completion times on a commodity laptop on the order of tens of microseconds.The paper identifies dataset or model choice and workflow orchestration or deployment as practical check points.Source: E. Discussion · p. 9
  • The injected-fault setup assigns fault classes to records, and treating unasserted dimensions as unconstrained means a consumer declaring nothing passes every check.A validator could instead report conditions it could not evaluate.Source: E. Discussion · p. 9
  • The taxonomy is not exhaustive, and the prevalence of its fault classes is unknown; detection rates are reported per injected group, not as real-world frequencies.The fault classes are grounded in published cases.Source: E. Discussion · p. 9
  • The measured transfer uses age, sex, and prior septic episode count, and the 9.2-point calibration error is smaller than a disjoint-population result would show.Patient identifiers were unavailable, so the cohorts could not measure transfer between definitions over the same patients.Source: E. Discussion · p. 9
  • The validator implements the SHACL Core subset used by compiled constraints, while its emitted shapes validate identically under pyshacl across twenty conforming and violating cases.Source: E. Discussion · p. 10

VI. CONCLUSION

  • Domain bounds detect a silent fault where lineage and descriptive records do not; the sepsis example ranked patients better out of bounds while predicting more than twice the deaths that occurred.The conclusion places detection before workflow execution.Source: §VI · p. 10
  • The paper proposes developing domain-bound profiles with domain communities, versioning them for automatic re-checks, and integrating checks into orchestration and schedulers.It also proposes asking models trained on bounded data about their training conditions.Source: §VI · p. 10
Loading 2609.30507v1…