PaperlayerView exact revision on arXiv

Source-linked AI summary

Implementation Guidelines for Data Quality Metrics

Philipp Jung, Katinka Becker, Felix Biessmann, Valerie Restat, Martin Seyferth, Daniel Schwabe, Lisa Ehrlinger

arXiv:2610.10919v1cs.DBcs.LG

TL;DR

ISO/IEC 25024 defines a syntactic-accuracy DQM, but implementation remains unclear, preventing practitioners from directly computing standardized DQMs and mapping tool-based measurements back to the standards. The paper classifies data-level ISO metrics by automability and provides implementation guidelines, realized in dqmeasure, which learns metric parameters from clean reference data when possible. On real-world and synthetic datasets, dqmeasure metrics detect injected errors, decline alongside downstream ML performance, and scale linearly with row count.

  • Problem

    ISO/IEC 25024 defines a syntactic-accuracy DQM, but implementation remains unclear, preventing practitioners from directly computing standardized DQMs and mapping tool-based measurements back to the standards.

    Source: §1 · p. 2

  • Method

    The paper classifies data-level ISO metrics by automability and provides implementation guidelines, realized in dqmeasure, which learns metric parameters from clean reference data when possible.

    Source: §1 · p. 2

  • Results

    On real-world and synthetic datasets, dqmeasure metrics detect injected errors, decline alongside downstream ML performance, and scale linearly with row count.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The guidelines and dqmeasure provide an implementation of ISO metrics for continuous DQ monitoring.

    Source: §1 · p. 2

    The work does not yet cover DQ dimensions and metrics requiring more than one table or information beyond the data itself.

    Source: §7 · p. 12

Abstract

from arXiv · show

Despite decades of data quality (DQ) research, a gap remains between DQ dimensions, such as accuracy or completeness, which the literature defines in textual form, and DQ tools, which typically implement low-level checks that are not aligned with these dimensions. ISO/IEC 25024 and ISO/IEC 5259 attempt to bridge this gap by defining DQ metrics for each dimension. However, these DQ metrics are hardly used, because the standards leave open how to implement them: for example, the metric for syntactic accuracy counts syntactically accurate values, but does not state how to decide that a value is syntactically accurate. This simply moves the problem to another level without solving it. As a result, DQ assessment currently cannot build on the standards. In this paper, we make the ISO DQ metrics executable. We classify all data-level metrics of both standards into (i) generalizable metrics that need no input beyond the data, (ii) parameterized metrics whose parameters can be learned from clean reference data or set by an expert, and (iii) non-generalizable metrics that need qualitative judgment and cannot be automated. For the metrics that can be automated, i.e., categories (i) and (ii), we propose implementation guidelines that resolve what the standards leave open. We realize the guidelines in dqmeasure, an open-source library of 20 metrics that learns these parameters from reference data instead of relying on manually defined rules. Our experiments on real-world and synthetic datasets show that the metric scores decrease monotonically with an increasing number of injected errors, decline together with downstream ML performance, and scale linearly with the number of rows, which enables automated DQ monitoring based on the standards.

1 Introduction

The paper addresses the gap between textual data-quality dimensions and tool-specific checks by making ISO/IEC 25024 and ISO/IEC 5259 metrics executable. It proposes implementation guidelines and dqmeasure, then evaluates the library on error detection, downstream ML performance, and scaling.

1 Introduction

  • DQ tools implement low-level checks that are not always aligned with textual DQ dimensions, while the ISO standards define metrics but leave implementation unclear.For syntactic accuracy, ISO/IEC 25024 specifies a metric but not how to decide whether a value is syntactically accurate.Source: §1 · pp. 1–2
  • The paper classifies standard metrics by automability and proposes guidelines for turning textual definitions into executable metrics.The classification distinguishes metrics requiring no external input, those requiring parameters or domain knowledge, and those that can be automated.Source: §1 · p. 2
  • Across real-world and synthetic datasets, dqmeasure metrics detect injected errors, decline alongside downstream ML performance, and scale linearly with row count.The evaluation targets automated DQ monitoring based on the standards.Source: §1 · p. 2
  • dqmeasure implements 20 metrics, learning parameters from clean reference data rather than relying on manually defined rules.Experts can still set parameters; the library also includes an LLM-based semantic-accuracy metric and can be used in continuous-integration pipelines and monitoring.Source: §1 · p. 2

2 Related Work

The related work reviews definitions of DQ dimensions and metrics, commonly used validation tools, and prior efforts to connect standards with implementation. It distinguishes this paper's focus on implementing existing ISO metrics from work on defining metrics, organizing measurement, or manually creating business rules.

2.1 Defining Data Quality Dimensions

  • DQ dimensions characterize data quality, while metrics quantify a dimension with a numerical score; this paper focuses on data-level metrics.The literature proposes many dimensions and classifications, with definitions and interpretations differing across sources.Source: §2.1 · p. 2

2.2 Data Quality Metrics Implementations

  • Common validation tools expose low-level checks and leave users to supply rules and parameters, so checks may not map to DQ dimensions or standards.Prior work reports that ad hoc metrics are difficult to compare across tools or map back to a dimension.Source: §2.2 · pp. 2–3

2.3 Bridging Standards and Implementations

  • Prior work describes metric definitions and measurement processes, but not how to implement the metrics.ISO/IEC 25021 specifies quality measure elements, while other work proposes measurement models and organizational methodologies.Source: §2.3 · p. 3
  • Business-rule methods for applying ISO standards still require manual, time-consuming rule creation; this paper instead classifies existing metrics, learns parameters from reference data when possible, and proposes implementation guidelines.The guidelines are tool-independent, and dqmeasure serves as their reference implementation.Source: §2.3 · p. 3, §2.2 · p. 3

3 From Definitions to Executable Metrics

The paper models executable DQMs, classifies ISO metrics by the inputs needed to compute them, and gives five guidelines for turning textual definitions into executable metrics. Its framework distinguishes metrics that need no parameters, metrics that can use learned or expert-set parameters, and metrics requiring qualitative judgment.

3.1 A Model of Executable DQMs

  • Most ISO DQMs are ratios of elements satisfying a requirement to elements for which it applies, made executable by specifying the subject, units, reference, and condition.The subject is a column or table; units are the counted elements, and the reference holds parameters needed to evaluate the condition.Source: §3.1 · p. 3
  • Some DQMs use units beyond table cells or need row context, while distribution comparison falls outside the common ratio model and is handled separately.Scores are oriented so higher values indicate better quality, and an empty set of units produces an undefined score.Source: §3.1 · pp. 3–4

3.2 A Generalizability-Based Classification Scheme for DQMs

  • The classification groups DQMs as generalizable without parameters, parameterized using reference data or expert input, or non-generalizable because they require qualitative judgment.Generalizable metrics can be implemented directly when required information is present in the data; non-generalizable metrics such as user accessibility cannot be automated.Source: §3.2 · p. 4
  • The implementation covers 20 category G or K metrics across accuracy, completeness, consistency, currentness, and timeliness, with defaults chosen to support out-of-the-box use.The implementation assumes required information is available in the data; experts may set parameters instead of learning them from reference data.Source: §3.2 · p. 4

3.3 Implementation Guidelines

  • The guidelines specify a metric's subject, units, condition, and reference; classify it; then parameterize category K metrics using learnable references, expert overrides, and changeable defaults.A DQM should be executable without manual configuration.Source: §3.3 · pp. 4, 6
  • The final guidelines harmonize scores and missing-value handling, return per-unit results where applicable, and validate metrics by testing their response to relevant and unrelated errors.Validation also documents meaningful data types and columns, and is especially important when rules or references are generated automatically.Source: §3.3 · p. 6

4 A Reference Implementation for DQMs

dqmeasure implements selected ISO DQMs using a common API and the paper's implementation guidelines. It supports metrics with and without parameters, learns parameters from clean reference data when possible, and harmonizes scores for reporting.

4 A Reference Implementation for DQMs

  • dqmeasure implements 20 selected DQMs, follows the scikit-learn API, and operates on Narwhals-compatible dataframes.Its documentation specifies the subject, unit, condition, and reference, along with other mandatory specifications such as scale type.Source: §4 · p. 6
  • Seven of the 20 DQMs are category G and need no parameters, while 13 are category K and require parameterization.Category G metrics can be used directly.Source: §4 · p. 6
  • Category K parameters can be set by users or learned from clean reference data, ranging from accepted values to mined semantic rules and LLM-assisted value judgments.Defaults for method and threshold choices allow use without manual configuration.Source: §4 · p. 6
  • Scores are oriented from 0 to 1 with higher values indicating better quality; undefined scores are NaN, and per-cell or per-row condition results are returned when applicable.Risk metrics report 1 − X; Value completeness also returns per-row scores corresponding to Record completeness.Source: §4 · p. 6
  • Distribution of data values is handled outside the common executable-DQM model: fit learns a reference distribution, and score returns one minus its distance from the observed distribution.This metric has no per-unit results.Source: §4 · p. 6
  • The evaluation injects errors into one fold, learns DQM parameters on clean folds, and scores the corrupted fold while training and evaluating an ML model on the same folds.The figure presents these steps as the experimental procedure.Source: Figure 3 · p. 7

5 Evaluation and Results

Across error-injection, downstream-ML, and runtime experiments, dqmeasure’s metrics generally tracked increasing data damage and scaled to large tables. Their responses depended on the metric’s assumptions and data, and expert judgment remained necessary to select suitable metrics.

5.1 Data Quality Measurement

  • Response breadth varied: distribution-derived metrics reacted to several error types, while completeness metrics generally reacted only to missing values.Value occurrence completeness and Data value distribution compare learned training-column distributions with test-column distributions; swap affected both moderately (−0.142 each).Source: §5.1 · p. 8
  • Data record consistency showed no response because the injected scenarios created no duplicate records; it could not be evaluated when every row contained a missing value.Source: §5.1 · p. 8
  • On HOSP, DQ scores generally fell or stayed constant as error rates rose, but Risk of data inconsistency rose because errors created unique values.Its uniqueness assumption fits identifiers, not HOSP’s categorical columns.Source: §5.1 · p. 8, Figure 4 · p. 9
  • Eight of fifteen DQMs learn parameters from clean reference data, seven need none, and every tested error scenario is detected by at least four DQMs.Experts still need to decide which metrics suit a dataset; Risk of data inconsistency, for example, says little about DQ on HOSP.Source: §5.1 · p. 8

5.2 Data Quality Impact on ML Tasks

  • At 50% corruption, Syntactic data accuracy (−0.025), Data format consistency (−0.007), and Data value distribution (−0.048) showed smaller score changes.These metrics assess one column at a time, and averaging across 36 affected columns shrinks their changes because most receive only missing values.Source: §5.2 · p. 9
  • ROC-AUC fell from 0.879±0.003 on clean data to 0.776±0.036 at 50% corruption, while reacting DQMs also declined with increasing error rates.The comparison is reported for the SynTabFall fall-risk prediction task.Source: §5.2 · pp. 9–10, Table 5 · p. 10, Figure 5 · p. 10

5.3 Runtime Evaluation

  • Semantic data accuracy was excluded from the large-table runtime comparison; at 30.5 ± 4.6 ms per request, its extrapolated full-table runtime was (1.25 ± 0.19) × 10^6 s.The full-table estimate is an extrapolation from measurements on 300, 500, 700, and 900 rows.Source: §5.3 · p. 11
  • All DQMs except Semantic data accuracy scale linearly and score 41 million rows in under two minutes; Update frequency grows slightly faster than linearly.Update frequency takes 17.2 ± 0.9 s on the full table.Source: §5.3 · p. 11

6 Discussion and Open Challenges

The discussion identifies where implementation requires interpretation, explains how clean reference data enables parameter learning, and emphasizes that executable metrics may still be unsuitable for particular datasets. It also outlines limitations and open challenges in automation, aggregation, and runtime scaling.

Lessons from applying the guidelines

  • The standards leave implementation choices, inconsistent uses of “data item,” and differing score orientations; the guidelines resolve these issues as prerequisites for comparing implementations.The discussion links these resolutions to IG1, IG3, and IG4.Source: §6 · p. 11
  • A DQM can be executable yet unsuitable for a dataset: Risk of data inconsistency rises with errors on HOSP, while several metrics are uninformative or inapplicable on SynTabFall.On SynTabFall, Record completeness is 0 at every error rate, Data record consistency cannot be evaluated, and Semantic consistency finds no functional dependency in the reference data.Source: §6 · p. 11

Reference data as an enabler

  • Clean-reference parameter learning reduces reliance on manual rules, but learned metrics inherit reference-data blind spots and some still require expert knowledge.When reference data is unavailable or potentially corrupted, the authors recommend manually defined expert knowledge.Source: §6 · p. 11

Runtime scaling

  • DQMs’ fit and score scale linearly with rows and columns, while fixed per-call overhead dominates runtime below around 105 rows.The column-scaling panel covers only DQMs that read more than one column.Source: Figure 6 · p. 12

Limitations

  • Injected-error tests may not reflect errors in practice, and the LLM-based Semantic data accuracy metric is limited to small tables or samples by runtime.A wrong value in reference data can escape detection, while valid values absent from the reference can be treated as invalid.Source: §6 · p. 12

Challenge 1: Automation with LLMs

  • LLMs and heuristics could further automate DQM categorization and parameterization, shifting effort toward validating metrics, including those now requiring qualitative judgment.The authors present this as a possible direction, not an evaluated result.Source: §6 · p. 12

Challenge 2: Aggregation of DQM scores

  • Aggregating scores could simplify large reports because DQMs share a scale and direction, but aggressive aggregation may obscure meaning and tradeoffs.The authors suggest lower-dimensional projection as one possible way to reduce complexity while preserving relevant tradeoffs.Source: §6 · p. 12

7 Conclusion

The paper presents executable DQMs and implementation guidelines for ISO/IEC 25024 and ISO/IEC 5259, realized in dqmeasure. It reports error detection, scores declining alongside data quality and ML performance, and linear scaling with rows, while identifying extensions as future work.

Conclusion

  • The authors classify ISO metrics by whether they need only data, parameters, or expert knowledge, and present five guidelines and dqmeasure, a 20-metric library that learns parameters from clean reference data.They report that dqmeasure detects injected errors, scores decline with diminishing DQ and ML performance, and metrics scale linearly with rows.Source: §7 · p. 12

Future work

  • Future work will extend dqmeasure to uncovered dimensions and metrics, examine LLM support for categorization and parameterization, and test the guidelines beyond ISO standards.Source: §7 · p. 12

8 Artifacts

The authors provide the source code, dqmeasure library, and instructions to reproduce the measurements, including access to all three datasets.

Artifacts

  • The public GitHub repository contains dqmeasure, reproduction instructions, and access to all three datasets.Source: §8 · p. 13
Loading 2610.10919v1…