Source-linked AI summary
A Survey of Data Quality Measurement and Monitoring Tools
Lisa Ehrlinger, Elisa Rusz, Wolfram Wöß
TL;DR
Data quality measurement supports trustworthy data-driven decisions, yet organizations often validate quality manually or ad hoc. This survey examines current tools’ profiling, metric-based measurement, and continuous monitoring capabilities, finding substantial gaps between research concepts and implementations.
Problem
Data quality measurement is needed for trustworthy data-driven decisions, but many companies still validate quality with Excel, Access, or manual ad hoc procedures without long-term strategies.
Method
The survey investigates data profiling, data quality metrics, and true continuous data quality monitoring, including metric definitions and aggregation levels.
Results
Most surveyed tools support data profiling to some extent, but no tool implements a wide range of research-proposed data quality metrics; general-purpose monitoring is usually a premium feature.
Takeaways & Limitations
Current data quality tools have substantial potential for functional enhancement, particularly in multi-column profiling, dependency discovery, generally applicable metrics, and accessible monitoring.
Takeaways & Limitations
The evaluation could not include tools dedicated to SAP because the researchers had no access to an SAP installation.
Abstract
from arXiv · showhide
High-quality data is key to interpretable and trustworthy data analytics and the basis for meaningful data-driven decisions. In practical scenarios, data quality is typically associated with data preprocessing, profiling, and cleansing for subsequent tasks like data integration or data analytics. However, from a scientific perspective, a lot of research has been published about the measurement (i.e., the detection) of data quality issues and different generally applicable data quality dimensions and metrics have been discussed. In this work, we close the gap between research into data quality measurement and practical implementations by investigating the functional scope of current data quality tools. With a systematic search, we identified 667 software tools dedicated to "data quality", from which we evaluated 13 tools with respect to three functionality areas: (1) data profiling, (2) data quality measurement in terms of metrics, and (3) continuous data quality monitoring. We selected the evaluated tools with regard to pre-defined exclusion criteria to ensure that they are domain-independent, provide the investigated functions, and are evaluable freely or as trial. This survey aims at a comprehensive overview on state-of-the-art data quality tools and reveals potential for their functional enhancement. Additionally, the results allow a critical discussion on concepts, which are widely accepted in research, but hardly implemented in any tool observed, for example, generally applicable data quality metrics.
1 Introduction
Data quality measurement supports trustworthy data-driven decisions, yet organizations often validate quality manually or with ad hoc tools. This survey examines current tools and finds substantial gaps in profiling, general-purpose metrics, and continuous monitoring.
- Motivation: 84 % of US CEOs are concerned about data quality, and poor quality is associated with average annual organizational losses of $15 million.The paper frames data quality as relevant to operational excellence and decisions made by both machines and people.
- Motivation: 66 % of companies use Excel or Access for validation, while 63 % determine data quality manually and ad hoc without a long-term strategy.The authors state that comprehensive data quality management requires a continuously executed strategy and appropriate automation tools.
- Main findings: Most surveyed tools supported data profiling, but multi-column profiling and dependency discovery remained areas for functional enhancement.The finding is explicitly framed relative to the state of research on data profiling.
- Main findings: No evaluated tool implemented a broad range of data quality metrics for the most important research dimensions without drawbacks such as attribute-level limits, gold-standard requirements, or implementation errors.The reported metric implementations may lack aggregation, depend on unavailable reference data, or contain errors.
- Main findings: In general-purpose tools, continuous data-quality monitoring was generally a premium feature, while Apache Griffin and MobyDQ automated rules but lacked predefined functions and profiling capabilities.The open-source exceptions were dedicated monitoring tools rather than comprehensive data-quality solutions.
2 Survey Methodology
The survey used a systematic protocol combining prior surveys, academic and open-source research, and random Google searches, then screened and deduplicated candidate tools. It evaluated the resulting tools against profiling, measurement, and continuous-monitoring requirements, ultimately assessing 13 tools.
- Survey protocol: The protocol was derived from Kitchenham’s systematic-review methodology, with some review steps omitted because the target was tool implementations rather than primary research papers.The omitted steps concerned quality assessment, data extraction, and synthesis of extracted research data.
- Research gap: The literature lacked a systematic detailed evaluation of state-of-the-art tools’ data-quality measurement and monitoring, which this survey set out to provide.Earlier studies were vendor-focused, nonsystematic, outdated, or centered on technical features and data cleansing rather than measurement functionality.
- Evaluation framework: The survey addressed which profiling capabilities, data-quality dimensions and metrics, and continuous monitoring functions current tools support.A requirements catalog mapped these research questions to concrete technical requirements.
- Search strategy: The search strategy combined tools from previous surveys, a systematic search for academic and open-source tools, and a random Google search using the same term combinations.The strategy aimed to establish a comprehensive list of candidate data-quality tools.
- Search strategy: The systematic search used “data quality,” “information quality,” and “tool,” while excluding “assessment” and “monitoring” to avoid omitting tools that used different terminology.“Information quality” was treated as a synonym for “data quality” to increase search coverage.
- Tool identification: 1,298 systematic-search results yielded 567 tool references; adding 110 tools from related surveys and 43 from Google produced 720 candidates, reduced to 667 after deduplication.The search counts include duplicate references to the same tool.
3 Theory on Data Quality and Evaluation Framework
The framework treats data quality as a context-dependent, multidimensional concept measured through metrics and supported by profiling and monitoring. It uses these concepts to define an evaluation of tools across data profiling, measurement, and continuous monitoring.
- Concepts: Data quality is commonly understood as “fitness for use,” making it subjective and context-dependent rather than universally fixed.
- Core activities: Data profiling analyzes datasets to collect metadata such as distinct values, missing values, data types, and value patterns before measurement or monitoring.
- Dimensions and metrics: Data quality measurement traditionally combines dimensions such as accuracy, completeness, and timeliness with metrics that quantify their fulfillment.
- Core activities: Continuous DQ monitoring means ongoing measurement of data quality, distinct from data monitoring that continuously checks rules.The survey includes monitoring because prior studies found no observed tools with monitoring functionality, while tool websites indicated such capabilities.
- Dimensions and metrics: Metrics map quality dimensions to numerical values and may be aggregated from value-level through column, record, table, and database levels.Weighted arithmetic means can aggregate results between levels.
- Dimensions and metrics: The survey observes metrics for accuracy, completeness, timeliness, and consistency, while noting that literature provides many more formulations than those discussed.The accuracy treatment includes Hinrichs’s QGen metric, which can be aggregated across multiple levels using expert-defined attribute weights.
4 Data Quality Tool Evaluation
The evaluation section examines selected data quality tools against the survey’s requirements catalog and discusses how each requirement is met.
- The section evaluates selected data quality tools using the requirements catalog introduced in the evaluation framework.
- The investigation covers each tool’s fulfillment of the defined evaluation requirements.
- The evaluation combines tool descriptions with discussion of their performance against the framework’s requirements.
4.1 Selected Data Quality Tools
The survey selected tools for detailed evaluation and documented their discovery, provenance, access conditions, functionality, usability, and support. The set included profiling-focused, comprehensive, and continuous-monitoring tools.
- Selection: 17 data quality tools were selected for detailed evaluation, but three SAP-based tools were excluded from the survey because no SAP installation was available.
- Selection: The selected tools were described by discovery source, vendor, licensing, evaluated version, delivery method, functionality focus, usability, and customer support.
- Tool profiles: Aggregate Profiler is a freely available open-source tool dedicated to data profiling and was found through both Springer and Google search results.The evaluated version was 6.2.4.
- Tool profiles: Apache Griffin was included because it continuously measures Big Data quality in batch-based settings, despite offering no data profiling and not being a comprehensive DQ solution.
- Tool profiles: Apache Griffin’s interface was considered intuitive after installation, supporting domain-specific accuracy metrics plus scheduling and monitoring, while completeness metrics were planned for later versions.Two experienced computer scientists required over a week to complete the installation.
4.1.3 Ataccama ONE
Ataccama’s data quality products were identified across multiple search sources and later consolidated into Ataccama ONE. The survey also records the related DataCleaner and DataHub product lineage.
- Ataccama: Ataccama Data Quality Center and Master Data Center had been evaluated in an earlier survey, while DQ Analyzer appeared in two other surveys.
- Ataccama: Gartner additionally mentioned Ataccama’s DQ Issue Tracker and DQ Dashboard in 2016.
- Ataccama: Since 2017, Ataccama consolidated its separate data quality solutions into Ataccama ONE.
- Related products: DataCleaner and DataHub originated with Human Inference, whose products later passed through Neopost, Quadient, and EDM Media Group.
4.1.5 Datamartist by nModal Solutions Inc.
Datamartist is a commercial tool focused on data profiling and transformation, evaluated through a 30-day trial. The supplied passages also describe Experian’s DQ products and report favorable overall assessments of one evaluated tool.
- Datamartist by nModal Solutions Inc.: Datamartist is dedicated to data profiling and data transformation.The evaluation used the free-to-download 30-day trial of version 1.7.9, including complete Pro edition features.
- Experian: Experian offers Cleanse and Pandora, with Aperture Data Studio introduced as Pandora’s planned replacement.Cleanse targets one-time data cleansing, while Pandora is described as the more comprehensive solution.
- Evaluation experience: The survey reported one tool’s overall assessments as best, alongside a good but technically oriented UI and timely, target-oriented customer support.
4.1.7 Informatica Data Quality
The supplied passages cover Informatica Data Quality, InfoZoom, and IBM InfoSphere Information Server for Data Quality. Informatica was evaluated through trials, whereas IBM ISDQ could not be evaluated because installation failed without vendor support.
- Informatica Data Quality: Informatica Data Quality was evaluated using two 30-day trial licences covering Developer, Analyst, and Administrator components.These components served desktop developers, web-based business users, and task scheduling, respectively.
- IBM InfoSphere Information Server for Data Quality: IBM InfoSphere Information Server for Data Quality could not be evaluated because installation stopped at an unresolved missing-file error.The researchers could not resolve the issue within the project timeframe and received no IBM support or specific installation instructions for the files.
- InfoZoom: InfoZoom is a commercial data-profiling tool using in-memory analytics, evaluated with its IZDQ extension under a six-month licence.
4.1.10 MobyDQ
MobyDQ is an open-source framework for automated data-quality checks, measurements, metrics, and anomaly alerts, but it does not provide data profiling. The supplied passages also describe MetricDoc for OpenRefine and monitoring in Oracle EDQ.
- MobyDQ: MobyDQ automates data-quality checks during processing, stores measurements and metric results, and triggers anomaly alerts.It is free and open source, and the evaluated version was installed quickly using detailed GitHub documentation.
- MobyDQ: MobyDQ does not provide data-profiling functionality because it focuses on creating, applying, and automating data-quality checks.
- Related tools: OpenRefine’s MetricDoc extension assesses data quality with customizable, reusable metrics and immediate visual feedback, while Oracle EDQ provides profiling and some monitoring.
4.1.13 Talend Open Studio for Data Quality
Talend Open Studio for Data Quality offers profiling, cleansing, business-rule management, and Big Data support, but its free edition lacks monitoring. The supplied passages also note SAS’s DataFlux-based product and limited machine-learning capabilities.
- Talend Open Studio for Data Quality: Talend Open Studio for Data Quality supports Big Data analysis and provides data profiling, cleansing, business-rule management, and a UI comparable to several commercial tools.The evaluation used version 6.5.1.
- Talend Open Studio for Data Quality: The free Talend version does not support data-quality monitoring, which is exclusive to the Enterprise edition.A free trial of Talend Data Management Platform was unavailable because such access was considered unlikely for the customer contact.
- SAS Data Quality: SAS Data Quality is based on the acquired DataFlux company’s product dfPower, and the evaluation did not find powerful machine-learning capabilities.
4.1.15 Data Quality Solutions dedicated to SAP
SAP-focused data quality tools are practically relevant but were not included in the evaluation because the authors lacked access to an SAP installation. The survey instead describes several SAP-oriented solutions and their functional roles.
- 4.1.15 Data Quality Solutions dedicated to SAP: The evaluation excluded SAP-specific data quality tools because the authors had no access to an SAP installation.The survey nevertheless describes relevant SAP-focused tools identified through its systematic search.
- 4.1.15 Data Quality Solutions dedicated to SAP: SAP Information Steward combines profiling, business-rule execution, and scorecard-based data quality monitoring.Its documented profiling functions include statistics, histograms, data types, and dependencies, with additional domain-specific functions.
- 4.1.15 Data Quality Solutions dedicated to SAP: ISO Professional Services provides an SAP-embedded data governance solution that reuses user-defined business rules from the SAP environment.Its Scarus Data Quality Server constitutes the core data quality component described in the passage.
- 4.1.15 Data Quality Solutions dedicated to SAP: BackOffice Associates offers dspMonitor for profiling, monitoring, and data quality checks, while dspCompose supports cleansing and data quality workflow management.The suite is dedicated to master data management, and dspCompose operates as an add-on for dspMonitor or SAP Information Steward.
4.2 Comparison of Data Profiling, DQ Measurement, and Monitoring Capabilities
Data profiling is broadly available for basic single-column analysis, but advanced multi-column and dependency discovery remain limited. DQ measurement tools implement few generally applicable metrics, with important restrictions in aggregation, required standards, and metric validity.
- Overall profiling coverage: Most tools support basic single-column profiling, whereas dependency discovery and multi-column profiling appear only in isolated cases.Cardinalities are broadly covered, but advanced profiling capabilities are rarely supported satisfactorily.
- Single-column profiling: Basic profiling results vary in completeness: percentage cardinalities, equi-depth or equi-height histograms, and out-of-the-box pattern and domain discovery are not consistently available.Most tools provide equi-width histograms, while only Talend OS supports Benford’s law and only three tools explicitly support quartiles.
- Single-column profiling: Value-length profiling usually reports minimum, maximum, and average lengths, but InfoZoom reports only the maximum and SAS and Talend OS restrict the feature to strings.Ataccama ONE additionally provides the median value length.
- Semantic and dependency analysis: Semantic-domain displays differ across tools, while functional-dependency discovery may require manual attribute selection rather than genuine automation.Talend OS displays matching rows, Ataccama ONE received partial ratings for semantic types and domains, and Talend OS only partially fulfills functional-dependency discovery.
- Dependency discovery: Dependency discovery has the lowest profiling coverage, with Experian Pandora and Informatica DQ providing the strongest support.Informatica DQ offers full unique column combination detection, while Experian Pandora provides primary- and foreign-key inference for inclusion dependencies.
- Advanced multi-column profiling: No investigated tool supports association rule mining, and advanced multi-column profiling is rarely satisfactory beyond widely supported duplicate detection.The evaluation specifically excluded related tools that may be installed alongside the investigated DQ tools.
5 Survey Discussion and Lessons Learned
The survey finds substantial variation and gaps in current data quality tools: profiling is common, but advanced profiling, broad metrics, aggregation, interpretability, and monitoring remain limited. It therefore questions whether research-defined data quality dimensions and metrics provide a practical basis for automated measurement.
- Market overview: 667 tools were identified, but the surveyed market remains difficult to interpret because tool capabilities and scope vary widely.About 50.82% of identified tools were domain-specific, while the evaluated selection included commercial, closed-source, free, and open-source tools.
- Continuous monitoring: DQ monitoring exists, but general-purpose tools typically reserve it for paid professional editions.Apache Griffin and MobyDQ automate rules, but lack pre-defined functions and data-profiling capabilities.
- DQ measurement: No surveyed tool implemented a broad range of research-proposed metrics across important data quality dimensions.Observed implementations may be limited to attribute-level calculations, require unavailable gold standards, or contain implementation errors.
- DQ measurement: Most tools mainly implement completeness and uniqueness, while aggregation and schema-level dimensions are largely absent.Informatica DQ aggregates column-level metrics to the table level, but no investigated tool provided higher-level prefabricated aggregation.
- Implications: The survey argues that generally applicable dimension-level metrics have limited practical relevance and that directly measurable aspects deserve greater focus.The paper highlights missing data and duplicate detection as core aspects that can be measured automatically without relying on abstract dimensions.
- Automation and interpretability: Current tools often require users to create and adjust measurements manually, while profiling outputs may omit thresholds or distance functions.The authors call for more automated, meaningful profiling across tables and data sources with clearly declared parameters and algorithms.
6 Conclusion and Outlook
The survey systematically maps data quality tools and addresses the limited prior evidence on generic metrics and monitoring capabilities. Its outlook emphasizes practice-oriented measurement, automated profiling, explicit parameters, and time-series analysis, while noting that the market continues to change.
- Survey scope: 667 tools were found, 17 were extracted for deeper investigation, and 13 were evaluated against 43 requirements covering profiling, measurement, and monitoring.The requirements were organized into three functional categories.
- Contribution: The survey fills gaps in prior work by examining generic data quality metrics and monitoring capabilities in the analyzed tools.Earlier surveys had not investigated either capability systematically.
- Outlook: Future work targets a practical methodology centered on directly measurable aspects rather than abstract dimensions lacking common understanding.The planned direction also includes automated out-of-the-box profiling with clearly declared parameters and time-series analysis of monitoring data.
- Market outlook: The survey reports that leading vendors have thousands of customers but identifies continued demand for broader data quality tool adoption.Reported customer counts range from 2,700 for SAS to 7,200 for Experian.
A Appendix: Data Profiling Test Cases
The appendix lists the Northwind-based test cases used to evaluate data profiling requirements, spanning cardinalities, distributions, types, dependencies, correlations, anomalies, and duplicate detection.
- Test basis: The evaluation used a modernized Northwind database and linked each test case to a data-profiling requirement.The test cases operationalized the profiling category for tool comparison.
- Single-column profiling: Cardinality tests measured rows, null values, distinct values, distinct-value ratios, histograms, minimum and maximum values, and constancy.These tests covered Product, Supplier.Fax, Customer.Country, and OrderItem.UnitPrice.
- Distributions and types: Additional profiling tests examined quartiles, first-digit distributions, basic and DBMS-specific types, value lengths, digit counts, decimal counts, patterns, semantic types, and domains.These cases focused on ProductName, UnitPrice, isDiscontinued, Customer.Country, and OrderItem.UnitPrice.
- Anomalies and duplicates: The appendix also tested clustering, very high values, and exact or relaxed duplicates.These cases targeted Order.TotalAmount, Product.UnitPrices, and Customer names.