Source-linked AI summary
Sherlock: A Deep Learning Approach to Semantic Data Type Detection
Madelon Hulsebos, Kevin Hu, Michiel Bakker, Emanuel Zgraggen, Arvind Satyanarayan, Tim Kraska, Çağatay Demiralp, César Hidalgo
TL;DR
Semantic type detection is important for data cleaning, schema matching, and data discovery, yet matching-based systems are limited by dirty data and narrow type coverage. Sherlock uses a multi-input neural network trained on large-scale VizNet columns represented by diverse features. It achieves a support-weighted F1-score of 0.89, exceeding machine learning, matching-based, and crowdsourced benchmarks.
Problem
Existing matching-based semantic type detectors are often not robust to dirty data, support limited types, and underperform for types without strict validation, despite semantic types’ importance to data science tasks.
Method
Sherlock formulates semantic type detection as multiclass classification using 1,588 column features and trains a multi-input neural network on VizNet columns matched to semantic types.
Results
0.89 support-weighted F1-score exceeded decision tree, random forest, matching-based, and crowdsourced-annotation benchmarks.
Takeaways & Limitations
Real-world columns provide training examples for detecting many semantic types at scale and include dirty values that may support robustness.
Takeaways & Limitations
Training-data quality was manually verified, leaving an opportunity to assess and label data quality at scale.
Abstract
from arXiv · showhide
Correctly detecting the semantic type of data columns is crucial for data science tasks such as automated data cleaning, schema matching, and data discovery. Existing data preparation and analysis systems rely on dictionary lookups and regular expression matching to detect semantic types. However, these matching-based approaches often are not robust to dirty data and only detect a limited number of types. We introduce Sherlock, a multi-input deep neural network for detecting semantic types. We train Sherlock on $686,765$ data columns retrieved from the VizNet corpus by matching $78$ semantic types from DBpedia to column headers. We characterize each matched column with $1,588$ features describing the statistical properties, character distributions, word embeddings, and paragraph vectors of column values. Sherlock achieves a support-weighted F$_1$ score of $0.89$, exceeding that of machine learning baselines, dictionary and regular expression benchmarks, and the consensus of crowdsourced annotations.
1 INTRODUCTION
Semantic type detection supports data cleaning, schema matching, and data discovery, but matching-based systems are limited by dirty data and narrow type coverage. Sherlock addresses this gap with a multi-input neural network trained on large-scale real-world columns and outperforms several benchmarks.
- Semantic types support automated data cleaning, schema matching, and data discovery by describing columns through real-world concepts.
- Matching-based approaches use regular expressions and dictionaries but are often fragile to malformed data, limited in supported types, and weak for types without strict validation.
- Sherlock is a deep learning approach that formulates semantic type detection as multiclass classification over column features.
- 0.89 support-weighted F1-score exceeded decision tree, random forest, matching-based, and crowdsourced-annotation benchmarks.
- The authors open source the data, code, and trained model to support benchmarking and integration into existing systems.
- 686,765 columns from VizNet were matched to 78 semantic types and described with 1,588 features.
2 RELATED WORK
Prior semantic type detection includes commercial and open-source matching systems, ontology-based methods, feature-based models, probabilistic approaches, and synthesized systems. Sherlock is positioned against these approaches using learned benchmarks representative of practical matching-based systems.
- Commercial and open source: Commercial and open-source systems commonly use regular expressions, dictionary lookups, or heuristics to detect a limited set of semantic types.
- Ontology-based: Ontology-based methods use web data, knowledge bases, column values, and headers to map columns to semantic types.
- Feature-based: Feature-based approaches describe numerical and textual columns with statistical, textual, and similarity features before training predictive models.
- Probabilistic: Probabilistic approaches predict value types, column types, or relationships using conditional random fields or probabilistic graphical models.
- Synthesized: Synthesized approaches combine existing detectors or generate type-detection logic from open-source repositories, but their implementations were unavailable for benchmarking.
3 DATA
Sherlock’s dataset is built by matching semantic-type labels to columns in VizNet, then filtering and representing columns with diverse fixed-length features. The resulting corpus contains 686,765 columns across 78 semantic types.
- 3.1 Source Corpus: The study begins with semantic types derived by matching DBpedia properties to WebTables columns.
- 3.1 Source Corpus: VizNet supplies diverse real-world datasets, and header matching produced 6,146,940 columns for 275 considered types.
- 3.1 Source Corpus: Manual verification found that most matched columns were plausibly described by their corresponding semantic type.
- 3.2 Feature Extraction: Each column is represented by global statistics, aggregated character distributions, pretrained word embeddings, and self-trained paragraph vectors.
- 3.2 Feature Extraction: Word embeddings characterize semantic content using vectors for words found in a pretrained GloVe dictionary, aggregated across column values.
- 3.2 Feature Extraction: Paragraph vectors represent each column as a paragraph and its values as words using the Distributed Bag of Words Paragraph Vector method.
- 3.3 Filtering and Preprocessing: 686,765 columns across 78 semantic types remained after excluding rare types and types with substantial missing GloVe coverage.
- 3.3 Filtering and Preprocessing: The final representation contains 1,588 features after adding an indicator for successful word-embedding extraction and imputing missing values.
4 METHODS
Sherlock uses a multi-input neural network to combine heterogeneous column features for semantic type detection, alongside simpler machine-learning, matching-based, and crowdsourced benchmarks. The evaluation uses fixed data splits, support-weighted F1, and prediction-time cost measurements.
- 4.1 Sherlock: A Multi-input Neural Network: Sherlock uses separate subnetworks for feature categories and combines their compressed outputs with statistical features in a primary network.The subnetworks produce fixed-dimensional outputs, which are concatenated with statistical features before final classification.
- 4.1 Sherlock: A Multi-input Neural Network: The network uses ReLU hidden layers, dropout, weight decay, and a final softmax whose highest-confidence class becomes the prediction.The selected hidden-layer sizes are 300 for character-level, 200 for word-embedding, and 400 for paragraph-vector subnetworks.
- 4.2 Benchmarks: The benchmarks include a decision tree, a 10-tree random forest, a value-or-header dictionary, learned regular expressions, and crowdsourced annotations.The decision tree represents simpler prior machine-learning models, while the dictionary and regular-expression systems represent matching-based approaches.
- 4.2 Benchmarks: Dictionary benchmarking maps 78,000 value-type pairs from the 1,000 most frequent values per type and classifies unseen columns by their most frequent matches.At test time, 1,000 randomly selected column values are compared with dictionary entries.
- Evaluation: Evaluation uses 60/20/20 training, validation, and testing splits, support-weighted F1, crowdsourcing bootstrap estimates, and prediction-time time and space measurements.Support weighting uses the number of test-set columns per class; computational cost is reported in seconds per sample and megabytes.
5 RESULTS
Sherlock is evaluated against machine-learning, matching-based, and crowdsourced benchmarks, then analyzed by type, feature contribution, feature importance, and confidence-based rejection. The results show accuracy gains from learned models, complementary feature sets, and a trade-off between predictive performance and computational efficiency.
- 5.1 Benchmark Results: Machine-learning models significantly outperform matching-based and crowdsourced consensus benchmarks in support-weighted F1 score.The benchmarks include decision tree, random forest, dictionary-based, and learned regular-expression models.
- 5.1 Benchmark Results: Sherlock significantly outperforms the decision tree baseline, while random forest remains competitive.Tree-based benchmarks may be suitable when feature and prediction interpretability matter.
- 5.1 Benchmark Results: Matching-based benchmarks are significantly smaller and faster than both machine-learning models, motivating a hybrid strategy for easy and ambiguous types.The proposed hybrid combines matching-based models for easy types with machine-learning models for more ambiguous types.
- 5.2 Performance for Individual Types: High-performing types contain finite valid-value sets or consistent character patterns, whereas numerical and overlapping values challenge sales and ranking detection.Grades and industry often have finite valid values; birth date and ISBN often follow consistent character patterns.
- 5.3 Feature Contributions: Each isolated feature set performs worse than the full model, while word embeddings, character distributions, and paragraph vectors outperform global statistics in isolation.The full model’s advantage supports combining feature categories, although the comparison may reflect differing feature counts.
- 5.4 Feature Importances: The most important decision-tree features are the number of values and maximum value length, with character patterns also helping distinguish types.The minimum number of hyphens and whether all values contain commas are highlighted among character-level features.
- 5.5 Rejection Curves: A 10% rejection threshold raises Sherlock’s F1 score to approximately 0.95, trading lower error for greater expert annotation capacity.The rejection curves assess performance when retaining only higher-confidence samples.
6 DISCUSSION
The discussion presents Sherlock as a high-performing semantic type detector while identifying opportunities to improve its training data, type coverage, features, and benchmarking.
- A multi-input neural network demonstrates high predictive performance compared with machine learning, matching-based, and crowdsourced benchmarks.The authors attribute potential robustness partly to dirty values in real-world test examples.
- Sherlock is distributed as a Python library and with open-source scripts for integration, alternative semantic types, and retraining on domain-specific data.The scripts support developers using enterprise or research data ecologies.
- Future work includes improving training-data quantity and quality, expanding semantic-type coverage, enriching features, and developing shared benchmarks.The paper also proposes features capturing column relationships, dataset context, and source context.
- Training-data diversity may improve Sherlock's robustness and generalizability beyond the four-source VizNet corpus.Suggested sources include Kaggle, R datasets, and the ClueWeb Excel-spreadsheet crawl.
- Prediction quality depends on correspondence between training data and unseen testing data, motivating scalable assessment of training-data quality.The authors verified data quality through manual inspection and propose combining crowdsourcing with active learning.
- The study's 78 semantic types remain a small subset of the entities and hierarchies available in DBpedia, WordNet, and Knowledge Graph.The authors note that existing knowledge bases offer substantially broader semantic-type coverage.
- A data-driven selection of semantic types could maximize training samples while better matching specific use cases and data ecologies.This approach uses column-header frequency when a relevant ontology is unavailable.
- The absence of a publicly available benchmark with code limited comparisons, prompting the authors to approximate state-of-the-art data-system benchmarks.They open-source the data and code to support future shared benchmarks.
7 CONCLUSION
The conclusion frames semantic type detection as important for data science and presents Sherlock as progress toward a machine-learning approach.
- Correctly detecting semantic types is critical to many important data science tasks.
- Sherlock provides a step forward toward semantic type detection using machine learning models and large-scale data repositories.
A.1 Supplemental Tables
The supplemental tables document the semantic types included in the study and the global statistical features used in its analysis.
- Table 7 lists the 78 semantic types included in the study.
- Table 8 describes 27 global statistical features and marks features included in Venetis et al. [33].
A.2 Mechanical Turk Demographics
The Mechanical Turk participant sample includes demographic, educational, work-frequency, and age information for 390 participants.
- 390 participants were surveyed, with 57.18% male and 0.43% female.
- The reported education levels ranged from some high school without a diploma to doctorate or professional degree.
- Participants reported working with data daily, weekly, monthly, annually, or never.
- The largest reported age group was 24-34, comprising 60.3% of participants.