PaperlayerView exact revision on arXiv

Source-linked AI summary

Annotating Columns with Pre-trained Language Models

Yoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Çağatay Demiralp, Chen Chen, Wang-Chiew Tan

arXiv:2104.01785v2cs.DBcs.CL

TL;DR

Many tables lack semantic column types and relationships, despite the cost of manual annotation. Doduo uses a pre-trained language model and multi-task learning to predict both from table values and context. It achieves state-of-the-art results on two benchmarks and reports competitive performance with limited input or training data.

  • Problem

    Tables often lack semantic column types and relations, although this metadata is useful and manual annotation can be expensive.

    Source: §1 · p. 1

  • Method

    Doduo serializes an entire table for a pre-trained Transformer and uses multi-task learning to predict column types and relations from shared, contextual representations.

    Source: §1 · p. 2

  • Results

    Doduo establishes new state-of-the-art performance on two benchmarks and achieves competitive performance with only 8 tokens per column or about 50% of training data.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The released toolbox was effective in a real-world data science case study, including on out-of-domain data.

    Source: §1 · p. 2, §8 · p. 9

    Doduo represents cell values as strings split into tokens, and direct support for numeric data remains important future work.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Inferring meta information about tables, such as column headers or relationships between columns, is an active research topic in data management as we find many tables are missing some of this information. In this paper, we study the problem of annotating table columns (i.e., predicting column types and the relationships between columns) using only information from the table itself. We develop a multi-task learning framework (called Doduo) based on pre-trained language models, which takes the entire table as input and predicts column types/relations using a single model. Experimental results show that Doduo establishes new state-of-the-art performance on two benchmarks for the column type prediction and column relation prediction tasks with up to 4.0% and 11.9% improvements, respectively. We report that Doduo can already outperform the previous state-of-the-art performance with a minimal number of tokens, only 8 tokens per column. We release a toolbox (https://github.com/megagonlabs/doduo) and confirm the effectiveness of Doduo on a real-world data science problem through a case study.

1 INTRODUCTION

Table columns often lack semantic types and relationships, although this metadata supports data-management tasks and can be expensive to annotate manually. Doduo jointly predicts both from table values, using table context and multi-task learning.

1 INTRODUCTION

  • Semantic column types and relations are often missing, while manual annotation can be expensive; such metadata supports data quality control, schema matching, and data discovery.Semantic types provide finer-grained information than standard database types, and relations describe connections between column meanings.Source: §1 · p. 1
  • Column types and relations are interdependent: knowing a person and location type can help predict a relation such as place_of_birth, but may not distinguish birth from death.Column values may also require both within-column and cross-column context to infer their semantic type.Source: §1 · p. 2, Figure 2 · p. 2
  • Doduo serializes all columns in a table for a Transformer and uses column representations for type prediction and pairs of representations for relation prediction.Its multi-task framework shares knowledge between the tasks; it can achieve competitive performance with only 8 tokens per column or less than half the training data.Source: §1 · p. 2
  • Doduo establishes new state-of-the-art performance on WikiTable and VizNet, with improvements up to 4.0% and 11.9%, respectively.The paper also reports a real-world data science case study and releases a toolbox.Source: §1 · p. 2

2 RELATED WORK

Prior work predicts semantic column types, relations between columns, or both, using neural models, external knowledge, and table context. Doduo is compared with these approaches as a unified cell-value-based model.

2 RELATED WORK

  • Column-type systems include Sherlock, which uses features from individual columns, and Sato, which adds table context and structured output; other methods use external knowledge bases.Doduo is compared with Sherlock, Sato, and TURL, with table context and token-level interactions among columns distinguishing it from the first two.Source: §2 · pp. 2–3
  • Column-relation research assigns semantic labels to column pairs, using Open IE, knowledge-base triples, language-model ranking, or graph representations.A relation such as place_of_birth connects person and location columns.Source: §2 · pp. 2–3
  • Pre-trained language models have been applied to table understanding, including annotation, parsing, text-to-SQL, and models combining table content with information from related tables.Unlike TURL, Doduo relies on cell values rather than additional metadata such as table headers for pretraining.Source: §2 · p. 3

3 BACKGROUND

The background defines column type and relation prediction, describes the use of pre-trained language models, and introduces multi-task learning as the framework for sharing parameters across tasks.

3 BACKGROUND

  • The background section introduces the two annotation tasks and gives an overview of language models and fine-tuning for column annotation.It sets up the formal task definitions and the multi-task framework used in the paper.Source: §3 · p. 3

3.1 Problem Formulation

  • Column type prediction assigns each table column a semantic type, while column relation prediction assigns a semantic relation to a pair of columns.The formulation treats cell values as strings split into tokens; direct support for numeric data is identified as important future work.Source: §3.1 · p. 3

3.2 Pre-trained Language Models

  • Pre-trained language models create context-dependent token representations and are fine-tuned with task-specific data and output layers.The paper uses BERT Base but states that Doduo is independent of the choice of pre-trained model.Source: §3.2 · p. 4

3.3 Multi-task learning

  • Multi-task learning optimizes an objective over multiple tasks, based on the assumption that they share knowledge and can benefit from a common model.The paper uses hard parameter sharing and a joint framework with one base model and separate output layers.Source: §3.3 · p. 4

4 MODEL

The model section contrasts independent single-column classification with Doduo’s table-wise serialization and contextualized column representations. It then describes separate task-specific training objectives and the challenges of choosing additional tasks.

4.1 Single-column Model

  • Independent sequence classification cannot capture table context, which is important for column annotation.The single-column model treats columns in the same table as independent sequences.Source: §4.1 · p. 5

4.2 Table Serialization

  • Doduo serializes an entire table and inserts one [CLS] token per column, enabling predictions for each column in the input sequence.The example serialization places [CLS] before each column’s values and [SEP] at the end.Source: §4.2 · p. 5

4.3 Contextualized Column Representations

  • A Transformer encodes the serialized table into contextualized token embeddings, and each column’s [CLS] embedding serves as its column representation.Those representations incorporate table context; the type-prediction layer uses a column representation, while the relation layer uses a pair of them.Source: §4.3 · pp. 5–6
  • For relation prediction, Doduo concatenates the contextualized representations of a column pair before applying a relation-prediction layer.The model computes the predicted relation from the concatenated pair representation.Source: §4.3 · p. 6

4.4 Learning from Multiple Tasks

  • Doduo switches tasks each epoch, using task-specific objectives and optimizers to handle imbalanced training data without balancing objective terms with a hyperparameter.The authors also state that adding tasks may not improve performance if tasks share too little knowledge or noisy labels propagate.Source: §4.4 · p. 6
  • Finding relevant additional tasks is challenging, and testing them is left for future work.The authors cite insufficient shared knowledge and propagation of noisy training labels as possible reasons added tasks may not improve performance.Source: §4.4 · p. 6

5 EVALUATION

Evaluation on WikiTable and VizNet shows Doduo outperforming prior methods across column type and relation prediction benchmarks. The experiments also examine the role of table context, metadata, and numeric-column performance.

5.1 Dataset

  • WikiTable provides multi-label column types and relations, while VizNet provides single-label column types; evaluation uses the benchmarks’ existing splits.Source: §5.1 · p. 6

5.2 Baselines

  • Comparisons use a table-value-pretrained TURL variant, Sherlock’s column features, and Sato’s added table-context features and CRF layer.Source: §5.2 · p. 6

5.4 Main Results

  • On the WikiTable dataset, Doduo significantly outperforms state-of-the-art TURL in micro F1 for column type prediction and column relation prediction, with improvements of 4.0% and 0.9%, respectively.Source: §5.4 · p. 7

5.3 Experimental Settings

  • WikiTable uses binary cross-entropy and micro F1; VizNet uses cross-entropy and both micro and macro F1.Source: §5.3 · p. 7

5.4 Main Results

  • Doduo’s full self-attention outperforms TURL’s visibility-matrix design; the authors infer direct cross-column and cross-row token intersections help column annotation.Source: §5.4 · p. 7
  • Appending column names slightly improves Doduo and makes it competitive with TURL using metadata; the authors report Doduo performs more robustly without metadata.Source: §5.4 · p. 7
  • 11.9% micro F1 and 6.7% macro F1 improvements over Sherlock and Sato on VizNet establish new state-of-the-art performance.Doduo was trained only for column type prediction on VizNet because relation labels were unavailable.Source: Table 4 · p. 7, §5.4 · p. 7
  • 86.9% average F1 across VizNet’s 15 most numeric types is slightly above 84.6% overall macro F1, despite low ranking (33.21%) and capacity (62.55%) scores.The authors suggest Transformers may recognize digit patterns in numeric values.Source: Table 5 · p. 7, §5.4 · p. 7

6 ANALYSIS

Ablations show that Doduo’s multi-task learning and multi-column architecture both contribute to performance. Its data-efficiency analyses find strong results with fewer labels and input tokens, while noting a boundary for very wide tables.

6.1 Ablation Analysis

  • On WikiTable, removing multi-task learning degrades both tasks, while the single-column variant performs significantly worse, supporting benefits from shared learning and table context.Dosolo omits multi-task learning; DosoloSCol also uses only the target column or column pair.Source: §6.1 · pp. 7–8, Table 6 · p. 8
  • On VizNet, Doduo significantly outperforms the single-column variant, while DosoloSCol still outperforms Sato, which uses LDA features for table context.The ablation results are reported for the VizNet Full dataset.Source: Table 7 · p. 8, §6.1 · p. 8
  • Random row shuffling causes only subtle degradation, and random column shuffling produces no substantial difference on WikiTable.The authors conjecture that fine-tuning aligns the model’s position embeddings with table structure.Source: §6.1 · p. 8

6.2 Data Efficiency

  • With half the training data, Doduo exceeds 0.9 F1 on both tasks, beats TURL on column-type prediction, and is comparable on column-relation prediction.It consistently outperforms Dosolo across the tested training-data sizes.Source: §6.2 · p. 8, Figure 4 · p. 8
  • With 8 tokens per column, Doduo outperforms TURL on WikiTable column-type prediction; relation prediction requires 32 tokens to beat TURL.On VizNet, Doduo with 8 tokens per column scores 92.5 F1 versus Sato’s 88.4 F1.Source: Table 8 · p. 8, §6.2 · p. 8
  • Sixteen-column support may not suffice for tables beyond WebTables; the authors suggest clustering relevant columns and applying Doduo to each cluster.Doduo can still use partial table context in this setting.Source: §6.2 · pp. 8–9

7 CASE STUDY: CLUSTERING COLUMNS

The case study uses Doduo embeddings to cluster semantically similar columns in an enterprise HR database. Doduo has the best reported clustering Precision and F1, and the results indicate transfer from Wikipedia tables to this enterprise setting.

Case study setting

  • The in-production HR database contains semantically similar columns with different names, while column descriptions are missing for many columns or unreliable.The scenario focuses on job-search and company-review analysis.Source: §7 · p. 9

Case study method

  • The workflow creates Doduo contextualized embeddings for columns and applies a clustering algorithm; column names and descriptions are used only to construct ground truth.The ground-truth clusters are manually refined, and all embedding methods use the same k-means algorithm.Source: §7 · p. 9

Case study evaluation

  • The evaluation compares Doduo embeddings and predicted types with fastText name/value embeddings and COMA and DistributionBased schema matching.Cluster quality is measured with Homogeneity (Precision), Completeness (Recall), and V-Measure (F1).Source: §7 · p. 9

Case study results

  • Doduo’s contextualized column embeddings achieve the best clustering Precision and F1, and outperform both schema-matching approaches in the case study.The model uses neither column names nor descriptions as input and was trained on Wikipedia tables, supporting transfer to the enterprise database setting.Source: Table 9 · p. 9, §7 · p. 9

8 CONCLUSION

The conclusion presents Doduo as a unified column-annotation framework using pre-trained Transformers and multi-task learning. It reports benchmark improvements, data efficiency, and a successful real-world case study.

Conclusion

  • Doduo combines pre-trained Transformer language models and multi-task learning in a unified column-annotation framework.Source: §8 · p. 9
  • Across two benchmarks, Doduo achieves state-of-the-art performance and competitive results using only 8 tokens per column or about 50% of training data.Source: §8 · p. 9
  • A case study supports Doduo’s effectiveness on a real-world data-science problem, and the authors present a toolbox for researchers and data scientists.Source: §8 · p. 9

A.1 Detailed Results

Detailed results describe where Doduo’s advantages appear across classes and relation types. It performs robustly on several low-frequency VizNet types and tends to improve results for less distinguishable labels.

A.1 Detailed Results

  • On VizNet, Doduo improves over Sato on both dataset variants, especially single-column tables, and performs robustly on religion, education, and organisation.Sato has zero or very poor F1 for these classes; the first-fold training counts are 24, 22, and 14, respectively.Source: §A.1 · p. 12
  • On WikiTable, Doduo tends to outperform Dosolo for less distinguishable types and relations, including artist versus writer and place-of-birth versus place-lived.Source: §A.1 · p. 12

A.2 Input Token Length

Doduo outperforms Sato on VizNet with only 8 maximum tokens per column. Its attention analysis also indicates that column representations can depend asymmetrically on other column types.

A.2 Input Token Length

  • With 8 maximum tokens per column, Doduo outperforms the state-of-the-art method Sato on VizNet.The comparison is reported for the VizNet dataset; the passage does not provide a numerical margin.Source: §A.2 · p. 12

A.4 Inter-column Dependency

  • In the last Transformer block, the analysis aggregates attention between [CLS] tokens across VizNet multi-column tables to form a column-type dependency matrix.The analysis uses attention weights from all heads and averages the matrices across tables.Source: §A.4 · p. 12
  • The attention analysis finds that column-type dependencies can be asymmetric: age relies on origin, while the reverse direction has low attention weight.The matrix is normalized to account for column-type co-occurrence, and higher or lower scores indicate more or less influence.Source: §A.4 · p. 12
  • The heatmap indicates that gender representations draw information from country, whereas gender appears less important for origin representations.Higher values encode stronger dependency from the column type on the y-axis to the type on the x-axis.Source: §A.4 · p. 12

A.5 Language Model Probing

The probing analysis tests whether pretrained BERT contains knowledge relevant to column annotation. Template-based perplexity results vary across column types and relations, and the authors report useful factual knowledge in the pretrained model while noting cases where Doduo does not benefit from it.

A.5 Language Model Probing

  • The authors hypothesize that pretraining may boost Doduo by providing relevant knowledge, and test BERT's knowledge using template-based probing.The motivation draws on prior work reporting factual knowledge in pretrained language models.Source: §A.5 · p. 12
  • The analysis scores candidate column types by filling a template and checking whether the true label has the lowest perplexity under vanilla BERT.For the type analysis, it selects single-token labels: 80 WikiTable types and 75 VizNet types.Source: §A.5 · pp. 13–14
  • A randomly initialized model with Doduo's architecture achieved approximately zero F1 when trained only on target-task data.The authors consider the model too large to train on that data without pretraining.Source: §A.5 · p. 14
  • For WikiTable column types, probing varies: government.election ranks 6.74 on average with lower-than-average PPL, while royalty.kingdom is nearly always ranked last.The authors suggest the latter result could reflect the lower frequency of “kingdom” in the pretraining corpus.Source: §A.5 · p. 14
  • On WikiTable, probing performance is higher for factual relations such as place_of_birth and position, with less separation between the top-five and bottom relations than for types.The authors attribute the smaller separation partly to relation templates having three blanks, which more often produce unnatural sentences.Source: §A.5 · p. 14
  • On VizNet, Doduo outperforms Sato for all top-five probed column types, while birthPlace and nationality are among the few types where it underperforms.The authors say these results suggest Doduo may not benefit from relevant pretrained factual knowledge for those column types.Source: Figure 5 · p. 13, §A.5 · p. 14
  • Vanilla BERT probing confirms that the pretrained model stores factual knowledge useful for column annotation, which the authors link to Doduo's gains over Sato.The BERT model used for probing was not fine-tuned on WikiTable or VizNet.Source: §A.5 · p. 14

B LIMITATIONS AND FUTURE WORK

The authors identify assumptions about table inputs and open questions for extending Doduo beyond the evaluated setting. They also flag uncertainty about which related tasks benefit from multi-task learning.

B LIMITATIONS AND FUTURE WORK

  • Doduo uses table values without metadata, although metadata can complement table semantics; determining when it is essential remains open.The authors note that spreadsheets and data frames often lack meaningful headers or captions.Source: B LIMITATIONS AND FUTURE WORK · p. 15
  • Doduo assumes a self-contained table and was not tested on normalized relational tables that split information across tables.Such settings may require inter-table relations and signals outside the target table; joint modeling is identified as future work.Source: B LIMITATIONS AND FUTURE WORK · p. 15
  • The framework assumes table values are correct and clean, while real-world inputs may contain missing or misplaced values.The authors identify mitigating the influence of incorrect values as future work.Source: B LIMITATIONS AND FUTURE WORK · p. 15
  • Although multi-task learning helps column annotation, it remains unclear which related tasks are useful and how task similarity relates to transfer benefits.The authors describe their study of this issue as preliminary.Source: B LIMITATIONS AND FUTURE WORK · p. 15

Related papers

Loading 2104.01785v2…