PaperlayerView exact revision on arXiv

Source-linked AI summary

BERTMap: A BERT-based Ontology Alignment System

Yuan He, Jiaoyan Chen, Denvar Antonyrajah, Ian Horrocks

arXiv:2112.02682v4cs.AIcs.CLcs.LG

TL;DR

Ontology matching needs to connect semantically related entities despite differences in labels and context, while prior approaches have relied on surface forms, non-contextual embeddings, or ad-hoc features. BERTMap fine-tunes BERT on ontology-derived label pairs to predict mappings and refines them using ontology structure and logic. Across biomedical tasks, it often outperforms LogMap and AML, though not on every task.

  • Problem

    Prior ontology-matching methods have relied on surface-form lexical matching, non-contextual embeddings, or ad-hoc feature engineering, and contextual BERT models had not been sufficiently investigated for the task.

    Source: Introduction · p. 1

  • Method

    BERTMap fine-tunes a BERT classifier on ontology-derived synonym and non-synonym label pairs, then extends and repairs predicted mappings using ontology structure and logic.

    Source: Introduction · p. 1, Text Semantics Corpora · p. 2, Mapping Extension · p. 4

  • Results

    BERTMap achieves higher F1 than all baselines on FMA-SNOMED and FMA-SNOMED+, but lower F1 than LogMap and AML on FMA-NCI.

    Source: Results · p. 5

  • Takeaways & Limitations

    BERTMap works using the ontologies being aligned and can be further improved with given mappings or complementary sources.

    Source: Conclusion and Future Work · p. 7

    Evaluation covers three biomedical alignment tasks; the authors identify evaluation with more large-scale industrial data as future work.

    Source: Introduction · p. 1, Conclusion and Future Work · p. 7

Abstract

from arXiv · show

Ontology alignment (a.k.a ontology matching (OM)) plays a critical role in knowledge integration. Owing to the success of machine learning in many domains, it has been applied in OM. However, the existing methods, which often adopt ad-hoc feature engineering or non-contextual word embeddings, have not yet outperformed rule-based systems especially in an unsupervised setting. In this paper, we propose a novel OM system named BERTMap which can support both unsupervised and semi-supervised settings. It first predicts mappings using a classifier based on fine-tuning the contextual embedding model BERT on text semantics corpora extracted from ontologies, and then refines the mappings through extension and repair by utilizing the ontology structure and logic. Our evaluation with three alignment tasks on biomedical ontologies demonstrates that BERTMap can often perform better than the leading OM systems LogMap and AML.

Introduction

Ontology alignment connects semantically related entities across ontologies, but surface-form matching and prior machine-learning approaches have limitations. BERTMap combines BERT-based prediction with ontology-structure and logic-based refinement, and often outperforms LogMap and AML on the evaluated biomedical tasks.

Introduction

  • Ontology alignment matches semantically related entities across ontologies, where a mapping usually expresses equivalence or subsumption.Different ontologies may use different names for the same class, such as “muscle layer” and “muscularis propria.”Source: Introduction · p. 1
  • Prior embedding-based methods use non-contextual representations, while feature-engineered methods rely on ad-hoc designs and many annotated samples; BERT had not been sufficiently investigated for ontology matching.Traditional systems also rely on lexical matching that compares surface forms and cannot capture word semantics.Source: Introduction · p. 1
  • BERTMap fine-tunes BERT to predict mappings, then uses ontology structure and logic to extend predictions and remove mappings that cause logical inconsistency.Candidate mappings are extracted using sub-word inverted indices before classification.Source: Introduction · p. 1
  • BERTMap is evaluated on FMA-SNOMED, FMA-NCI, and an extended FMA-SNOMED task, and can often outperform LogMap and AML.The extended task adds more complete labels from the original SNOMED ontology.Source: Introduction · p. 1

Preliminaries

The paper formulates alignment as scoring equivalence between named classes across two ontologies, then extending and repairing those scores into determined mappings. Its BERT preliminaries explain how contextual token representations are learned and adapted for downstream tasks.

Problem Formulation

  • For two ontologies, the task is to score equivalence between pairs of named classes, then extend and repair the scored mappings to produce determined mappings.Each score lies in [0, 1] and indicates the degree of equivalence.Source: Problem Formulation · p. 2

BERT: Pre-Training and Fine-Tuning

  • BERT uses bidirectional transformer encoders and self-attention to produce token representations informed by the whole input sequence, distinguishing different occurrences of the same word.Pre-training uses Masked Language Modelling and Next Sentence Prediction.Source: BERT: Pre-Training and Fine-Tuning · p. 2
  • For a downstream task, pre-trained BERT is paired with task-specific layers and fine-tuned, typically with a few epochs and a moderate number of training samples.The input may be one sentence or a sentence pair, depending on the task.Source: BERT: Pre-Training and Fine-Tuning · p. 2

BERTMap

BERTMap trains a BERT classifier on ontology-derived synonym and non-synonym label pairs, uses sub-word indices to select mapping candidates, and refines predictions through structural extension and logical repair.

Corpus Construction and BERT Fine-Tuning

  • The training corpora pair labels from the same or equivalent classes as synonyms and labels from distinct classes as non-synonyms, using intra-ontology, cross-ontology, and optional complementary sources.For hard negative examples, sibling classes are assumed disjoint; complementary data comes from same-domain auxiliary ontologies with labels shared with the input ontologies.Source: Text Semantics Corpora · p. 2
  • BERT is fine-tuned with a binary classifier on the synonym and non-synonym pairs; its output score indicates how likely the input label pair is synonymous.The corpus combination uses abbreviations such as io, co, and cp, while ids denotes identity synonyms.Source: Fine-tuning · pp. 2–3, Text Semantics Corpora · p. 2

Mapping Prediction

  • Sub-word inverted indices select target classes sharing tokens with a source class, and the top k candidates are scored, reducing search complexity to O(kn) from O(n^2).BERTMap uses BERT’s built-in WordPiece tokenizer; its scoring combines exact label matching and averaged BERT synonym scores.Source: Candidate Selection · p. 3, Mapping Score Computation · p. 3

Mapping Refinement

  • Mapping extension iteratively adds high-scoring superclass and subclass pairs related to existing mappings, retaining only new pairs scoring at least κ = 0.9.The process stops when no new mappings are found; results were reported as insensitive to κ.Source: Mapping Extension · p. 4
  • Mapping repair removes mappings that would create logical conflicts after ontology integration, and extension and repair can improve performance without excessive time cost.The adopted repair method computes an approximate repair that is a subset of a diagnosis and leaves only a small number of unsatisfiable classes.Source: Mapping Repair · p. 4

Evaluation

The evaluation tests BERTMap on FMA-SNOMED, FMA-SNOMED+, and FMA-NCI under unsupervised and semi-supervised settings. It outperforms all baselines on the two SNOMED tasks but trails LogMap and AML on FMA-NCI.

Experiment Settings

  • The evaluation covers FMA-SNOMED and FMA-NCI OAEI LargeBio tasks, plus FMA-SNOMED+ with labels added from a newer SNOMED version.Additional labels are used for fine-tuning alone in FMA-SNOMED, and for both fine-tuning and prediction in FMA-SNOMED+.Source: Datasets and Tasks · p. 4
  • Systems are evaluated using Precision, Recall, and Macro-F1, with held-out reference mappings split into validation and test sets for each supervision setting.The unsupervised split uses 10% validation and 90% test mappings; the semi-supervised split uses 20% training, 10% validation, and 70% test mappings.Source: Evaluation Metrics · p. 4
  • BERTMap varies supervision, identity synonyms, complementary corpora, and mapping extension and repair; validation selects mapping thresholds before repair, against internal and external baselines.The external baselines include LogMap and AML, and the settings use Bio-Clinical BERT fine-tuned on synonym and non-synonym pairs.Source: BERTMap Settings · p. 5, Baselines · p. 5

Results

  • Semi-supervised +co generally improves over unsupervised; +cp helps especially with deficient labels, +ids slightly helps or makes no difference, and extension and repair provide consistent but modest gains.The authors suggest gains from extension and repair may be modest because prediction already performs well.Source: Results · p. 5
  • BERTMap achieves higher F1 than all baselines on FMA-SNOMED and FMA-SNOMED+, but lower F1 than LogMap and AML on FMA-NCI.The results compare both unsupervised and semi-supervised settings.Source: Results · p. 5, Table 2 · p. 6, Table 3 · p. 6
  • Across BERTMap models, validation selects λ = 0.999; as λ rises, precision increases significantly, recall drops slightly, and F1 peaks at that threshold.Most settings select the same hyperparameters, including τ = src2tgt.Source: Results · pp. 5, 7, Figure 2 · p. 7
  • BERTMap retrieves reference mappings missed by LogMap and AML, including implicit connections such as “third cervical” to “C3” and “wall” to “membrane”.The examples illustrate contextual embeddings capturing connections beyond traditional lexical matching.Source: Results · p. 7, Table 5 · p. 7

Related Work

Ontology matching systems combine lexical, structural, and logical methods, but surface-form matching misses word semantics. Learning-based approaches introduce semantic representations, yet can depend on feature engineering or labeled mappings, while prior BERT approaches have reported limitations.

Related Work

  • Classic systems such as LogMap and AML combine lexical matching with structural extension and logic-based repair, but their lexical matching uses surface forms and ignores word semantics.BERTMap is described as using a similar architecture while incorporating textual semantics and context.Source: Related Work · p. 7
  • Supervised ontology-matching approaches use features, class embeddings, or context, but often require complex engineering and many high-quality labeled mappings.Distant supervision and sample transfer have been investigated, though sample quality varies and limits performance.Source: Related Work · p. 7
  • Unsupervised approaches such as ERSOM and DeepAlign refine embeddings to compute class similarity but do not consider word contexts..Source: Related Work · p. 7
  • A preliminary BERT investigation reported weaker results for token-embedding similarity than string matching, while its fine-tuned approach had lower mean reciprocal rank than Fast-Text and lacked gold-standard evaluation data.The fine-tuned approach had higher coverage, but Precision, Recall, and F1 were not computed.Source: Related Work · p. 7

Conclusion and Future Work

BERTMap combines contextual text semantics with ontology structure and logic for mapping prediction and refinement. The authors report that it works using only the ontologies to be aligned, can be improved with additional sources, and identify industrial-scale evaluation as future work.

Conclusion and Future Work

  • BERTMap combines a BERT-based predictor using contextual semantics and sub-word inverted indices with ontology-based mapping extension and repair.The authors state that extension further improves recall and repair further improves precision.Source: Conclusion and Future Work · p. 7
  • BERTMap works with only the ontologies being aligned and can be further improved using given mappings and/or complementary sources.Source: Conclusion and Future Work · p. 7
  • The authors plan to evaluate BERTMap on more large-scale industrial data and consider BERT-based ontology embeddings and additional ways to integrate prediction, extension, and repair.Source: Conclusion and Future Work · p. 7

A Full Ablation Results of Mapping Thresholds on the Validation Mapping Sets

The figures plot validation precision, recall, and Macro-F1 across mapping-score thresholds for BERTMap settings and alignment tasks. The authors note that validation results are generally worse than test results because smaller evaluation sets require more positive mappings to be ignored while the number of negative mappings stays the same.

A Full Ablation Results of Mapping Thresholds on the Validation Mapping Sets

  • Validation precision, recall, and Macro-F1 are plotted against threshold λ for BERTMap settings across FMA-NCI, FMA-SNOMED, and FMA-SNOMED+.The figures include combined and directional results; the FMA-SNOMED figure also includes settings with complementary corpora.Source: A Full Ablation Results of Mapping… · pp. 8–9
  • Validation results are generally worse than test results because smaller evaluation sets require more positive mappings to be ignored while the number of negative mappings stays the same.Source: A Full Ablation Results of Mapping… · p. 9
  • Figure 3 plots FMA-NCI validation metrics across thresholds below 1, marking the maximum F1 with a red vertical line.Threshold 1 is excluded because it represents the string-match result.Source: Figure 3 · p. 10

Related papers

Loading 2112.02682v4…