Source-linked AI summary
Do General NLP Embeddings Capture Ontological Reasoning?
Hamed Babaei Giglou, Jennifer D'Souza, Sören Auer
TL;DR
General-purpose embeddings perform well on linguistic similarity, but their sensitivity to symbolic ontological structure remains unclear. AVA evaluates this question with logic-sensitive ontology perturbations and contrastive triplets. The results show moderate pretrained discrimination, substantial fine-tuning gains with weak downstream transfer, and evidence of perturbation-specific pattern learning.
Problem
Standard similarity benchmarks rarely test whether lexically similar statements differ in ontology-level relational semantics, leaving this capability unclear.
Method
AVA combines structured ontology perturbations with contrastive objectives to evaluate ontology-aware discrimination and cross-ontology transfer.
Results
Fine-tuning substantially improves triplet discrimination, but gains transfer only weakly to downstream ontology engineering tasks.
Takeaways & Limitations
High contrastive discrimination accuracy does not necessarily indicate transferable ontology understanding; models may learn perturbation-specific structural patterns.
Takeaways & Limitations
AVA evaluates ontology discrimination rather than inference or entailment, so its performance is not evidence of logical reasoning ability.
Abstract
from arXiv · showhide
General-purpose NLP embedding models perform well on linguistic tasks, but their ability to capture symbolic ontological structure remains unclear. We introduce AVA, a systematic framework for evaluating whether embeddings distinguish logic-sensitive relational semantics in ontologies and knowledge graphs. AVA comprises 171,007 contrastive triplets derived from 163 heterogeneous ontologies using hierarchy inversion, relation substitution, and disjointness injection. Each triplet contains an ontology statement, a semantically equivalent paraphrase, and a logic-sensitive hard negative with contradictory relational meaning. We evaluate more than 25 state-of-the-art embedding models and find substantial limitations: the best model achieves only 0.739 triplet accuracy, while hard negative accuracy falls to 0.135. Fine-tuning improves discrimination by a large margin but transfers poorly to downstream Semantic Web tasks, including taxonomy discovery and ontology alignment. Further analysis suggests that improvements stem partly from perturbation-specific pattern recognition rather than robust ontological understanding. These findings reveal a persistent gap between linguistic representation learning and ontology-level discrimination, challenging the assumption that strong NLP benchmark performance translates to Semantic Web competence.
1 Introduction
Strong NLP similarity performance does not establish sensitivity to ontology-level relational semantics. AVA evaluates this gap and finds moderate discrimination, weak transfer, and evidence that high benchmark accuracy may reflect perturbation-specific patterns.
- Motivation: Standard STS benchmarks rarely test whether lexically similar statements differ in subclass, domain/range, or disjointness semantics.This leaves unclear whether strong STS performance reflects sensitivity to symbolic ontological knowledge.
- Findings: General-purpose embeddings show moderate ontology-sensitive similarity performance and substantial degradation on hard negatives.The experiments identify limitations despite strong performance on conventional linguistic tasks.
- Findings: Contrastive fine-tuning dramatically improves triplet discrimination, with hyperbolic objectives achieving near-perfect ranking accuracy.These gains are reported alongside weak transfer to downstream ontology engineering tasks.
- AVA: AVA introduces a large-scale benchmark using structured logic-sensitive perturbations to evaluate ontology-aware semantic similarity.The contribution includes evaluating modern embedding models and contrastive objectives across ontology-sensitive distinctions.
- Implication: High contrastive discrimination accuracy does not necessarily imply transferable ontology understanding for Semantic Web applications.The paper presents this optimization–generalization gap as a limitation of current embedding-based approaches.
2 AVA
AVA combines ontology-derived perturbations with contrastive objectives to test whether embeddings distinguish relational semantics beyond surface similarity. Its pipeline extracts bounded ontology subgraphs, generates hard-negative triplets, filters them, and trains models with Euclidean or hyperbolic losses.
- Framework: AVA combines logic-sensitive ontology perturbations with contrastive objectives for ontology-aware discrimination and cross-ontology transfer.The framework evaluates whether embeddings capture relational semantics beyond surface lexical similarity.
- Ontology Graph Extraction: The dataset uses 163 heterogeneous ontologies spanning biomedical, geospatial, social, engineering, and schema domains.Graphs represent OWL classes and properties connected by structural OWL/RDFS relations.
- Ontology Graph Extraction: Two-hop subgraphs retain 3–20 nodes and yield 50,548 unique structures after deduplication.The two-hop radius captures sibling and grandparent relationships that single-hop neighborhoods often miss.
- Contrastive Triplet Generation: An LLM generates contrastive triplets containing asserted anchors, semantically equivalent paraphrases, and logic-sensitive negatives from structured subgraph prompts.The generation constraints cover hierarchy, domain/range, equivalence, and disjointness statements.
- Post-Processing: Post-processing isolates anchor–negative pairs above 90 token-set similarity as hard negatives and discards indistinguishable positive–negative pairs.The pipeline also deduplicates samples at the anchor level.
- Contrastive Objectives: Cosine triplet loss separates anchor-positive from anchor-negative similarities using a margin of 0.3, while hyperbolic loss replaces Euclidean distance with Poincaré-ball distance.The objectives are designed to evaluate ontology-aware discrimination and represent hierarchical structure.
3 Results
Results show that general-purpose embeddings struggle with ontology-sensitive distinctions, while contrastive fine-tuning sharply improves triplet discrimination but transfers weakly to taxonomy discovery and ontology alignment.
- Pre-trained embeddings: 0.739 triplet accuracy is the best pre-trained result, achieved by Qwen3-Embedding-0.6B, while Llama-Embed-Nemotron-8B reaches only 0.217.Llama-Embed-Nemotron-8B also records 0.135 hard negative accuracy.
- Pre-trained embeddings: Hard negative accuracy drops substantially across models, indicating difficulty distinguishing ontology-consistent statements from highly similar contradictory statements.The evaluation ranks the positive above the negative using cosine similarity.
- Contrastive fine-tuning: 0.989 triplet accuracy is achieved by MPNET-base with hyperbolic triplet loss, up from 0.636 before fine-tuning.Its hard negative accuracy rises from 0.427 to 0.980.
- Downstream transfer: Fine-tuning often degrades taxonomy discovery performance, with standard triplet loss consistently reducing recall across most ontology benchmarks.The comparison covers MPNET-base and MiniLM-L6 models before and after fine-tuning.
- Downstream transfer: Hyperbolic loss provides only modest ontology-alignment improvements on several benchmarks, while other datasets remain largely unchanged.DPO substantially reduces alignment accuracy across all evaluation datasets.
- Interpretation: The gap between near-perfect triplet ranking and small downstream gains suggests that discrimination accuracy and ontology generalization are distinct evaluation objectives.Models may learn perturbation-specific decision boundaries without acquiring transferable hierarchical and logical representations.
4 Discussion
The discussion identifies benchmark and task boundaries that constrain how AVA results should be interpreted. It also highlights intrinsic difficulty in taxonomy discovery and the possibility of generator-specific artifacts.
- Benchmark limitations: Single-generator construction may introduce stylistic or distributional artifacts shared with the strongest encoder family.The authors propose multiple generators and broader cross-generator validation as future safeguards.
- Downstream-task limitations: Taxonomy discovery is intrinsically difficult when candidate classes exhibit substantial semantic overlap.This overlap may partly explain the persistently low absolute recall across models, which ontology-aware fine-tuning only partially mitigates.
- Scope of AVA: AVA evaluates ontology discrimination, not logical reasoning or entailment.Testing inference would require assessing whether models derive unstated consequences such as A⊑C from A⊑B and B⊑C.
5 Conclusion
The conclusion separates strong similarity and retrieval performance from reliable ontology-level discrimination. It leaves open whether embedding optimization alone can capture formal logical constraints.
- Strong similarity and retrieval performance does not necessarily imply reliable discrimination between ontology-consistent and ontology-violating statements.
- Embedding-based models may capture statistical patterns while failing to reflect formal logical constraints in ontology engineering.
- Whether embedding optimization alone is sufficient, or explicit logical mechanisms are needed, remains unclear.
GenAI Disclosure
The manuscript reports that generative AI tools were used for language editing, while the authors developed and verified the scientific content.
- Chat-GPT and Gemini were used solely for grammar, spelling, and readability checks.
- The authors reviewed and adapted suggested changes for accuracy and appropriateness.
- The authors state that the scientific content, research design, analysis, and conclusions were developed and verified without AI involvement.