Source-linked AI summary
Commonsense Knowledge Base Completion with Structural and Semantic Context
Chaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, Yejin Choi
TL;DR
Commonsense KB completion must handle free-form nodes that produce large, sparse graphs where conventional methods struggle. The paper combines graph-based structural context, semantic densification, and language-model representations, finding that language transfer generally works best while graph structure adds complementary benefits, especially on subgraphs.
Problem
Commonsense KGs are much larger and sparser than conventional KBs, challenging completion models that implicitly assume densely connected graphs.
Method
The paper combines GCN-based local structural context and semantic similarity links with BERT representations transferred from node text.
Results
Transfer learning is generally more effective than graph-structure learning, while graph structure provides complementary gains, especially when training on subgraphs.
Takeaways & Limitations
BERT representations substantially improve completion, and graph embeddings can boost performance when subgraph training makes structural modeling efficient.
Takeaways & Limitations
COMET scores for ATOMIC were computed on a smaller 2000-tuple evaluation set because of computational limitations.
Abstract
from arXiv · showhide
Automatic KB completion for commonsense knowledge graphs (e.g., ATOMIC and ConceptNet) poses unique challenges compared to the much studied conventional knowledge bases (e.g., Freebase). Commonsense knowledge graphs use free-form text to represent nodes, resulting in orders of magnitude more nodes compared to conventional KBs (18x more nodes in ATOMIC compared to Freebase (FB15K-237)). Importantly, this implies significantly sparser graph structures - a major challenge for existing KB completion methods that assume densely connected graphs over a relatively smaller set of nodes. In this paper, we present novel KB completion models that can address these challenges by exploiting the structural and semantic context of nodes. Specifically, we investigate two key ideas: (1) learning from local graph structure, using graph convolutional networks and automatic graph densification and (2) transfer learning from pre-trained language models to knowledge graphs for enhanced contextual representation of knowledge. We describe our method to incorporate information from both these sources in a joint model and provide the first empirical results for KB completion on ATOMIC and evaluation with ranking metrics on ConceptNet. Our results demonstrate the effectiveness of language model representations in boosting link prediction performance and the advantages of learning from local graph structure (+1.5 points in MRR for ConceptNet) when training on subgraphs for computational efficiency. Further analysis on model predictions shines light on the types of commonsense knowledge that language models capture well.
1 Introduction and Motivation
Commonsense KB completion is difficult because free-form, semantically diverse nodes create much larger and sparser graphs than conventional KBs. The paper addresses this with structural and semantic context, reporting complementary benefits from graph structure and language-model transfer.
- Challenge: Commonsense KGs contain free-form, semantically diverse nodes, creating orders-of-magnitude larger and substantially sparser graphs than conventional KBs.FB15K-237 has 100x the density of ConceptNet and ATOMIC.
- Challenge: Existing KB completion models can degrade quickly as graph density decreases, as illustrated by ConvTransE on FB15K-237.
- Approach: The paper investigates graph convolutional networks with automatic graph densification to learn local structural context.
- Approach: It also fine-tunes pretrained language models for knowledge graphs to improve contextual representations of node text.
- Contributions: The work provides first empirical KB-completion results on ATOMIC, ranking evaluation on ConceptNet, and analysis of commonsense knowledge captured by language models.
- Findings: Transfer learning is generally more effective, while graph structure provides complementary gains, especially with computationally efficient subgraph training.
2 Knowledge Graphs
The paper evaluates commonsense completion on ConceptNet and ATOMIC alongside FB15K-237, emphasizing sparse graph structure and density-related performance degradation. The datasets differ in the type and textual form of commonsense knowledge they represent.
- Graph structure: Figure 2 bins nodes by in-degree to illustrate the relative sparseness of commonsense KGs versus FB15K-237.
- Graph structure: Figure 3 shows decreasing ConvTransE KB-completion scores as FB15K-237 graph density decreases on a logarithmic scale.
- Knowledge graphs: ConceptNet contains general commonsense facts, while ATOMIC represents social commonsense about day-to-day events.
- Knowledge graphs: ConceptNet nodes average 2.85 words, and its development and test sets contain 1200 tuples each.
- Knowledge graphs: ATOMIC encodes effects, needs, intents, and attributes of actors in events, with node phrases averaging 4.40 words.
- Graph statistics: Table 1 defines training-set graph density as D = V N(N−1), using the number of edges and nodes.
3 Machine Commonsense Completion
The model combines graph structure and language-based semantics for commonsense KB completion, using GCNs, semantic graph densification, and BERT representations. A convolutional decoder scores candidate tuples, while subgraph sampling addresses the computational cost of full-graph training.
- Model overview: The architecture investigates transfer learning from language to knowledge graphs and learning from graph structure, using BERT representations and GCN-based embeddings.Synthetic semantic-similarity links enrich sparse graph connectivity for GCN message passing.
- Learning from Graph Structure: A GCN encodes each node from its local neighborhood by propagating relation-aware messages across multiple layers.The model accounts for both relation types and the relative importance of neighboring nodes during aggregation.
- Graph Densification: Semantic graph densification adds synthetic similarity edges computed from pairwise cosine similarity between fine-tuned BERT node representations.Thresholds produce 122,618 similarity edges for CN-100K and 89,682 for ATOMIC; these edges are used for graph embeddings but not decoder scoring.
- Convolutional Decoder: The decoder uses graph embeddings, BERT embeddings, or their concatenation to score candidate tuples with a convolutional architecture and binary cross-entropy loss.Relation embeddings are dimension-matched with node embeddings so they can be stacked before convolution.
- Efficient Training: Full-graph training is memory-intensive, so the model samples smaller subgraphs; uniform random edge sampling performs best among the tested criteria.A GCN-plus-BERT ATOMIC model requires approximately 30GB of memory and 8–10 days of training on a Quadro RTX 8000 GPU.
4 Experimental Setup
The experiments compare embedding-based completion models, the generative COMET model, and proposed ConvTransE variants using ranking-based evaluation. Proposed models combine convolutional decoding with GCN, similarity-induced links, and BERT representations.
- Evaluation: Ranking evaluation uses HITS and Mean Reciprocal Rank after filtering other valid entities and averaging rankings in both directions.The directions rank e2 given (e1, rel) and e1 given (e2, rel−).
- Models: The study evaluates DistMult, ComplEx, ConvE, ConvTransE, and COMET for commonsense knowledge-base completion.COMET generates target phrases, whereas the other listed models are completion baselines or discriminative decoders.
- Evaluation: COMET is scored only in the forward direction because it was not trained with inverse relations, using total and normalized negative log-likelihoods.The total score can favor shorter sequences, while the normalized score can favor longer sequences.
- Proposed models: All proposed models use the CONVTRANSE decoder, with GCN, similarity-induced links, BERT, or concatenated graph-and-BERT embeddings defining model variants.GCN provides graph embeddings, while BERT supplies phrase representations for node inputs.
- Training: Training lasts at least 200 epochs and selects the checkpoint with the highest development-set MRR.Development MRR is checked every 10 epochs for CN-100K and every 30 epochs for ATOMIC.
5 Results and Discussion
Results show that BERT representations strongly help ConceptNet but are less effective for ATOMIC, while graph structure and similarity-induced links provide complementary gains under subgraph sampling. Generative models are difficult to repurpose for tuple ranking, and human evaluation addresses semantic similarities missed by automated metrics.
- BERT representations: BERT improves CN-100K MRR by approximately 19–35 points but is less effective for ATOMIC.The result is attributed to BERT’s strength with taxonomic relations and ATOMIC’s larger node set and more complex relation types.
- BERT representations: BERT most effectively captures taxonomic relations such as MadeOf and PartOf, along with temporal relations including HasPrerequisite and ReceivesAction.The analysis links this pattern to the higher frequency of taxonomic knowledge in large text corpora.
- Graph structure: Local graph structure improves performance with subgraph sampling, but its gains fade when the full graph is used.Graph embeddings provide benefits when training the entire graph in memory is infeasible.
- Graph densification: Similarity-induced edges improve graph embedding learning under subgraph sampling, with the best CN-100K MRR from SIM+GCN+BERT+CONVTRANSE.For ATOMIC, SIM+GCN+CONVTRANSE exceeds BERT+CONVTRANSE by +1.5 MRR points.
- Generative models: Generative COMET models are difficult to repurpose for ranking tuples because log-likelihood is an imperfect estimate of tuple truth.COMET remains faster to train, uses less storage memory, and is transductive.
- Human evaluation: Human evaluation measures the average percentage of valid tuples among top-10 candidates because automated metrics miss semantically similar false negatives.The evaluation uses three Amazon Mechanical Turk annotators on a sampled development subset.
6 Related Work
Prior commonsense KB-completion work includes graph traversal, embedding, recurrent, and dimensionality-reduction approaches. This paper builds on convolutional decoding and GCN neighborhood aggregation while addressing commonsense graph sparsity through graph densification and ranking-based evaluation.
- Existing KB completion: Embedding-based KB-completion methods scale better but typically assume sufficient relation instances and relatively high node degree, conditions commonsense KGs often lack.This motivates methods that incorporate additional structural or semantic context.
- Graph-based methods: The paper uses a convolutional decoder with GCNs to incorporate each node’s neighborhood into its representation.Its approach differs from related GCN-ConvTransE work by weighting neighbors, densifying sparse graphs, and jointly training graph and language representations.
- Commonsense KB completion: Earlier commonsense KB-completion work encodes tuples with Bi-LSTMs and predicts binary classification scores, whereas this paper argues for ranking-based training and evaluation.Related ConceptNet work also explored predicting new knowledge through dimensionality reduction.
- Downstream relevance: Commonsense KB completion is connected to downstream applications including information retrieval, question answering, and reading comprehension.A survey summarizes the use of commonsense KBs in downstream tasks.
7 Conclusion
The paper develops commonsense KB completion models enriched with structural and semantic context, showing benefits from language-model representations, local graph structure, and semantic similarity edges.
- The proposed models combine GCN-derived structural context with language-model representations to address commonsense KG completion.A progressive masking strategy integrates both information sources efficiently.
- BERT-based node representations provide significant improvements, especially when graph text resembles pre-training corpora.
- Graph embeddings encode rich local context and boost performance when models train on subgraphs.
- Semantic similarity edges can augment commonsense graphs and help link completion.
9 Supplementary Material
The supplementary material reports implementation settings for BERT, baseline models, and graph-based models, and presents qualitative prediction figures for ConceptNet and ATOMIC.
- Implementation Details: BERT-Large was fine-tuned with masked language modeling using sequence length 64, batch size 32, learning rate 3e-5, and 0.1 warmup proportion.
- Implementation Details: Baseline batch sizes and learning rates were tuned from four candidate values while other hyperparameters retained their defaults.The tested batch sizes were 128, 256, and 512; the tested learning rates were 0.001, 0.002, and 0.003.
- Implementation Details: The graph convolutional network used two layers with 200-dimensional input and output embeddings, while subgraph sampling used batches of 30000 edges.Using three, four, or five GCN layers did not produce significant improvements.
- Qualitative Analysis: Figures 6 and 7 show randomly sampled top-1 predictions with human validity annotations for ConceptNet CN-100K and ATOMIC, respectively.The ATOMIC figure clarifies that “x” denotes PersonX and “o” denotes other actors.
- Qualitative Analysis: Figure 8 compares the top-scoring relations produced by BERT+CONVTRANSE and SIM+GCN+CONVTRANSE.