JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs

• Limited graph-text parallel data makes it difficult for typical neural generation models to learn entity- and relation-to-token alignments from scratch.Source: §1 · p. 1
• Existing pre-trained approaches can neglect graph structure during encoding and lack explicit graph-text alignment objectives.Full attention does not explicitly consider relations between input entities, while text-reconstruction tasks are not designed to model complex graph-text alignments.Source: §1 · p. 1
• JointGT adds structure-aware semantic aggregation at each Transformer layer and three pre-training tasks for graph-text alignment.The tasks cover graph-enhanced text reconstruction, text-enhanced graph reconstruction, and embedding-space alignment via Optimal Transport.Source: §1 · p. 2
• Experiments evaluate JointGT on WebNLG, WebQuestions, and PathQuestions.Source: §1 · p. 2
• JointGT achieves new state-of-the-art performance on KG-to-text generation.Source: §1 · p. 2

• KG-to-text research includes graph neural networks and graph Transformers to improve representations of linearized input graphs [1, 2, 3, 4].Source: §2 · p. 2
• Recent KG-to-text generation research mainly covers encoder modification to reduce structural-information loss from linearized graphs, including graph neural networks and graph Transformers, alongside unsupervised training objectives.Source: §2 · p. 2
• Pre-trained language models including GPT, BART, and T5 have been fine-tuned on graph-to-text datasets [8, 9, 10, 11].Source: §2 · p. 2
• Graph-enhanced pre-trained models for NLU incorporate knowledge graphs to support understanding of entities and relations in text [12, 13, 14, 15].JointGT differs by focusing on graph encoding and sequence decoding for KG-to-text generation rather than graph-text joint encoding for NLU.Source: §2 · p. 2

3.1 Task Definition and Model Overview

• JointGT starts from a linearized graph and a text sequence, then jointly learns graph and text representations with discrete-space reconstruction and continuous-space matching.The model is built on pre-trained encoder-decoder models such as BART and T5.Source: §3.1 · p. 3

3.2 Model Structure

• At each Transformer encoder layer, mean pooling extracts entity and relation representations from vanilla self-attention states, and graph-aware aggregation and a residual layer combine structural and contextual information.The model derives entity and relation embeddings from contextual representations, a design the authors say helps it generalize to new entities and relations when fine-tuned on datasets with a different knowledge graph.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

3.3.1 Graph Enhanced Text Reconstruction

• With the complete graph as input, graph enhanced text reconstruction recovers the masked text sequence.Entity words are masked with 40% probability and other words with 20% probability; consecutive mask tokens are merged.Source: §3.3.1 · pp. 4–5

3.3.2 Text Enhanced Graph Reconstruction

• With complete text and a graph corrupted by masked entities and relations, text enhanced graph reconstruction recovers the missing graph tokens.Entities and relations are masked with probabilities of 40% and 20%, respectively.Source: §3.3.2 · p. 5

3.3.3 Graph-Text Embedding Alignment

• Optimal Transport aligns graph-side entity and relation representations with text representations in the embedding space.The cost uses cosine distance between contextual embedding vectors; IPOT approximates the OT distance because exact minimization over the transport plan is computationally intractable.Source: §3.3.3 · p. 5

4.1 Pre-training Dataset and Implementation

• KGTEXT supplies 7M graph-text pairs, with texts crawled from English Wikidump and graphs acquired from Wikidata using entity hyperlinks.Table 2 reports dataset statistics including entity and relation counts, data splits, average triples, and average text length.Source: §4.1 · p. 5, Table 2 · p. 6

4.2 Fine-Tuning Settings

• Table 1 reports results on the three benchmarks and includes parameter counts; its markers indicate significance relative to BART and T5.Table 3 reports WebNLG(U) human-evaluation win, lose, and tie percentages, with Fleiss’ Kappa indicating moderate agreement.Source: Table 1 · p. 6, Table 3 · p. 7

4.1 Pre-training Dataset and Implementation

• JointGT uses BART-base or T5-base parameters, pre-trains for one epoch, and runs on three NVIDIA Quadro RTX 6000 GPUs.The reported pre-training times are 44 hours for JointGT (BART) and 69 hours for JointGT (T5).Source: §4.1 · p. 6

4.2 Fine-Tuning Settings

• Fine-tuning benchmarks are WebNLG, WebQuestions, and PathQuestions, covering graph description and knowledge-base question generation.WebNLG has Unconstrained and Constrained splits; PathQuestions uses two-hop or three-hop paths between entities.Source: §4.2 · pp. 6–7

4.3 Baselines

• The comparison uses pre-trained baselines KGPT, BART, and T5, alongside task-specific models without pre-training.The task-specific baselines are Seq2Seq with copying or delexicalisation for WebNLG, and G2S for WebQuestions and PathQuestions.Source: §4.3 · p. 7
• KGPT uses the same pre-training dataset as JointGT and directly uses KG-to-text generation as its pre-training task.Source: §4.3 · p. 7
• Baseline results are reprinted when the datasets match; otherwise, baselines are implemented from released code and parameters, with reported means over five runs.Source: §4.3 · p. 7

4.4 Automatic Evaluation

• JointGT with BART or T5 outperforms the corresponding vanilla model on most metrics and achieves state-of-the-art performance on all evaluated datasets.The automatic metrics are BLEU, METEOR, and ROUGE-L.Source: §4.4 · p. 7

4.5 Human Evaluation

• JointGT beats BART and T5 in both fluency and adequacy in pairwise human evaluation, with moderate annotator agreement (0.4 ≤ κ ≤ 0.6).The evaluation sampled 100 WebNLG(U) test graphs and used three annotators per pair.Source: §4.5 · p. 7

4.6.1 Encoder Structure

• On WebNLG(U), the proposed encoder outperforms SeqEnc and RelEnc, and its advantage over RelEnc is attributed to contextual initialization of entity and relation representations.The authors link this design to better generalization during fine-tuning.Source: §4.6.1 · p. 8, Table 4 · p. 8
• The encoder’s improvement margin over alternatives is more evident for WebNLG(U) graphs with more input triples.The comparison divides the test set by triple count and reports BLEU scores.Source: §4.6.1 · p. 8, Table 5 · p. 8

4.6.2 Pre-Training Task

• Each proposed pre-training task contributes to performance; graph-enhanced text reconstruction is the most important of the three, and replacing the task set with BART or KGPT pre-training causes an apparent drop.The authors attribute the stronger role of graph-enhanced text reconstruction to its direct conditional-generation supervision of the decoder.Source: §4.6.2 · p. 8, Table 6 · p. 8

4.7 Few-Shot Learning

• The few-shot setting uses only a few training instances for fine-tuning while keeping the model structure fixed.JointGT’s pre-training tasks are compared with BART and KGPT pre-training tasks.Source: §4.7 · pp. 8–9
• JointGT’s pre-training tasks perform better than the BART and KGPT alternatives, especially when the fine-tuning data is limited.The comparison reports BLEU scores across different proportions of WebNLG(U).Source: §4.7 · p. 9, Table 7 · p. 9
• The authors interpret the few-shot results as evidence that their tasks capture graph-text alignments and support generalization with only a few training samples.Source: §4.7 · p. 9

4.8 Case Study

• BART’s example omits the triple stating that Acharya Institute of Technology offers Tennis.Source: §4.8 · p. 9
• T5’s example omits the International Tennis Federation governing-body triple and generates a fact not present in the graph.Source: §4.8 · p. 9
• Both JointGT variants include all input triples and express relations between entity pairs more faithfully in the example.Source: §4.8 · p. 9

5 Conclusion

• JointGT is a graph-text joint representation model for KG-to-text generation.Source: §5 · p. 9
• Its structure-aware semantic aggregation module is plugged into vanilla Transformer layers to preserve input-graph structure.Source: §5 · p. 9
• Three pre-training tasks learn graph-text alignments in discrete vocabulary and continuous embedding spaces, and experiments report that JointGT outperforms state-of-the-art pre-trained NLG models across various KG-to-text datasets.Source: §5 · p. 9

A IPOT Algorithm

• IPOT iteratively approximates Optimal Transport distance and computes the transport plan T.The algorithm is applied to the graph and text sequences and their corresponding embedding vectors.Source: A IPOT Algorithm · p. 12
• Algorithm 1 presents the IPOT implementation used to calculate T for the graph and text embeddings.The graph sequence is paired with graph embedding vectors; the text sequence X is paired with embeddings S.Source: A IPOT Algorithm · p. 12
• The IPOT algorithm uses β = 1.0, K = 1, and N = 10 as hyper-parameters.The settings follow Chen et al. (2020a).Source: A IPOT Algorithm · p. 12

B Hyper-Parameter Setting

• Tables 8–10 present the pre-training and fine-tuning search spaces and the best downstream assignments.The best pre-training assignment is described in the main content.Source: Table 8 · p. 12, Table 9 · p. 13, Table 10 · p. 13
• The models were implemented with Huggingface’s Transformers, and reported hyper-parameters were consistent with its codes.The settings include search spaces and best assignments for pre-training and downstream fine-tuning.Source: B Hyper-Parameter Setting · pp. 12–13
• 10 pre-training hyper-parameter trials were manually searched using BLEU on WebNLG(U) validation after fine-tuning as the selection criterion.The selected pre-training assignment is described in the main content.Source: B Hyper-Parameter Setting · p. 13
• 20 downstream fine-tuning hyper-parameter trials were manually searched, with BLEU as the criterion across all downstream tasks.Source: B Hyper-Parameter Setting · p. 13