Source-linked AI summary
Densely Connected Graph Convolutional Networks for Graph-to-Sequence Learning
Zhijiang Guo, Yan Zhang, Zhiyang Teng, Wei Lu
TL;DR
Graph-to-sequence learning seeks graph representations that support text generation, while deep GCNs have been difficult to train beyond shallow architectures. The paper introduces DCGCNs with dense connectivity, and reports state-of-the-art results on AMR-to-text generation and syntax-based neural machine translation.
Problem
Prior GCN approaches used shallow architectures to capture local structure, while deeper GCNs remained limited despite the value of capturing non-local graph information.
Source: §1 · p. 2Method
DCGCNs introduce dense connections between layers so deeper graph convolutions can combine local and non-local graph information.
Source: §1 · p. 2Results
DCGCNs outperform state-of-the-art models on AMR-to-text generation and syntax-based neural machine translation, with AMR gains of 2 BLEU points on LDC2015E86 and 4.3 BLEU points on LDC2017T10.
Source: §1 · p. 2, §6 · p. 12Takeaways & Limitations
The authors conclude that dense connectivity lets DCGCNs scale to significantly more layers without performance degradation or optimization difficulties, helping encode graph structure, especially for large graphs.
Source: §6 · pp. 12–13
Abstract
from arXiv · showhide
We focus on graph-to-sequence learning, which can be framed as transducing graph structures to sequences for text generation. To capture structural information associated with graphs, we investigate the problem of encoding graphs using graph convolutional networks (GCNs). Unlike various existing approaches where shallow architectures were used for capturing local structural information only, we introduce a dense connection strategy, proposing a novel Densely Connected Graph Convolutional Networks (DCGCNs). Such a deep architecture is able to integrate both local and non-local features to learn a better structural representation of a graph. Our model outperforms the state-of-the-art neural models significantly on AMRto-text generation and syntax-based neural machine translation.
1 Introduction
Graph-to-sequence learning uses graph structure to represent sentence semantics for text generation, but deep GCNs have been difficult to train effectively. The paper introduces dense connectivity to support deeper GCNs and reports improvements on AMR-to-text generation and syntax-based translation.
1 Introduction
- Graph-to-sequence learning uses graph representations of sentence semantics to learn representations useful for text generation.AMR is described as a directed, labeled graph whose nodes denote semantic concepts and whose edges denote relations.Source: §1 · p. 1
- Prior deep GCN efforts remained relatively shallow, reaching at most six layers in the cited experiments, while recurrence-based graph encoders offered an alternative.The paper notes that deeper convolutional architectures are attractive because they are highly parallelizable and more amenable to hardware acceleration.Source: §1 · pp. 1–2
- Dense connectivity gives each layer access to outputs from all preceding layers, allowing deeper GCNs to capture both local and non-local graph information.The authors propose this as a way to learn better graph representations than shallower GCNs.Source: §1 · p. 2
- DCGCNs surpass state-of-the-art neural models by 2 BLEU points on LDC2015E86 and 4.3 BLEU points on LDC2017T10 for AMR-to-text generation, and are consistently better on syntax-based translation.The syntax-based translation results are described as demonstrating effectiveness on a large training set.Source: §1 · p. 2
2 Densely Connected GCNs
The section presents DCGCNs as graph encoders that build node representations through neighborhood aggregation and dense connections. It also describes layer aggregation, parameter-efficient hidden dimensions, and graph attention as components of the approach.
2.1 GCNs
- A GCN updates each node from its neighboring nodes; L layers are needed to capture information L hops away.Earlier approaches include residual connections and layer aggregation, which combines node representations from all GCN layers.Source: §2.1 · p. 2
2 Densely Connected GCNs
- Each DCGCN block uses two densely connected sub-blocks with different layer counts, joined by a linear transformation and followed by a residual connection.The section also describes combining representations from different GCN layers with a linear combination layer and residual input.Source: Figure 2 · p. 3
2.2 Dense Connectivity
- Dense connectivity concatenates each node's initial representation with its outputs from all preceding layers, so deeper layers can access earlier information.Intermediate layers use small hidden dimensions, set to d/L, to improve parameter efficiency while concatenated outputs form the representation.Source: §2.2 · p. 3
2.3 Graph Attention
- Graph attention assigns different weights to nodes in a neighborhood and uses normalized attention coefficients to combine node representations without changing output dimensions.The coefficients are computed after a shared linear projection and use a LeakyReLU activation.Source: §2.3 · pp. 3–4
3 Graph-to-Sequence Model
The model encodes graphs directly with DCGCNs, augmenting graph structure and aggregating edge-specific information before an attention-based LSTM decoder generates text.
3 Graph-to-Sequence Model
- DCGCNs serve as a graph encoder that models graph structure directly, without linearizing the graph.The encoder is built from DCGCN blocks with two multi-layer sub-blocks per block.Source: §3 · p. 4, §3.1 · p. 4
3.1 Graph Encoder
- A linear combination layer filters representations from different DCGCN layers and adds a residual connection from the initial inputs.The layer combines densely connected outputs; another linear combination layer produces the final representations.Source: §3.1 · p. 4
3.2 Extended Levi Graph
- The extended Levi graph adds a global node connected to every node, and dependency trees also receive forward and backward sequential connections.The global-node representation initializes the decoder; positional encodings use minimum distance from the root, with the global node indexed as -1.Source: §3.2 · p. 5
3.3 Direction Aggregation
- Direction aggregation assigns separate weights and biases to edge types, then combines their representations through concatenation and a linear transformation.AMR graphs have four edge types, while dependency trees have six.Source: §3.3 · pp. 5–6
3.4 Decoder
- An attention-based LSTM decoder generates the sequence, initialized with the global-node representation and using a coverage mechanism.At each token, it considers attention memory, the previous token embedding, previous hidden state, previous context, and previous coverage.Source: §3.4 · p. 6
4 Experiments
Across AMR-to-text generation and syntax-based translation, DCGCN outperforms the reported neural baselines. Additional experiments find that dense connectivity supports deeper models and that DCGCN’s advantage grows on larger graphs.
4.1 Experimental Setup
- The experiments used AMR15 and AMR17 plus En-De and En-Cs translation datasets, transforming AMR graphs and dependency trees into extended Levi graphs.The AMR training sets contained about 16K–36K instances, compared with about 200K for NMT; models used no pretrained embeddings and beam size 10.Source: §4.1 · p. 6
4.2 Main Results on AMR-to-text Generation
- 27.6 BLEU on AMR17 made the single DCGCN a new single-model state of the art; it exceeded single Seq2SeqB by 5.9 BLEU points.The single model also beat Seq2SeqB’s ensemble by 1 BLEU point and used 3/5 and 1/9 the parameters of GGNN2Seq and Seq2SeqB, respectively.Source: §4.2 · p. 7, Table 2 · p. 7
- 28.2 BLEU without external data beat the reported non-neural AMR15 systems, while DCGCN also achieved 33.2 BLEU with 0.3M external data and a 35.3-BLEU ensemble.With 0.1M additional data, the single model scored 29.0 BLEU, exceeding Seq2SeqK and GraphLSTM trained with 0.2M additional data.Source: §4.2 · pp. 7–8, Table 3 · p. 7
4.3 Main Results on Syntax-based NMT
- On En-De and En-Cs, single DCGCN models scored 19.0 and 12.1 BLEU, respectively, exceeding the best GCN-based models by 2.7 and 2.5 points.Ensemble DCGCN scored 20.5 and 13.1 BLEU, above the syntax-based ensemble results of GGNN2Seq (19.6 and 11.7).Source: §4.3 · p. 8, Table 4 · p. 8
4.4 Additional Experiments
- DCGCN trained at 36 layers and improved with additional blocks, while deeper DCGCNs generally outperformed shallower ones at similar parameter budgets.At 18.4M parameters, DCGCN4 exceeded DCGCN3 by 1 BLEU point; across four budgets, DCGCN’s BLEU margins over the best equal-budget GCNs were 2.0, 2.7, 2.7, and 3.4.Source: §4.4 · p. 9, Table 7 · p. 9
- Removing dense connections progressively reduced AMR15 development BLEU from 25.5 to 24.8, 23.8, and 23.2, and removing all dense connections prevented convergence.Ablations also found that removing the global node or linear combination reduced B/C scores by 1.3/1.2 and 2/2.2 points, respectively; removing coverage reduced them by 1.7/2.4 points.Source: §4.4 · p. 10, Table 8 · p. 10, Table 9 · p. 10
4.5 Analysis and Discussion
- On small AMR graphs (|G| ≤30), DCGCN performed similarly to the baselines, while its advantage over them grew as graph size increased.In an example, DCGCN generated two important words missed by both residual-connection baselines; the authors attribute the difference to richer semantic information from long dependencies.Source: §4.5 · p. 11, Figure 7 · p. 11, Table 10 · p. 11
5 Related Work
Related work spans statistical and neural graph-to-sequence methods, while GCN research highlights a challenge: deeper networks can access more non-local information but often perform worse. DCGCN builds on dense connectivity to encode richer graph structure with a deeper GCN.
Graph-to-sequence learning
- Earlier graph-to-sequence work used statistical approaches, including tree and forest representations, hypergraphs, AMR linearization, and synchronous node replacement grammars.Examples include Lu et al. (2009), Lu and Ng (2011), Flanigan et al. (2016), Pourdamghani et al. (2016), Song et al. (2017), and Konstas et al. (2017).Source: §5 · pp. 11–12
- Neural graph encoders include GGNNs with Levi graphs and graph-state LSTMs, whereas this work uses GCNs with linear transformations rather than gated node-state updates.Sequence-based networks may lose graph structure through linearization; recurrent graph models exchange information between connected nodes to capture non-local information.Source: §5 · p. 12
- Prior systems combined GCNs with RNN or CNN encoders, while purely GCN graph encoders remained relatively shallow; DCGCN instead uses a deep GCN encoder alone.Bastings et al. (2017), Damonte and Cohen (2019), and Marcheggiani and Perez-Beltrachini (2018) are cited as related approaches.Source: §5 · p. 12
Densely connected convolutional networks
- DenseNets connect layers directly to one another to promote information flow, extending the bypass-path ideas used in Highway Networks and ResNets.Highway Networks use bypassing paths with gating units, while ResNets use identity mappings as bypasses.Source: §5 · p. 12
Graph convolutional networks
- GCNs aggregate first-order neighborhoods, but deeper GCNs often underperform despite access to non-local information; over-smoothed representations can make nodes from different clusters harder to distinguish.Related responses include layer aggregation and co-training or self-training mechanisms.Source: §5 · p. 12
6 Conclusion
DCGCNs outperform state-of-the-art models on AMR-to-text generation and syntax-based neural machine translation. Dense connectivity supports deeper GCNs without performance degradation, and the paper identifies further graph-learning and NLP applications to explore.
6 Conclusion
- DCGCNs outperform state-of-the-art models on AMR-to-text generation and syntax-based neural machine translation.The conclusion reports this result across both tasks.Source: §6 · p. 12
- Dense connectivity lets DCGCNs scale to significantly more layers without performance degradation or optimization difficulties, helping capture graph structure, especially for large graphs.The paper attributes these properties to its dense connectivity mechanism.Source: §6 · pp. 12–13
- Future work includes applying the framework to graph representation learning and exploring relation extraction and semantic role labeling.The paper presents these as potential applications of the proposed approach.Source: §6 · p. 13