Source-linked AI summary
Structured Neural Summarization
Patrick Fernandes, Miltiadis Allamanis, Marc Brockschmidt
TL;DR
Summarization requires condensing complex inputs while retaining their core semantics, but sequence encoders often struggle with long-distance relationships. The paper extends sequence encoders with graph components over additional relationships and evaluates the resulting hybrid models across three summarization tasks. The hybrid approach outperforms pure sequence and pure graph representations across the reported evaluation.
Problem
Sequence-based summarization systems often fail to handle long texts and long-distance relationships reliably, motivating models that represent additional structure.
Method
The paper extends standard sequence encoders with graph neural network components that process additional relationships while retaining sequential representations.
Results
The resulting hybrid sequence-graph models outperform pure sequence and pure graph-based representations across three summarization tasks.
Takeaways & Limitations
Explicit relationship modeling provides an inductive bias that improves the practical performance of existing deep learning systems across formal and natural-language summarization tasks.
Abstract
from arXiv · showhide
Summarization of long sequences into a concise statement is a core problem in natural language processing, requiring non-trivial understanding of the input. Based on the promising results of graph neural networks on highly structured data, we develop a framework to extend existing sequence encoders with a graph component that can reason about long-distance relationships in weakly structured data such as text. In an extensive evaluation, we show that the resulting hybrid sequence-graph models outperform both pure sequence models as well as pure graph models on a range of summarization tasks.
1 INTRODUCTION
The paper addresses failures of sequence encoders on long-distance relationships by augmenting them with graph reasoning over additional structure. This hybrid framework is evaluated across summarization tasks and is reported to outperform pure sequence and pure graph representations.
- Motivation: Standard sequence-to-sequence summarizers often fail to correctly handle long texts and long-distance relationships despite their theoretical capacity.Existing systems commonly use recurrent, convolutional, or self-attentive sequence encoders with attention and copying in the decoder.
- Approach: The proposed framework extends standard sequence encoders with a graph component that leverages additional relationships in sequence data.The approach uses graph structure to model relationships such as co-references while preserving the information carried by sequential representations.
- Evaluation: The hybrid sequence-graph models outperform baselines using pure sequence or pure graph-based representations.The framework is evaluated on three summarization tasks from the literature.
2 STRUCTURED SUMMARIZATION TASKS
The paper studies three summarization tasks that translate long structured sequences into shorter sequences while preserving meaning. They range from highly structured source-code tasks to less structured natural-language summarization.
- Task overview: The three tasks differ in structure and summarization properties but share translation from a long sequence to a shorter meaning-preserving sequence.The first two tasks summarize source code, while the final task applies the framework to natural language.
- METHODNAMING: METHODNAMING predicts a function or method name from source code as a sequence of subtokens.Method names average 2.9 subtokens in the Java dataset, and 33% of name subtokens can be copied from the source code.
- METHODDOC: METHODDOC predicts a succinct method-functionality description from source code, with longer targets and less direct copying than method names.The C# documentation targets average 19.1 tokens, and 19.4% of documentation tokens can be copied from code.
- NLSUMMARIZATION: NLSUMMARIZATION generates a novel abstractive natural-language summary from input text with less explicitly defined structure.The task treats text traditionally as sequences of sentences and words, then recasts it as structured summarization using additional relationships.
3 MODEL
The model combines a standard sequence encoder with a graph neural network to propagate information across relationships among sequence elements. The resulting representations remain compatible with an unmodified decoder and can incorporate graph-level information and additional nodes.
- Sequence GNNs: The resulting per-token representations can be passed to an unmodified decoder, while additional nodes can represent sentences or syntax-tree non-terminals.Additional node features may be initialized from sources such as embeddings of their labels.
- Gated Graph Neural Networks: Gated graph neural network dynamics are rolled out for a fixed number of timesteps, with final node states used as output representations.The node updates use a GRU recurrent cell, and the final-step states provide the graph neural network outputs.
- Sequence GNNs: A sequence encoder first produces per-element and sequence representations, which form the inputs to the graph neural network.The framework accepts any sequence encoder that maps a sequence to per-element representations and a sequence representation.
- Sequence GNNs: The graph component uses directed, typed relationships among sequence elements to update node representations through neural message passing.Each node sends transformed messages to neighbors, and incoming messages are aggregated before the node state is updated.
- Sequence GNNs: The graph outputs are combined with the sequence representation to produce the final sequence representations used by downstream models.A weighted averaging mechanism forms a graph-level representation, which is combined with the sequence representation through a learnable projection.
4 EVALUATION
The evaluation tests sequence GNNs across source-code naming, source-code documentation, and news summarization, comparing hybrid encoders with sequence, graph, and task-specific baselines. Results show strong gains on METHODNAMING, mixed effects on METHODDOC, and competitiveness on natural-language summarization.
- Evaluation setup: The evaluation covers METHODNAMING, METHODDOC, and CNN/DM natural-language summarization, using task-specific datasets and metrics.METHODNAMING uses F1; METHODDOC uses BLEU, F1, ROUGE-2, and ROUGE-L; CNN/DM uses ROUGE.
- Evaluation setup: The source-code graph connects identifier subtokens, full identifiers, parse-tree nodes, and recent lexical uses through typed edges.The graph uses INTOKEN, NEXTTOKEN, CHILD, and LASTLEXICALUSE relationships.
- Overall results: Across all tasks, hybrid sequence GNN encoders outperform pure sequence encoders.The authors report this aggregate comparison in their evaluation results.
- METHODNAMING: All GNN-augmented METHODNAMING models outperform the specialized state of the art using graph structure obtainable from programming-language parsers.Encoder and decoder configuration effects are reported as largely orthogonal.
- METHODDOC: On METHODDOC, graph augmentation improves BLEU but worsens ROUGE, while the graph-only configuration performs worse than a pure sequence model.The authors associate the mixed result with prediction length and speculate that insufficient propagation timesteps limit graph-only information flow.
- NLSUMMARIZATION: On CNN/DM, the same model used for structured code tasks is competitive with specialized natural-language models but remains behind the best See et al. configuration.The authors attribute the remaining gap to their simplistic decoder and training objective.
5 RELATED WORK
Prior summarization work improves sequence-to-sequence systems through attention, copying, and training objectives, while other research exposes linguistic or code structure explicitly. This paper differs by directly encoding simple relationships over tokenized text rather than using AMR graph rewriting or only sequence representations.
- Summarization models: Abstractive summarization systems use attention over input text to guide decoders that generate concise summaries.Pointer networks and copying mechanisms extend this sequence-to-sequence approach.
- Summarization models: Coverage tracking and reinforcement learning provide alternative training objectives targeting document coverage or measures such as ROUGE-L.The paper treats these objectives as orthogonal to its graph-augmented encoder.
- Structured representations: Earlier work exposes language structure with tree-based models, graph convolutional networks, named-entity graphs, and abstract meaning representation.These approaches apply structure to tasks including translation, question answering, and summarization.
- Structured representations: Unlike AMR-based summarization, this work directly encodes relatively simple relationships on tokenized text and does not formulate summarization as graph rewriting.The authors identify combining their encoder with richer AMR structures as a future direction.
- Source-code summarization: Source-code summarization research includes method naming, documentation prediction, sequence-to-sequence models, and models that expose syntax-tree structure.Linking text to code is also connected to applications such as code search, traceability, and redundant-comment detection.
6 DISCUSSION & CONCLUSIONS
The framework augments sequence encoders with graph components to exploit additional structure, improving diverse sequence models across three summarization tasks.
- The framework extends sequence encoders with a graph component that leverages rich additional structure.The authors identify explicit relationship modeling as a broadly applicable inductive bias for improving existing deep learning systems.
A CODE SUMMARIZATION SAMPLES
The code summarization samples present ground-truth behaviors alongside outputs from sequence and hybrid sequence-graph models.
- A CODE SUMMARIZATION SAMPLES: The samples compare ground-truth behavior with outputs from BiLSTM, BiLSTM+GNN, and pointer-augmented variants.The listed examples cover parameter handling, geometry construction, conversion, provider initialization, platform detection, and metric assertions.
- A CODE SUMMARIZATION SAMPLES: The examples show model-specific summaries of null-parameter construction and serialization behavior.The ground truth and model descriptions distinguish ordinary parameter handling from pointer-augmented null serialization.
- A CODE SUMMARIZATION SAMPLES: The examples also describe geometry creation from spatial values and standard well-known spatial formats.The corresponding summaries mention geometry points, XML-derived spatial geometry, and well-known values.
B NATURAL LANGUAGE SUMMARIZATION SAMPLES
The natural-language samples compare reference summaries with outputs from pointer-based and graph-augmented summarization models across news-style inputs.
- B NATURAL LANGUAGE SUMMARIZATION SAMPLES: For the Lil Wayne report, graph-augmented models preserve key facts about the shooting, including the lack of injuries and arrests.The BiLSTM+GNN outputs retain the interstate detail in one case, while the pointer variant retains the event and police-status facts.
- B NATURAL LANGUAGE SUMMARIZATION SAMPLES: For the Florian Thauvin report, the reference summary retains Marseille’s sale price, Tottenham’s interest, and monitoring by Chelsea and Valencia.The pointer-based model output captures Tottenham’s discussions, the squad omission, and Marseille’s £15m sale push.
- B NATURAL LANGUAGE SUMMARIZATION SAMPLES: The model outputs vary in factual coverage, with some retaining event details while others repeat or omit information.The examples include repeated squad-omission statements and a partially corrupted interstate name marked %UNK%.
C.1 C# DATASET
The C# dataset is extracted from open-source GitHub projects and divided into train, validation, and test splits.
- C.1 C# DATASET: 460,905 methods were extracted from open-source GitHub projects, including 55,635 with documentation comments.The dataset uses an 85-5-10% split, and the repository states are listed in Table 3.
- C.1 C# DATASET: Table 3 lists the projects included in the C# dataset in alphabetical order.The passage identifies the table as the project inventory for the dataset.
C.3 PYTHON METHOD DOCUMENTATION DATASET
The documentation dataset uses an existing Barone and Sennrich split, with substantial validation and test overlap with training documentation. The supplied table reference instead concerns extracted graphs from a Java method-naming dataset.
- The dataset split contains 106,065 training samples, 1,943 validation samples, and 1,937 test samples.
- 16.9% of validation samples and 15.3% of test samples have documentation identical to a training sample.The authors identify this overlap as a potential issue discussed by Lopes et al. and Allamanis.
- Table 4 reports extracted graph statistics for the Java method naming dataset of Alon et al. (2019).
C.4 GRAPH DATA STATISTICS
The paper presents graph characteristics for the datasets used in evaluation and refers to a consolidated table of dataset graph statistics.
- The paper presents the data characteristics of the graphs used across the datasets.
- Table 5 is titled “Graph Statistics For Datasets.”
- The supplied passages provide the table title but no individual graph-statistics values or comparisons.