Source-linked AI summary
Conditional Generation with a Question-Answering Blueprint
Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Anders Sandholm, Dipanjan Das, Mirella Lapata
TL;DR
Neural generation models can hallucinate and miss important content, while their black-box nature makes decisions difficult to inspect. This paper uses automatically generated QA-pair blueprints as plans for conditional generation, with models that incorporate them globally or sentence by sentence. Blueprint models are reported as more factual than non-planning alternatives, and are generally rated better than previous state-of-the-art models on summary quality.
Problem
Neural generation models can hallucinate, fail to cover important content, and obscure model decisions, complicating faithful and controllable conditional generation.
Source: §1 · p. 1Method
The paper automatically augments input-output examples with ordered QA-pair blueprints and trains Transformer models to generate using global or iterative plans.
Source: §1 · p. 2, §3.1 · p. 3Results
Blueprint models are generally perceived as significantly better than SIBERT and R2T-BART; the best-rated system varies across AQuaMuse, WikiCatSum, and SummScreen.
Source: §6.1 · p. 13Takeaways & Limitations
Incremental planning and generation offers a way to control output while avoiding decoding a global blueprint and summary in one sequence.
Source: §1 · p. 2Generating summaries in response to a set of user-provided questions is left to future work.
Source: §5.2 · p. 11
Abstract
from arXiv · showhide
The ability to convey relevant and faithful information is critical for many tasks in conditional generation and yet remains elusive for neural seq-to-seq models whose outputs often reveal hallucinations and fail to correctly cover important details. In this work, we advocate planning as a useful intermediate representation for rendering conditional generation less opaque and more grounded. Our work proposes a new conceptualization of text plans as a sequence of question-answer (QA) pairs. We enhance existing datasets (e.g., for summarization) with a QA blueprint operating as a proxy for both content selection (i.e.,~what to say) and planning (i.e.,~in what order). We obtain blueprints automatically by exploiting state-of-the-art question generation technology and convert input-output pairs into input-blueprint-output tuples. We develop Transformer-based models, each varying in how they incorporate the blueprint in the generated output (e.g., as a global plan or iteratively). Evaluation across metrics and datasets demonstrates that blueprint models are more factual than alternatives which do not resort to planning and allow tighter control of the generation output.
1 Introduction
Neural generation can hallucinate, miss salient content, and obscure model decisions, making faithful and controllable conditional generation difficult. The paper proposes automatically generated QA-pair plans and Transformer models that use them globally or iteratively; experiments report greater factuality than non-planning alternatives.
1 Introduction
- Neural generators can hallucinate, repeat themselves, and struggle to identify salient content; these challenges intensify with long-form text and very long inputs.Their black-box nature also makes decisions and error attribution difficult, while limited modularity constrains tailoring.Source: §1 · p. 1
- The proposed QA blueprint makes discourse questions explicit and uses ordered question-answer pairs as a proxy for content selection and planning.The pairs represent both what to say and in what order.Source: §1 · p. 2
- Existing input-output examples are augmented automatically with blueprints, producing input-blueprint-output tuples for training three Transformer-based generation models.The models vary in how they incorporate the blueprint into generation.Source: §1 · p. 2
- The iterative model plans and generates one sentence at a time, conditioning on the input and sentences already generated.It avoids decoding a global blueprint and summary in one sequence, reducing the risk of exceeding maximum decoder length.Source: §1 · p. 2
- Blueprint models are reported as more factual than alternatives without planning, while QA blueprints support tighter output control and explanations of model predictions.An erroneous plan is expected to yield an erroneous summary.Source: §1 · p. 2
2 Related Work
Prior work uses questions and answers to represent semantic content, discourse, and summarization signals, while planning methods often use input-specific plans. This paper instead adopts automatically obtained, unstructured QA pairs as a more elaborate plan representation for summarization.
2 Related Work
- QUD theories structure discourse through questions and sub-questions, and prior work has manually annotated QUDs or generated inquisitive questions.The paper draws on these ideas but does not directly implement a particular QUD theory.Source: §2 · pp. 2–3
QA Pairs as a Proxy for Annotation Labels
- QA pairs have represented semantic roles and discourse relations; this work uses them as propositional-content abstractions without targeting specific relation types or linguistic assumptions.Other summarization work uses QA pairs for overlap evaluation, training signals, salient-content identification, or hierarchically structured question-summary generation; this paper's pairs are automatic and unstructured.Source: QA Pairs as a Proxy for Annotation… · p. 3
2 Related Work
- Earlier encoder-decoder planning methods use input-specific plans, including record sequences, extracted sentence plans, symbolic plans, or paragraph plans; this work is closest to Narayan et al. (2021).The paper uses a more elaborate QA blueprint and interfaces decoding with plan generation similarly to Narayan et al. (2020).Source: QA Pairs as a Proxy for Annotation… · p. 3
3 Text Generation with Blueprints
The paper formulates generation around an ordered QA blueprint between input and output. It automatically generates and filters QA pairs, then uses global or incremental blueprint-conditioned models to generate text.
3.1 Problem Formulation
- The input d may be a document, multiple documents, dialogue history, or database tables; the output s is generated with an ordered QA blueprint b between them.The blueprint is a set of question-answer pairs, and existing input-output examples are augmented into (d, b, s) tuples.Source: §3.1 · p. 3
3.2 Blueprint Annotation
- QA pairs are generated for output text and filtered through round-trip consistency, proposition matching, and selection stages to form the blueprint.The text is split into propositions, with Rheme favoring a question whose answer is rightmost (longest answer breaks ties), then Coverage greedily selects non-overlapping pairs and orders them by answer position.Source: §3.2 · pp. 4–5
3.3 Blueprint Models
- The end-to-end model predicts a global QA blueprint and then the output in one sequence, using the blueprint and input to generate the output.The global blueprint functions as a macro-plan, but generating blueprint plus output increases sequence length by 220%; pipeline separation also risks error propagation from an inaccurate blueprint.Source: §3.3 · p. 5
- The multitask model predicts an answer plan and then generates either the output or corresponding questions, rather than decoding the full blueprint and output together.It mitigates the longer-sequence issue, but conditions output on answers alone rather than question-answer pairs; the plan is less detailed than the end-to-end model.Source: §3.3 · p. 5
- The iterative model predicts a blueprint for the next sentence and then generates that sentence, conditioning on the input and sentences generated so far.It creates sentence-level plans rather than a global macro-plan; each sentence uses its full QA blueprint, but not previously generated QA pairs.Source: §3.3 · p. 6
4 Experimental Setup
The experiments span long-input, multi-sentence generation tasks that vary in source type, information need, and summary style, and compare blueprint models with LONGT5 and T5 baselines. [No printed citation markers are needed for these setup claims.]
4.1 Datasets
- The evaluation covers long-input, multi-sentence outputs across varied source types, information needs, and summary styles.The datasets include multi-document web sources and dialogue transcripts, with both abstractive and extractive summaries.Source: §4.1 · p. 6
- AQuaMuSe inputs average 6.46 web documents, or 12,987 tokens, for query-focused long-form answer generation.Each document averages 2,008 tokens.Source: §4.1 · pp. 6–7
- WikiCatSum generates Wikipedia abstracts from topic-related webpages, with inputs truncated to 800 tokens.The dataset combines Films, Companies, and Animals subsets.Source: §4.1 · p. 7
- SummScreen-FD condenses book-length transcripts averaging 8,000 tokens into a few sentences, with plot details scattered through dialogue.The subset used covers episodes from 88 shows.Source: §4.1 · p. 7
- SummScreen-FD has 28.10 blueprint QA pairs on average, versus 8.16 for AQuaMuSe and 9.56 for WikiCatSum, and is more abstractive than either dataset.Its blueprint-augmented target length is 597.90, versus 291.28 for WikiCatSum and 272.68 for AQuaMuSe.Source: §4.1 · p. 7
4.2 Comparison Systems
- The comparison includes end-to-end, multitask, iterative, and two-stage blueprint models, alongside LONGT5 and T5 baselines.LONGT5 models use maximum input and output lengths of 4,096 and 512 tokens; the T5 baseline uses 1,024 and 256 tokens.Source: §4.2 · p. 7
5 Automatic Evaluation
Automatic evaluation compares summary quality, faithfulness, informativeness, and blueprint grounding across three datasets. ITERATIVE is consistently more faithful than the comparison systems, while results also reveal dataset-dependent trade-offs in informativeness and controllability.
5.1 Metrics
- Answering reference questions with predicted summaries measures informativeness, while answering predicted-blueprint questions measures grounding.Faithfulness is estimated with textual entailment; the entailment predictions correlate with human faithfulness ratings at Spearman’s ρ = 0.774.Source: §5.1 · p. 8
5.2 Results
- Blueprint variants slightly underperform LONGT5 on AQuaMuSe but score better on WikiCatSum and SummScreen-FD; MULTITASK leads blueprint models on WikiCatSum and ties ITERATIVE on SummScreen-FD.LONGT5 outperforms T5 on every task and sets a new state of the art on AQuaMuSe and SummScreen-FD.Source: §5.2 · pp. 8–9, Table 4 · p. 9
- On SummScreen-FD, predicted summaries answer only 7.59% of reference questions despite a maximum Rouge score of 31.88.The authors report that QA-based informativeness evaluation reveals task complexity that Rouge alone does not capture.Source: §5.2 · p. 9
- E2E and ITERATIVE generate text more grounded in their predicted blueprints than MULTITASK, which conditions on answer spans only.E2E generates its blueprint before the summary; ITERATIVE plans and generates one sentence at a time.Source: §5.2 · p. 10
- ITERATIVE is more faithful than E2E, MULTITASK, T5, and LONGT5 on all datasets, except its difference from E2E on AQuaMuSe is not significant.On SummScreen-FD, its ANLI score is 20.84, while E2E and MULTITASK score below 10.Source: Table 4 · p. 9, §5.2 · p. 10
- Changing a predicted QA blueprint can alter content and order without retraining, and removing QA pairs can produce a shorter summary.The paper also describes removing unanswerable pairs to control faithfulness.Source: §5.2 · pp. 10–11, Table 6 · p. 11, Table 7 · p. 11
- Removing QA pairs that cannot be answered from the input improves entailment scores across datasets, with ITERATIVE+drop performing best.These variants show a minor drop in Rouge and informativeness; gains are larger on abstractive WikiCatSum and SummScreen-FD than on mostly extractive AQuaMuSe.Source: §5.2 · p. 11
5.3 Ablation Studies
- Forming answer-question pairs performs better than predicting questions before answers; removing both Rheme and Coverage harms summary quality, while random ordering mainly harms blueprint quality.Removing either Rheme or Coverage alone has a small negative impact on summaries but not blueprints.Source: §5.3 · pp. 11–12
6 Human-based Evaluation
Human evaluations assessed summary quality, blueprint quality, and how closely summaries followed their blueprints. Results varied by criterion and dataset: E2E led in blueprint coherence, while ITERATIVE summaries were most grounded.
6.1 Summary Quality
- LONGT5 and its blueprint variants were generally rated above SIBERT and R2T-BART, while the top-rated system varied by dataset.LONGT5 ranked best on AQuaMuse, E2E on WikiCatSum, and ITERATIVE on SummScreen; some pairwise differences were not statistically significant.Source: Table 11 · p. 12, §6.1 · p. 13
6 Human-based Evaluation
- The studies assessed summary informativeness and fluency, blueprint coherence and redundancy, and blueprint-to-summary coverage, contradictions, and added information.Summary quality was judged on 100 instances from three datasets, with 5,400 annotations and 97.11% annotator agreement.Source: §6.1 · p. 13, §6.2 · p. 13, §6.3 · p. 14
6.2 Blueprint Quality
- E2E generated the most coherent blueprints, while ITERATIVE blueprints were generally the most redundant.E2E's coherence advantage was statistically significant over comparison systems except gold-standard blueprints.Source: §6.2 · p. 13, Table 12 · p. 13
6.3 Blueprint Grounded Generation
- Across datasets, ITERATIVE summaries were most grounded, with the fewest blueprint questions absent from or contradicted by the summary and the least new information.ITERATIVE+drop was slightly less grounded in human judgments, although its automatic entailment scores improved on all three datasets.Source: §6.3 · p. 14
7 Conclusion
The paper represents plans as ordered question-answer pairs and develops models that use global blueprints or plan and generate sentence by sentence. Across three challenging datasets, blueprint models were more informative than unplanned sequence-to-sequence approaches, and ITERATIVE was the strongest choice for grounded generation.
7 Conclusion
- Across three challenging datasets, blueprint models were more informative than vanilla sequence-to-sequence approaches, and ITERATIVE was the best choice for grounded generation.The authors also identify ITERATIVE as a promising direction for long-form generation.Source: §7 · p. 14
- Blueprint predictions can be inspected and errors traced to the plan, which offers an intuitive way for users to influence output by editing blueprint length or content.The paper also suggests blueprints may help retrieve and organize important content across many or very long inputs.Source: §7 · p. 14