Faithfulness-Aware Decoding Strategies for Abstractive Summarization

• Across XSum and CNN/DM, beam search produces the most faithful summaries, while randomness from nucleus sampling hurts faithfulness.Source: §1 · p. 1
• The ranking and lookahead methods significantly improve faithfulness over baseline decoding strategies, according to four automatic metrics and human evaluation.Source: §1 · pp. 1–2
• Up to 1/6 of teacher decoding time is needed by a distilled student to generate summaries with similar faithfulness.The student uses greedy decoding.Source: §1 · p. 2
• After two iterations, the student model can generate more faithful summaries and outperform the original teacher model.Source: §1 · p. 2

• The study asks whether beam search’s broader candidate exploration improves faithfulness and whether randomness from sampling affects it.Source: §2 · p. 2
• Greedy decoding selects the most probable next token, while beam search retains the top k hypotheses at each step.Source: §2 · p. 3
• Nucleus sampling randomly selects among top tokens within a cumulative-probability threshold p; larger p allows more diverse output.A small p approaches greedy decoding.Source: §2 · p. 3

3 Faithfulness-Aware Decoding Strategies

• The two approaches target paths that base decoding may not select for faithfulness: ranking chooses among explored candidates, while lookahead guides search during generation.Source: §3 · p. 3

3.1 Ranking with Faithfulness Metrics

• Faithfulness-based ranking selects the highest-scoring summary from beam candidates, including candidates ranked lower by the model score.Source: §3.1 · p. 3

3.2 Lookahead

• Lookahead adds a reference-free faithfulness score for full future summaries, allowing exploration beyond the candidates constrained by beam reranking.The candidate summaries begin with the tokens generated so far; full summaries are used because the metrics expect full inputs.Source: §3.2 · pp. 3–4

3.3 Combining Ranking and Lookahead

• BEAM+LOOKAHEAD+RANKING first generates faithful beam candidates with lookahead, then selects among them using ranking.Source: §3.3 · p. 4

4 Efficient Decoding via Distillation

• Distillation transfers faithfulness-aware decoding behavior to a student that can generate similarly faithful summaries with greedy decoding.The stated aim is to improve decoding time rather than downsize the model.Source: §4 · p. 4
• Student training combines cross-entropy on original references with cross-entropy on teacher-generated summaries, weighted by λ.The teacher supplies additional reference summaries for the student to imitate.Source: §4 · p. 4
• A trained student can become the next teacher, generating additional references for another student in iterative distillation.The next teacher uses the proposed faithfulness-aware decoding methods to create those references.Source: §4 · p. 4

5.1 Datasets and Models

• Experiments use BART-large on XSum and CNN/DM; the same experiment with PEGASUS is reported in Appendix B.The BART-large checkpoint is 406M parameters.Source: §5.1 · pp. 4–5

5.2 Evaluation Metrics

• Summary quality is evaluated with ROUGE-L and BERTScore, while faithfulness is evaluated with BS-Fact, FactCC, DAE, and QuestEval.BS-Fact uses BERTScore precision against the source document rather than the reference summary.Source: §5.2 · p. 5

5.3 Human Evaluation Setup

• Human faithfulness ratings use three judgments per summary and majority voting, with fully factual summaries counted as the faithfulness score.The setup samples 200 examples from each dataset and evaluates summaries from greedy, sampling, beam, ranking, and lookahead systems.Source: §5.3 · p. 5
• Human informativeness is measured with best-worst scaling, using three annotators to choose the most and least informative summaries.For each dataset, 200 articles are evaluated across five systems; a system’s score is its best-selection percentage minus its worst-selection percentage.Source: §5.3 · p. 5

5.4 Decoding Setting Details

• Basic decoding compares greedy search, beam search with k = 10, and nucleus sampling with p = 0.9; ranking uses a composite faithfulness metric, and lookahead uses BS-Fact.The composite metric combines faithfulness metrics; lookahead uses greedy search to generate future summaries and is applied to greedy and beam search.Source: §5.4 · p. 5
• Beam search trends best and nucleus sampling worst in faithfulness, while larger beams add faithful candidates but only slightly improve the top beam.The baseline table reports the general comparison; Figure 3 distinguishes the best available candidate from the top beam.Source: Table 1 · p. 6, Figure 3 · p. 6

6.1 Baseline Decoding Results

• Larger beams improve all faithfulness scores in the search-space analysis, but the standard baseline comparison favors beam search except on XSum FactCC; nucleus sampling underperforms greedy decoding.Source: §6.1 · p. 6

6.2 Faithfulness-Aware Decoding Results

• Applying ranking on top of beam search improves faithfulness significantly over beam search, as measured by all faithfulness metrics; specifically, QuestEval reaches 62.57.Source: §6.2 · p. 6, Table 2 · p. 7
• Lookahead improves faithfulness over its base decoding strategy across all faithfulness metrics, and combining lookahead with ranking further improves FactCC, DAE, and QuestEval.Source: §6.2 · p. 7

6.1 Baseline Decoding Results

• Faithfulness-aware decoding lowers ROUGE-L, while beam search improves ROUGE-L and BERTScore over greedy on XSum but not CNN/DM.Source: §6.1 · p. 6, §6.2 · p. 7

6.3 Human Evaluation Results

• Human raters favored BEAM+LOOKAHEAD for faithfulness and informativeness; on XSum it scored 56.5, five points above BEAM+RANKING, while CNN/DM had no major errors.Source: §6.3 · p. 7, Table 3 · p. 8

6.4 Abstractiveness

• On 200 XSum examples, BEAM+LOOKAHEAD's relative faithfulness gain (29.89%) was four times its abstractiveness decrease (7.27%); adding ABSTR raises abstractiveness at a small faithfulness cost.Source: §6.4 · p. 8, Figure 4 · p. 8

6.5 Distillation

• Distilled students approach teacher performance while cutting ranking time from 0.77 to 0.47 seconds and lookahead from 3 to 0.49 seconds per example.Source: §6.5 · p. 8, Table 5 · p. 9
• After two iterations, the greedy-decoding student outperforms the original BEAM+RANKING teacher on all faithfulness metrics, with further iterations continuing to improve faithfulness.Source: §6.5 · p. 9

• Earlier decoding analyses considered diversity and repetitiveness, but did not explicitly analyze faithfulness.Source: §7 · p. 9
• Unlike conventional soft-target distillation, this approach uses generated summaries as hard labels with interpolation, targeting decoding cost rather than model-size reduction.Source: §7 · p. 9
• With BEAM+RANKING as teacher, two distillation iterations let the student outperform the teacher on faithfulness, and further iterations continue to improve it.Source: Table 6 · p. 9

8 Conclusion

• The paper analyzes decoding strategies’ effects on faithfulness and introduces ranking and lookahead methods to improve it over base decoding.It also reports a distillation approach in which a student generates summaries of similar faithfulness with minimal decoding time.Source: §8 · p. 9
• Distillation trains a student on summaries from faithfulness-aware methods, enabling similar faithfulness with minimal decoding time.The approach can be iterative.Source: §8 · p. 9
• Future work could extend faithfulness analysis beyond summarization and combine heuristics that cover other aspects and styles.Source: §8 · p. 9

9 Limitations

• Lookahead requires heavy computation, especially when combined with beam search for base decoding and future-summary generation.Source: §9 · p. 9
• The paper offers distillation as a way to improve decoding speed.Source: §9 · p. 10
• Online decoding later discards generated future summaries and their faithfulness scores, much like pruned beam candidates.Storing useful future summaries is suggested as a possible direction for reusing them during decoding.Source: §9 · p. 10

10 Ethical Impact

• The paper notes that faithfulness metrics may have under-studied biases, and that optimizing other metrics could also serve malicious intents.The proposed methods also require large computations, while distillation is presented as a possible mitigation for environmental impact.Source: §10 · p. 10

A.1 Datasets

• The experiments use English news articles and summaries from XSum and CNN/DM, with the passage reporting their train, validation, and test split sizes.Source: §A.1 · p. 12

A.2 Metrics

• The appendix includes Mechanical Turk annotation instructions for factual consistency and informativeness, and specifies ROBERTA-LARGE for BS and BS-Fact and sentence-level errors for DAE.Source: Figure 5 · p. 13, Figure 6 · p. 13, §A.2 · p. 13

A.3 Decoding Details

• Lookahead heuristics are applied to the top five model-scored tokens at each step; the heuristic weight is tuned on development data, and runs take 33 hours on XSum and about 70 hours on CNN/DM.Source: §A.3 · p. 13

B Full Analysis

• Increasing beam size improves faithfulness while increasing nucleus-sampling p degrades it for BART-Large, with a similar trend on PEGASUS-Large.Source: Table 7 · p. 14, B Full Analysis · pp. 14–15, Table 8 · p. 14

A.4 Human Evaluation Details

• Faithfulness annotation agreement was α=0.63 on CNN/DM and 0.57 on XSum; informativeness agreement was 0.22 and 0.32, respectively.Source: §A.4 · p. 14

C Lookahead Methods

• Beam-based lookahead added no gains; the authors suspect future beam paths may be unreachable because base decoding selects only top tokens.Source: Table 9 · p. 15, C Lookahead Methods · p. 15

D Composite Metric

• The composite metric increased partial correlations in all reported settings except XSum Spearman correlation.It was trained on FACTCOLLECT and tuned on FRANK; its component weights were 0.29 for FactCC, -0.29 for DAE, 1.97 for BS-Fact, and 0.94 for QuestEval.Source: D Composite Metric · p. 15
• Ranking with the composite metric produced consistent gains across faithfulness metrics on WikiHow, a domain beyond XSum and CNN/DM.Source: D Composite Metric · p. 15
• Scoring only partially generated summaries (l=0) does not effectively estimate faithfulness, unlike evaluating the full future summary.Source: Table 12 · p. 16

E Ablations

• For XSum, scoring only the current partial summary yields mixed results and slight improvements on BS-Fact, DAE, and QuestEval; substantial gains require using the full future summary.The l = 0 setting scores already-generated partial summaries, while l = full incorporates the future.Source: E Ablations · p. 16
• The composite metric achieves similarly good scores across the considered faithfulness metrics, although optimizing an individual metric performs best on that metric.Table 13 presents ranking results with different faithfulness metrics.Source: E Ablations · p. 16, Table 13 · p. 16

F Abstractiveness

• Across CNN/DM and XSum, faithfulness gains outweigh decreases in abstractiveness, while BEAM+RANKING achieves the highest average score.The average uses all faithfulness metrics; DAE is inverted so higher scores indicate greater faithfulness.Source: Figure 9 · p. 17, Figure 10 · p. 17, F Abstractiveness · p. 17

E Ablations

• On XSum, lookahead maintains a higher BS-Fact score especially at t = 5–50 and consistently improves DAE across summary lengths versus greedy decoding.The authors report that greedy decoding's BS-Fact score dips between t = 5 and t = 40, while lookahead can prevent that dip.Source: E Ablations · pp. 16–17

F Abstractiveness

• BEAM+LOOKAHEAD+ABSTR slightly raises MINT while improving faithfulness in the human-annotation CNN/DM examples.Figure 8 describes this tradeoff on 200 CNN/DM examples.Source: F Abstractiveness · p. 17, Figure 8 · p. 17