PaperlayerView exact revision on arXiv

Source-linked AI summary

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, Navdeep Jaitly

arXiv:2401.16380v1cs.CL

TL;DR

Web-data curation strategies are often undocumented and costly to evaluate, so this paper introduces WRAP, which rephrases web documents into different styles for language-model pre-training. The authors report that synthetic-data training achieved equivalent models with 5x less data or 3x less compute.

  • Problem

    Web-data curation strategies remain largely undocumented, while documenting practical improvements requires expensive repeated pre-training.

    Source: §1 · p. 1

  • Method

    WRAP uses an off-the-shelf LLM to rephrase web-corpus documents into different styles for language-model pre-training.

    Source: §1 · p. 2

  • Results

    Across 13 zero-shot tasks and 21 Pile domains, synthetic-data training used 5x less data or 3x less compute for equivalent models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Synthetic rephrases offer value when high-quality data is scarce and may support generalization to domains and styles underrepresented in pre-training.

    Source: §8 · p. 14

    Enforcing diversity in generated rephrases remains challenging.

    Source: §8 · p. 14

Abstract

from arXiv · show

Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web. In this work, we propose Web Rephrase Augmented Pre-training ($\textbf{WRAP}$) that uses an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as "like Wikipedia" or in "question-answer format" to jointly pre-train LLMs on real and synthetic rephrases. First, we show that using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training by $\sim3x$. At the same pre-training compute budget, it improves perplexity by more than 10% on average across different subsets of the Pile, and improves zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we investigate the impact of the re-phrasing style on the performance of the model, offering insights into how the composition of the training data can impact the performance of LLMs in OOD settings. Our gains are attributed to the fact that re-phrased synthetic data has higher utility than just real data because it (i) incorporates style diversity that closely reflects downstream evaluation style, and (ii) has higher 'quality' than web-scraped data.

1 Introduction

WRAP addresses the cost and data limits of web-based pre-training by rephrasing web documents into varied styles and training on real and synthetic text. The authors report that this approach can reduce the data or compute needed to train equivalent models and improve evaluation performance.

1 Introduction

  • Web data curation is costly to study, while scaling laws call for linearly increasing compute and data even as high-quality data is limited and long pre-training is prohibitively expensive.Existing synthetic-data generation can also be opaque and costly, requiring GPT-3.5 prompting for billions of tokens, and may create knowledge bias through targeted topic selection.Source: §1 · pp. 1–2
  • WRAP uses an off-the-shelf pre-trained LLM to rephrase web-corpus documents into different styles, then pre-trains on a mixture of real and synthetic data.The rephrasing model can be an open-source 1.8B/7B LLM rather than GPT3.5; rephrasing does not rely on the model as a knowledge bank.Source: §1 · p. 2, Figure 1 · p. 2
  • Rephrasing preserves web information and its natural diversity, avoiding reliance on generated knowledge that may introduce factual errors or data biases; style alone can produce downstream gains.The approach targets the cost and data-bias challenges of synthetic-data curation by using the web's naturally diverse content.Source: §1 · p. 2
  • On C4, synthetic-data pre-training trained equivalent models with 5x lesser data or 3x lesser compute across 13 zero-shot tasks and 21 Pile language-modeling domains.The authors also report that synthetic-data models outperformed TinyLlama models trained on 3 trillion tokens (10x data and compute) across several zero-shot Q/A tasks; a 350M model trained on 15% of C4 outperformed a 1.3B model trained on all of C4.Source: §1 · p. 2, Figure 1 · p. 2
  • WRAP models trained on combinations of real and synthetic rephrases reduced perplexity by ∼50% on the Pile.The paper also reports improved zero-shot question-answer accuracy, while Figure 1 describes zero-shot performance and weighted average perplexity across 21 Pile sub-domains.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Related work examines how compute, data quantity, and data quality shape language-model pre-training, alongside growing use of synthetic data for language and multimodal training. Prior findings show both benefits and risks from repeated or generated data.

Neural Scaling Laws for Language Models

  • Scaling-law studies link compute-optimal model and data sizes, while repeated-data research finds diminishing utility beyond four epochs and possible overfitting (Hoffmann et al., 2022; Muennighoff et al., 2023; Xue et al., 2023).Chinchilla scaling indicated a linear relationship between model size and required training data, and suggested earlier models such as Gopher were undertrained.Source: §2 · p. 3

Dataset Selection

  • Pre-training dataset selection uses heuristics, domain reweighting, deduplication, and filtering, but remains an active, high-impact, understudied area (Brown et al., 2020; Xie et al., 2023; Abbas et al., 2023).Prior approaches also filtered documents using textual features and developed methods for removing low-quality data in language and image-language datasets.Source: §2 · p. 3

Data Augmentation and synthetic data

3 WRAP: Web Rephrase Augmented Pretraining

WRAP uses the natural diversity of web articles and smaller instruction-tuned models to create high-quality paraphrases in varied styles, then pre-trains on both synthetic and real text.

3 WRAP: Web Rephrase Augmented Pretraining

  • WRAP leverages the natural diversity of web articles so smaller LLMs than GPT-3.5 can produce high-quality paraphrases of noisy, unstructured text, addressing generation cost and data-diversity challenges.Source: §3 · p. 4

3.1 Rephrasing the Web

  • To address loosely evidenced benefits of high-quality text and web’s lack of QA or conversational data, WRAP tests Easy, Medium, Hard, and Q/A rephrasing styles.Source: §3.1 · p. 4
  • A frozen Mistral-7B instruction-tuned model rephrases C4 documents; human feedback shaped the Medium prompt, and examples are capped at 300 tokens to limit information loss.The prompt was iteratively refined by comparing outputs from medium-sized LLMs with GPT-4.Source: §3.1 · p. 4
  • WRAP samples real and synthetic data in a 1:1 ratio so models learn from synthetic text while retaining exposure to noisy web text, including typos and linguistic errors.Source: §3.1 · p. 4

3.2 Implementation Details

  • The decoder-only transformer models span 128M, 350M, and 1.3B parameters; XL models are trained for 300k steps with a one-million-token batch size unless otherwise specified.The XL model has 24 layers, 16 attention heads, and a hidden dimension of 2048.Source: §3.2 · pp. 4–5
  • For a 1.3B LLM trained for 300B tokens, WRAP (C4 + QA-85B) outperforms C4 models trained on 2x real data in Pile perplexity.The comparison is reported in Figure 2.Source: Figure 2 · p. 5

4 Perplexity Evaluation

On average across Pile subsets, WRAP substantially lowers perplexity versus real-data-only training and reaches lower perplexity much earlier. Evaluation uses Pile domains because WRAP and C4-only training optimize different text distributions.

Evaluation setup

  • Models are compared on 21 Pile sub-domains rather than C4 because WRAP and C4-only training optimize different text distributions.The Pile subsets use the first 10,000 documents per domain; training on real and synthetic styles costs less than 1 perplexity on C4 validation.Source: §4 · p. 5

Data Complexity

  • On average across Pile subsets, WRAP improves perplexity by 50% over models trained on real data alone.Models trained for 150B tokens and even 350M models outperform full-C4 training for 300B tokens; on ArXiv and HackerNews, perplexity is nearly 3x lower.Source: §4 · p. 6

Learning Speed

  • At 10B tokens, WRAP’s average Pile perplexity is lower than C4 training’s after 15 checkpoints, suggesting a 15x pre-training speed-up.The passage defers learning-speed discussion to zero-shot tasks for more meaningful comparisons.Source: §4 · p. 6

5 Zero-shot Tasks

Across 13 zero-shot benchmarks, WRAP’s synthetic-plus-real training improves average performance on General Understanding tasks, while synthetic data helps pre-training faster but does not impart new knowledge. Results also show that gains vary by task and can diminish when real data is added.

5.1 Datasets

  • The evaluation covers 13 zero-shot benchmarks across general understanding and specialized knowledge, including reasoning, language comprehension, science, medicine, mathematics, and logic.Source: §5.1 · p. 6

5.2 Results

  • Comparisons use real-token counts because documents can be rephrased multiple times, making total corpus size less meaningful; Synthetic+Real experiments augment the same number of synthetic rephrases.Source: §5.2 · pp. 6–7
  • 49.4% vs. 47.4%: Synthetic+C4 outperforms the 85B-token real-C4 split on average across General Understanding tasks.The models trained on Synthetic+C4 scored 49.4% on average, compared with 47.4% for the real-C4 split.Source: §5.2 · p. 7, Table 1 · p. 7
  • 44.6% vs. 43.5%: Pythia (300B) exceeds C4 (85B), while synthetic data does not impart new knowledge and mainly helps pre-train faster.The paper attributes the difference to the larger dataset exposing the model to more knowledge.Source: §5.2 · p. 7, Table 2 · p. 7
  • Table 3 reports that adding real data improves performance when pre-training on Medium or Wikipedia-style paraphrases.Source: Table 3 · p. 8
  • RefinedWeb (320B) beats RefinedWeb (160B) by only 0.2%, while TinyLlama (1T tokens) performs comparably to WRAP with 85B raw web tokens.The authors describe these results as evidence that improvements from adding more real data eventually saturate.Source: §5.2 · p. 8
  • 44.0%: Synthetic (85B) scores higher than any other model on TruthfulQA; adding real C4 lowers Synthetic+C4 to 40.5%, down 4%.The authors suggest instruction-tuned models may correct misconceptions during rephrasing; they also report continued benefits from combining C4 and synthetic rephrases on HellaSwag and BoolQ.Source: §5.2 · p. 8

6 Analysis and Ablations

The ablations show that synthetic rephrases work best alongside real C4, while their benefits depend on style, rephraser quality, and evaluation domain. Rephrases preserve meaning while changing style.

6.1 Data Combination Analysis

  • Real C4 improves zero-shot performance and specialized-task results, while synthetic-only training raises perplexity on several Pile domains.Synthetic text lacks some special tokens and styles found in web scrapes, which particularly affects domains such as OWT, Hackernews, Philpapers, and Gutenberg.Source: §6.1 · pp. 8–9, Table 4 · p. 9
  • Multiple-style mixtures slightly improve average Pile perplexity but underperform C4 plus Q/A on zero-shot QA tasks.Q/A and Wikipedia rephrases help on some domains; Stackexchange benefits from Q/A-style synthetic data.Source: §6.1 · p. 9, Table 5 · p. 9, Figure 4 · p. 10

6.2 Method Ablations

  • Qwen-1.8B and Mistral-7B rephrases achieve lower perplexity than Vicuna-13B, while fine-tuned T5-base performs worse; all beat real C4 alone.The comparison trained 345M models for 30B tokens using data generated with the same prompt.Source: §6.2 · p. 9, Figure 5 · p. 11
  • Models trained on synonym-replacement or random-deletion augmentations perform significantly worse than models trained on real and synthetic data.The comparison used 350M-parameter models trained for 15B tokens, with a 1.5B-token pool that would otherwise be repeated around 10 times.Source: §6.2 · p. 10, Figure 6 · p. 11
  • An oracle choosing the best synthetic style per domain improves perplexity by 16%, although no single style is best across all domains.Matching synthetic-data style to the evaluation domain improves performance, but selecting the best style in advance is impractical.Source: §6.2 · p. 10, Figure 7 · p. 12
  • Real-synthetic pairs have higher cosine similarity than the tested baselines, indicating rephrases preserve meaning without adding information.The baselines include two random C4 samples, a continuation comparison, and two halves of the same sample.Source: §6.2 · p. 12, Figure 8 · p. 12

7 Limitations and Opportunities

The cost analysis argues that synthetic rephrasing can be economically viable even against real-data training, particularly as generation becomes more efficient and reusable. The remaining opportunity is qualified by an unresolved question about diversity in paraphrase-generated content.

7.1 Cost Analysis

  • WRAP applies to low-resource settings, where gathering more in-domain data is not an option, and data-rich settings such as Common Crawl, where synthetic data’s viability remains an open question.Source: §7.1 · p. 12
  • The 3-trillion-token TinyLlama model underperforms joint real/synthetic training and resembles real-only 300B-token models, suggesting limited longer-training gains at 350M/1.3B parameters, though larger models may benefit.Source: §7.1 · p. 13
  • Generating 85B tokens costs about 25K GPU hours; by comparison, 300B-token training costs about 6k GPU hours, while training a 13B model takes about 30K GPU hours.The generation estimate uses Mistral-7B on a single A100; the training estimate assumes 64 A100s.Source: §7.1 · p. 13
  • At 1.3B scale, Qwen rephrasing and projected inference optimizations can make synthetic-data pretraining cheaper than real-only training; at 13B scale, a 3-10x training-cost reduction can absorb generation overhead.Qwen-1.8B provides 3x higher token throughput, and speculative decoding and optimized inference suggest another 3-5x generation-cost improvement.Source: §7.1 · p. 13
  • Synthetic data generation is a one-time investment reusable across model scales and fully parallelizable on single GPUs, unlike training’s need for a large cluster with fast inter-node connectivity.Source: §7.1 · p. 13

7.2 Diversity of Synthetic Generations

  • Rephrasing aims to mitigate diversity problems in novel-content generation, but its impact on style and knowledge diversity remains unassessed; instruction-tuned AI-assisted writing may reduce content diversity.Prior work seeded generation with selected topics or scenarios (Li et al., 2023b;c).Source: §7.2 · p. 13

8 Conclusion … A.2.1 Pile Weighted Average Ratios

The conclusion presents synthetic rephrases as a way to improve data efficiency and generalization, while the appendices specify the training corpus and Pile perplexity evaluation data and weighting.

8 Conclusion

  • Synthetic data can encode desirable attributes such as fairness, bias, and style in training data, offering an alternative to adjusting the training algorithm for alignment.The paper frames its contribution as a guide to using synthetic styles in LLM pre-training.Source: §8 · p. 13
  • Synthetic rephrases can provide more value than repeating existing data when high-quality data is scarce and can support generalization to other domains and underrepresented styles.The conclusion identifies generator quality, the real-to-synthetic data balance, and diminishing returns across epochs as open design questions.Source: §8 · p. 14
  • Synthetic data generation remains costly and requires strong language models, while enforcing diversity is challenging.Using the web's natural diversity for rephrases limits new knowledge learning and improves training through higher-quality inputs; the paper reports potential efficiency gains in compute and data size.Source: §8 · p. 14

A.1 Training Dataset

  • C4 is the primary pre-training corpus: a curated English CommonCrawl dataset comprising over 170 billion tokens, selected for its size and quality.The passage notes C4's use in openly available pre-training corpora, including The Pile and RedPajama.Source: §A.1 · p. 21
  • Refined Web serves as a more stringently filtered web-data comparison; experiments use its first 3,050 files for 300B tokens and its first 1,650 files to account for multiple epochs.The comparison tests synthetic rephrases against high-quality web data reported to achieve performance similar to curated datasets (Penedo et al., 2023).Source: §A.1 · p. 21

A.2 Pile Perplexity Evaluation

  • Pile perplexity evaluation uses 20 subsets, excludes non-English Europarl, and takes 10,000 samples per subset split into documents of at most 1,024 tokens.Reported average perplexity is weighted across domains using the ratios in Table 7.Source: §A.2 · p. 21

A.2.1 Pile Weighted Average Ratios

  • The Table 7 weighting ratios come from the first 10,000 Pile validation documents and differ slightly from Gao et al. (2020), though most are similar.Source: §A.2.1 · p. 21

A.3 Zero-shot Evaluation Dataset · B Filtering Details for Synthetic Data

The zero-shot evaluation covers 13 benchmarks divided between specialized knowledge and general understanding; the synthetic-data appendix also describes filtering paraphrases with extraneous introductions. Together, these sections specify evaluation coverage and identify an output-cleaning issue.

A.3 Zero-shot Evaluation Dataset

  • The evaluation uses 13 zero-shot benchmarks across Specialized Knowledge and General Understanding.The two subsets assess domain-specific expertise and broader cognitive, language-understanding, and common-sense skills.Source: §A.3 · pp. 21–22
  • Specialized Knowledge includes ARC Challenge, SciQ, PubMedQA, and MathQA for science, biomedical, and mathematical reasoning.ARC Challenge uses difficult grade 3–9 science exam questions requiring higher-order reasoning (Clark et al., 2018); SciQ assesses scientific understanding (Johannes Welbl, 2017), PubMedQA biomedical understanding (Jin et al., 2019), and MathQA mathematical problem-solving (Amini et al., 2019).Source: §A.3 · p. 21
  • MMLU assesses expertise across specialized subjects spanning professional domains and academia (Hendrycks et al., 2021).Source: §A.3 · p. 22
  • General Understanding includes ARC Easy, BoolQ, Winogrande, and PIQA, testing easier science questions, reading comprehension, pronoun disambiguation, and practical physical common sense.ARC Easy questions are less challenging and require less advanced reasoning than ARC Challenge (Clark et al., 2018); BoolQ (Clark et al., 2019), Winogrande (ai2, 2019), and PIQA (Bisk et al., 2020) assess the other listed skills.Source: §A.3 · p. 22
  • HellaSwag tests coherent scenario completion, TruthfulQA factual truthfulness, and OpenBookQA factual knowledge and reasoning (Mihaylov et al., 2018).Source: §A.3 · p. 22
  • LogiQA-2 tests understanding and application of logical constructs and principles.Source: §A.3 · p. 22

B Filtering Details for Synthetic Data

  • Synthetic paraphrases can contain extraneous introductions such as “Here’s a paraphrase...” or keywords such as “high-quality English,” prompting the authors to develop a filtering method.Source: B Filtering Details for Synthetic Data · p. 23

B.1 Methodology … C.3 Syntactic Properties

The appendix describes a filter for unwanted paraphrase content and compares synthetic data with C4 and Pile data. Medium-style rephrases have higher reading level and syntactic complexity than QA-style rephrases, while semantic similarity and diversity vary by comparison.

B.1 Methodology

  • The unwanted-content filter splits text into sentences, checks the first sentence for delimiters, removes preceding unwanted content, and drops a paraphrase if flagged keywords remain without an applicable edit.The procedure checks text before delimiters such as a paragraph break or colon.Source: §B.1 · p. 23
  • Manual inspection found that fewer than 0.1% of sentences contained unwanted elements after filtering.Source: §B.1 · p. 23

C Properties of Synthetic Corpus

  • The synthetic-corpus analysis compares semantic similarity, syntactic complexity, and diversity to investigate information leakage, style capture, and data-quality attributes related to generalization.Source: C Properties of Synthetic Corpus · p. 23

C.1 Experimental Setup

  • Comparisons use up to 1,000 documents per dataset, paired synthetic and real C4 samples, and Pile test samples; quality statistics exclude outliers beyond two standard deviations.When a Pile subset has fewer than 1,000 samples, samples are split; distribution figures use KDEs over 1,000 values.Source: §C.1 · p. 23

C.2 Semantic Properties

  • Synthetic rephrases have similar average cosine similarity but lower spread than true MRPC rephrases.The paper denotes the real-paraphrase comparison as RealP and compares it with synthetic rephrases and sentence splits R1 and R2.Source: §C.2 · p. 24, Figure 9 · p. 24
  • 7-8 (C4), 10 (medium), and 6 (QA) are the reported Flesch–Kincaid reading levels; medium matches Wikipedia, while QA sentences are shorter after splitting into questions and answers.The passage attributes QA’s reduced reading level to question-and-answer splitting, which creates shorter sentences than the original text and medium-style rephrases.Source: §C.2 · p. 24, Figure 10 · p. 24
  • Medium-style diversity is similar to most Pile subsets, whereas QA has low type-token ratio, matching Ubuntu, GitHub, and math datasets with repetitive question-and-answer formatting.The passage connects lower metric values to QA’s repeated “Question” and “Answer” format.Source: Figure 10 · p. 24, §C.2 · p. 25

C.3 Syntactic Properties

  • Medium-style rephrases increase mean dependency-tree depth and mean dependency distance, while QA style reduces both measures and syntactic complexity.The paper treats tree depth and dependency distance as measures of syntactic difficulty.Source: §C.3 · p. 25, Figure 11 · p. 25

D Evaluation Metrics · E Additional Results for Smaller Model and Token Sizes

Evaluation uses macro token-level perplexity on encoded texts, calculated from the dataset’s accumulated loss and total token count. The metric is capped at 20 to stabilize it when losses are high.

D Evaluation Metrics

  • The evaluation metric is macro token-level perplexity computed over a batch of encoded texts.The supplied passages do not include the displayed equation or further results for smaller models and token sizes.Source: D Evaluation Metrics · p. 25
  • The calculation uses accumulated dataset loss L and total token count T.L is cumulative loss over all shifted logits and labels; T is the dataset’s total number of tokens.Source: D Evaluation Metrics · p. 25
  • The metric’s calculation uses the exponential function.The supplied passages identify exp as the exponential function but omit the displayed equation.Source: D Evaluation Metrics · p. 25
  • The value 20 is an upper limit used to stabilize the metric when loss values are high.This implementation detail applies in cases of high loss values.Source: D Evaluation Metrics · p. 25

E.1 Results for 350M Models Trained for 75B Tokens · E.2 Results for 1.3B Models Trained for 150B Tokens

At both smaller scales, QA rephrases improve task performance and lower Pile perplexity relative to C4-only training. Medium rephrases also improve results, while matching QA style provides further gains at 350M and yields similar findings at 1.3B.

E.1 Results for 350M Models Trained for 75B Tokens

  • A 350M GPT-2-medium model trained for 75B tokens has much lower Pile perplexity averaged across 21 domains than a model trained only on C4.Source: §E.1 · pp. 25–26, Figure 12 · p. 26
  • Adding QA rephrases at 350M improves general-understanding task performance by 1.5% and specialized-knowledge task performance by roughly 3%.The specialized-knowledge results cover tasks requiring domain knowledge such as science, medicine, mathematics, and logic.Source: §E.1 · p. 26, Table 8 · p. 26
  • At 350M, medium rephrases improve performance over C4-only training, while QA rephrases improve it further by better matching the target style.The passage attributes the medium-rephrase improvement to their higher quality.Source: §E.1 · p. 26

E.2 Results for 1.3B Models Trained for 150B Tokens

  • At 1.3B parameters trained for 150B tokens, QA rephrases lower average Pile perplexity across 20 domains versus C4-only training, even versus 1.3B C4-only models trained twice as long.The 150B-token training uses half as many steps.Source: §E.2 · p. 26
  • Adding QA rephrases at 1.3B improves specialized-knowledge task performance by 2% and general-understanding task performance by roughly 2.5%.Source: §E.2 · p. 26
  • At 1.3B, medium rephrases produce findings similar to those in the other small-scale experiments.Source: §E.2 · p. 26

F LLM Leaderboard Few-shot Results … Original

Models trained with synthetic rephrases retain improvements in few-shot evaluation, and the appendix documents the prompts and examples used to produce rephrased data.

F LLM Leaderboard Few-shot Results

  • Across six few-shot OpenLLMLeaderboard tasks, models trained for 300B and 350B tokens improved substantially on ARC-Challenge, matched zero-shot gains on TruthfulQA, were comparable on other datasets, and outperformed Falcon-1.3B and Pythia-1.4B.The models corresponded to roughly 85B and 100B unique C4 tokens, respectively.Source: F LLM Leaderboard Few-shot Results · pp. 26–27

G Rephrase Prompt Templates

  • Prompts for generating synthetic C4 versions with Mistral-7B are detailed, with slight variations for other frozen LLMs and no prompt for T5.Source: G Rephrase Prompt Templates · p. 29

Easy Style

  • Easy style prompts ask for paraphrases using very small vocabulary and extremely simple sentences that a toddler can understand.Source: Easy Style · p. 29

Hard Style

  • Hard style prompts request terse, abstruse paraphrases, replacing simple words and phrases with rare, complex language for erudite scholars.Source: Hard Style · p. 29

Medium Style

  • Medium style prompts ask for diverse, high-quality English paraphrases resembling sentences in Wikipedia.Source: Medium Style · p. 29

Q/A Style

  • Q/A style prompts convert narratives into conversational formats with multiple “Question:” and “Answer:” tags.Source: Q/A Style · p. 29

H Rephrase Examples

Original

  • C4 examples show Mistral-7B rephrasing source passages, including a recipe description expanded into a more detailed version that retains its ingredients and preparation steps.Source: Original · pp. 31–32

Related papers

Loading 2401.16380v1…