PaperlayerView exact revision on arXiv

Source-linked AI summary

WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpus

Hongjing Qian, Yutao Zhu, Zhicheng Dou, Haoqi Gu, Xinyu Zhang, Zheng Liu, Ruofei Lai, Zhao Cao, Jian-Yun Nie, Ji-Rong Wen

arXiv:2304.04358v1cs.CLcs.AI

TL;DR

WEBBRAIN addresses the challenge of generating factual articles for queries when Web information is scattered and changes quickly. It mines Web evidence, constructs Wikipedia-based datasets, and introduces ReGen; the paper reports that ReGen outperforms all baselines in automatic and human evaluation.

  • Problem

    Web information is scattered and evolves quickly, making it challenging to find expected information, while manual editing curbs Wikipedia's coverage.

    Source: §1 · p. 1

  • Method

    The paper defines WEBBRAIN as generating short factual articles for queries from mined Web evidence and introduces ReGen with evidence-retrieval and generation strategies.

    Source: §1 · pp. 1–2

  • Results

    ReGen outperforms all baselines in both automatic and human evaluation.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The work is an early step toward automatically building Wikipedia pages for unseen topics.

    Source: I LIMITATION AND FUTURE WORK · p. 24

    The formulation permits sentences that rely only on PLM language knowledge and have no supporting evidence.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

In this paper, we introduce a new NLP task -- generating short factual articles with references for queries by mining supporting evidence from the Web. In this task, called WebBrain, the ultimate goal is to generate a fluent, informative, and factually-correct short article (e.g., a Wikipedia article) for a factual query unseen in Wikipedia. To enable experiments on WebBrain, we construct a large-scale dataset WebBrain-Raw by extracting English Wikipedia articles and their crawlable Wikipedia references. WebBrain-Raw is ten times larger than the previous biggest peer dataset, which can greatly benefit the research community. From WebBrain-Raw, we construct two task-specific datasets: WebBrain-R and WebBrain-G, which are used to train in-domain retriever and generator, respectively. Besides, we empirically analyze the performances of the current state-of-the-art NLP techniques on WebBrain and introduce a new framework ReGen, which enhances the generation factualness by improved evidence retrieval and task-specific pre-training for generation. Experiment results show that ReGen outperforms all baselines in both automatic and human evaluations.

1 INTRODUCTION

WEBBRAIN aims to generate short factual articles for queries by mining supporting evidence from the Web, addressing the difficulty of finding information across a scattered, rapidly evolving Web. The paper introduces WebBrain-Raw and ReGen, which targets factual correctness through retrieval and generation strategies.

1 INTRODUCTION

  • Wikipedia's reliance on manual editing curbs its coverage, while its articles offer credible answers with claims supported by references.The paper frames automatic article generation as a way to explore broader information needs.Source: §1 · p. 1
  • WEBBRAIN generates short factual articles for queries by mining Web evidence to support their claims.A potential target is the first section of a new Wiki page.Source: §1 · p. 1
  • WebBrain-Raw extracts English Wikipedia articles and their corresponding reference articles, and is about ten times larger than the previous biggest peer dataset.The paper identifies WikiSum as that peer dataset.Source: §1 · p. 2
  • ReGen outperforms all baselines in both automatic and human evaluation.The framework enhances factual correctness through topic-consistent evidence retrieval, citation marks during generation, and fact-oriented generation pre-training.Source: §1 · pp. 2–3

2 RELATED WORK

The related-work discussion situates WEBBRAIN among language-model generation, Wikipedia-based tasks, and retrieval-augmented generation. It distinguishes the task by requiring Web evidence mining for factual articles rather than relying only on model knowledge or prepared documents.

2 RELATED WORK

  • Pre-trained language models are typically pre-trained on large corpora and fine-tuned for downstream tasks; several paradigms support text generation.The passage names masked, left-to-right, prefix, and encoder-decoder language models.Source: §2 · p. 3
  • Prior Wikipedia-related tasks include knowledge-intensive NLP, multi-document summarization, Wiki-page construction, information updates, and citation-quality refinement.The cited examples include WikiSum, WikiWrite, Fruit, and Sphere.Source: §2 · p. 3
  • Unlike retrieval-augmented QA, which usually returns a span or sentence, WEBBRAIN mines Web evidence to generate a comprehensive short article.Unlike multi-document summarization, it must mine useful evidence before generation rather than summarize already-prepared documents.Source: §2 · p. 3
  • WEBBRAIN-Raw is significantly larger than existing Wikipedia datasets.Table 1 presents the datasets built on Wikipedia.Source: Table 1 · p. 4

3 TASK: WEBBRAIN

WEBBRAIN generates a short article for a query using prior knowledge that combines a language model with supporting evidence. The paper builds its data foundation from Wikipedia articles and references, then selects and filters reference material to create datasets for training retrievers and generators.

3.1 TASK DEFINITION

  • Given query q, the task generates a short article Tq grounded in prior knowledge Θ.For WEBBRAIN, Θ combines implicit PLM knowledge θ with explicit knowledge γ from evidence mined from an external Web corpus.Source: §3.1 · p. 4
  • The formulation expects each generated sentence to be supported by a subset of retrieved evidence, while allowing common-sense sentences to rely only on PLM knowledge.For such sentences, the corresponding evidence subset is empty.Source: §3.1 · p. 4

3.2 DATA COLLECTION

  • WEBBRAIN-Raw extracts Wikipedia pages and crawlable references to provide a large-scale data foundation for the task.The paper notes that earlier datasets did not provide full-size Wikipedia pages with their corresponding references.Source: §3.2 · p. 4
  • Reference cleaning removes short or non-English pages and filters invalid content, including HTML-only text, long sentences, tables, and URLs.The dataset also uses PST to select references whose terms overlap with the query and cited article sentence.Source: §3.2 · p. 5
  • For generation targets, the authors use the first Wiki-article section and select the reference passage with the highest PST value.Reference articles are split into overlapping sentence-based passages with a maximum length of 256 tokens.Source: §3.2 · p. 5
  • WebBrain-G provides five supporting passages per query, while WebBrain-R pairs supporting passages with four BM25-mined negative passages.The datasets are constructed to train the generator and retriever, respectively.Source: §3.2 · p. 5

4 FRAMEWORK: REGEN

ReGen retrieves supporting Web evidence with a sparse lexical retriever, then generates an article from the query and references using a FiD generator and sentence-reference warm-up.

4.1 RETRIEVER

  • SPLADE encodes text as sparse lexical representations and is selected for entity-centric queries for exact-entity matching, generalizability, and contextual query expansion.Source: §4.1 · p. 6
  • Training uses mined negative passages and in-batch negatives with a contrastive LCE loss.Source: §4.1 · p. 6
  • Asymmetric regularization keeps query representations semantically rich while making document indices sparser and more efficient, with low-importance tokens ignored.Source: §4.1 · p. 6
  • The retriever filters documents with dT(d1,di)>0.9 as topically irrelevant and removes later documents with distance <0.1 as redundant.Source: §4.1 · p. 6

4.2 GENERATOR

  • FiD encodes each query-reference sequence independently, then decodes over concatenated representations to fuse evidence into output text.Source: §4.2 · p. 7
  • The generator is trained by minimizing the negative log-likelihood of the ground-truth text.Source: §4.2 · p. 7
  • Warm-up trains on query-plus-single-reference inputs to generate corresponding sentences before full multi-reference generation, creating an easier-to-harder curriculum.Source: §4.2 · p. 7

5 EXPERIMENTS

ReGen achieves the best results across the reported metrics, while experiments show that retrieval quality and reference quantity shape generation quality. The discussion also compares ReGen with GPT-3 and finds stronger text-similarity and factualness results for ReGen.

5.1 SETTINGS

  • The evaluation uses n-gram overlap metrics, QAGS and Triple Score for factualness, and human ratings of fluency, informativeness, and faithfulness.The model comparisons include BART, GPT-2, vanilla FiD+BART, and ReGen; retrieval comparisons include BM25, DPR, and SPLADE.Source: §5.1 · pp. 7–8

5.2 OVERALL RESULTS

  • ReGen in both sizes achieves the best results on all reported metrics, and the warm-up strategy improves performance over FiD, especially for the larger model.The authors report gains over BART and GPT-2, particularly on QAGS and Triple Score.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 DISCUSSION

  • ReGen surpasses BM25, DPR (RetroMAE), and SPLADE on most end-to-end metrics and all reported retrieval metrics.Compared with oracle references, retrieval increases BLEU-1 but lowers the other generation metrics, underscoring retrieval quality as a challenge.Source: §5.3 · p. 8, Table 4 · p. 9
  • With zero to five retrieved references, BLEU and text length generally increase, while ROUGE and Triple Score fluctuate; ROUGE is best with three references.The authors attribute the pattern to noise in retrieved references and note that generating long, faithful text remains challenging even with references.Source: §5.3 · p. 9, Figure 2 · p. 9
  • With zero to five ground-truth references, BART and GPT-2 performance decreases, while FiD and ReGen improve on BLEU and Triple Score; ROUGE suggests more noise.Targets with more references are usually longer, and references provide more information for generation.Source: §5.3 · p. 9, Figure 3 · p. 10
  • On 100 test samples, ReGen outperforms GPT-3 (Prompt) by a large margin on BLEU, METEOR, and ROUGE, with a clear improvement on factualness metrics as well.GPT-3 uses a zero-shot Wikipedia-style prompt, while ReGen receives the query and retriever-provided evidence.Source: §5.3 · pp. 9–10, Figure 4 · p. 10

6 CONCLUSION

The conclusion presents WebBrain as a task for generating factual descriptions of queries from evidence mined from a large web corpus. It reports the construction of WebBrain-Raw, weaknesses in current techniques on factualness, and ReGen’s advantage over baselines in automatic and human evaluations.

6 CONCLUSION

  • The paper introduces WEBRAIN, constructs WebBrain-Raw from Wikipedia articles and references, and proposes ReGen, which outperforms all baselines on automatic and human evaluations.The authors report that current NLP techniques often fail to maintain factualness on WEBRAIN.Source: §6 · p. 10

A IMPLEMENTATION DETAILS

Implementation covers retriever and generator training as well as the construction of their task-specific datasets.

A IMPLEMENTATION DETAILS

  • The retriever is trained for five epochs from efficient-splade-V-large, while SPLADE and DPR baselines use their official settings on WebBrain-R.The setup specifies batch size 192, query and passage length limits of 16 and 256, and a learning rate of 2e-5.Source: A IMPLEMENTATION DETAILS · p. 18
  • WebBrain-R keeps the highest-PST passage per reference document, pairs selected passages with Wiki titles, and mines negative passages from BM25’s top 30–50 results.Four passages are randomly selected from those results for each query.Source: A IMPLEMENTATION DETAILS · p. 18
  • WebBrain-G uses each selected reference passage with the Wiki title as input and the page’s first section as the generation target.Source: A IMPLEMENTATION DETAILS · p. 18

B HUMAN EVALUATION

The human evaluation compares generated texts for fluency, informativeness, and faithfulness through blinded pairwise judgments. Reference mark-enhanced decoding improves refer scores, with less effect on other metrics, but remains an early inference-stage exploration.

B HUMAN EVALUATION

  • Human annotators compare model outputs pairwise for fluency, informativeness, and faithfulness, with model identities hidden and outputs shuffled.Table 5 reports the human annotation results; its caption identifies the first three pairs as base models.Source: B HUMAN EVALUATION · p. 18, Table 5 · p. 19

C REFERENCE MARK CORRECTION

  • Reference mark-enhanced decoding greatly improves refer scores but has less influence on other metrics.Figure 5 evaluates the strategy's influence; the authors say it labels references more accurately.Source: C REFERENCE MARK CORRECTION · pp. 18–19, Figure 5 · p. 20
  • The authors describe reference mark correction as an early exploration: it runs only at inference, and evaluating reference-mark accuracy remains unresolved.They leave training-stage use and evaluation of reference-mark accuracy for future work.Source: C REFERENCE MARK CORRECTION · p. 20

D MOTIVATION AND IMPACT OF REFERENCE PASSAGE SELECTION

WebBrain-R uses a manageable selected passage corpus for retriever training and evaluation, while WebBrain-G supplies relevant, oracle-like references for generator training. End-to-end results indicate that selected-set evaluation reflects performance on the full corpus, and ReGen's retriever exceeds BM25 on two reported metrics there.

D MOTIVATION AND IMPACT OF REFERENCE PASSAGE SELECTION

  • WebBrain-G selects relevant, oracle-like references to help the generator learn to extract useful information, although this filtering may not be practical in deployment.The authors also test end-to-end generation with retrieved evidence from the full reference passage set of about 204 million passages.Source: D MOTIVATION AND IMPACT OF REFERENCE… · p. 20
  • Using the full passage set, ReGen’s retriever surpasses BM25 on BLEU-1 and METEOR, generalizing better in the large-scale passage set and finding more informative passages.Source: D MOTIVATION AND IMPACT OF REFERENCE… · p. 21
  • Despite the full set's much larger size, retrievers show similar performance tendencies, so selected-set evaluation reflects performance on the full set.The selected corpus is kept at million-level scale to make retriever training-phase experiments efficient.Source: D MOTIVATION AND IMPACT OF REFERENCE… · pp. 20–21
  • Although ReGen's generator is trained with oracle passages, it generalizes to generation from retrieved passages.Source: D MOTIVATION AND IMPACT OF REFERENCE… · p. 21

E EVALUATION OF SEMANTIC SIMILARITY

Semantic-similarity evaluation finds ReGen (large) strongest among the compared models, with recall particularly high. In a separate GPT-3 comparison, Wikipedia-style prompting improves naturalness and factualness, while ReGen scores higher on reported text-similarity and factuality metrics.

E EVALUATION OF SEMANTIC SIMILARITY

  • BERTScore and BARTScore supplement n-gram overlap metrics to evaluate semantic-level text similarity.The evaluation reports BERTScore precision, recall, and F1; BARTScore is an average log-likelihood, so its values are negative.Source: E EVALUATION OF SEMANTIC SIMILARITY · pp. 21–22
  • ReGen (large) achieves the best results among models on semantic-similarity evaluation, with notably higher recall than other models.The authors say this is consistent with Table 3 and that the recall suggests more effective coverage of ground-truth content.Source: E EVALUATION OF SEMANTIC SIMILARITY · p. 22

F ADDITIONAL PERFORMANCE COMPARISON WITH GPT-3

G CASE STUDY

Case studies show that references can make generated text more faithful, while additional references can increase factualness but also introduce noise and hallucinations. The demonstration system handles entity and some natural-language queries, but heuristic full-page generation can produce section titles without a coherent taxonomy.

G CASE STUDY

  • With one reference, FiD and ReGen generate more faithful text than BART; GPT-3 can be fluent and informative yet produce mostly incorrect facts in that case.The case study also reports that ReGen better captures clues from the reference.Source: G CASE STUDY · pp. 22–23, Table 12 · p. 23
  • With more retrieved references, ReGen's outputs become longer and more factual, but irrelevant references and mixed information can introduce noise and hallucinations.The authors note that these issues leave room for improvement in existing methods.Source: G CASE STUDY · p. 23

H SYSTEM DEMONSTRATION

  • For an entity-style query, the demonstration generates relevant content, and reference mark-enhanced decoding makes source marks more accurate and text more readable.The example query is “Google Translate.”Source: H SYSTEM DEMONSTRATION · p. 23
  • Although trained on entity queries, ReGen generalizes to some natural-language queries with more specific information-seeking intent.For “how Google Translate work,” the generated text is closer to the entity's intrinsic functionality than for “Google Translate.”Source: H SYSTEM DEMONSTRATION · pp. 23–24
  • Heuristic full-page generation can introduce section titles that lack a reasonable taxonomy for the query.The example “translate tool” is broader than “Google Translate.”Source: H SYSTEM DEMONSTRATION · p. 24

I LIMITATION AND FUTURE WORK

The authors frame short-article generation as an early step toward automatically building Wikipedia pages for unseen topics, and identify limitations in retrieval, reference-mark accuracy, and article length.

I LIMITATION AND FUTURE WORK

  • Short factual articles are an early step toward automatically building Wikipedia pages for unseen topics.The study also collects a dataset, evaluates existing models, and proposes strategies.Source: I LIMITATION AND FUTURE WORK · p. 24
  • ReGen performs worse with retrieved evidence than with oracle references, leaving retrieval quality and generation from imperfect references as open problems.The authors identify both improving the retriever and generating better results from non-perfect references as future problems.Source: I LIMITATION AND FUTURE WORK · p. 24
  • With FiD as its backbone, ReGen has difficulty determining reference marks accurately, and existing metrics cannot measure that accuracy.The authors plan to design new attention mechanisms to facilitate reference-mark judgment.Source: I LIMITATION AND FUTURE WORK · p. 24
  • The study targets short articles, while generating the more information-rich full Wikipedia pages remains a future challenge.The authors describe long-article generation as another challenge for future work.Source: I LIMITATION AND FUTURE WORK · p. 24
Loading 2304.04358v1…