PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparison of Common Crawl News & GDELT

Ameir El Ouadi, David Beskow

arXiv:2610.00587v1cs.IR

TL;DR

Worldwide news data matters for technical research and for understanding people, places, organizations, and events, but the source coverage of news datasets can differ. This paper compares GDELT and CC-News across dataset characteristics; despite similar article volumes, their news sources overlap little.

  • Problem

    The paper examines how two widely used worldwide-news datasets differ in coverage and content, an important comparison because news data supports technical research and understanding global events.

    Source: §I · p. 1

  • Method

    The authors compare CC-News and GDELT using temporal, URL and domain, and language analyses of data collected from January through June 2023.

    Source: §III · p. 2

  • Results

    Despite similar article counts (103,932,961 CC-News and 96,411,230 GDELT), the datasets share only 4,626 domains, or 5.4% of news sources.

    Source: §III · p. 2

  • Takeaways & Limitations

    Researchers should weigh the datasets’ distinct characteristics against their study requirements and may use both to draw on their different strengths.

    Source: §IV · p. 3

    Topic-based classification of articles is left for future work to better analyze dataset composition.

    Source: §IV · p. 3

Abstract

from arXiv · show

The corpus of worldwide news is important for natural language processing, knowledge graphs, large language models, and other technical efforts. Additionally, this corpus is important for understanding the people, places, organizations, and events that interact in real-time every day. This paper compares two news datasets used for these tasks today, namely the Global Database of Events, Language, and Tone (GDELT) and Common Crawl News. Our research highlights the strengths and limitations of each dataset, analyzing their content and coverage. Notably, while GDELT relies on broadcasts, prints, and web news from across the globe, Common Crawl focuses on news sites from around the world gathered through web crawling. Our analysis revealed considerable differences in where the two datasets gather their news sources.

I. INTRODUCTION

Worldwide news data can inform research on people, places, and events, and this paper compares GDELT with Common Crawl News to assess their coverage and content. The datasets differ in how they collect news and what data they provide.

  • GDELT collects broadcast, print, and web news from across the globe in over 100 languages, with an archive dating to January 1979.Source: §I · p. 1
  • GDELT uses CAMEO coding to classify news events and structures metadata about people, locations, organizations, events, and images.It provides event and entity metadata rather than raw text; users must re-hydrate URLs to obtain the text.Source: §I · p. 1
  • Common Crawl stores crawled web pages, metadata, and text extracts; its CC-News stream, launched in 2016, contains articles from news sites worldwide.Source: §I · p. 1
  • The paper compares publicly available GDELT and CC-News, examining their coverage, content, strengths, and limitations.The comparison focuses on two widely used news datasets with different collection approaches.Source: §I · p. 1

II. LITERATURE REVIEW

The literature traces news-based computational journalism from early event-data projects through retrieval collections and automated event coding. It also describes Common Crawl as an accessible web corpus, while noting that crawls can contain undesirable content.

  • WEIS and COPDAB helped establish the use of news data in international politics research.WEIS was created by Charles McClelland, and Edward Azar’s COPDAB followed it.Source: §II · p. 1
  • TREC campaigns used newswire collections for ad-hoc retrieval, often with hundreds of thousands of documents from prominent news organizations.Source: §II · pp. 1–2
  • KEDS automated news-content analysis to produce event data, while ICEWS used KEDS coding for a crisis early-warning project; GDELT is a later open event-data repository.The text distinguishes GDELT’s open access from ICEWS, which was limited to official government use.Source: §II · p. 2
  • Common Crawl’s accessibility supports NLP, machine learning, and derived corpora; CC-News provides regularly published news crawls, which can contain undesirable content [16].Source: §II · p. 2

III. METHODOLOGY AND RESULTS

The study compares CC-News and GDELT using temporal, URL/domain, and language analyses. Their article counts were similar, but their source domains differed sharply, while eight of their top ten languages were shared.

  • The study collected both datasets for the stated six-month period and analyzed temporal patterns, URLs, domains, and languages.Source: §III · p. 2
  • 103,932,961 CC-News and 96,411,230 GDELT articles corresponded to only 4,626 shared domains, or 5.4% of sources.Source: §III · p. 2, Figure 2 · p. 2
  • One domain overlaps across the top ten most frequent domains, and six across the top 100; 17.11% of CC-News domains appear in GDELT versus 7.31% vice versa.Source: §III · p. 2, Table I · p. 3
  • GDELT’s top ten domains account for 5.9% and domains contributing ten or fewer articles for 46%, versus 7.7% and 28% in CC-News.Source: §III · p. 2
  • English accounts for 36% of GDELT articles and 44% of CC-News articles; eight of the top ten languages are shared.Source: §III · pp. 2–3
  • CC-News includes nontraditional sources such as Nike, contributing over 92,000 articles, and Reddit, as well as adult-content-related domains.Source: §III · p. 3

IV. CONCLUSION AND FUTURE WORK

The clearest differences are in news sources: GDELT spans traditional worldwide sources, while CC-News offers higher volume and a perspective on other web-based news. The paper urges researchers to match dataset choice to study needs, while identifying topic-level analysis and possible CC-Bot access constraints as future concerns.

IV. CONCLUSION AND FUTURE WORK

  • The authors frame the datasets as freely available sources of extensive data amid restrictions on many web archives.Those restrictions include limits on public access, technology, or copyright.Source: §IV · p. 3
  • The comparative analysis examines dataset size, diversity, news source quality, and language distribution.The authors present the work as offering insights for computational journalism and social science.Source: §IV · p. 3
  • GDELT spans traditional news sources worldwide, while CC-News offers higher volume and a perspective on other web-based news; their largest differences are in news sources.The conclusion identifies source differences as the largest contrast between the datasets.Source: §IV · p. 3
  • Researchers should weigh each dataset’s distinct characteristics against their study requirements, potentially using both to leverage their unique characteristics.The datasets offer different strengths and limitations.Source: §IV · p. 3
  • Future work will classify articles by topic to better analyze each dataset’s content composition.The authors also note that large language models may increase these datasets’ value while potentially limiting availability, for example as more news websites block CC-Bot.Source: §IV · p. 3
Loading 2610.00587v1…