PaperlayerView exact revision on arXiv

Source-linked AI summary

WiDiff: Extracting Changes from Wikidata's Edit History

Carolina Cortés, Lisa Ehrlinger, Lorena Etcheverry, Felix Naumann

arXiv:2609.08508v1cs.DB

TL;DR

Wikidata’s available change formats do not support analytical queries over its complete edit history. WiDiff extracts that history into a queryable dataset; its analysis reports that only 0.07% of rank promotions and 0.14% of deprecations were reverted.

  • Problem

    Wikidata’s available change formats do not support analytical queries over its complete edit history, despite its diverse and extensive edit activity.

    Source: §1 · p. 2

  • Method

    WiDiff extracts Wikidata’s complete edit history and stores the changes in a relational database with a unified interface for large-scale analytical queries.

    Source: §8 · p. 13

  • Results

    Only 0.07% of rank promotions and 0.14% of rank deprecations were reverted.

    Source: §7.2 · p. 13

  • Takeaways & Limitations

    The extracted changes from the June 2025 dump are publicly available as a dataset.

    Source: §8 · p. 13

    Qualifier and reference values lack stable cross-revision identifiers, so WiDiff captures only their CREATE and DELETE operations.

    Source: §4 · p. 5

Abstract

from arXiv · show

Knowledge graphs have become a key resource for integrating heterogeneous data and powering downstream tasks such as question answering, entity linking, and semantic search. They are built and maintained incrementally, either (i) fully automated, e.g., YAGO, (ii) semiautomatically with community oversight, e.g., DBpedia, or (iii) manually through collaborative editing, e.g., Wikidata. Understanding the evolution of knowledge graphs is essential as changes may reflect real-world updates, error corrections, or noise introduced by vandalism, all of which affect the reliability of downstream applications. Among openly available knowledge graphs, Wikidata is the most challenging case to study evolution, with over 120 million entities edited by humans and bots and an edit history spanning more than a decade. Although Wikidata exposes change data in various formats (e.g., periodic dumps and real-time event streams), none support analytical queries over the complete edit history. Therefore, we present WiDiff, a tool that extracts changes from Wikidata's complete edit history and provides a unified interface for large-scale analytical queries over it.

1 Introduction

Knowledge-graph changes can reflect real-world events, data-quality work, or vandalism, and unmanaged changes can hinder analysis and reuse. WiDiff addresses the lack of a maintained tool for large-scale queries over Wikidata’s complete edit history.

1 Introduction

  • Unmanaged knowledge-graph changes can introduce inconsistencies and contradictions that hinder analysis and reuse.Changes may reflect real-world events or corrections, as well as noise such as vandalism.Source: §1 · p. 1
  • Wikidata’s human- and bot-edited history exceeds 2 billion edits, but its available change formats do not support analytical queries over the complete history.As of August 2025, Wikidata represents more than 120 million items and receives between 300 and 1,100 edits per minute.Source: §1 · pp. 1–2
  • Earlier systems and datasets have limitations, and none provides a maintained, up-to-date tool or dataset for large-scale queries over all Wikidata edits.The cited approaches include a deprecated monitoring framework, an unreleased dataset, and an unavailable endpoint with limited demonstrated coverage.Source: §1 · p. 2
  • WiDiff is an open-source tool that extracts Wikidata’s complete edit history and offers a unified interface for large-scale analytical queries.The edit history can support data-quality assessment, studies of knowledge-graph evolution, and edit suggestions.Source: §1 · pp. 2–3

2 Background on Wikidata’s Data Structure

Wikidata organizes information as entity pages containing RDF statements, with annotations and ranks that provide context about sources, validity, and preferred values. Its data types are grouped in this work by their JSON types.

2 Background on Wikidata’s Data Structure

  • Wikidata pages correspond to items or properties and contain statements modeled as RDF triples with a subject, predicate, and object.Items may represent individuals or classes.Source: §2 · p. 3
  • References attach supporting sources to statements as sets of property-value pairs, while qualifiers add context as a property-value pair.A qualifier can specify the time constraining a statement’s temporal validity.Source: §2 · p. 3
  • Statement ranks—normal, preferred, or deprecated—preserve formerly valid statements and mark preferred values among alternatives.Ranks let Wikidata represent both current and historical aspects of the real world.Source: §2 · p. 3
  • This work groups Wikidata’s 18 data types by JSON type, yielding string, quantity, time, entity, and globecoordinate categories.For example, Wikidata’s quantity data type maps to the JSON type quantity, while geo-shape maps to string.Source: §2 · p. 3

3 Preliminaries

The paper distinguishes an editor’s action from the resulting entity version, the computed difference between versions, and an individual change over time. These terms describe different units of editing activity.

3 Preliminaries

  • An edit is a concrete action an editor can perform on a Wikidata entity, such as adding a statement.Source: §3 · p. 4
  • A revision is a timestamped version of a Wikidata entity created when a user submits an edit.Source: §3 · p. 4
  • A diff is the additions, deletions, and updates needed to transform one temporally consecutive revision of an entity into the next.Source: §3 · p. 4
  • A change is an event in which a statement value, rank, qualifier statement value, or reference statement value differs between two points in time.Source: §3 · p. 4

4 Change Data Model

WiDiff processes Wikidata’s historical XML dumps by comparing entity revisions and storing changes rather than full graph snapshots. Its extended change-cube model records revision metadata and represents changes at several statement granularities, with a specific tracking constraint for qualifiers and references.

4 Change Data Model

  • Only Wikidata’s XML dumps provide the complete edit history; each entity revision contains metadata and a JSON snapshot of the entity’s state.Source: §4 · p. 4
  • Because full graph snapshots are impractical at Wikidata’s scale, WiDiff stores changes to statements as a change log.The passage cites more than 120 million entities and over 300 edits per minute as scale factors.Source: §4 · p. 4
  • WiDiff extends the change cube to represent value, rank, qualifier-value, and reference-value changes, adding revision metadata, old and new values, data types, operation type, and granularity-specific fields.Source: §4 · p. 4, Figure 1 · p. 5
  • A change to a statement’s property is represented as deletion of the original and creation of a new property.Source: §4 · p. 5
  • Statement-value and rank changes are uniquely identified by revision_id, property_id, and value_id.Source: §4 · p. 5
  • Because qualifier and reference values lack stable identifiers across revisions, WiDiff captures only their CREATE and DELETE operations and hashes values to avoid duplicates.Two qualifier or reference values are considered distinct only when their hashes differ.Source: §4 · p. 5

5 WiDiff

WiDiff processes Wikidata history dumps by comparing entity snapshots and storing extracted changes in a relational database. Its configurable pipeline supports analyses across several change types and entity subsets.

5 WiDiff

  • WiDiff’s architecture covers dump input, parsing and change extraction, storage, and configuration.Figure 2 summarizes these stages, described in Sections 5.1–5.5.Source: §5 · pp. 5–6, Figure 2 · p. 6

5.1 Input Data

  • Input pages-meta-history files contain sequences of revisions, each pairing metadata with a full snapshot of the page’s state.For entity pages, snapshots include labels, descriptions, aliases, and statements with ranks, qualifiers, references, and sitelinks.Source: §5.1 · pp. 6–7

5.2 File Parsing

  • WiDiff compares consecutive revisions, incrementally parsing XML through page and output queues, worker processes, and a database writer.It processes QID entities and filters out other pages and Wikidata sandboxes.Source: §5.2 · p. 7

5.3 Change extraction

  • For statement changes, WiDiff classifies added and removed properties as creates and deletes, and compares shared properties to detect value and rank updates.It also compares consecutive revisions for label and description changes, recording create, delete, or update operations.Source: §5.3 · pp. 7–8
  • SHA-1 hashes identify qualifier values and reference contents for change detection; set differences identify additions and deletions.Reference-level hashes use the set of statements within each reference, so references with identical content receive the same hash; this handles exact duplicates.Source: §5.3 · pp. 8–9

5.4 Storage Layer

  • PostgreSQL loads and indexes in 155s, versus 178s to serialize tuples to RDF before QLever’s 171s loading and indexing; the overall difference is 2.25x.QLever compressed indexes and vocabularies to 3.5GB, while PostgreSQL occupied 6.4GB; the authors choose relational storage because the change log has a fixed schema and grows append-only.Source: §5.4 · p. 9
  • The storage schema uses separate tables for revisions and value, rank, qualifier, and reference changes.A fixed schema follows Wikidata’s statement structure rather than varying by entity or evolving over time.Source: §5.4 · p. 9

5.5 Configuration

  • Configuration sets processing language and parallelism, filters entity subsets, and stores those subsets in separate tables for focused queries.Users can exclude astronomical objects, scholarly articles, and entities below a configurable value-change threshold.Source: §5.5 · p. 10

6 Change Data

The authors applied WiDiff to the June 2025 Wikidata dump and report the scale and runtime of extraction. They also describe the resulting dataset after applying entity filters.

6 Change Data

  • 6.1 days of wall-clock time were required to process 120,794,819 entities and 2,312,348,608 revisions, yielding 4.43 TB of compressed extracted data.Average throughput was 59 entities and 1,126 revisions per second; peak memory for a single file reached 181.5 GB across parallel processes.Source: §6 · pp. 10–11
  • 84% of astronomical objects and 56% of scholarly articles were never updated; 89% and 91% of their value changes, respectively, were statement creations.For scholarly articles, 70% of revisions were made by bots; 36,137,153 entities had fewer than ten value changes.Source: §6 · p. 11
  • 31,057,469 entities with more than ten value changes, excluding scholarly articles and astronomical objects, span revisions from October 2012 to June 2025.This filtered dataset contains 743,418,656 revisions, 709,506,539 value changes, 616,278,002 rank changes, 1,877,386,505 reference statement changes, and 182,486,518 qualifier statement changes.Source: §6 · p. 11

7 Insights into Wikidata’s Change Activity

WiDiff’s change-level view reveals editing patterns that revision counts alone can miss, while rank changes often lack documented reasons. The section uses these data to characterize activity across entities and rank changes.

7.1 Distribution of Value Changes and Revisions

  • 23 value changes and 24 revisions per entity on average; 73% had fewer than 23 changes, while 67% had fewer than 24 revisions.Among 546,654 entities with a single revision, that revision contained an average of 13 value changes.Source: §7.1 · p. 11, Figure 3 · p. 12, Figure 4 · p. 13
  • The Call of the Wild had 14,302 revisions but changes to only 61 properties, while P747 accumulated 22,238 value changes.A revision can bundle multiple edits, so revision counts can understate editing activity.Source: §7.1 · pp. 11–12
  • Revision metadata, including comments, can identify likely imports; Universe’s “Import” comment followed Africa’s first revision by hours.Both entities’ revisions occurred on October 29, 2012; the comment suggests an automated import.Source: §7.1 · p. 12

7.2 Deprecation and Promotion of statements’ ranks

  • Only 35% of rank deprecations had a reason qualifier; among these, redirect (19%) and withdrawn identifier value (15%) were most common, followed by link rot (9%), unreliable source (5%), and conflation (4%).The passage identifies the latter reasons as link rot, an unreliable source, and conflation of distinct concepts.Source: §7.2 · p. 12
  • Only 4% of statements with a preferred rank had a reason qualifier; the most common reasons were most precise value (62%), most recent value (22%), and currently valid value (6%).Source: §7.2 · p. 13
  • Only 0.07% of rank promotions and 0.14% of deprecations were reverted; deprecations were 71% registered-user and 21% bot edits, while promotions were 58% bot and 40% registered-user edits.Source: §7.2 · p. 13

8 Conclusion

WiDiff extracts Wikidata’s complete edit history from historical dumps into a relational database for large-scale analytical queries, and the June 2025 extracted-change dataset is publicly available. The conclusion identifies current coverage limits and future extensions, including broader language and knowledge-graph support.

8 Conclusion

  • WiDiff extracts Wikidata’s complete edit history into a relational database and provides a unified interface for large-scale analytical queries.The dataset extracted from the June 2025 dump is publicly available.Source: §8 · p. 13
  • The current tool does not extract entity aliases or sitelinks and supports changes to labels and descriptions in only one language.Source: §8 · p. 13
  • A reverted edit is a later change that restores the value to its prior state within four weeks.The definition applies to changes to a statement value or rank.Source: §8 · p. 13
  • The authors identify multilingual label and description changes and extension to other knowledge graphs such as DBpedia as future work.Different history formats complicate extension; DBpedia’s four-month snapshots lose changes occurring between snapshots.Source: §8 · p. 14
Loading 2609.08508v1…