PaperlayerView exact revision on arXiv

Source-linked AI summary

Quality change: norm or exception? Measurement, Analysis and Detection of Quality Change in Wikipedia

Paramita Das, Bhanu Prakash Reddy Guda, Sasi Bhusan Seelaboyina, Soumya Sarkar, Animesh Mukherjee

arXiv:2111.01496v1cs.SIcs.IR

TL;DR

Automated approaches can miss article-specific dynamics in Wikipedia quality, motivating a study of quality over time and prediction of changes. The authors model quality changes with change-point detection and find both widespread stability and rapid shifts, alongside promising results from a content-based HYBRID method.

  • Problem

    Automated approaches can fail to capture dynamic quality changes in individual articles, leaving a gap between manual assessment and automated prediction.

    Source: §1 · p. 3

  • Method

    The authors study temporal quality evolution and model quality-change detection using features and standard change-point detection algorithms.

    Source: §1 · p. 3, §9 · p. 30

  • Results

    More than 50% of articles had no quality change over their lifespan; HYBRID with content-based features achieved covering 0.79, precision 0.68, and recall 0.69.

    Source: §6.3 · p. 22, §9 · p. 30

  • Takeaways & Limitations

    Talk-page discussions can contextualize quality switches, with editor conflict accompanying a demotion and consensus preceding a promotion in two cases.

    Source: §8.1 · p. 27

    The models do not explain their predictions as feature dimensions increase.

    Source: §8.4 · p. 29

Abstract

from arXiv · show

Wikipedia has been turned into an immensely popular crowd-sourced encyclopedia for information dissemination on numerous versatile topics in the form of subscription free content. It allows anyone to contribute so that the articles remain comprehensive and updated. For enrichment of content without compromising standards, the Wikipedia community enumerates a detailed set of guidelines, which should be followed. Based on these, articles are categorized into several quality classes by the Wikipedia editors with increasing adherence to guidelines. This quality assessment task by editors is laborious as well as demands platform expertise. As a first objective, in this paper, we study evolution of a Wikipedia article with respect to such quality scales. Our results show novel non-intuitive patterns emerging from this exploration. As a second objective we attempt to develop an automated data driven approach for the detection of the early signals influencing the quality change of articles. We posit this as a change point detection problem whereby we represent an article as a time series of consecutive revisions and encode every revision by a set of intuitive features. Finally, various change point detection algorithms are used to efficiently and accurately detect the future change points. We also perform various ablation studies to understand which group of features are most effective in identifying the change points. To the best of our knowledge, this is the first work that rigorously explores English Wikipedia article quality life cycle from the perspective of quality indicators and provides a novel unsupervised page level approach to detect quality switch, which can help in automatic content monitoring in Wikipedia thus contributing significantly to the CSCW community.

1 INTRODUCTION

The paper studies how Wikipedia article quality evolves and how those changes might be detected automatically. It reports varied quality trajectories and explores page-level change-point detection using article and collaboration features.

1 INTRODUCTION

  • Quality changes over time, while manual assessment is laborious and automated assessment can introduce noise and miss individual article dynamics.The paper positions temporal quality analysis as a way to understand Wikipedia and other peer-production systems.Source: §1 · pp. 2–3

1.2 Our contributions

  • The paper examines quality life cycles in a random sample of 30,000 English Wikipedia articles spanning more than 14 years, using editor-assigned quality labels.The authors contrast their use of explicit editor assessments with earlier work relying on predicted quality labels.Source: §1 · p. 3, §1.2 · p. 3

1.1 Research questions

  • The study asks how articles transition among quality states and which article characteristics can help forecast upcoming quality changes.It frames the challenge as bridging manual assessment and automated methods that may miss each article’s dynamic changes.Source: §1.1 · p. 3, §1 · p. 3

1.3 Key observations

  • Almost half of sampled articles remain in one quality class throughout their lifetimes, usually at low quality, while others improve, fluctuate, or cycle between classes.Some pages make sudden jumps between very low and very high quality classes.Source: §1.3 · p. 4
  • Article content attributes are the main aggregate quality signal, but talk-page activity combined with readability features predicts quality change points even better.The paper reports that these features outperform end-to-end quality prediction approaches such as ORES.Source: §1.3 · p. 4
  • The authors suggest their findings could contribute to Wikipedia’s ongoing effort to promote participatory machine learning.Source: §1.3 · p. 4

2 RELATED WORK

Prior work examines collaboration, motivation, and content quality in Wikipedia, while this paper focuses on quality’s temporal evolution. It treats article quality as a multivariate time series segmentation problem and applies offline change-point methods.

2 RELATED WORK

  • The related-work discussion places the study within broader research on collective production and the organizational factors that can shape article quality.Source: §2 · p. 4

2.1 Longitudinal analysis of Wikipedia

  • Longitudinal Wikipedia research has studied collaboration, conflict, editor roles, motivation, and how participation relates to article quality.Related work also considers anonymous editors, bots, and policies intended to retain contributors.Source: §2.1 · pp. 4–5

2.2 Content quality assessment in Wikipedia

  • Prior automatic quality assessment includes supervised feature engineering, sequence and distributional models, and network-based approaches; this paper adds longitudinal analysis of changing assessments.Source: §2.2 · p. 5

2.3 Change point detection

  • The paper treats quality change as offline signal segmentation, detecting change points after considering the entire article time series.Its multivariate revision features motivate methods that can identify multiple change points; it tests parametric BinSeg and PELT alongside nonparametric ECP.Source: §2.3 · p. 6

3 BACKGROUND AND DATA

The study uses Wikipedia’s editor-assigned quality classes and revision histories to examine article characteristics and quality evolution. It samples articles with talk pages, preprocesses article text, and groups classes for analysis.

3.1 Wikipedia article quality assessment scheme

  • Wikipedia’s editor-assigned classes range from FA at the highest quality to Stub at the lowest, and editors record assessments on article talk pages.The community uses review processes to award FA or GA status, while many articles miss peer review.Source: §3.1 · p. 6

3.2 Dataset Description

  • The dataset includes 30,000 English Wikipedia articles with corresponding talk pages and their full revision histories, sampled to approximately balance quality classes.All FA and A articles were included because they are limited in number; the authors report category-distribution similarity to English Wikipedia.Source: §3.2 · p. 7

3.3 Preprocessing

  • Preprocessing converts wiki markup into plain English-like text and uses parsed tokens to identify structural and content features such as headings, links, references, and infoboxes.Source: §3.3 · pp. 7–8

3.4 Merging of quality classes

  • Because quality criteria are qualitative and some classes are sparse, the authors merge classes into four ordered groups: FA, AGA, BC, and SS.The resulting order preserves Wikipedia’s predefined quality hierarchy.Source: §3.4 · p. 8

3.5 Basic characteristics of the quality classes

  • High-quality articles (FA, AGA) have more user views, revisions, and collaborators than low-quality SS articles.The section compares high- and low-quality article characteristics across topics, views, revisions, and collaborators.Source: §3.5 · pp. 8–9

4 TEMPORAL EVOLUTION OF ARTICLE QUALITY

The analysis classifies article quality trajectories as promotion, demotion, both, no change, or cyclic switches. It finds long promotion delays, rare demotions, many one-time assessments, and cycles that may reflect editorial conflict.

4.1 Only promotion

  • 42.97% of articles experienced one or more promotions, but SS→FA promotion took more than five years on average; AGA→FA was fastest.A page that moved through multiple classes was counted in each corresponding promotion subcategory.Source: §4.1 · p. 10

4.2 Only demotion

  • 221 articles underwent one or more demotions; FA→BC cases averaged more than two years between assessments, and manual inspection found long periods without quality contribution.The authors characterize demotion as uncommon in the dataset.Source: §4.2 · p. 10

4.3 Both promotion and demotion

  • 1,407 articles experienced both promotion and demotion during their revision histories.Source: §4.3 · p. 11

4.4 No change in quality

  • 51.73% of articles were evaluated only once; these 15,949 pages were mostly SS, and quality usually remained fixed, with changes rare.The authors describe the large one-time-assessment share as evidence that many articles may be overlooked by the current assessment framework.Source: §4.4 · p. 11, Figure 4 · p. 12, Figure 5 · p. 12

4.5 Cyclic switch of qualities

  • 1,286 articles exhibited cyclic switches, mostly of length three (1,258); average turnaround was 920.89 days, while 180 articles had cycles under 15 days.The distribution of articles by number of switches decays quickly.Source: §4.5 · p. 12, Figure 6 · p. 13
  • Cyclic-switch counts declined exponentially with switch length; common reversals involved BC/SS or BC/AGA, and such cycles may reflect editor conflicts, while bots accounted for 0.05%.Within one article, illustrated FA→FA cycles varied between three and eight state changes; black lines in Figure 7 mark assigned quality rather than switching behavior.Source: §4.5 · pp. 12–13, Figure 7 · p. 13

5 DETECTION OF QUALITY CHANGE POINTS

This section develops an unsupervised, feature-based approach to detecting article quality change points, motivated by the need for timely signals for Wikipedia editors. It represents each article as a monthly time series, examines quality indicators, and applies change-point detection and evaluation methods.

5.1 The quality indicators

  • Each revision is represented by 34 features spanning editor contributions, editing activity, and article content; editor experience and engagement are omitted.The authors omit experience and engagement to reduce noise in representing pages individually.Source: §5.1 · pp. 14–16

5.1.4 Correlation among the features.

  • Across the 34 indicators, features within the same category are positively correlated and features across categories are negatively correlated, usually with weak correlations.Most coefficients fall between -0.08 and 0.08, except for semantically close features within a category.Source: §5.1.4 · p. 16, Figure 8 · p. 17

5.1.5 Analysis of temporal changes in the features.

  • Around FA demotions, registered editors and main-page revisions decline while article length rises; the three features’ mean values shift at the change point.The analysis covers 54 pages that fell from FA to a lower quality class; the plotted values are means across those pages.Source: §5.1.5 · pp. 17–18, Figure 9 · p. 18

5.2.1 The time-series settings.

  • The method treats each article as a monthly time series, uses the latest revision in each month, and defines quality-change months as ground-truth change points.The representation has one d-dimensional feature vector per timestamp; added endpoints are for evaluation convenience, not actual change points.Source: §5.2.1 · p. 19

5.2.2 The CPD algorithms.

  • The algorithms minimize segment loss plus a penalty on change-point count, grouping timestamps with the same article quality into contiguous segments.The point at which quality changes belongs to the new segment.Source: §5.2.2 · p. 19
  • The study applies offline multivariate change-point algorithms, including Binary Segmentation, PELT, and ECP, to detect changes in the feature series.Binary Segmentation greedily splits the series; PELT minimizes the cost over possible change points; ECP estimates locations through hierarchical clustering.Source: §5.2.2 · pp. 19–20

5.3 Evaluation

  • Evaluation uses partition covering and change-point precision and recall, with a margin of error and one-to-one matching to avoid double-counting detections.Covering assesses predicted versus ground-truth partitions; the classification metrics treat a prediction as a true positive when it falls within the margin of a ground-truth point.Source: §5.3 · p. 20

6 EXPERIMENT AND RESULTS

The experiments evaluate unsupervised change-point detection on monthly article revision series and compare feature combinations and algorithms. The reported results favor the HYBRID method on content features overall, while deeper analyses identify useful feature combinations and classwise patterns.

6.1 Experimental setup

  • The experiment retains 14,872 of 30,826 articles with multiple ground-truth change points and represents each article by monthly latest revisions.Monthly sampling retains 95% of quality change points; the data are split 80:20 for training and testing.Source: §6.1 · pp. 21–22

6.2 Hyperparameter settings

  • PELT and Binary Segmentation use penalty values of 1, ECP performs best with min_size 5, and precision and recall allow a ±5-month margin.Hyperparameters are tuned on the training set, then metrics are aggregated over test articles.Source: §6.2 · p. 22

6.3 Key results

  • On content features, HYBRID achieves the best reported results: covering 0.79, precision 0.68, and recall 0.69.Among individual algorithms, PELT has the best three-metric compromise: covering 0.6, precision 0.37, and recall 0.61.Source: §6.3 · p. 22, Table 7 · p. 23

6.4 Additional experiments

  • For articles with at least three change points or a latest FA rating, ECP performs considerably better than on the full dataset across feature combinations.These subsets contain 1,982 and 3,190 articles, respectively.Source: §6.4 · pp. 22–23, Table 8 · p. 23

6.5 Ablation study

  • Masking any single feature produces no significant metric drop, while content-based features alone yield the strongest overall category-level performance.The individual-feature masking analysis reports the three largest changes for PELT.Source: §6.3 · p. 22, §6.5 · p. 23, Table 9 · p. 24
  • A selected G7 combination of activity and content features performs better in deeper analysis, especially talk-page activity measures and the “difficult words” readability feature.The cited activity measures include revision intervals and revision counts for talk and article pages.Source: §6.5 · p. 24
  • G3 and G5 content-feature groups with “difficult words” achieve the highest covering gain, a pattern not apparent in aggregate results.The authors also identify talk-page revision intervals and counts plus readability features as crucial in the deeper analysis.Source: §6.5 · p. 24
  • Across quality classes, G5 yields the best covering, while G7 wins for most performance metrics.Source: §6.5 · p. 24, Table 11 · p. 25

7 COMPARISON OF CPD METHODS WITH ORES

The paper compares its HYBRID change-point method with ORES by evaluating both on article revisions and applying the same margin around editor-marked change points. HYBRID outperforms ORES on both evaluated page sets.

7.1 The experiment

  • For ORES, each revision’s predicted quality class is compared with the previous revision; a class change is treated as a predicted change point.The comparison uses the same margin of error as the CPD predictions when scoring against editor-marked change points.Source: §7.1 · p. 25

7.2 Result

  • The comparison finds that HYBRID outperforms ORES on both pages with at least one ground-truth change point and pages whose latest class is FA.Source: §7.2 · p. 26, Table 12 · p. 26

8 DISCUSSION

The discussion finds that quality change is uncommon for many Wikipedia articles, yet some experience rapid shifts and cyclic switches. It connects these patterns to editor coordination, content features, and potential community-facing monitoring, while noting that model predictions lack explanations.

8.1 Further insights from study

  • 51.73% of articles had no subsequent quality change after their first shift, while cyclic switches sometimes revisited original states with low time lags.Lower-quality articles showed 64.61% higher stagnancy than higher-quality articles; demotions had three times lower turnaround time than promotions.Source: §8.1 · pp. 26–27
  • Rapid cyclic quality switches resemble Wikipedia edit wars, while editors’ coordination and communication are identified as influential in quality maintenance.The authors connect conflict and consensus among contributors and reviewers with diverse quality-life-cycle trends.Source: §8.1 · p. 27
  • Content features are generally most predictive, but combinations with activity features can perform competitively or better across quality classes.Difficult-word presence and talk-page revision activity, including elapsed time between revisions, are highlighted as important features.Source: §8.1 · p. 27
  • In two revisited cases, the approach detected quality-switch points within the accepted error margin, coinciding with actual changes primarily triggered by abrupt content changes.The examples involved conflict over demoting Catholic Church and consensus to promote Hurricane Hazel.Source: §8.1 · pp. 27–28

8.2 Design implications

  • The proposed pipeline is described as open to community adaptation through feature and threshold changes, with open-sourced data and models intended to support further development.The authors frame this as a socio-infrastructural service that would not require their explicit intervention in community governance.Source: §8.2 · p. 29
  • The authors envision community-facing alerts and participatory machine learning as ways to support collaborative practices and faster dispute resolution.They suggest alerts could facilitate resolution of disputes currently handled through Requests for Comments.Source: §8.2 · p. 29

8.3 Quality change: norm or exception?

  • More than 50% of articles experience no quality change over their lifespans, and quality changes are more frequent on pages with more views, edits, and collaborators.The discussion connects this long-tail pattern to concerns about preferential treatment and systemic bias in coverage.Source: §8.3 · p. 29

8.4 Limitations

  • A key limitation is that predictions from higher-dimensional feature sets lack explanation; the authors propose a community process and contextual-mistake database as future remedies.The database is intended to help the community understand misclassification reasons and support development of explainable models.Source: §8.4 · pp. 29–30

9 CONCLUSION

The paper examines quality trajectories in 30,000 Wikipedia articles and models quality changes as change points in revision time series. It reports that quality change is uncommon but sometimes rapid and cyclic, and proposes an alert system as a future direction.

9 CONCLUSION

  • More than 50% of articles showed no quality change over their lifespans, while some underwent multiple rapid shifts, including cyclic switches.The authors describe these cyclic changes as possible quality-switch wars.Source: §9 · p. 30
  • The authors modeled quality changes as change points using intuitive features and standard change point detection algorithms, reporting decent evaluation scores.They state that content-based features alone were sufficient to trigger the best results.Source: §9 · p. 30
  • The authors propose a full-fledged Wikipedia alert system to support more informed, proactive, and timely decisions about article quality changes.Future work includes multilingual extension, more complex models, and deployment of a monitoring system.Source: §9 · p. 30

A.1 Turn around time

Turn-around times vary across quality-change patterns and article transitions, with longer quality jumps associated with delayed assessment. Articles in the SS category with no quality change were assessed immediately after creation.

A.1 Turn around time

  • For articles with no quality change, turn-around-time statistics are similar across categories except SS pages, which were assessed immediately after creation.Source: §A.1 · p. 34, Figure 12 · p. 34
  • Longer quality jumps, such as SS-to-FA promotions or FA-to-BC demotions, show evidence of delayed quality assessment.Turn-around times are not consistent across articles for all quality-change patterns.Source: §A.1 · p. 33, Figure 12 · p. 34

A.2 Temporal patterns of the different features

For a sample of 54 pages, temporal trends across 34 features changed significantly at the point of quality shifts. The appendix figures plot contribution- and activity-based features separately from content-based features.

A.2 Temporal patterns of the different features

  • Significant temporal-trend changes were observed at quality-shift junctions across all 34 features in the 54-page sample.Figures 13 and 14 plot contribution- and activity-based features, and content-based features, respectively.Source: §A.2 · p. 34, Figure 13 · p. 35, Figure 14 · p. 36
  • Figure 13 plots temporal trends in contribution-based and activity-based features for the 54 sample articles.Source: Figure 13 · p. 35
  • Figure 14 plots temporal patterns in content-based features for the 54 sample articles.Source: Figure 14 · p. 36
Loading 2111.01496v1…