PaperlayerView exact revision on arXiv

Source-linked AI summary

"We Need a Woman in Music": Exploring Wikipedia's Values on Article Priority

Mo Houtti, Isaac Johnson, Joel Cepeda, Soumya Khandelwal, Aviral Bhatnagar, Loren Terveen

arXiv:2208.08426v1cs.CYcs.HC

TL;DR

Wikipedia has more article-improvement work than available editor effort, while recommender systems have often prioritized individual interests rather than broader community values. This paper analyzes Vital-article discussions as evidence of encyclopedia priorities and examines how prioritization relates to global content balance. It finds that priority reflects both article importance and effects on Wikipedia’s overall composition, including gender and geographical balance.

  • Problem

    Wikipedia has more article-improvement work than available editor effort, and recommender systems often prioritize individual interests rather than the encyclopedia’s broader values.

    Source: §1 · pp. 1–2

  • Method

    The authors analyze Vital-article discussions, compare their findings with List of articles every Wikipedia should have, and examine prioritization methods in relation to gender and geographical balance.

    Source: §3.1 · p. 6, §4.2 · p. 8, §9 · p. 15

  • Results

    Article priority reflects both inherent importance and effects on Wikipedia’s global composition, including gender and geographical balance.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper argues that recommender-system evaluation should consider the global list of recommendations and the encyclopedia’s values, not only individual interests.

    Source: §1 · p. 3, §14 · p. 26

    The qualitative participant population is not representative, limiting claims about the relative weights of the identified criteria.

    Source: §13.5 · p. 25

Abstract

from arXiv · show

Wikipedia -- like most peer production communities -- suffers from a basic problem: the amount of work that needs to be done (articles to be created and improved) exceeds the available resources (editor effort). Recommender systems have been deployed to address this problem, but they have tended to recommend work tasks that match individuals' personal interests, ignoring more global community values. In English Wikipedia, discussion about Vital articles constitutes a proxy for community values about the types of articles that are most important, and should therefore be prioritized for improvement. We first analyzed these discussions, finding that an article's priority is considered a function of 1) its inherent importance and 2) its effects on Wikipedia's global composition. One important example of the second consideration is balance, including along the dimensions of gender and geography. We then conducted a quantitative analysis evaluating how four different article prioritization methods -- two from prior research -- would affect Wikipedia's overall balance on these two dimensions; we found significant differences among the methods. We discuss the implications of our results, including particularly how they can guide the design of recommender systems that take into account community values, not just individuals' interests.

1 INTRODUCTION

Wikipedia must allocate scarce editor effort across millions of articles, while existing recommender systems often prioritize individual interests rather than the encyclopedia’s broader values. The paper uses Vital-article discussions to identify those values and compares prioritization methods by their implications for Wikipedia’s global content balance.

1 INTRODUCTION

  • Editor effort is scarce relative to Wikipedia’s work: as of July 2020, only 5,822 of more than 6 million articles had the highest quality rating.The paper frames this as an allocation problem across potential article-improvement tasks.Source: §1 · pp. 1–2
  • Recommender systems can direct editor effort, but personalized recommendations foreground editors’ interests, which may differ from reader demand and encyclopedia-wide values.The paper argues that prioritization should also account for what volunteer editors will actually want to follow.Source: §1 · p. 2
  • The study examines Vital-article discussions as a way to identify encyclopedia values, drawing on experienced editors and a smaller analysis of the related List of articles every Wikipedia should have project.It then compares four prioritization methods, including PageRank, pageviews, edit count, and Vital articles, for their effects on gender and geographical balance.Source: §1 · pp. 2–3
  • Article priority reflects both inherent importance and effects on Wikipedia’s global composition, with gender and geographical balance among the global criteria.The findings motivate evaluating recommendation lists as a whole rather than judging each recommendation set in isolation.Source: §1 · p. 3

2 RELATED WORK

Prior work has used recommender systems to route editor effort, but has not focused on directing Wikipedians toward content prioritized by the encyclopedia. This paper draws on multi-stakeholder recommendation and treats the encyclopedia as a stakeholder whose values can be studied through experienced editors’ work.

2 RELATED WORK

  • SuggestBot’s personalized recommendations based on predicted interests increased editor content output four times versus random recommendations; Wulczyn et al.’s personalized recommendations doubled engagement, while no research had focused on directing effort toward encyclopedia-priority content.Warncke-Wang et al. examined alignment between content quality and readership, not routing editing tasks using reader demand.Source: §2 · pp. 3–5
  • The multi-stakeholder framework distinguishes consumers, providers, and the system, corresponding on Wikipedia to readers, editors, and the encyclopedia.The paper focuses on the encyclopedia as a stakeholder in task-routing design.Source: §2 · p. 4
  • Experienced Wikipedians’ concern for the encyclopedia as a whole motivates using Vital-article discussions as a proxy for its article-priority values [26].The authors describe this as a useful but imperfect source for understanding values beyond individual editors’ interests.Source: §2 · p. 4
  • PageRank is a common prior method for measuring article importance, appearing in 14 of more than 50 papers in an informal review.Prior examples use it to rank papers, authors, topics, universities, and historical figures.Source: §2 · pp. 4–5
  • The quantitative analysis compares PageRank and pageviews with Vital articles and edit counts, examining gender and geographical balance and possible effects on quality-content composition.The authors also consider how editor demographics may shape these comparisons.Source: §2 · p. 5

3 BACKGROUND: VITAL ARTICLES

Vital articles is a volunteer project that ranks subjects for improvement and uses talk-page discussion to establish inclusion. Its participants are highly experienced but not representative of Wikipedia editors overall.

3 BACKGROUND: VITAL ARTICLES

  • Editors describe Vital articles as a way to establish a hierarchy and identify Wikipedia-wide articles needing attention, reflecting the scarcity of effort to improve them.The list is intended specifically for English Wikipedia, unlike the language-neutral List of articles every Wikipedia should have.Source: §3 · pp. 5–6
  • The paper has no clear empirical measure of Vital articles’ actual impact on English Wikipedia, and states that measuring it is beyond scope.Editors generally considered the project useful, but that judgment is not an empirical impact measure.Source: §3 · p. 6
  • Vital articles organizes subjects into five importance levels, from the 10 highest-priority articles at level 1 to the top 50,000 at level 5.Importance is defined in this project as the desirability of bringing an article up to Featured Article status; articles within a level are not ranked against one another.Source: §3 · p. 6
  • Talk-page discussion and consensus govern proposed additions or removals, making the discussions a source of evidence about the values editors invoke when prioritizing articles.The project’s home page says articles should not be added or removed without talk-page consensus.Source: §3 · p. 6

3.1 Participant Demographics

  • Vital-article discussants are highly experienced but not representative; the authors therefore do not treat the amount of discussion as evidence of a value’s importance.Among 794 identified participants, the median English Wikipedia edit count was 6,419, compared with 2 for all registered accounts.Source: §3.1 · p. 6
  • Only 7.1% of Vital-article participants reporting gender identified as women, versus 12.9% of Wikipedia editors overall, but reporting rates differed sharply.Forty percent of Vital-article participants reported gender, compared with 2% of Wikipedians overall; the authors note men may be more likely to report gender.Source: §3.1 · p. 7
  • List of articles every Wikipedia should have provides a comparison group of experienced editors with a different project and language-edition composition.The authors use its discussions to substantiate values found in Vital articles and assess whether they extend beyond one project’s membership.Source: §3.1 · p. 7

4 METHODS — RQ1

The authors use thematic analysis of Vital articles discussions, with a List of articles every Wikipedia should have analysis as a robustness check, to identify criteria for prioritizing articles. Those criteria concern both an article’s inherent importance and its effect on the composition of quality content across Wikipedia.

5 RESULTS — RQ1

  • Editors’ prioritization criteria fall into inherent-importance and global-composition categories, and systematic analysis surfaced gender balance among the criteria.The analysis also found criteria beyond those in the Vital articles FAQ.Source: §5 · p. 8

5.1 Importance Criteria

  • Everyday relevance and cultural significance support inherent importance, from topics such as Math and Sleep to Michael Jackson, the Sun, and Sport.The paper defines importance criteria as arguments about an article’s inherent worth or value.Source: §5.1 · p. 8, §5.1.1 · p. 9, §5.1.2 · p. 9

5.1.3 Historical Significance.; 5.1.4 Enduring Significance.

  • Historical significance and enduring significance are distinct: editors value historical consequence while also seeking lasting significance and resisting recency bias.The coronavirus pandemic was accepted based on expected long-term effects on everyday life, without relying on historical significance.Source: §5.1.4 · p. 10, §5.1.3 · p. 10

5.1.5 Breadth.

  • Broader articles are considered more important than narrower, more specific articles.Editors used breadth both to argue for a different Vital articles level and to prefer broader coverage over a narrower topic.Source: §5.1.5 · p. 10

5.2 Global Criteria; 5.2.1 Balance.

  • Global criteria require judging articles in the context of the list as a whole; balance concerns especially address geographic and gender disparities.Editors cited Western-centric coverage and proposed adding a female-dominated activity to balance Sport.Source: §5.2 · p. 10, §5.2.1 · pp. 10–11

5.2.1 Balance.; 5.2.2 Non-redundancy.

  • Editors argue against redundant articles when their important content is already covered, while completeness supports covering important topic areas equitably.The paper distinguishes completeness—ensuring an area is covered—from balance between areas.Source: §5.2.2 · p. 11, §5.2.3 · p. 11

5.3 Criteria in Conflict

  • Editors weigh competing criteria in discussion, and consensus on proposals reflects collectively judged value despite disagreement; editor interest was not used to justify priority.The Michael Jackson discussion combined cultural, enduring, historical, and balance arguments.Source: §5.3 · pp. 11–12

5.4 Results of Robustness Check

  • The LoA analysis found the same criteria, supporting their relevance beyond Vital articles, while geography received greater emphasis in its all-language-edition context.The gender-balance concern also appeared, including an editor’s account of overlooking Menstruation amid Wikipedia’s editor demographics.Source: §5.4 · p. 12

6 PRIORITIZATION METHODS

The paper compares four ways to prioritize English Wikipedia articles for editor effort: PageRank, pageviews, edit count, and Vital articles (VA).

6 PRIORITIZATION METHODS

  • The study compares PageRank, pageviews, edit count, and VA as four possible ways to prioritize articles for recommender systems directing editor effort.PageRank and pageviews approximate article importance, edit count reflects past edits, and VA serves as a proxy for the encyclopedia’s values as implemented by Wikipedians.Source: §6 · p. 12

6.1 PageRank

  • PageRank is computed over links between English Wikipedia articles as of July 31, 2020, excluding template links.The authors use PySpark’s PageRank implementation and exclude templates to avoid bias toward articles automatically linked from many articles.Source: §6.1 · p. 12

6.2 Pageviews

  • Pageviews ranks articles by total user pageviews from August 2019 through July 2020.Source: §6.2 · p. 13

6.3 Edit count

  • Edit count ranks articles using unique edits by registered non-bot users as of July 2020.The authors exclude bots and unregistered users to focus on where Wikipedians’ efforts have historically been directed.Source: §6.3 · p. 13

6.4 Vital articles

  • The VA dataset contains 44,118 unique articles as of July 31, 2020, with each article assigned only to its highest VA level.Although higher-level articles also appear in lower levels, the analysis assigns each article to just one level to distinguish levels.Source: §6.4 · p. 13

7 METHODS — RQ2

The authors use nDCG to assess how strongly each prioritization method represents topics organized through WikiProjects, then report the top ten projects for each method.

7 METHODS — RQ2

  • For each WikiProject with at least 100 articles, the authors calculate DCG using article relevance at each rank, discounted by log2(i + 1).An article is relevant when it belongs to the WikiProject.Source: §7 · p. 13
  • They normalize DCG by ideal DCG, defined as the score if the top p articles all belonged to the WikiProject.The resulting nDCG ranges from 0 to 1.Source: §7 · p. 13
  • For VA, articles in the same level are tied, and a rank’s relevance reflects the proportion of articles at that level belonging to the WikiProject.Source: §7 · p. 13
  • The analysis reports the top 10 WikiProjects for each of VA, PageRank, pageviews, and edit count; higher nDCG means stronger representation near the top of a ranking.The table presents the top 10 WikiProjects by nDCG for each prioritization method.Source: §7 · pp. 13–14, Table 2 · p. 14

8 RESULTS — RQ2

The four prioritization methods surface different topic patterns: VA emphasizes core topics, while pageviews and PageRank show distinct topic skews, and edit count spans varied topics but favors narrower ones.

8 RESULTS — RQ2

  • VA’s best-represented WikiProjects cover core topics such as Anthropology, Philosophy, and Science, indicating broad and diverse topic representation.Source: §8 · p. 14
  • Edit count’s leading WikiProjects span varied areas, but narrower topics such as Rock music rank highly instead of broader Music.The authors conclude that editor focus is not centered on topics as broad as might be preferable for the encyclopedia.Source: §8 · p. 14
  • Pageviews strongly represents people and popular culture, and biographies in its ranking include women more highly than under the other prioritization methods.WikiProject:Women ranks above WikiProject:Biography among the best-represented projects.Source: §8 · p. 14
  • PageRank leans heavily toward countries and cities, possibly because linking conventions give countries out-sized PageRank scores.The authors suggest that town articles’ bot-generated links to countries and biographies’ links to birth cities may contribute.Source: §8 · p. 14

9 METHODS — RQ3

The authors compare gender and geographical composition across prioritization methods by dividing their top 50,000 articles into VA-sized levels and classifying biographies and article geography.

9 METHODS — RQ3

  • The top 50,000 articles for each method are divided into five levels of 10, 100, 1,000, 10,000, and 50,000 articles, following VA’s exponentially growing levels.The authors use this scheme because rank differences are treated as less meaningful farther down a list, and it allows comparison with VA.Source: §9 · p. 15
  • Gender breakdowns are calculated for biographies using Wikidata, with statistics limited to cisgender men and women.Each method had fewer than 70 biographies of gender-fluid, non-binary, and transgender individuals, too few for statistically meaningful trends in representation.Source: §9 · p. 15
  • Geographical breakdowns use Wikidata country data and Wikimedia Foundation Global North/South classifications; articles without coordinates or spanning both regions are region-neutral.Source: §9 · p. 15

10 RESULTS — RQ3

The prioritization methods differ in how they represent women and geographic regions. Vital Articles (VA) favors region-neutral and Global South content, while the metric-based rankings vary across these dimensions.

10.2 Geographical Representation

  • VA has the most region-neutral content and, among non-region-neutral articles, the strongest Global South representation; edit count leads metric-based rankings on both measures.VA is 100% region-neutral at level 1; its level-5 share is 51.7%, close to Wikipedia’s 50.7%. PageRank is least region-neutral but exceeds pageviews in Global South representation at levels 1–4.Source: §10.2 · p. 16

10.1 Gender Representation

  • Pageviews leads overall in women’s representation; edit count and pageviews exceed the platform baseline, while VA and PageRank fall below, with VA at just 8.9% women at level 3.Edit count reaches 32.1% women at level 2, while pageviews’ three level-1 biographies are all men who had recently been in the news.Source: §10.1 · p. 16, Figure 2 · p. 17

11 METHODS — RQ4

The authors project how different prioritization methods could change the composition of Featured articles over five years. Their projection assumes the recent rate of Featured-article additions continues and that articles are upgraded in ranking order.

11 METHODS — RQ4

  • Featured articles are Wikipedia’s highest quality rating, and their composition matters because the Main Page features one each day.The rating is based on strict criteria; as of July 2020, 5,822 articles had reached it, and the Main Page receives about 15 million hits per day.Source: §11 · p. 17
  • The projection assumes the 1,257 Featured articles added from July 2015 to July 2020 will be matched over the next five years, through 2025.Source: §11 · p. 17
  • For each method, articles are grouped into levels and upgraded to Featured status from higher to lower levels, using the same gender and geographical classifications as before.Source: §11 · p. 18

12 RESULTS — RQ4

The five-year projection produces different changes in Featured-article composition across prioritization methods. Pageviews has the largest positive effect on women’s representation, while VA uniquely increases region-neutral content.

12.1 Changes to Gender Composition

  • Pageviews raises women’s share of Featured-article biographies from 15.4% to 21.4%; edit count raises it to 16.4%, while PageRank and VA lower it to 15.1% and 14.7%.Source: §12.1 · p. 18, Figure 4 · p. 19
  • VA’s negative effect on women’s representation would begin to reverse only at level 5, which would take approximately four decades to reach at the current rate.The authors consider it unlikely that higher levels would remain static long enough for level 5 to become necessary for prioritization.Source: §12.1 · p. 19

12.2 Changes to Geographical Composition

  • All methods modestly increase Global South representation; PageRank raises it from 3.9% to 5.7% but reduces region-neutral articles, while VA raises region-neutral content to 63.1% and is the only method to increase it.VA’s Global South share is 4.4%, and its Global North representation is lower than under the other methods.Source: §12.2 · pp. 19–20, Figure 5 · p. 20

13 DISCUSSION

The discussion argues that task-routing should account for both editors’ individual interests and the encyclopedia’s values. It connects article priority to global content composition and examines how proxy metrics, participant demographics, and study limitations shape that goal.

13.1 Task-Routing on Wikipedia Needs to be Multi-Stakeholder-Focused

  • Editors’ individual interests can conflict with collective goals such as balance, non-redundancy, and completeness, so task-routing should take a multi-stakeholder approach even to serve editors’ broader interests.The authors do not identify which specific multi-stakeholder algorithmic approach is best for Wikipedia.Source: §13.1 · p. 21

13.2 Global Criteria Reveal the Importance of Aggregate Diversity

  • Because priority partly depends on effects on the global composition of quality content, recommender evaluation should assess the combined recommendations across editors, not each user’s list alone.Aggregate diversity concerns the diversity of recommendations presented to all users; similar post-processing approaches may preserve recommendation accuracy while raising aggregate diversity.Source: §13.2 · pp. 21–22

13.3 Evaluating Proxy Metrics for Operationalizing Importance

  • PageRank and pageviews fall short as importance proxies: PageRank would reduce gender balance, while pageviews would do least to improve geographical balance.Pageviews most heavily surfaces popular-culture articles; PageRank reflects Wikipedia’s linking conventions, including location links, rather than authoritativeness in a meaningful sense.Source: §13.3 · p. 22, §13.3.1 · p. 22, §13.3.2 · p. 23

13.3.1 Pageviews as Proxy for Importance.

  • Pageviews may reflect access patterns and popular-culture interest rather than readers’ priorities for content quality, so equal views of two articles need not indicate equal importance to readers.Google search and Main Page exposure can affect pageviews; the authors question whether this metric meaningfully approximates reader values.Source: §13.3.1 · pp. 22–23

13.3.2 PageRank as Proxy for Importance.

  • PageRank’s location-heavy results arise partly because Wikipedia articles commonly link towns to countries and biographies to birthplaces, while editor-driven metrics may also reflect feasibility factors such as source availability.The authors argue that recommendations should target low-quality items needing work, where implicit importance data may be sparse; explicit assessments are time-consuming to crowd-source.Source: §13.3.2 · p. 23, §13.3.3 · p. 23

13.3.3 Broader Challenges in Operationalizing Importance.

  • VA prioritizes enduring significance through a slow manual process, which may constrain recommender adaptability and impose time-based limits on what can be prioritized.The authors caution that VA’s avoidance of recency bias may reflect this process constraint rather than encyclopedia values.Source: §13.3.3 · pp. 23–24

13.4.1 Gender Bias.

  • Using Vital Articles (VA) to direct effort may reverse the trend toward greater gender parity; only 7.1% of identified VA discussion participants listing a gender identified as women, versus 12.9% of English Wikipedia editors overall.The authors suggest proposal-creation requirements may exacerbate self-focus bias, despite participants’ stated support for gender balance.Source: §13.4.1 · p. 24

13.4.2 Geographical Bias.

  • VA represents the Global South more than alternative methods despite editors’ concentration in the Global North, while the authors’ proposed historical-power explanation remains tentative.The qualitative sample is not representative of the full community, preventing claims about the relative weights of prioritization criteria; quantitative analyses also assume editors follow methods perfectly.Source: §13.4.2 · pp. 24–25, §13.5 · p. 25

14 CONCLUSION

The conclusion reports that existing article-importance approaches fall short when judged against English Wikipedia’s values. It highlights challenges for automating editor-effort direction and the role of community gender demographics in realizing those values.

14 CONCLUSION

  • Existing approaches for determining article importance fall short when evaluated through English Wikipedia’s values.Source: §14 · p. 26
  • Automating the direction of editor effort on Wikipedia presents several challenges.Source: §14 · p. 26
  • The community’s gender demographics can prevent it from actualizing the encyclopedia’s values, lending further credibility to feminist critiques of Wikipedia.Source: §14 · p. 26
Loading 2208.08426v1…