PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Users' Security and Privacy Concerns About Third-Party Community Bots on Discord

Kofi Ampomah, Tu Le

arXiv:2610.36626v1cs.CRcs.HC

TL;DR

The paper studies security and privacy concerns raised in reviews of Discord community bots, where an administrator’s installation can affect members who did not take part in the decision and standardized disclosures are lacking. It analyzes reviews and bot metadata with a four-stage pipeline and builds BotLens for auditing. S&P discussion varies across bot types, while star ratings and conventional metadata features do not fully capture it.

Abstract

from arXiv · show

Third-party community-based bots are central to how social platforms such as Discord work. They moderate content, manage roles, play music, and support AI-assisted conversations across millions of servers. Yet when a bot is installed in a server, it can access messages, user profiles, and server metadata concerning members who did not participate in the installation decision, with no standardized way to communicate the associated security and privacy implications. In this paper, we analyze 23,102 third-party Discord bots and 183,142 user reviews spanning 49 detected languages and five years of feedback (2021-2026). We employ LLM-based knowledge distillation and fine-tune a DeBERTa-based classifier to identify candidate security and privacy (S&P)-relevant reviews. After manually validating the complete classifier-positive set, we retain 8,832 reviews identified as S&P-relevant, representing an observed pipeline yield of 4.82%, and map them to a 31-theme S&P taxonomy extending prior work. Our results show that S&P-related reviews focus heavily on Data Sharing (30.5%) and Data Collection (16.2%). Confession and anonymity bots have pipeline-identified SPRRs above 50%, more than ten times the ecosystem-wide pipeline yield of 4.82%. Star ratings may obscure the frequency and valence of S&P-related discussion, while conventional metadata features, such as permission count and administrator status, show only weak associations with user-reported S&P concerns. We then build BotLens, a proof-of-concept auditing framework that combines review-derived S&P signals with bot metadata to provide on-demand decision support for the community-based bot ecosystem. BotLens exposes six tools through a Model Context Protocol server, supporting bot-level checkups, server-wide comparison, and integration with AI agents.

I. INTRODUCTION

A server administrator’s bot installation can expose data and interactions belonging to members who did not authorize it, while those members may not know what a bot can access or how it uses their data. The paper addresses limited standardized disclosure and research on community-bot S&P concerns by analyzing user reviews and developing BotLens for on-demand auditing.

I. INTRODUCTION

  • A single administrator’s installation can expose data and interactions from potentially thousands of members, who may have little control over the third-party service.Bots’ ongoing access can depend on permission bitfields and privileged Gateway Intents, which members and even approving administrators may misunderstand.Source: §I · p. 1
  • Community-bot marketplaces lack a standardized disclosure framework, and only 0.77% of bots in the dataset reference a privacy policy in their listings.User reviews offer complementary, unprompted, in-situ evidence, but concerns in this ecosystem remain underexplored and no tool provides structured, on-demand access to these signals.Source: §I · p. 2

D. Our Contribution to Research on Community Bots

  • BotLens combines review-derived S&P signals with bot metadata in a six-tool MCP server for on-demand bot-level and server-wide auditing.The study also examines how requested permissions and other metadata associate with user-reported concerns.Source: §I · p. 2

B. User Perceptions of Bots

C. User Review Analysis

  • The paper extends a 20-theme Android-review taxonomy with 11 community-bot-specific themes and combines S&P review classification with thematic and affective-expression analysis.This complements prior review-analysis methods, which have used reviews to study privacy concerns and their relationship with app evaluations,,, [42],.Source: C. User Review Analysis · p. 3

D. Our Contribution to Research on Community Bots

  • Prior bot research emphasizes permission models, protocol vulnerabilities, and technical analysis, while less is known about concerns Discord users publicly report.Existing mobile-app review frameworks do not capture bot-specific risks such as administrator-mediated installation, Gateway Intents, and server-wide message access.Source: D. Our Contribution to Research on… · p. 3

III. METHODOLOGY

The paper builds a four-stage pipeline that collects Discord bot reviews and metadata, develops a Discord-adapted S&P taxonomy, trains a classifier through LLM knowledge distillation, and uses the resulting labels for analysis and auditing. Its taxonomy combines existing review categories with themes derived from bot-specific concerns.

III. METHODOLOGY

  • The four-stage pipeline develops a taxonomy, trains a classifier through knowledge distillation, labels S&P reviews with taxonomy and affective-expression tags, then analyzes reviews and metadata for BotLens auditing.It processes Top.gg reviews and bot metadata to support the paper’s research questions and auditing tool.Source: §III · p. 3

A. Data Collection and Preprocessing

  • 183,142 valid reviews remained after preprocessing 201,733 raw reviews spanning January 2021 through February 2026 and identifying 49 distinct languages.Valid non-English reviews were translated to English before the text was merged with bot metadata.Source: A. Data Collection and Preprocessing · p. 3

B. Stage 1: S&P Taxonomy Development

  • The taxonomy combines Akgul et al.’s 20-theme Android-review coding frame with 11 new bot-ecosystem themes derived through iterative open coding of 500 S&P-relevant reviews.Discord documentation informed interpretation of platform terminology, not the generation of candidate codes.Source: B. Stage 1: S&P Taxonomy Development · pp. 3–4
  • The 31-theme taxonomy groups concerns into Data Practices, User Rights & Controls, Security & Safety, and Discord-specific Risks.Discord-specific themes include Over-Listening, Permission Scope Mismatch, Message Content Access, and Gateway Intent Abuse.Source: B. Stage 1: S&P Taxonomy Development · p. 4

C. Stage 2: Knowledge Distillation

  • Stage 2 uses an LLM teacher to label reviews and a fine-tuned smaller student model, confining LLM inference to an initial labeling pass.The taxonomy guides teacher classifications, and the teacher output includes an S&P label, confidence, reasoning, and identified theme.Source: C. Stage 2: Knowledge Distillation · p. 4, 1) LLM for Labeling · p. 4

1) LLM for Labeling:

  • 30% teacher labeling achieved the highest observed F1 among the evaluated budgets, at F1 = 0.938.The five evaluated budgets were 10%, 20%, 30%, 40%, and 45%; because each used different sampled subsets and evaluation splits, this is exploratory rather than evidence of a universally optimal budget.Source: 1) LLM for Labeling · p. 4, Table II · p. 4
  • Incremental labeling reduced total labeling time from approximately 200 hours to 54 hours on the RTX 5090.The 10% labeled set served as the base, with only newly selected samples labeled at each higher percentage.Source: 1) LLM for Labeling · p. 4

2) Active Learning:

The pipeline tests active learning, validates teacher labels, trains and evaluates a student classifier, and applies it to the full review corpus for manual validation and structured analysis. Its metrics describe pipeline-identified discussion and theme prevalence, not verified vulnerabilities or bot-level risk.

2) Active Learning:

  • At the 10% labeling level, active learning raised F1 from 0.73 to 0.87 with 923 additional labels; at 30%, it raised F1 from 0.938 to 0.941.The loop selects reviews with prediction probabilities closest to 0.5, adds Gaussian noise to uncertainty scores, and uses majority labels from three LLM passes; the result is limited to the evaluated configuration.Source: 2) Active Learning · pp. 4–5, Table III · p. 5

3) Teacher Model Labeling Validation:

  • Two researchers independently annotated a stratified 10% sample of 5,494 teacher-labeled reviews, with Cohen’s κ of 0.74 between researchers and 0.71 and 0.84 against Llama 3.Raw agreement exceeded 99.8% across the three comparisons, and disagreements were adjudicated; the resulting human annotations are called reference labels, not ground truth.Source: 3) Teacher Model Labeling Validation · p. 5

4) Student Model Training:

  • On the selected 30% labeling condition’s 98-review evaluation split, precision, recall, and F1 were each 0.938.There were 16 S&P-relevant examples, with one false positive and one false negative; the 95% Wilson intervals for precision and recall were approximately 0.72–0.99, reflecting uncertainty from the small positive sample.Source: 4) Student Model Training · p. 5

D. Stage 3: Classification

  • The trained student classifier processes the full corpus, enriching detected S&P reviews with taxonomy and affective-expression labels for later analysis.The student model is DeBERTa-v3large, trained with a 0.8 classification threshold to minimize false positives.Source: D. Stage 3: Classification · p. 5, 4) Student Model Training · p. 5

1) S&P Review Identification:

  • All 8,832 classifier-positive candidates were manually validated and retained as S&P-relevant reviews, a pipeline yield of 4.82% of 183,142 reviews.Validation assessed relevance, not the factual correctness of reviewers’ claims, and does not rule out relevant reviews among classifier-negative reviews.Source: 1) S&P Review Identification · p. 5

2) Taxonomy Assignment:

  • Each retained review receives one primary theme from the 31-theme taxonomy; quantitative analyses use exactly one primary theme per review.Secondary themes were retained only as descriptive notes, and outputs that could not be matched to the finalized taxonomy were marked “unassigned.”Source: 2) Taxonomy Assignment · p. 5

3) Affective-Expression Classification:

  • Affective-expression labels characterize language in reviews, not users’ underlying emotions or longer-term moods.GoEmotions’ 28 labels are aggregated into eight categories, using a multi-label threshold of 0.3.Source: 3) Affective-Expression Classification · pp. 5–6

E. Stage 4: Data Analysis

  • SPRR divides classifier-identified, manually validated S&P-relevant reviews by total reviews at the aggregation level; TPR shows theme distribution across assigned S&P reviews, totaling 100% excluding unassigned reviews.Temporal trends track SPRRt at 30-day intervals over 2021–2026 using ordinary least-squares regression. Association tests relate bot characteristics to SPRR using Pearson r and Spearman ρ.Source: E. Stage 4: Data Analysis · p. 6

IV. FINDINGS

Across the review corpus, S&P discussion rose to a mid-2023 peak before declining, while S&P reviews were more polarized in star ratings. Data Practices dominated the taxonomy, with Data Sharing the most prevalent individual theme.

A. Aggregate Security and Privacy Trend Analysis

  • SPRR rose from roughly 4% in early 2021 to around 8% in mid-2023, then fell to about 3% by early 2026.The full-window linear trend was modestly downward (R2 = 0.35).Source: Figure 2 · p. 6, A. Aggregate Security and Privacy… · p. 6
  • 50.3% of S&P reviews received five stars and 24.4% one star, while 81.3% of non-S&P reviews received five stars.S&P reviews were more often associated with highly positive or highly negative ratings than with mid-range ratings; ratings reflect writers’ reported experiences and perceptions, not verified violations.Source: A. Aggregate Security and Privacy… · pp. 6–7, Figure 3 · p. 7

B. RQ1: S&P Themes Across the Bot Ecosystem

2) Theme-Level Analysis:

3) SPRR Across Bot Tags and Categories:

Pipeline-identified S&P review rates were especially high for anonymity and confession bots. S&P themes and affective-language patterns also varied across bot functions and categories.

3) SPRR Across Bot Tags and Categories:

  • Among bot tags with at least 100 reviews, secret had the highest SPRR, 55.4% (112/202); confessions-logs (55.1%), secrets (53.8%), and anonymous (53.0%) also exceeded 50%, with these rates exceeding ecosystem-wide pipeline yield by more than ten times.Source: 3) SPRR Across Bot Tags and Categories · p. 7

4) Theme Variation Across Bot Categories:

  • Data Sharing dominated most bot categories, but AI bots had larger Data Collection and Data Exposure shares, while antinuke and antiraid bots had more Data Security discussion.Community and economy bots had especially prominent Data Sharing (∼35%); 247-music showed elevated Data Collection, and autoroles showed more Anonymity/Identification concerns.Source: Figure 7 · p. 8, 4) Theme Variation Across Bot Categories · p. 8

2) Affective Expressions by Bot Category:

  • Neutral labels accounted for roughly 40% to 60% across the top eight S&P themes, while Data Exposure had the largest Anger (∼10%) and Confusion (∼8%) shares.DM Privacy also showed notable Anger; Data Deletion had comparatively more Joy, and Data Security more Surprise.Source: Figure 8 · p. 8, 2) Affective Expressions by Bot Category · p. 8
  • Community bots had the highest proportion of negative affective-expression labels, with Anger at roughly 12%, followed by Sadness and Confusion.Anime and autoroles also had comparatively elevated negative-label proportions; automoderation and antinuke had the lowest, while AI bots showed a mixed distribution.Source: 2) Affective Expressions by Bot Category · pp. 8–9, Figure 9 · p. 9

3) Connecting Affective Expressions to Star Ratings:

Affective labels describe review text rather than underlying emotions, and metadata analyses show only limited relationships with S&P discussion. Server count was the notable exception, though its association was observational.

3) Connecting Affective Expressions to Star Ratings:

  • Neutral and Joy were common across themes, while Anger and Confusion appeared more often in Data Exposure and DM Privacy reviews.These labels characterize textual affect, not users’ underlying emotions or decision-making processes.Source: 3) Connecting Affective Expressions to… · p. 9

D. RQ3: Permissions, Metadata, and S&P Concerns

  • The bot-level metadata analyses covered 1,457 bots with at least five reviews and at least one S&P-relevant review; server-count analysis covered 896 bots disclosing server counts.The features examined included permissions, server count, administrator status, and code transparency.Source: D. RQ3: Permissions, Metadata, and S&P… · p. 9

1) Permission Count and SPRR:

  • Permission count had no significant linear relationship with SPRR (r = −0.031, p = 0.320), and rates across permission-count bins were non-monotonic.The elevated rate among low-permission bots partly reflected confession and anonymity bots that handle sensitive content.Source: Figure 10 · p. 9, 1) Permission Count and SPRR · p. 9

2) High-Risk Permissions:

  • No individual high-risk permission showed a strong association with S&P discussion; the largest reported gap was for MODERATE MEMBERS (7.81% vs. 5.61%).Nine of twelve permissions had higher SPRR when requested, while three showed the opposite pattern.Source: 2) High-Risk Permissions · p. 9, Figure 11 · p. 10

3) Admin vs. Non-Admin Bots:

  • Admin and non-admin bots had nearly identical SPRR: 5.69% (n=425) versus 5.64% (n=1,032), with similarly composed theme profiles.The paper suggests S&P discussion is more associated with bot function and category than administrator permission.Source: 3) Admin vs. Non-Admin Bots · p. 9, Figure 12 · p. 10

4) Server Count and SPRR:

  • 11.89% of bots in the 100 to 1K server bin had S&P-relevant reviews, the highest rate across the five bins.The smallest and largest bins had the lowest rates; the analysis is observational and does not establish causation between server count and S&P discussion.Source: 4) Server Count and SPRR · pp. 9–10, Figure 13 · p. 10

5) Code Transparency:

  • Bots without a linked public GitHub repository had a slightly higher SPRR than those with one: 5.81% (n=1,221) versus 4.91% (n=236).The difference does not establish that GitHub links reduce S&P problems; the data do not distinguish possible explanations such as perceived trust or development practices.Source: 5) Code Transparency · p. 10

6) Metadata Correlation Matrix:

Visible metadata were generally weak predictors of S&P discussion, and privacy-policy references were rare in bot listings. BotLens combines review-derived signals with metadata in a proof-of-concept auditing framework.

V. AUDITING FRAMEWORK FOR COMMUNITY BOTS

  • BotLens is a proof-of-concept auditing framework that translates review-derived S&P signals and bot metadata into a tool for community bots.Its motivation is that users may miss relevant reviews when platforms sort by recency or helpfulness rather than topic.Source: §V · p. 10, A. Motivation · pp. 10–11

6) Metadata Correlation Matrix:

  • Server count had the strongest metadata correlation with SPRR (ρ = −0.34), while permission count, high-risk permissions, average rating, admin status, and GitHub availability had weak correlations.Permission count and high-risk permission count were strongly correlated (ρ = 0.90); only 177 of 23,102 listings referenced a privacy policy (0.77%).Source: 6) Metadata Correlation Matrix · p. 10, Figure 14 · p. 11

1) Offline classification pipeline:

  • The offline pipeline processes 183,142 reviews and precomputes each review’s S&P label and taxonomy tags, plus each bot’s SPRR, permission profile, and sentiment composition.It includes scraping, cleaning, LLM labeling, DeBERTa classification, taxonomy and affective-expression labeling, permission decoding, and aggregation.Source: 1) Offline classification pipeline · p. 11

2) MCP query server:

  • BotLens exposes six MCP tools for bot signals, taxonomy breakdowns, representative reviews, bot comparisons, high-salience bot lists, and server-wide scans.Because classifications are precomputed, queries use indexed lookups rather than model inference and return structured summaries in milliseconds.Source: 2) MCP query server · p. 11

1) Use Case 1 - Checking a specific bot:

  • A Wick checkup returned SPRR, rating, permissions, themes, and sentiment composition without a platform login and in under 2 ms.Wick had a 4.3-star rating and 28.9% SPRR; its review-derived signals supplied S&P information not captured by the aggregate rating alone.Source: Figure 15 · p. 11, 1) Use Case 1 - Checking a specific bot · p. 11

2) Use Case 2 - Auditing an entire server:

BotLens compares installed bots using review-derived S&P signals, permissions, and sentiment. The discussion finds that S&P review signals vary by bot function, while ratings and permission metadata alone offer limited insight.

2) Use Case 2 - Auditing an entire server:

  • Three of six bots in the server-wide example have SPRR above 10%, and sentiment composition distinguishes predominantly critical from predominantly positive S&P discussion.The comparison report also presents reported S&P themes and requested permissions.Source: 2) Use Case 2 - Auditing an entire… · p. 11, Figure 16 · p. 12

D. Time Performance

  • Across 22 queries, scan_server_bots averaged 14.7 ms and peaked at 31.3 ms; its session-wide mean was 5.0 ms.get_bot_signals, get_taxonomy_breakdown, and get_review_samples averaged 1.7 ms, 1.6 ms, and 1.9 ms, respectively.Source: D. Time Performance · pp. 11–12

1) S&P discussion is meaningful but unevenly distributed:

  • 4.82% of reviews were identified and retained as S&P-relevant, while confession and anonymity bots exceed 50% SPRR.These are observed pipeline rates, not estimates of true prevalence.Source: 1) S&P discussion is meaningful but… · p. 12

2) Bot function shapes the disposition of S&P concerns:

  • SPRR varies with bot function and data sensitivity; moderation and governance bots raise distinct concerns from confession and anonymity bots.The discussion recommends function-specific disclosures and, for moderation and governance bots, least privilege, explanations of automated decisions, and appeal mechanisms.Source: 2) Bot function shapes the disposition… · p. 12

3) Star ratings do not capture S&P discussion salience:

  • Star ratings may obscure S&P feedback: Wick has 28.9% SPRR and 4.3 stars, while Auttaja has 23.5% SPRR and 3.3 stars.The authors caution that aggregate ratings should not be interpreted as endorsements of S&P practices.Source: 3) Star ratings do not capture S&P… · p. 12

4) Permission metadata is a weak standalone predictor:

  • Admin and non-admin bots have nearly identical SPRR (5.69% vs. 5.64%), and permission metadata is only weakly associated with reported concerns.The authors conclude that permission auditing alone provides an incomplete picture.Source: 4) Permission metadata is a weak… · p. 12

5) User reviews capture what policies miss:

  • Only 0.77% of bots reference a privacy policy in Top.gg listings, so reviews can complement policy analysis while remaining self-reported and unverified.Prior work reported 4.35% for Discord chatbots that request permissions.Source: 5) User reviews capture what policies… · p. 12

6) Affective-expression patterns complement star ratings:

  • Neutral and Joy are common across several S&P themes, while Anger and Confusion appear more often for Data Exposure and DM Privacy.These language patterns are not direct evidence of underlying emotions or binary S&P judgments.Source: 6) Affective-expression patterns… · p. 12

B. Recommendations

The recommendations call for S&P signals alongside ratings and permissions, and for disclosures tailored to bot function. The authors qualify these recommendations because the evidence is review-based, setting-specific, and model-assisted, while BotLens remains an unevaluated proof of concept.

1) For Social Platforms with Community-based Bots:

  • Platforms could pair star ratings with a dedicated S&P review indicator and category-specific disclosures for bots handling sensitive content, without treating elevated SPRR as verified risk.The indicator could report pipeline-identified S&P discussion and distinguish concerns from positive feedback.Source: 1) For Social Platforms with… · pp. 12–13

2) For Bot Developers:

  • Developers should request only necessary permissions and clearly explain data access; Positive Privacy accounts for 2.2% of S&P reviews.Source: 2) For Bot Developers · p. 13

3) For Administrators and Users:

  • Administrators should consider review-derived signals alongside permissions; BotLens surfaces SPRR, themes, sentiment, and permissions to identify bots warranting closer examination.The authors also flag comparatively high SPRR among bots deployed in 100–1K servers for closer examination.Source: 3) For Administrators and Users · p. 13

1) Data Source:

  • The Top.gg review sample may not represent the broader population, and its self-reported claims were not verified against historical bot versions, configurations, logs, or incident context.Findings therefore characterize review writers’ reported experiences and perceptions, not confirmed bot behavior or verified violations.Source: 1) Data Source · p. 13

2) Manual Validation and SPRR Interpretation:

  • The 4.82% SPRR is observed pipeline yield, not true prevalence; the 0.30% teacher-positive rate applies only to the 54,942-review subset.Agreement measures establish consistency, not labeling sensitivity, dataset completeness, or ground truth.Source: 2) Manual Validation and SPRR… · p. 13

3) Generalizability Beyond Discord:

  • SPRR values reflect Top.gg’s reviewer population and bot mix, while nine taxonomy themes depend on Discord-specific mechanisms.Other platforms would need different extensions, though the four theme groups and 20 themes inherited from prior work remain applicable.Source: 3) Generalizability Beyond Discord · p. 13

4) LLM for Labeling and Taxonomy Assignment:

  • Residual labeling errors may propagate: language identification and translation are automated, and taxonomy and affective-expression labels lack independent evaluation against dedicated ground truth.The resulting distributions are interpreted as model-assisted descriptive findings.Source: 4) LLM for Labeling and Taxonomy… · p. 13

5) Real-Time and In-Situ Auditing:

  • BotLens is a feasibility proof of concept, not a completeness claim; it neither verifies review claims nor determines bot safety, and its decision-making impact was not evaluated.Source: 5) Real-Time and In-Situ Auditing · p. 13

VII. CONCLUSION

The conclusion presents a baseline for auditing community bots and BotLens as decision support, while emphasizing that review-derived signals describe reported experiences rather than verified behavior. The ethics discussion addresses the effects on users, administrators, developers, platforms, and researchers, and describes safeguards and scope limits.

VII. CONCLUSION

  • Confession and anonymity bots show especially prevalent S&P discussion, while bot function and content sensitivity are more informative than permission count or administrator status alone.The conclusion characterizes the latter metadata features as having only weak associations with user-reported concerns.Source: §VII · pp. 13–14

B. Potential Harm and Mitigation

A. Stakeholder Identification and Impact Analysis

  • BotLens gives administrators review-derived SPRR, S&P themes, sentiment, and permission signals to identify bots warranting closer examination, not to establish risk or misconduct.This is decision support when ratings or deployment counts provide little information about S&P practices.Source: A. Stakeholder Identification and… · p. 14
  • Because the analysis does not establish misconduct, naming bots may create reputational harm; reported patterns are framed as signals for examination, not verdicts.The authors say named bots illustrate patterns rather than single out developers.Source: A. Stakeholder Identification and… · p. 14
  • The findings identify communication gaps that platforms could address through clearer permission disclosures, more visible privacy policies, better-organized reviews, or decision support.The study used public pages and rate limits and did not access authenticated or private pages.Source: A. Stakeholder Identification and… · p. 14
  • The work contributes a method, taxonomy, and dataset for studying user-reported S&P concerns in community-bot ecosystems at scale.The approach may also apply to other bot marketplaces, messaging platforms, or AI-agent ecosystems with public feedback.Source: A. Stakeholder Identification and… · p. 14

B. Potential Harm and Mitigation

  • A high SPRR reflects how often the pipeline identifies S&P topics in reviews, not whether claims are correct or a bot is unsafe; residual classifier error may remain.The authors also caution that a low SPRR does not guarantee safety and treat BotLens as decision support rather than a verdict.Source: B. Potential Harm and Mitigation · p. 14

C. Decision and Research Justification

  • The authors justify the work as a balance of potential costs and benefits, citing public-data use, removal of identifying details, aggregation, and cautious framing as harm-reduction measures.They identify the gap between bot capabilities and what server members can observe as the motivation.Source: C. Decision and Research Justification · pp. 14–15

APPENDIX

The appendix specifies BotLens’s auditing flow, reports affective-expression patterns across detected languages and an open-versus-closed-source SPRR comparison, and presents the complete taxonomy.

A. BotLens Auditing Algorithm

  • BotLens discovers APP-tagged bots in an administrator’s authenticated server session, aggregates review-derived signals or falls back to a permission audit, and returns bots ranked by SPRR discussion salience.For bots with reviews, the procedure computes SPRR from the fraction of their reviews identified as S&P-relevant and sorts taxonomy flags by frequency.Source: A. BotLens Auditing Algorithm · p. 16

B. Affective Expressions by Language

  • Neutral and Joy are the largest affective-expression categories in all six language groups, which together account for 94.2% of S&P-relevant reviews; translated-text labels limit interpretation of magnitude.Figure 17 covers languages with at least 100 S&P-relevant reviews; the authors do not interpret the differences as differences in writers’ expression.Source: Figure 17 · p. 17, B. Affective Expressions by Language · p. 17

C. SPRR by Code Transparency

D. Complete Security and Privacy Taxonomy

ARTIFACT APPENDIX

The artifact provides the study’s pipeline outputs, taxonomy, datasets, and BotLens server, with instructions for installation and reproducing selected analyses. Its experiments report corpus statistics and the paper’s major empirical claims.

ARTIFACT APPENDIX

  • The artifact packages the analysis pipeline, 31-theme taxonomy, datasets underlying reported results, and BotLens auditing server.The distributed data includes classified reviews, the S&P-relevant subset with theme and affective labels, bot listing metadata, annotator workbooks, and reference figures.Source: ARTIFACT APPENDIX · p. 19, 4) Benchmarks · p. 19

1) How to access:

  • The archive is available on Zenodo and GitHub; its data and code use CC BY-NC-SA 4.0 and MIT licenses, respectively.The appendix provides checksum and setup instructions for the archive.Source: 1) How to access · p. 19

2) Hardware dependencies:

  • Reproducing the distributed-output experiments requires no GPU; the appendix specifies an eight-core, 16 GB machine and about 1 GB of additional working space.Regenerating the raw-crawl pipeline stages instead requires an NVIDIA GPU with at least 16 GB VRAM.Source: 2) Hardware dependencies · p. 19

3) Software dependencies:

  • The artifact uses Python 3.11 on Linux, macOS, or Windows, with dependencies pinned in a lock file.Ollama and CUDA are required only for the raw-crawl stages outside the reproduced experiments.Source: 3) Software dependencies · p. 19

B. Artifact Installation & Configuration

  • The smoke-test command completes within seconds without data, GPU, or network access; the stated expected outcome is 46 passing checksums and 17 tests.The appendix describes this as the minimal installation check.Source: B. Artifact Installation & Configuration · p. 19

C. Major Claims •

  • 183,142 reviews across 23,102 bots yielded 8,832 pipeline-identified S&P-relevant reviews (4.82%); 177 bots (0.77%) reference a privacy policy.These are the artifact’s E1 corpus and metadata counts.Source: C. Major Claims • · p. 19, 1) Experiment (E1) · p. 19
  • Data Practices account for 71.3% of concerns, led by Data Sharing (30.5%) and Data Collection (16.2%), while confession and anonymity tags exceed 50% SPRR.These are the artifact’s E2 major-claim values.Source: C. Major Claims • · p. 19
  • The artifact’s major claims also report weak predictive value for requested permissions and administrator status, substantial annotator agreement with teacher-generated labels, and six BotLens tools with low-millisecond indexed lookups.The associated experiments are E3, E4, and E5.Source: C. Major Claims • · p. 19

2) Experiment (E2):

The appendix describes experiments that regenerate the thematic, affective, metadata, agreement, and BotLens analyses. It also documents reproducibility boundaries, a corrected theme lookup, and a scanner that evaluators are told not to run.

2) Experiment (E2):

  • E2 regenerates the thematic and affective-expression distributions underlying RQ1 and RQ2, with figures 2–9 written to the output directory.The artifact notes one documented departure from the published figures; the remaining three theme groups reproduce exactly.Source: 2) Experiment (E2) · pp. 19–20

3) Experiment (E3):

  • Permission count and SPRR are not linearly associated (r = −0.031, p = 0.320); administrator and non-administrator bots differ negligibly, while server count has the strongest association (Spearman ρ = −0.34).The reported subset is n = 1,457.Source: 3) Experiment (E3) · p. 20

4) Experiment (E4):

  • Cohen’s κ is 0.7401 between annotators and 0.7136 and 0.8383 between each annotator and the teacher model, across N = 5,494 items.Raw agreement exceeded 99.8% throughout.Source: 4) Experiment (E4) · p. 20

5) Experiment (E5):

  • BotLens exposes six tools and returns under 1 ms per call for every tool in the reported evaluation; three of six scanned bots exceed 10% SPRR.Latency depends on the evaluating machine, and the claim concerns indexed lookups with precomputed classifications rather than specific timings.Source: 5) Experiment (E5) · p. 20

E. Notes

  • One corrected theme-lookup key moves a review into Data Practices, changing Figure 4’s counts to n = 6,295 (71.3%) and n = 251 (2.8%); no other value is affected.The paper reported n = 6,294 and n = 252 (2.9%) for those groups.Source: E. Notes · p. 20
  • The distributed artifact cannot reproduce the active-learning column of Table III, and pipeline Stages 0–6 are records of methodology rather than rerunnable steps.Those stages require the raw crawl, a local Ollama instance, and a GPU; the scripts generate figures 2–14, 17, and 18.Source: E. Notes · p. 20
  • Evaluators are told not to run the Discord scanner because it authenticates as a user account, contrary to Discord’s Terms of Service; a plain-text bot-name input is offered instead.No paper result depends on the scanner.Source: E. Notes · p. 20
Loading 2610.36626v1…