PaperlayerView exact revision on arXiv

Source-linked AI summary

Echoes in the Algorithm: Analyzing the Fidelity of User Preferences Against Realized Platform Reach

Emelia Hughes, Tim Weninger

arXiv:2609.09365v1cs.HC

TL;DR

When popularity cues recede, users still infer which TikTok videos reached broader audiences. TokOrNot compares preference and reach judgments with view-based outcomes; judgments align only modestly with higher-view outcomes, while preference and prediction align much more strongly.

  • Problem

    The paper asks how people judge which content succeeds when popularity cues are hidden, delayed, or less visible.

    Source: §1 · p. 1

  • Method

    The study uses TokOrNot’s pairwise task to relate preference to higher-view outcomes and treats public view counts as a proxy for realized reach.

    Source: §3.2 · p. 5, §6 · p. 14

  • Results

    In the pairwise TikTok task, judgments were only modestly aligned with higher-view outcomes, while preference and prediction aligned much more strongly.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Users continue inferring reach when visible metrics recede, drawing on personal preference, qualitative cues, and informal theories about platform and audience dynamics.

    Source: §6 · p. 14

    The authors identify fixed question order and within-trial anchoring as the task structure’s most important limitation.

    Source: §5.3 · p. 13

Abstract

from arXiv · show

What does popular content look like when platforms withhold the usual cues? On TikTok, users still form impressions about which videos are taking off even when likes and view counts are hidden, delayed, or pushed to the margins of the interface. We study this problem through TokOrNot, a web-based game in which participants compared pairs of TikTok videos and reported (i) which one they preferred and (ii) which one they believed had reached a larger audience. We benchmark these judgments against verified public view counts, which we use as a bounded proxy for realized platform reach. Across 3,513 judgments from 363 participants, participants identified the higher-reach video only modestly above chance (56.75%, 95% CI: 56.01-58.55). Preference aligned with the higher-view video at a similar rate, while preference and prediction matched in 83.48% of trials (95% CI: 83.12-85.95). Performance also varied across content categories. Taken together, these results do not suggest that users can reliably read platform success from content alone. Instead, they point to a looser and more uncertain interpretive process in which reach judgments often track personal taste or other weak heuristics when explicit popularity cues are absent. We discuss the implications for algorithmic literacy and for interface designs that reduce visible metrics without leaving users to infer reach from uneven or idiosyncratic cues alone.

1 Introduction

The paper asks how people interpret reach when popularity metrics are hidden, and benchmarks pairwise judgments against public view counts. Participants were only modestly accurate, while preferences and reach predictions often aligned.

1 Introduction

  • Users continue forming impressions of which videos are taking off when view counts and likes are hidden, delayed, or pushed to the margins.Without explicit popularity cues, reach judgments may draw on content cues, personal taste, genre expectations, and assumptions about what spreads.Source: §1 · pp. 1–2
  • TokOrNot asks participants to choose their preferred video and predict which of two metric-hidden TikTok videos reached a larger audience.Verified public view counts serve as a bounded proxy for realized reach.Source: §1 · p. 2
  • The study examines whether people’s preferences align with higher-reach content, how accurately they identify it, and whether predictions follow preference.It also compares view-based outcomes with aggregated human judgments.Source: §1 · p. 3
  • 56.75% of judgments identified the higher-reach video, while preference and prediction matched in 83.48% of trials.The authors conclude that users cannot reliably read platform success from content alone; judgments often track personal taste or other weak heuristics.Source: §1 · p. 3
  • The paper contributes a pairwise empirical benchmark and an outcome-legibility framing for interpreting platform success when obvious popularity cues are reduced.It motivates interface support that helps users orient to reach without simply restoring precise counters.Source: §1 · p. 3

2 Related Work

Related work connects users’ informal theories about platforms to popularity cues and recommendation-driven culture. TokOrNot adapts comparative crowd-judgment methods to benchmark relative reach in short-form video.

2 Related Work

  • Across social media, users develop informal explanations for why content appears and build folk theories about the For You Page and what content it rewards, interpreting platform behavior through repeated encounters with trends, repetition, and personalization.Folk theories also develop around content moderation and shadowbanning.Source: §2 · pp. 3–4
  • Hiding or softening popularity metrics changes the evidence users can draw on, rather than eliminating interpretation.Users may then infer popularity from weaker signals such as polish, genre familiarity, affect, or trend resemblance.Source: §2 · p. 3
  • TikTok popularity judgments are entangled with recommendation and participatory culture, including expectations about what content the platform rewards.In metric-hidden settings, cultural fit may serve as a reach cue alongside explicit popularity signals.Source: §2 · pp. 3–4
  • TokOrNot builds on prior comparative-choice approaches rather than claiming a wholly new paradigm; it moves judgments into a recommendation-driven short-video setting and benchmarks them against realized platform reach proxied by public view counts.Source: §2 · p. 4

3 Methodology

TokOrNot uses controlled pairwise judgments to examine how people infer relative TikTok reach without visible engagement metrics. The methodology benchmarks judgments against verified public views while treating those counts as a bounded proxy, not a direct measure of algorithmic exposure.

3 Methodology

  • The preregistered online experiment used repeated pairwise judgments to study reach in a controlled comparison, not to simulate TikTok’s sequential For You Page.Showing two candidate videos side-by-side holds exposure constant for the comparison.Source: §3 · p. 4

3.1 Video and Pair Collection

  • Videos were grouped into five view-count buckets and paired within content category, reducing sensitivity to small count fluctuations and broad cross-genre taste differences.The analysis covered nine content categories.Source: §3.1 · p. 5

3.2 Measures

  • Verified public views served as a bounded reach proxy, but they capture recorded attention rather than algorithmic exposure alone.The measure relies on assumptions about exposure, stabilization, and comparability within pairs; the higher-view video is the pair-level reference for higher reach.Source: §3.2 · p. 5

3.3 Participants and Procedure

  • 363 participants contributed 3,513 judgments across 1,978 video pairs, with up to ten metric-hidden trials per participant and preference elicited before reach prediction.That fixed order leaves preference–reach alignment open to within-trial anchoring or consistency pressure, so it is not clean evidence of a stable heuristic.Source: §3.3 · pp. 5–6, Figure 2 · p. 6

3.4 Data Analysis

  • Trial-level logistic models tested reach-prediction correctness and preference substitution, with two-way clustering by participant and pair and participant-cluster bootstrap intervals for headline rates.Robustness checks included alternative specifications and exclusion of the first trial.Source: §3.4 · p. 7
  • For preference conditioning, model comparison tested whether pref_popular predicted accuracy; RQ4 used Survey Equivalence to compare view-based signals with aggregated human judgments.Regression models also assessed robustness to reach gap, trial order, and content category.Source: §3.4 · pp. 6–7

4 Results

In paired-video judgments, participants recognized higher reach and preferred higher-reach videos at similar rates, while their preferences and reach predictions often matched. Category, model, early-trial, and participant-level analyses further qualify these patterns.

4.1 Accuracy, Preference–Reach Alignment, and Preference–Prediction Matching (RQ1–RQ3)

  • Across 3,513 trials from 363 participants, accuracy and preference–reach alignment were each 56.75% (95% CI: 56.01–58.55), versus 83.48% preference–prediction matching (95% CI: 83.12–85.95).The overall rates are shown in Fig. 3.Source: §4.1 · p. 7, Figure 3 · p. 8
  • Accuracy ranged from 50.00% for Singing to 67.16% for Random, while preference–prediction matching ranged from 74.55% in Dancing to 91.04% in Random; category differences in correctness were not significant (χ2 = 12.10, df = 8, p = .15).The category differences are descriptive, not strong evidence of systematic category effects.Source: Figure 3 · p. 8, §4.1 · p. 8
  • Accuracy was 68.4% for high–low pairs and 61.7% for medium–viral pairs, but evidence for a reach-gap effect remained weak (chi-squared p = .063; regression β = 0.044, p = .111).Larger contrasts were descriptively associated with better accuracy, but gap size did not reliably improve accuracy in this dataset.Source: Figure 4 · p. 8, §4.1 · pp. 8–9

4.2 Modeling Correctness and Preference–Prediction Matching (RQ2–RQ3)

  • Reach gap and trial index were non-significant in the correctness model, and reach gap and trial index were also non-significant in the preference–prediction matching model.The authors note that these models do not rule out within-trial anchoring because preference was elicited immediately before reach judgment.Source: §4.2 · p. 9
  • Adding whether participants preferred the higher-view video improved correctness-model fit (ΔAIC = −1800.47), indicating a strong association, not proof that preference determines reach inference.The authors interpret preference alignment as explanatory for correctness within this task.Source: §4.2 · p. 9

4.3 Survey Equivalence and Early-Trial Bounding (RQ4)

  • TikTok’s view-based outcome aligned with aggregated human judgments at a level comparable to roughly two human raters, rather than a large or especially reliable crowd signal.Fig. 5 compares performance as the number of human raters increases from one to five.Source: Figure 5 · p. 10, §4.3 · p. 10
  • Accuracy and preference–prediction matching were similar in the first trials and across all trials, bounding across-trial drift without resolving possible within-trial anchoring.Preference was always elicited before reach prediction.Source: §4.3 · pp. 10–11, Figure 6 · p. 11

4.4 Participant-Level Patterns (Exploratory)

  • Daily users had 60.5% accuracy and 58.8% preference–reach alignment, versus non-users’ 57.3% and 55.9%; these participant-level patterns are descriptive and should be interpreted cautiously.Participants with three or more years of experience had 60.2% accuracy, compared with 52.0% among those with three to twelve months of use.Source: §4.4 · p. 12

5 Discussion

When visible popularity cues recede, people still judge reach, but those judgments align only loosely with view-based outcomes and are often entangled with personal taste and weak cues. The discussion proposes bounded orientation as a design direction while emphasizing limits in the study’s proxy, task, sample, and question order.

5.1 Interpreting Reach under Reduced Metrics

  • Participants’ reach judgments were only loosely calibrated to realized reach: they identified higher-view videos modestly above chance, while preference and reach judgments aligned much more often.These results do not suggest that users can reliably read reach from content alone.Source: §5.1 · p. 12
  • TikTok’s recommendation-driven short-video setting makes genre conventions, trend recognition, and cultural fit relevant alongside production cues, with judgments entangled in personal expectations.The contribution is not that users can cleanly identify successful TikTok content, but that reach judgments can become entangled with personal and genre-shaped expectations.Source: §5.1 · p. 12

5.2 Bounded Orientation as a Design Implication

  • Bounded orientation proposes lightweight, contextual support for interpreting reach without restoring precise metrics or claiming exact explanations for video performance.The goal is to reduce the interpretive vacuum when counters disappear but popularity judgments remain socially important.Source: §5.2 · p. 12
  • The authors frame bounded orientation as a middle position, but the study does not evaluate such interventions or show they would fix problems associated with visible metrics.Possible supports include coarse or delayed ranges, contextual reminders, and genre-aware prompts.Source: §5.2 · p. 13

5.3 Limitations and Threats to Validity

  • Because participants reported preference before reach, preference–prediction matching may reflect within-trial anchoring; early-trial analyses do not resolve this concern.The authors identify task structure as the most important limitation and suggest randomizing or separating the questions.Source: §5.3 · p. 13
  • Public view counts are a bounded reach proxy that may also reflect creator audience, video age, prior notoriety, and off-platform traffic, not internal ranking logic.The findings therefore concern calibration to a public attention outcome, not perception of TikTok’s internal ranking logic.Source: §5.3 · p. 13
  • The paired comparison task holds exposure constant but strips away feed context, repeated account exposure, comments, and ambient familiarity, limiting its ecological scope.It captures one bounded form of reach inference rather than everyday platform interpretation in full.Source: §5.3 · p. 13
  • The U.S.-based Prolific sample was older than TikTok’s core users, so the observed rates should not be treated as characterizing TikTok users as a whole.A younger or more internationally distributed sample might reason differently about the same pairs.Source: §5.3 · p. 13

6 Conclusion

The conclusion emphasizes that users continue inferring reach when visible metrics recede, but their judgments align only modestly with higher-view outcomes in the pairwise TikTok task. The paper benchmarks judgments against observable public reach and suggests future work on other content regimes, samples, methods, and interpretive supports.

6 Conclusion

  • In the pairwise TikTok task, reach judgments were only modestly aligned with the higher-view outcome, while preference and prediction aligned much more strongly.The findings suggest realized platform reach is not easily legible from content alone when explicit popularity cues recede.Source: §6 · p. 14
  • The paper’s main contribution is benchmarking user judgments against observable platform reach proxied by public view counts, rather than TikTok’s internal ranking logic.This outcome-centered framing focuses on what users can infer from the interface while distribution mechanisms remain opaque.Source: §6 · p. 14
  • Metric reduction does not eliminate interpretation; users may instead rely on weaker, more personal, and more uneven cues when trying to understand success.The conclusion treats this as a challenge for understanding what “popular” means on-platform.Source: §6 · p. 14
  • Future studies could test other content regimes and more diverse samples, randomize question order, compare visible- and hidden-metric conditions, and evaluate lightweight supports.Suggested supports include coarse metric ranges or delayed visibility rather than precise counters.Source: §6 · p. 14
Loading 2609.09365v1…