Source-linked AI summary
AutoLike: Auditing Social Media Recommendations through User Interactions
Hieu Le, Salma Elmalaki, Zubair Shafiq, Athina Markopoulou
TL;DR
Social media recommendation algorithms can spread misinformation and harmful content, motivating audits of what they serve. AutoLike formulates auditing as reinforcement learning, and demonstrates that a streamlined version can drive TikTok toward content related to specified topics and sentiments.
Problem
Recommendation algorithms personalize content but can also spread misinformation and harmful content.
Method
AutoLike is a reinforcement learning framework that audits social media recommendations by driving the recommendation system toward particular content through user interactions.
Results
AutoLike can drive TikTok’s algorithm to serve content related to a given topic and sentiment, including 2× as much negative mental health content.
Takeaways & Limitations
AutoLike has the potential to assist regulators in auditing recommendation systems for problematic content.
Takeaways & Limitations
TikTok content classifiers are complex and should be evaluated before integration with AutoLike.
Abstract
from arXiv · showhide
Modern social media platforms, such as TikTok, Facebook, and YouTube, rely on recommendation systems to personalize content for users based on user interactions with endless streams of content, such as "For You" pages. However, these complex algorithms can inadvertently deliver problematic content related to self-harm, mental health, and eating disorders. We introduce AutoLike, a framework to audit recommendation systems in social media platforms for topics of interest and their sentiments. To automate the process, we formulate the problem as a reinforcement learning problem. AutoLike drives the recommendation system to serve a particular type of content through interactions (e.g., liking). We apply the AutoLike framework to the TikTok platform as a case study. We evaluate how well AutoLike identifies TikTok content automatically across nine topics of interest; and conduct eight experiments to demonstrate how well it drives TikTok's recommendation system towards particular topics and sentiments. AutoLike has the potential to assist regulators in auditing recommendation systems for problematic content. (Warning: This paper contains qualitative examples that may be viewed as offensive or harmful.)
1 Introduction
Social media recommendation systems can deliver harmful content, motivating technical auditing approaches. AutoLike formulates auditing as reinforcement learning that learns interactions to steer recommendation feeds toward selected topics and sentiments, with TikTok results showing increased delivery of targeted content.
- Motivation: Recommendation systems can spread misinformation and harmful content, while policy and platform responses remain unsettled, creating a need for technical auditing.Technical approaches can support auditing, inform developers and regulators, and identify violations for enforcement.
- The AutoLike Framework: AutoLike automatically audits social media recommendation algorithms by studying how user interactions affect content delivered through a For You Page.The framework treats the recommendation system as an environment and focuses on interactions such as liking, watching, sharing, and swiping.
- The AutoLike Framework: AutoLike produces pathways of recommended content and user actions that examiners can analyze to understand recommendation-system behavior.The framework can apply to social media platforms where users interact with a For You Page and can support auditing beyond topic and sentiment dimensions.
- The AutoLike Framework: AutoLike formulates auditing as reinforcement learning, using topic and sentiment inputs, interaction actions, and rewards based on how closely recommendations match the target.The agent learns a policy over time to select actions that efficiently drive the recommendation system toward the specified content.
- Case Study: Auditing TikTok: On TikTok, AutoLike can drive the platform to serve 2× as much negative mental health content as a controlled experiment.The case study applies the framework to TikTok and demonstrates steering toward particular content characteristics.
- Potential Applications and Impact: AutoLike may help regulators, developers, and researchers audit harmful recommendations, improve algorithm transparency, and examine which interactions expose or avoid targeted content.Developers can use these findings to inform policy and responsible algorithm design, while researchers can extend the framework to other platforms and dimensions of interest.
2 The AutoLike Framework
AutoLike models recommendation auditing as reinforcement learning: an agent uses user-like interactions to steer a platform’s recommendation system toward specified topic and sentiment goals. It produces interaction pathways that reveal how recommended content changes over time.
- Formulation Overview: AutoLike treats iterative user interactions with recommended content as a reinforcement-learning problem because each action affects later personalization.The agent interacts with the recommendation system as a black box, receiving rewards based on progress toward the desired content state.
- Outputs: The framework outputs sequences of recommended content and agent actions, creating pathways that can be analyzed to understand how interactions drive the recommendation system.These pathways represent the content-action trajectory produced during the agent’s interaction with the FYP.
- Actions and Policy: At each time step, the policy selects an available action, such as liking, watching, bookmarking, reposting, or skipping, without knowing its effect on the recommendation system beforehand.The action space depends on the platform and content shown, while the policy balances exploration against exploitation using state-action values.
- States and the End Goal: The agent represents each recommended item as a two-dimensional state containing topic relevance and sentiment, with an end goal specified by the user.Topic and sentiment scores range over discretized values from 0 to 1; sentiment 0 denotes positive content and 1 denotes negative content.
- States and the End Goal: AutoLike supports auditing harmful recommendations by emphasizing negative content dimensions, including distinctions such as pro-eating-disorder versus recovery content.The framework can include additional user-defined dimensions, but this paper focuses on negative aspects of recommended content.
- Reward Function and State-Action Values: The proximity-based reward increases as the next state approaches the user’s end goal, and Q-learning estimates which actions maximize cumulative reward over the time horizon.The reward is bounded in [0,1] and can use a normalized distance between the next state and the goal; temporal-difference learning does not require knowledge of the recommendation system.
3 AutoLike Implementation for TikTok
The TikTok implementation runs AutoLike on a live Android app, extracting and classifying recommended videos before selecting interactions through the learned policy. Its platform-specific workflow includes automated UI actions, content transcription, state construction, and practical account and clipboard workarounds.
- Implementation Overview: AutoLike runs in a live TikTok Android setting so it can capture how recommendation behavior changes through interactions over time.The implementation uses a rooted Pixel 3 device, while the framework’s core components are intended to remain platform-independent.
- Actions and Action Space: The TikTok agent’s action space includes skipping, watching, liking, bookmarking, and reposting, automated through UIAutomator2 and validated against available interface elements.Skipping uses a swipe, watching waits for the video duration, and the remaining actions use selectors derived from the app’s UI hierarchy.
- Environment: TikTok: The implementation requires manually incubating new TikTok accounts for 2–4 days before features such as liking and following become available.The authors refresh the FYP and device identifiers to make the feed fresh, while skipping ads to reduce AutoLike’s impact on TikTok’s ad ecosystem.
- Extracting TikTok Data: For each recommendation, AutoLike obtains the TikTok URL, extracts metadata and video content, transcribes audio with Whisper large-v2 [24], and concatenates it with the description.The text is cleaned to remove common hashtags and exact topic matches before classification.
- Determine the State: TikTok: Because Android blocks clipboard reads, AutoLike retrieves copied TikTok URLs by navigating through the home screen and search bar before pasting the URL.The paper notes network-traffic inspection as an alternative retrieval method.
- Determine the State: A zero-shot NLI classifier assigns topic and negative-sentiment confidence scores, which AutoLike converts into a discretized ⟨Topic, Sentiment⟩ state.For example, confidence scores ⟨0.06, 0.07⟩ map to the nearest valid state ⟨0.1, 0.1⟩.
4 Evaluation
AutoLike is evaluated on TikTok through automated classification and interaction experiments, showing strong topic targeting but weaker sentiment targeting and classification for some sensitive topics.
- 4.1 Classification of TikTok Content: Zero-shot classification performed well for several hashtag-topic pairs but struggled with eating-disorder and self-harm hashtags.The classifier worked for mental health, discrimination, physical violence, adult content, and binge-eating examples, but often labeled #ed as mental health and performed poorly on #self-hm, #sh, and #shawareness.
- 4.2 Deploying AutoLike on TikTok’s “For You” Page: AutoLike drove 2–3× as much on-topic content as control for benign topics, while sentiment-only targeting showed no discernible difference.At approximately step 100, Pets reached three times the control’s on-topic content and Sports doubled it; rates persisted during testing.
- 4.2 Deploying AutoLike on TikTok’s “For You” Page: AutoLike’s reinforcement-learning deployment requires processing thousands of TikToks and uses confidence thresholds to select interactions.The reported stopping condition required liking at least four of the last ten TikToks, with classification confidence above 0.5 for individual dimensions.
- 4.2 Deploying AutoLike on TikTok’s “For You” Page: Joint topic-and-sentiment targeting produced 1.5–2× more related content than control, with sad-content gains varying by topic specificity.Sad Pets produced 1.4× more sad content, whereas the more specific Sad Cats produced around 2.2× more.
- 4.2 Deploying AutoLike on TikTok’s “For You” Page: At the end of the combined-dimension experiment, AutoLike drove TikTok to serve 2× as much relevant content as control, increasing to 3× overall.The experiment also surfaced harmful content involving mental health, body image, loss and death, depression, and self-harm.
5 Related Work
Prior audits use sock puppets, user studies, donated data, and personalization frameworks to examine recommendation behavior. AutoLike instead learns interactions through reinforcement learning and can operate from a fresh FYP.
- Auditing Recommendation Systems: Related audits have measured radicalized or sexual recommendations, demographic personalization, eating-disorder content, election information, and user engagement across platforms.Approaches include sock puppets, surveys, interviews, commercial browsing panels, and data donations.
- Measuring Personalization: A separate TikTok personalization framework found that personalized content appeared 30% to 50% of the time, with liking and following producing the most personalized content.That framework inferred personalization from shared hashtags, creators, and user interactions.
- AutoLike vs. Prior Work: AutoLike is a reinforcement-learning framework that learns which actions to take to drive recommendations toward a specified topic and sentiment.Its reward function favors content of interest, and the agent learns through experience rather than following fixed action scripts.
- AutoLike vs. Prior Work: Unlike many sock-puppet systems using hardcoded interactions or prior searches, AutoLike can learn from scratch on a fresh FYP and still drive the recommendation system.The framework is deployed on TikTok’s mobile app and uses a real Android device rather than an emulator.
6 Conclusion & Future Directions
AutoLike audits social-media recommendation systems by learning from automated interactions to steer TikTok toward specified topics and sentiments. The framework identifies TikToks and demonstrates topic- and sentiment-directed recommendation steering, while supporting extensions across actions, platforms, users, and state dimensions.
- Conclusion: Recommendation systems provide personalized discovery but can also spread misinformation and harmful content, motivating audits of their delivered content.AutoLike focuses on recommendation behavior revealed through user interactions rather than only static user interests.
- Conclusion: AutoLike uses automated interactions and reinforcement learning to learn how to drive recommendation algorithms toward content matching a selected topic and sentiment.The framework models user actions such as liking and watching as interactions through which it learns an efficient steering strategy.
- Conclusion: The TikTok case study evaluates TikTok classification and shows that a streamlined AutoLike can drive recommendations toward a specified topic and sentiment.The implementation targets Android devices and the mobile TikTok app.
- Future Directions: Future work includes testing actions beyond liking and skipping, applying AutoLike to other platforms and specialized user groups, and adding dimensions such as truthfulness and intent.The authors also propose combining implicit interaction-based interests with explicitly declared interests and user attributes.
- Future Directions: The authors plan to release AutoLike software and collected datasets to support community extensions of the framework.They describe the release as a starting point for further development.
A Ethical Considerations
The study’s sensitive-content experiments were automated and designed to reduce platform impact. Manual review was conducted by the authors with consent, while the streamlined and reusable design reduced the number of processed TikToks.
- Ethical Considerations: All sensitive-content experiments were automated without real human users, and authors manually evaluating materials provided consent and reported no adverse effects.The experiments concerned topics including mental health, eating disorders, and self-harm.
- Ethical Considerations: AutoLike’s deployment aimed to reduce platform impact by automatically skipping ads and using a streamlined design that processed 200 TikToks per experiment instead of potentially tens of thousands.The controlled experiment was designed for reuse, further minimizing impact on TikTok.