Source-linked AI summary
State of AI: An Empirical 100 Trillion Token Study with OpenRouter
Malika Aubakirova, Alex Atallah, Chris Clark, Justin Summerville, Anjney Midha
TL;DR
Systematic evidence about how LLMs are used in practice remains limited, despite rapid changes in model capabilities and deployment. The paper analyzes OpenRouter metadata across models, tasks, geographies, costs, and retention, finding heterogeneous adoption, substantial creative roleplay, rising agentic inference, and persistent early-user cohorts.
Problem
Systematic evidence about real-world LLM use remains limited because existing accounts emphasize qualitative demonstrations or benchmark performance rather than large-scale behavioral data.
Method
The study analyzes anonymized OpenRouter usage metadata across models, tasks, geographies, costs, and time, using observational platform data and proxy measures.
Results
The study finds a heterogeneous multi-model ecosystem, substantial roleplay usage, rising agentic inference, increasing global adoption, and foundational retention cohorts.
Takeaways & Limitations
Real-world LLM engagement is complex and multifaceted, with users selecting models across contexts rather than relying on a single dominant system.
Takeaways & Limitations
The study covers one platform and finite time window, excludes some deployment settings, and uses proxies for agentic inference and user geography.
Abstract
from arXiv · showhide
The past year has marked a turning point in the evolution and real-world use of large language models (LLMs). With the release of the first widely adopted reasoning model, o1, on December 5th, 2024, the field shifted from single-pass pattern generation to multi-step deliberation inference, accelerating deployment, experimentation, and new classes of applications. As this shift unfolded at a rapid pace, our empirical understanding of how these models have actually been used in practice has lagged behind. In this work, we leverage the OpenRouter platform, which is an AI inference provider across a wide variety of LLMs, to analyze over 100 trillion tokens of real-world LLM interactions across tasks, geographies, and time. In our empirical study, we observe substantial adoption of open-weight models, the outsized popularity of creative roleplay (beyond just the productivity tasks many assume dominate) and coding assistance categories, plus the rise of agentic inference. Furthermore, our retention analysis identifies foundational cohorts: early users whose engagement persists far longer than later cohorts. We term this phenomenon the Cinderella "Glass Slipper" effect. These findings underscore that the way developers and end-users engage with LLMs "in the wild" is complex and multifaceted. We discuss implications for model builders, AI developers, and infrastructure providers, and outline how a data-driven understanding of usage can inform better design and deployment of LLM systems.
1 Introduction
The paper addresses limited systematic evidence about real-world LLM use by analyzing OpenRouter activity across models, tasks, geographies, costs, and retention. It examines adoption of open models, task diversity, agentic inference, and persistent user cohorts.
- Systematic evidence about real-world LLM usage remains limited beyond qualitative demonstrations and benchmark performance.
- OpenRouter provides a cross-model view of which systems users choose for tasks and how behavior varies across regions and time.
- The study compares adoption patterns for open-weight and proprietary models.
- The paper investigates agentic inference through multi-step, tool-assisted interactions rather than single-turn model use.
- It analyzes usage by application category, including programming, roleplay, and translation, to identify major activity domains.
- The analyses also compare geographic uptake and effective cost against usage while highlighting unexpected real-world usage patterns.Effective cost is based on average input plus output tokens and accounts for caching effects.
- Retention analysis identifies foundational cohorts whose early fit between user needs and model characteristics sustains engagement over time, termed the Cinderella “Glass Slipper” effect.
2 Data and Methodology
The study analyzes anonymized OpenRouter metadata to characterize model usage, token workloads, content categories, geography, and time. Its methodology preserves privacy but relies on platform coverage, opt-in classification, predefined categories, billing geography, and uneven time windows.
- The dataset contains anonymized request-level metadata for billions of prompt–completion pairs from a global user base over approximately two years.
- The analysis uses timing, model, provider, token, performance, geography, latency, streaming, cancellation, and tool-calling metadata without prompt or completion text.
- OpenRouter supports more than 300+ active models from over 60 providers, but the observational dataset reflects platform availability, pricing, and user preferences.
- Content classification maps hierarchical GoogleTagClassifier labels into study-defined buckets such as Programming, Roleplay, Translation, Education, and Literature/Creative Writing.
- Predefined taxonomy limits how novel or cross-domain behaviors are categorized, and some requests receive multiple labels.
- Model origins, source status, and token variants are explicitly distinguished, with total tokens defined as prompt plus completion tokens.
- Geographic analysis uses billing location as a proxy, although third-party billing and shared enterprise accounts may not match actual user locations.
- Most analyses cover a rolling 13-month period, while category analyses begin later because consistent tagging became available only in mid-2025.
3 Open vs. Closed Source Models
Open-weight models steadily expanded on OpenRouter, reaching roughly one-third of usage by late 2025, while proprietary models remained largest. The open ecosystem also became more competitive, with usage shifting toward medium and large models and fragmenting across several sustained contenders.
- Approximately one-third of OpenRouter usage came from open-source models by late 2025, while proprietary models retained the largest share.
- Chinese open-source models rose from 1.2% weekly usage in late 2024 to nearly 30% in some weeks, averaging 13.0% over the year.
- New open models such as Qwen, MiniMax, Kimi K2, and GPT-OSS often reached production-scale adoption within weeks of release.
- No single open model exceeded 25% of OSS tokens by late 2025, as usage spread across five to seven models.
- Open-source leadership depends on continued improvement, as successive DeepSeek releases retained share while stagnant models tended to lose ground.
- Small-model usage declined on OpenRouter while medium and large models captured more value, although self-hosted small-model activity is not observed.
- The large-model segment remained pluralistic, with several contenders sustaining meaningful usage rather than consolidating around one model.
4 The Rise of Agentic Inference
LLM usage is shifting from isolated text completion toward reasoning-intensive, tool-integrated workflows with longer contexts and especially complex programming tasks. These trends indicate a growing agent-like structure in real-world inference.
- Reasoning models: More than 50% of total tokens were routed through reasoning-optimized models by 2025, up from a negligible share in early Q1.The trend reflects expanding model capabilities and user demand for systems that manage task state and multi-step work.
- Reasoning models: Grok Code Fast 1 led reasoning-model token traffic, ahead of Gemini 2.5 Pro and Gemini 2.5 Flash.The leading models changed rapidly, with Grok Code Fast 1 gaining share after Gemini 2.5 Pro had led weeks earlier.
- Tool use: Tool-call metrics capture successful invocations, whereas input-tool metrics capture merely available tools and therefore count more requests.The distinction separates realized tool use from potential tool availability.
- Context and sequence growth: Average prompt length grew roughly fourfold, from about 1.5K to over 6K tokens, while completions nearly tripled from about 150 to 400.Programming workloads are the dominant driver of prompt growth, with code-related requests routinely exceeding 20K input tokens.
- Context and sequence growth: Average sequence length more than tripled from under 2,000 tokens in late 2023 to over 5,400 by late 2025.Programming prompts average 3–4 times the token length of general-purpose prompts, linking longer interactions to software workflows.
- Agentic inference: The median request increasingly forms part of a structured agent-like loop involving external tools, stateful reasoning, and longer contexts.The paper identifies latency, tool handling, context support, and robustness as increasingly important infrastructure requirements.
5 Categories: How Are People Using LLMs?
Real-world LLM usage clusters around programming, roleplay, and personal assistance, while many specialized domains remain diffuse. Provider profiles differ substantially, ranging from coding-focused to general-purpose or consumer-oriented usage.
- Major categories: Programming grew from roughly 11% of total token volume in early 2025 to over 50% in recent weeks.Anthropic’s Claude series dominated programming-related spend for most of the observed period, while competitors gained share later.
- Major categories: Roleplay is dominated by Games/Roleplaying Games at nearly 60% of roleplay tokens, followed by Writers Resources at 15.6% and Adult content at 15.4%.The distribution indicates structured interactive fiction, scenario generation, and personal fantasy use.
- Major categories: Over two-thirds of programming traffic is labeled Programming/Other, with Development Tools accounting for 26.4%.The mix spans debugging, logic work, script drafting, and other general-purpose coding tasks.
- Long-tail categories: Health is highly fragmented, with no sub-tag exceeding 25%, while finance, academia, and legal are also diffuse.These domains distribute activity across varied intents rather than a single dominant workflow.
- Provider profiles: Claude usage exceeded 80% in Programming and Technology combined, whereas Google’s usage was broader and its coding share declined to roughly 18%.Provider-level distributions range from Claude’s coding focus to Google’s more diverse information-oriented profile.
- Provider profiles: xAI usage was often over 80% Programming before broadening in late November, while DeepSeek emphasized roleplay, casual chat, and entertainment and Qwen allocated 40–60% to programming.These profiles reflect sharply different provider and model-family usage compositions.
6 Geography: How LLM Usage Differs Across Regions
LLM usage is increasingly global but remains regionally and linguistically uneven in the OpenRouter user base. Asia’s spending share rose substantially, while English remained dominant alongside a meaningful multilingual tail.
- Scope: The geographic figures describe OpenRouter’s user base and provide one snapshot of regional engagement rather than a complete measure of global usage.The paper frames the regional analysis as a view of local usage and spend within the platform.
- Regional distribution: Asia’s share of global spend rose from roughly 13% in the earliest weeks to approximately 31% in the most recent period.North America remained the largest region, while Europe contributed a stable mid-teens to low-twenties share.
- Language distribution: English accounted for more than 80% of all tokens, while Simplified Chinese represented nearly 5%.Chinese, Russian, and Spanish formed a meaningful multilingual tail in detected prompt language.
- Implications: Cross-regional usability across languages, compliance regimes, and deployment settings is becoming important as adoption is both global and locally optimized.The paper presents this as an implication for model builders and infrastructure operators.
7 Analysis of LLM User Retention
Retention analysis reveals foundational cohorts whose users remain engaged far longer than later cohorts, linking durable adoption to a persistent workload–model fit. The paper names this pattern the Cinderella “Glass Slipper” effect.
- Foundational cohorts: A small set of early user cohorts shows durable retention beneath generally high churn and rapid cohort decay.These foundational cohorts represent persistent workload–model fits rather than merely early adoption.
- Foundational cohorts: The June 2025 Gemini 2.5 Pro cohort and May 2025 Claude 4 Sonnet cohort retained approximately 40% of users at Month 5.The paper associates these cohorts with breakthroughs such as reasoning fidelity or tool-use stability.
- Cohort variation: GPT-4o Mini exhibited a dominant July 2024 foundational cohort, while later cohorts clustered at the bottom of its retention chart.The pattern is presented as a narrow launch window for establishing a sticky workload–model fit.
- Cohort variation: Gemini 2.0 Flash and Llama 4 Maverick showed no high-performing foundational cohort, with every cohort performing similarly poorly.The paper interprets this as evidence that no frontier workload fit was established for these models.
- Cohort variation: DeepSeek cohorts displayed resurrection jumps, with retention rising after initial churn for selected cohorts.Examples include DeepSeek R1’s April 2025 cohort around Month 3 and DeepSeek Chat V3-0324’s July 2025 cohort around Month 2.
- Implications: The Glass Slipper framework treats retention as a signal of capability inflection and a narrow temporal window for durable adoption.When a model solves a critical workload first, users may embed it across pipelines and face high switching friction.
8 Cost vs. Usage Dynamics
LLM workloads separate into distinct cost–usage segments: roleplay and programming drive high-volume demand, while technology combines high cost with high usage. Pricing alone weakly predicts demand because capability, reliability, and task value shape model selection.
- Market Segmentation: High-usage, low-cost workloads include roleplay and programming, making both conversational entertainment and professional productivity major volume drivers.Programming has the highest usage volume, while roleplay nearly rivals it.
- Market Segmentation: Lower-volume, high-cost workloads include finance, academia, health, and marketing, where domain specificity and reliability are important.These categories are characterized as specialized professional applications rather than mass-market uses.
- Market Segmentation: Technology is a high-cost, high-usage outlier, indicating a market for complex answers where users may accept premium pricing.The passage identifies system architecture and advanced technical queries as examples of potentially high-value demand.
- Pricing and Demand: 10% lower prices correspond to only about a 0.5–0.7% increase in usage, indicating weak aggregate price sensitivity despite substantial market segmentation.The relationship is reported from a log–log usage-versus-cost analysis.
- Model Archetypes: Model usage reflects multiple archetypes, including premium leaders, efficient giants, long-tail models, and premium specialists.Premium leaders combine high cost and high usage, while premium specialists combine high cost with low usage.
- Model Selection: Users balance price against quality, reliability, capability, and integration, so cheaper models can scale while expensive models retain demand for high-value tasks.The market remains differentiated rather than fully commoditized.
9 Discussion
The discussion presents LLM usage as heterogeneous across models, applications, geographies, costs, and retention patterns. It highlights open-model adoption, creative roleplay, agentic inference, global demand, and workload–model fit as central features of deployment.
- A Multi-Model Ecosystem: No single model dominates all usage: closed and open models both capture substantial shares, with DeepSeek and Qwen sometimes exceeding 30% collectively.The authors characterize this as a model-agnostic and heterogeneous ecosystem.
- Usage Diversity Beyond Productivity: Over half of open-source model usage is devoted to roleplay and storytelling, challenging the assumption that LLMs are used mainly for productivity.The discussion also notes substantial early casual and creative use on proprietary platforms.
- Agents vs Humans: Agentic inference is rising through multi-step queries, chained tool use, external-data access, and iterative refinement rather than single-turn responses.The discussion frames this shift as changing evaluation toward sustained reasoning and process-oriented performance.
- Geographic Outlook: Asia’s share of total token demand rose from about 13% to 31%, alongside stronger enterprise adoption and China’s emergence as a model producer.The discussion links this geographic shift to the need for multilingual and culturally adaptable systems.
- Cost vs. Usage Dynamics: Price alone explains little about usage because users trade off cost against reasoning quality, reliability, and breadth of capability.Open models tend to serve lower-cost, high-volume tasks, while closed models retain high-value workloads.
- Retention and the Cinderella Glass Slipper Phenomenon: Retention depends on early workload–model fit: foundational cohorts remain engaged after finding a capability that solves an important need.The paper names this persistent fit the Cinderella “Glass Slipper” phenomenon.
- Implications: Real-world usage varies by country, state, and use case, making large-scale behavioral measurement important for understanding LLM deployment.The authors position empirical usage analysis as a foundation for future study and deployment decisions.
10 Limitations
The study provides only a partial view of the broader LLM ecosystem because it analyzes one platform over a finite time window and uses proxy measures for some behaviors and geographies.
- Scope Boundary: The findings are limited to OpenRouter over a finite time window, excluding some enterprise, locally hosted, and closed internal deployments.The authors therefore describe the results as indicative behavioral patterns rather than definitive ecosystem measurements.
- Measurement Limits: Agentic inference is proxied through multi-step or tool-invocation calls, while geography is inferred from billing rather than verified location data.These proxies constrain how directly the analyses measure the underlying phenomena.
11 Conclusion
The conclusion depicts LLMs as a plural, globally distributed computational substrate whose use is shifting toward orchestrated, multi-step inference. It argues that future progress depends on empirical measurement and operational systems thinking rather than monolithic model bets.
- Computational Substrate: LLMs are becoming embedded in workflows, applications, and agentic systems across domains including programming and creative writing.The conclusion frames them as an increasingly important computational substrate for reasoning-like tasks.
- Reasoning and Evaluation: Reasoning is increasingly evaluated through planning, verification, process metrics, latency–cost tradeoffs, and success on tasks under orchestration.This follows the emergence of extended deliberation and tool use in o1-class systems.
- Plural Ecosystem: The ecosystem is structurally plural: users select among models using capability, latency, price, and trust rather than relying on one dominant provider.The conclusion presents this heterogeneity as a fundamental market property.
- Agentic Inference: Multi-step and tool-linked interactions shift inference from static completion toward dynamic orchestration across models, APIs, and tools.The conclusion labels this pattern agentic inference.
- Global Distribution: Asian usage is expanding, while China is emerging as both a model developer and exporter, reinforcing the global character of LLM infrastructure.The conclusion points to Moonshot AI, DeepSeek, and Qwen as examples.
- Future Direction: The field is moving toward systems thinking, instrumentation, and empirical usage analytics rather than monolithic model bets or leaderboard deltas.The next phase is described in terms of measuring task completion, reducing variance, and aligning behavior under distribution shifts.
Contributions
The study combined foundational platform and infrastructure work with experiment design, analysis, strategic guidance, and manuscript review. Contributors also supported implementation, testing, refinement, exploratory setup, and technical assistance.
- Contributions: The OpenRouter team provided the platform, infrastructure, datasets, and technical vision underlying the study.Alex Atallah, Chris Clark, and Louis Vichy contributed engineering groundwork and architectural direction.
- Contributions: Additional contributors supported implementation, testing, experimental refinement, exploratory setup, technical assistance, discussions, feedback, and manuscript review.
- Contributions: Malika Aubakirova led experiment design, implementation, data analysis, and preparation of the paper, while Anjney Midha shaped its framing and direction.
Appendix
The appendix includes a figure showing category sub-composition for the major domains of roleplay, technology, and programming. The figure indicates that all three domains have distinct internal patterns.
- Appendix: Figure 29 compares category sub-composition across the roleplay, technology, and programming domains.
- Appendix: All three domains exhibit distinct internal patterns.