PaperlayerView exact revision on arXiv

Source-linked AI summary

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, Shlok Natarajan, Etash Kumar Guha, Shang Zhu, Ben Athiwaratkun, John Hennessy, Azalia Mirhoseini, Christopher Ré

arXiv:2511.07885v6cs.DCcs.AIcs.CLcs.LG

TL;DR

Centralized cloud inference faces growing demand, while local models and accelerators may serve many queries within device power constraints. The paper introduces IPW and evaluates local inference across models, hardware, and over 1M queries. It finds broad local coverage and substantial efficiency improvement, supporting local inference as a complement to centralized infrastructure.

  • Problem

    The paper asks whether local inference can viably redistribute demand from centralized infrastructure by combining sufficient query capability with power-efficient execution.

  • Method

    The study evaluates 20+ local models, 8 accelerators, and over 1M single-turn chat and reasoning queries using IPW, task accuracy per unit of power.

  • Results

    Local models handle 88.7% of queries, while IPW improves 5.3× from 2023–2025 as locally serviceable coverage rises from 23.2% to 71.3%.

  • Takeaways & Limitations

    Local inference can serve a substantial subset of workloads as a practical complement to centralized infrastructure, with IPW tracking this transition.

  • Takeaways & Limitations

    Software-based power measurements can introduce 10–15% inaccuracies, though the study reports consistent relative comparisons across configurations.

Abstract

from arXiv · show

Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure. Demand growth strains this paradigm faster than providers can scale. Two advances create an opportunity to rethink it: small, local LMs (<=20B active parameters) now achieve competitive performance to frontier models on many tasks, and local accelerators (e.g., Apple M4 Max) can host these models at interactive latencies. This raises the question: can local inference viably redistribute demand from centralized infrastructure? This requires measuring both whether local LMs can accurately answer real-world queries and whether they can do so efficiently on power-constrained devices (e.g., laptops). We propose intelligence per watt (IPW), task accuracy per unit of power, as a unified metric for the capability and efficiency of local inference across model-accelerator configurations. We evaluate 20+ state-of-the-art local LMs, 8 hardware accelerators (local and cloud), and 1M real-world single-turn chat and reasoning queries. For each query, we measure accuracy (local LM win rate against frontier models), energy, latency, and power. We find three key results. First, local LMs successfully answer 88.7% of these queries, with accuracy varying by domain. Second, longitudinal analysis from 2023-2025 shows IPW improved 5.3x, driven by both algorithmic and accelerator advances, with locally-serviceable query coverage rising from 23.2% to 71.3%. Third, local accelerators achieve at least 1.4x lower IPW than cloud accelerators running identical models, revealing significant headroom for local accelerator optimization. These findings demonstrate that local inference can meaningfully redistribute demand from centralized infrastructure for a substantial subset of queries, with IPW serving as the critical metric for tracking this transition.

1 Introduction

Local inference is becoming a plausible complement to centralized cloud serving as small models and local accelerators improve. The paper introduces intelligence per watt (IPW) and evaluates local capability, efficiency, and workload redistribution across models, hardware, and real-world queries.

  • 1 Introduction: 5.3× IPW growth and rising local coverage support redistributing a substantial subset of inference demand from centralized infrastructure.Local coverage rises from 23.2% to 71.3%, while improvements reflect both model and hardware advances.
  • 1 Introduction: Small local models and capable local accelerators create the conditions for interactive on-device inference despite centralized systems’ growing resource demands.The paper frames this transition as analogous to computing moving from mainframes to personal computers when efficiency met device power constraints.
  • 1 Introduction: IPW measures task accuracy per unit of power, jointly capturing local-model capability and accelerator efficiency under constrained power budgets.The metric enables comparisons across model–accelerator configurations and tracks gains from model, post-training, and hardware innovations.
  • 1 Introduction: The empirical study asks how much demand local systems can serve, how IPW has evolved, and what savings hybrid local-cloud routing can achieve.Its questions cover query coverage over time, model-versus-accelerator progress, and compute, energy, and cost savings.
  • 1 Introduction: 88.7% of single-turn chat and reasoning queries can be handled by small local models, with coverage varying by domain.The study evaluates 20+ models, 8 accelerators, and more than 1M queries spanning 2023–2025.

2 Preliminaries

The paper formalizes local and cloud inference as model–accelerator systems connected by a routing function. It defines complementary power- and energy-based metrics to measure capability delivered relative to instantaneous draw and total query energy.

  • 2 Preliminaries: Local models have ≤20B active parameters and cloud models ≥100B, while routing assigns each query to either local or cloud inference.Active rather than total parameters determine per-query inference efficiency; total parameters primarily govern storage.
  • 2 Preliminaries: The infrastructure model distinguishes local accelerators such as Apple M4 and AMD Ryzen from cloud accelerators such as NVIDIA H200 and AMD MI300X.This abstraction supports comparisons across local and cloud model–hardware configurations.
  • 2 Preliminaries: Accuracy per watt divides average query accuracy by average power, measuring steady-state intelligence efficiency for a model–accelerator pair.The metric complements energy-based measures that incorporate generation latency.
  • 2 Preliminaries: Accuracy per joule divides average accuracy by average power multiplied by latency, capturing total energy consumed per query.For each query, P(m,h,q)·τ(m,h,q) represents energy in joules.
  • 2 Preliminaries: Power-based metrics reflect instantaneous hardware efficiency, whereas energy-based metrics account for both power consumption and response-generation latency.Together, these metrics provide complementary views of inference efficiency.

3 Dataset and Profiling Harness

The study combines naturalistic and standardized query datasets with broad model and accelerator coverage, then profiles each configuration using reproducible cross-platform telemetry. Its measurements include accuracy, latency, throughput, time-to-first-token, and power-related efficiency signals.

  • 3 Dataset and Profiling Harness: The dataset combines over 1M queries from naturalistic chat, general reasoning, knowledge, and expert-level reasoning benchmarks.Sampling includes 500K WILDCHAT, 500K NATURALREASONING, 12K MMLU PRO, and 26.5K SUPERGPQA queries.
  • 3 Dataset and Profiling Harness: The evaluation spans diverse local, workstation, datacenter, and smartphone-class accelerators, including Apple, NVIDIA, AMD, and SambaNova systems.The main accelerator set includes Apple M4 Max, multiple NVIDIA generations, AMD MI300X, and a smartphone-class Apple A18 Pro extension.
  • 3 Dataset and Profiling Harness: Models cover QWEN3, GPT-OSS, GEMMA3, and IBM GRANITE families across multiple sizes, alongside state-of-the-art cloud models.This supports comparisons across model generations and parameter scales.
  • 3 Dataset and Profiling Harness: For each query–model–hardware triple, the harness collects accuracy, latency, throughput, time-to-first-token, and related efficiency metrics.Responses are evaluated with LLM-as-a-judge or benchmark ground truths, depending on the dataset.
  • 3 Dataset and Profiling Harness: Software-based power measurements can introduce 10–15% inaccuracies, although the cross-platform approach provides consistent relative comparisons.The harness uses platform-specific telemetry because CODECARBON’s sampling and hardware support are insufficient for fine-grained cross-platform attribution.

4 Intelligence Efficiency Study

Across 2023–2025, local inference became substantially more capable and efficient, while model diversity and query routing expanded the share of workloads that can be handled locally.

  • Q1 Workload coverage: 88.7% overall coverage is achieved by routing queries across diverse local models, exceeding individual-model performance by exploiting complementary capabilities.Best-of-local improves coverage by 28.8 percentage points over QWEN3-14B and 16.3 points over GPT-OSS-120B; on reasoning benchmarks it can surpass best-of-cloud through best-of-N selection.
  • Q1 Workload coverage: Local coverage varies substantially by task: the best local LM reaches 88.9% on chat but 64.9% on natural reasoning, while technical disciplines remain harder.Standardized benchmarks show best-of-local coverage of 93.4% on MMLU PRO and 83.6% on SUPERGPQA, dropping to 60% for Architecture and Engineering.
  • Q2 Intelligence efficiency: 18.0× intelligence-per-joule improvement over 16 months decomposes into 3.0× from local models and 5.9× from local accelerators.The metric decomposition differs by efficiency axis: model progress dominates per-watt gains, whereas accelerator progress dominates per-joule gains because it also reduces latency.
  • Q2 Intelligence efficiency: 5.3× accuracy-per-watt improvement from 2023 to 2025 coincides with local query coverage rising from 23.2% to 71.3%.The progression spans MIXTRAL-8X7B on Quadro RTX 6000, LLAMA-3.1-8B on RTX 6000 ADA, and GPT-OSS-120B on Apple M4 Max.
  • Q2 Intelligence efficiency: Cloud accelerators retain a 1.40× IPW advantage over Apple M4 Max on identical workloads, with larger per-joule gaps because cloud hardware completes queries faster.The comparison uses batch size 1 for both local and cloud systems, so the gap is not attributed to batching.
  • Q3 Query routing: 64.3% energy, 61.8% compute, and 59.0% cost savings are achieved by an 80%-accurate router versus sending all queries to QWEN3-235B.The simulation routes 80.2M queries between four local models on Apple M4 Max and a batched cloud frontier baseline, capturing most oracle gains.

5 Conclusion and Key Takeaways

The study presents IPW as a framework for assessing local inference and finds substantial local query coverage, rapid efficiency gains, and practical benefits from hybrid routing. It also identifies model, hardware, routing, and workload directions for extending the analysis.

  • Key findings: 88.7% of single-turn chat and reasoning queries can be handled locally, while IPW improved 5.3× over two years through model and hardware advances.The study covers 20+ models, 8 accelerators, and 1M queries spanning 2023–2025.
  • Key findings: 60–80% aggregate energy, compute, and cost reductions are achievable with hybrid local-cloud routing at realistic routing accuracy.
  • Metric and scope: IPW unifies local-model capability and accelerator efficiency for evaluating whether local inference can redistribute demand from centralized infrastructure.
  • Practical takeaways: The paper recommends MoE models on memory-rich devices, aggressive FP4 quantization, and expanding local-model ensembles once router accuracy exceeds approximately 80%.It also identifies multimodal inference and hybrid local-cloud execution as future extensions.
  • Related work: The work is positioned alongside prior research on routing, collaborative local-cloud systems, and energy-aware AI benchmarking.

B.1 Dataset Curation

The dataset curation combines naturalistic chat and reasoning queries with standardized benchmarks, assigns queries to economic domains, and evaluates local solvability using judge-based or exact-answer comparisons. Coverage varies substantially by dataset and domain.

  • Dataset design: The curated evaluation combines naturalistic queries with standardized benchmarks to measure deployment behavior, knowledge breadth, and reasoning across domains.The study samples from WILDCHAT, NATURALREASONING, MMLU PRO, and SUPERGPQA.
  • Dataset design: Queries are mapped to 22 Anthropic Economic Index categories adapted from U.S. labor frameworks to support domain-level analysis.
  • Dataset composition: WILDCHAT is dominated by arts, design, sports, entertainment, and media queries, whereas NATURALREASONING emphasizes life, physical, and social science and computer and mathematical queries.
  • Solvability results: Solvability varies sharply by dataset: Architecture and engineering reaches 99.4% in WILDCHAT but only 41.5% in NATURALREASONING.WILDCHAT is generally above 94% across domains, while reasoning queries are more variable.
  • Evaluation: The evaluation uses LLM judges for open-ended tasks and exact multiple-choice comparisons for SUPERGPQA and MMLU PRO.The judge prompts distinguish subjective or open-ended criteria from objective technical correctness.
  • Domain analysis: Figure 5 compares local-model handling with cloud routing across economic domains using stacked fractions of single-turn queries.

C Limitations and Broader Impacts (Extended)

The study’s conclusions are bounded by telemetry precision, workload coverage, evaluation methodology, hardware and runtime choices, and the rapidly changing ecosystem. Broader impacts and future work emphasize extending analysis to agentic, long-context, and multimodal workloads.

  • Limitations: Software telemetry can introduce 10–15% measurement inaccuracies, so absolute energy values should be interpreted cautiously despite consistent relative comparisons.
  • Limitations: The main analysis focuses on single-turn chat and reasoning, while multi-turn conversations, agents, tool use, and long-context applications may have different routing tradeoffs.
  • Broader impacts: Local inference could benefit energy consumption, infrastructure cost, and access where cloud connectivity is unreliable, expensive, or undesirable for privacy reasons.
  • Future coverage: Inputs are capped at 32,000 characters, whereas production workloads may reach 100K–1M tokens with prefill energy and KV-cache pressure becoming binding constraints.
  • Future coverage: The study is text-only, leaving vision-language, audio, and video workloads with unmeasured modality-specific compute and memory costs.

D.2 Pushing the Local-AI Frontier

The paper identifies remaining barriers to local-AI expansion in hard reasoning, accelerator design, quantization, serving systems, and hybrid execution.

  • Reasoning frontier: 95% of Level 5 reasoning queries remain unsolved by current local LMs, leaving long-chain reasoning within local power budgets as an open challenge.The authors call for reasoning-focused post-training, architectures, and test-time compute strategies suited to local deployment.
  • Accelerator design: Local accelerators trail cloud accelerators by 1.4–7.4× in IPW on identical workloads, motivating NPU-first, sparsity-aware, and co-designed systems.Smartphone-class results nevertheless show approximately 7× higher IPW than workstation GPUs at approximately 12 W.
  • Quantization: FP16→FP4 reduces energy 3–3.5× while costing approximately 2.5 percentage points of accuracy per step, but sub-FP4 regimes require accuracy-preserving training and hardware support.INT2, ternary, and binary precision are presented as open quantization directions.
  • Serving and hybrid execution: Serving framework and batch size shift absolute IPW but preserve rankings, while hybrid local-cloud execution requires an expanded metric aggregating heterogeneous power and energy.The current IPW definition evaluates isolated model–accelerator pairs rather than split-query systems.
  • Routing and domains: Routing gains saturate near 80% of oracle improvement, while a 17.3-point best-single versus best-local gap makes ensemble diversity especially valuable.The practical rule is to expand the local-model ensemble after routing accuracy reaches roughly 80%.

E.2 How has local LM task coverage changed over different “difficulty” slices of the data

Local-model coverage improves sharply for easier chat and reasoning queries, but the hardest reasoning slice remains largely unsolved despite recent progress.

  • Chat difficulty: 2025 chat models achieve 98–99% success on Levels 1–3 and 92.6% on Level 4, with absolute gains reaching 76.4 percentage points.Difficulty levels are defined by the smallest SOTA model size needed to solve each query.
  • Reasoning difficulty: Reasoning progress is strong through Level 3 but slows sharply thereafter: Level 4 rises from 7.93% to 31.72%, while Level 5 improves only from 3.27% to 4.72%.The hardest reasoning problems remain a substantial frontier for local models.
  • Reasoning frontier: Level 5 reasoning remains a wide-open frontier, with only 51% success in the benchmark’s saturation pattern and 134 problems representing 16.5% of the reasoning dataset remaining 95% unsolved.Levels 1–2 are near-complete at 98–99%, while Levels 3–4 approach saturation at 85–92%.
  • Efficiency trends: Across April 2024–August 2025, the study tracks perplexity and accuracy normalized by joules per query across nine model families and varied GPU configurations.The cited figure evaluates intelligence efficiency across model–hardware pairs for single-turn chat and reasoning queries.
  • Quantization: FP16→FP4 quantization delivers 3–3.5× energy reductions with approximately 2.5 percentage points of accuracy degradation per precision step.The tradeoff is measured across eight QWEN3 and GEMMA models and three reasoning datasets; QWEN3-14B loses 5.5 points on SuperGPQA while reducing energy 3.23×.

E.6 How do SOTA open-source LMs compare to the SOTA closed-source LMs on Chat and Reasoning Queries?

Open-source frontier-scale models nearly match closed-source models on knowledge and expert reasoning benchmarks, but gaps widen on naturalistic reasoning and under local deployment constraints.

  • Benchmark comparison: QWEN3-235B-A22B trails closed-source models by 5.1% on MMLU Pro and 3.4% on SuperGPQA, but by 12.9% on NATURALREASONING.The comparison uses three closed-source frontier models and eight open-source models across chat and reasoning benchmarks.
  • Local deployment: The best deployment-constrained local model, QWEN3-14B, trails closed-source models by 11.8–13.2% across benchmarks, with the largest gap on NATURALREASONING.The smallest gap occurs on MMLU Pro, while naturalistic reasoning remains the most difficult comparison.
  • Evaluation scope: The evaluation covers WILDCHAT, NATURALREASONING, MMLU Pro, and SuperGPQA, with Qwen3-235B-A22B serving as the WILDCHAT reference model.The tables distinguish closed-source, all open-source, and local models with at most 20B active parameters.

E.7 How does performance on chat and reasoning queries connect to U.S. GDP?

GDP-weighted coverage is substantially higher for conversational tasks than for naturalistic reasoning, indicating that reasoning capability remains the main constraint on economically relevant local-model coverage.

  • Metric construction: GDP-weighted accuracy estimates addressable economic value by weighting model performance in each category by that sector’s contribution to 2024 U.S. GDP of $29.18T.The mapping uses 22 economic categories linked to U.S. Bureau of Economic Analysis GDP-by-industry data.
  • GDP-weighted coverage: WILDCHAT reaches 69.6% of U.S. GDP coverage with GPT-OSS-120B, whereas NATURALREASONING reaches only 23.3% with QWEN3-235B-A22.The corresponding task accuracies are 89.2% for chat and 69.3% for reasoning.
  • Economic opportunity: The 46.3-point gap between chat and reasoning coverage corresponds to an estimated $13.5T difference in addressable GDP-relevant tasks.The paper associates the disparity with weaker technical reasoning in sectors such as architecture, engineering, and physical sciences.
  • Scope and caveat: The GDP analysis is an upper-bound estimate rather than a realized-impact prediction because it treats benchmark accuracy as a proxy for automation and uses an approximate category-to-industry mapping.Deployment also depends on tools, trust, regulation, and the genuinely AI-amenable share of occupational work.

E.8 How do local accelerators compare to cloud accelerators in terms of intelligence efficiency?

Cloud accelerators are more efficient than local accelerators for identical models, especially when latency is included, although local model capability and deployability have improved substantially. Local deployment nevertheless offers system-level benefits and increasingly broad query coverage.

  • Cloud-versus-local efficiency: 1.40× lower intelligence per watt is achieved by Apple M4 Max than NVIDIA B200 and SambaNova SN40L on identical QWEN3 models.Across QWEN3 model sizes, NVIDIA B200 achieves 1.39×–1.40× higher intelligence per watt than M4 Max; SambaNova is more efficient on larger models.
  • Cloud-versus-local efficiency: 1.6×–2.3× higher intelligence per joule is achieved by NVIDIA B200 than Apple M4 Max, while SambaNova reaches 6.5×–7.4× higher efficiency.Per-joule comparisons incorporate both power consumption and generation latency, making cloud advantages larger than in per-watt comparisons.
  • Local capability and deployability: 78.2% win/tie rate on WILDCHAT and 80.9% accuracy on NATURALREASONING show rapid local-model capability gains from April 2024 to August 2025.WILDCHAT improved from 28.0% to 78.2%, while NATURALREASONING rose from 48.7% to 80.9%.
  • Local capability and deployability: 126.3× memory growth in consumer accelerators from 2012 to 2025 expanded local deployment of increasingly capable models.Consumer accelerators increased from roughly 10–20 GB in 2020 to 128–512 GB through unified-memory architectures.
  • System-level implications: 88.7% of correctly handled queries can bypass cloud compute, offsetting local per-query efficiency disadvantages through reduced infrastructure, latency, and API costs.The main comparisons use batch size 1 for realistic single-user local inference, while routing simulations use cloud batch size 16 and report 60–80% savings.

E.11 Are our findings consistent on smartphone-class accelerators?

Smartphone-class accelerators can serve lightweight local queries with high power efficiency, but memory, quantization, thermal limits, and latency constrain their broader use. Related multi-turn evaluations preserve the qualitative local-versus-cloud trade-offs while remaining preliminary.

  • Smartphone-class accelerators: 11.2–13.3×10^-3 accuracy per watt makes the iPhone 16 Pro approximately 7× more power-efficient than workstation GPUs running the same model.The advantage reflects the iPhone’s roughly 12 W SoC power envelope versus 220–240 W for workstation GPUs.
  • Smartphone-class accelerators: FP4 is essentially required for smartphone deployment, while FP8 is the practical ceiling for sustained interactive use within the iPhone’s 8 GB memory.FP4 causes approximately 6.5 percentage points of accuracy degradation on Qwen3-4B versus workstation FP4.
  • Smartphone-class accelerators: 9–26× higher per-query latency on the A18 Pro narrows or reverses its per-joule advantage despite lower power consumption.The A18 Pro’s 60 GB/s memory bandwidth contrasts with 546 GB/s on M4 Max.
  • Smartphone-class accelerators: Very-low-difficulty queries could be routed to mobile NPUs, potentially extending energy savings beyond the reported 60–80%, though full mobile-tier routing remains future work.The proposed extension is limited to lightweight queries and is not fully characterized in this study.
  • Multi-turn agentic workloads: 2.4–3.0× higher per-joule efficiency on cloud hardware persists on GAIA and TerminalBenchV2, while local hardware retains 3.7–3.8× higher per-watt efficiency.The strongest local model remains within approximately 2.2 percentage points of cloud accuracy on both benchmarks, but the 245-query evaluation is only a sanity check.
Loading 2511.07885v6…