PaperlayerView exact revision on arXiv

Source-linked AI summary

Sleep-time Compute: Beyond Inference Scaling at Test-time

Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, Joseph E. Gonzalez

arXiv:2504.13171v1cs.AIcs.CL

TL;DR

Test-time scaling can help LLMs solve difficult problems, but may impose high latency and inference cost, including repeated reasoning over shared context. This paper introduces sleep-time compute, which prepares context offline and reuses the resulting inferences at test time. It reports lower test-time compute for the same accuracy across two reasoning tasks, with further gains from scaling and amortizing sleep-time compute.

  • Problem

    Test-time compute can improve performance on difficult problems but adds substantial latency and cost, including redundant reasoning when related queries share context.

    Source: §1 · p. 1

  • Method

    Sleep-time compute uses offline inference to re-represent available context for later queries, allowing the representation to be reused across queries.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    About 5× less test-time compute achieves the same accuracy on Stateful GSM-Symbolic and Stateful AIME with sleep-time compute.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Amortizing sleep-time compute across related queries reduces average cost per question by 2.5×.

    Source: §1 · p. 3

    The experiments assume interactions have two phases, sleep-time and test-time, whereas real-world use can include context modifications between rounds and variable intervals between interactions.

    Source: §7 · p. 13

Abstract

from arXiv · show

Scaling test-time compute has emerged as a key ingredient for enabling large language models (LLMs) to solve difficult problems, but comes with high latency and inference cost. We introduce sleep-time compute, which allows models to "think" offline about contexts before queries are presented: by anticipating what queries users might ask and pre-computing useful quantities, we can significantly reduce the compute requirements at test-time. To demonstrate the efficacy of our method, we create modified versions of two reasoning tasks - Stateful GSM-Symbolic and Stateful AIME. We find that sleep-time compute can reduce the amount of test-time compute needed to achieve the same accuracy by ~ 5x on Stateful GSM-Symbolic and Stateful AIME and that by scaling sleep-time compute we can further increase accuracy by up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME. Furthermore, we introduce Multi-Query GSM-Symbolic, which extends GSM-Symbolic by including multiple related queries per context. By amortizing sleep-time compute across related queries about the same context using Multi-Query GSM-Symbolic, we can decrease the average cost per query by 2.5x. We then conduct additional analysis to understand when sleep-time compute is most effective, finding the predictability of the user query to be well correlated with the efficacy of sleep-time compute. Finally, we conduct a case-study of applying sleep-time compute to a realistic agentic SWE task.

1 Introduction

Test-time compute can improve LLM performance on difficult tasks, but incurs substantial latency and cost when related queries require repeated reasoning over shared context. Sleep-time compute prepares context offline, reducing test-time compute requirements and allowing its inferences to be reused across queries.

1 Introduction

  • The evaluation modifies two mathematical reasoning datasets by separating each problem into a context and a question.Source: §1 · p. 2
  • About 5× less test-time compute achieves the same accuracy on Stateful GSM-Symbolic and Stateful AIME with sleep-time compute.The paper reports a Pareto improvement in the test-time compute–accuracy curve.Source: §1 · p. 2
  • Scaling sleep-time compute shifts the accuracy up by 13% on Stateful GSM-Symbolic and 18% on Stateful AIME.Source: §1 · p. 3
  • Amortizing sleep-time compute across multiple queries about the same context reduces average cost per question by 2.5×.Source: §1 · p. 3
  • Sleep-time compute is more effective when the user query is more predictable from the context.Source: §1 · p. 3

2 Related Work

The paper places sleep-time compute alongside test-time scaling, speculative decoding, and pre-computation. It presents offline context processing as an alternative way to use compute before a query arrives.

2 Related Work

3 Sleep-time Compute

Standard test-time compute reasons over context and query together, so shared-context queries can trigger redundant work. Sleep-time compute preprocesses context before the query and reuses the resulting representation to answer with a smaller test-time budget.

3 Sleep-time Compute

  • With standard test-time compute, each query about the same context receives an independent reasoning process, even when queries are related.Because context is processed only after the query arrives, users may wait several minutes for an answer.Source: §3 · p. 4
  • Sleep-time compute transforms context c into a re-represented context c′ using offline inference, then answers q from c′ with a smaller test-time budget b than B.The paper implements the preprocessing by prompting the model to draw inferences and rewrite the context for possible test-time use.Source: §3 · p. 4
  • The re-represented context c′ can be shared across queries about the same context, amortizing the preprocessing compute and reducing total cost.Source: §3 · p. 4

4 Experimental Setup

The experimental setup evaluates sleep-time compute on reasoning and test-time-scaling benchmarks using stateful datasets, selected models, and a standard test-time-compute baseline.

4 Experimental Setup

  • The experiments select datasets that are standard LLM reasoning and test-time-scaling benchmarks and show gains from scaling test-time compute with state-of-the-art models.Source: §4.1 · p. 4

4.1 Datasets

  • Stateful GSM-Symbolic and Stateful AIME are created by splitting existing problems into context and question; Figure 2 illustrates the GSM-Symbolic split.GSM-Symbolic uses P1 and P2 splits, while Stateful AIME contains 60 questions from AIME 2024 and 2025; Multi-Query GSM-Symbolic adds generated questions to shared contexts.Source: §4.1 · pp. 4–5, Figure 2 · p. 5

4.2 Models and Baselines

  • The study uses GPT-4o-mini and GPT-4o on GSM-Symbolic, and o1, o3-mini, Claude Sonnet 3.7 Extended Thinking, and Deepseek-R1 on AIME.Source: §4.2 · p. 5
  • The main baseline presents both context and query to the model for the first time at test time.A context-only baseline is also used to check whether the query is trivially predictable from context.Source: §4.2 · p. 5

5 Experiments and Results

The experiments test whether sleep-time compute improves test-time compute–accuracy trade-offs, can be scaled or amortized across queries, and helps more when queries are predictable. Across the supplied results, it improves the trade-off, with benefits varying by task, budget, and query predictability.

5.1 Improving Pareto Test-Time Trade-off with sleep-time compute

  • Sleep-time compute reaches comparable accuracy with 5× less test-time compute on Stateful GSM-Symbolic and shifts the test-time compute–accuracy frontier on Stateful AIME, though o1 shows limited gains.On Stateful GSM-Symbolic, the advantage is strongest at lower test-time budgets; at higher budgets, the baseline slightly outperforms it.Source: Figure 3 · p. 6, §5.1 · p. 6, Figure 4 · p. 7
  • Sleep-time compute generally outperforms parallel test-time scaling with pass@k at the same test-time token budget across the evaluated tasks and models.The comparison uses pass@k with oracle access to a ground-truth verifier, an assumption not made by sleep-time compute.Source: §5.1 · p. 7, Figure 5 · p. 8

5.2 Scaling up sleep-time compute

  • Up to 13% higher performance on Stateful GSM-Symbolic and 18% on Stateful AIME follows from scaling sleep-time compute, with 5 parallel generations generally outperforming 10.The GSM-Symbolic gains are largest on more difficult tasks with stronger models; the passage suggests additional sleep-time compute can benefit tasks with more complicated contexts.Source: §5.2 · p. 8, Figure 7 · p. 10, Figure 8 · p. 10

5.3 Amortizing sleep-time compute across queries with shared context

  • Up to 2.5× lower average cost per query is achieved with 10 queries per context, compared with the single-query baseline.The analysis weights test-time tokens at 10 times the cost of sleep-time tokens; savings depend on having multiple questions per context.Source: §5.3 · pp. 8–9, Figure 9 · p. 11

5.4 Predictable queries benefit more from sleep-time compute

  • The accuracy gap between sleep-time compute and standard test-time compute widens as questions become more predictable from context on both GSM8K-Symbolic P1 and P2.Predictability is measured using the question’s log-probability given the context under the Llama2-70B base model.Source: §5.4 · pp. 9, 10, 12, Figure 10 · p. 11

6 A Case Study of Sleep-time Compute for Agentic SWE

The SWE-Features case study evaluates sleep-time compute on multi-file feature-development tasks. It helps at lower test-time budgets, but standard test-time compute can perform better at higher budgets.

6 A Case Study of Sleep-time Compute for Agentic SWE

  • SWE-Features targets tasks requiring edits across multiple repository files and introduces new features; sleep-time agents explore the repository before working on the target pull request.The benchmark draws from pull requests modifying at least three files, and related pull requests provide context.Source: §6 · p. 12
  • Performance is evaluated by comparing the predicted and ground-truth sets of modified files using F1 score.Source: §6 · p. 12
  • At lower test-time compute budgets, sleep-time compute achieves up to roughly a 1.5× decrease in test-time tokens, while standard test-time compute can perform better at high budgets.At high budgets, standard test-time compute has higher precision and comparable recall; the authors hypothesize sleep-time agents edit more files after exploring more of the repository.Source: §6 · p. 12

7 Discussion and Limitations

The discussion identifies query predictability and the simplifying two-phase interaction assumption as boundaries for sleep-time compute. It also frames the method as natural-language representation learning and suggests future work on synthetic-data generation.

7 Discussion and Limitations

  • Sleep-time compute is less effective when queries are difficult to predict from context or unrelated to it; standard test-time scaling may be preferable in those settings.Future work could identify contexts with predictable questions and allocate compute between sleep-time and test-time.Source: §7 · p. 13
  • Experiments assume interactions divide into sleep-time and test-time, while real use may include multiple interaction rounds, context changes, and variable idle periods.The paper calls for extending the paradigm to handle these scenarios.Source: §7 · p. 13
  • The approach represents context in natural language to make it more useful for answering later queries, unlike traditional representation learning in parameter or activation space.The paper proposes further exploration of natural-language representations.Source: §7 · p. 13
  • Sleep-time compute could help amortize the cost of generating synthetic pretraining data across related queries, or its output could itself serve as synthetic data.The passage motivates this direction by noting that generating pretraining-scale synthetic data can be expensive.Source: §7 · p. 13

C Details on Multi-Query GSM-Symbolic

  • Multi-Query GSM-Symbolic samples one instance per template, separates it into context and question, and synthetically generates additional questions.The appendix provides an example with multiple questions about the same context.Source: Table 1 · p. 19

D SWE-Features Details

SWE-Features details describe a small multi-file feature benchmark, its related-pull-request context, and file-set F1 evaluation. The appendix also reports that explicit step tracking improves agent performance, especially under low budgets.

D SWE-Features Details

  • SWE-Features examples add new functionality spanning multiple files and are filtered for self-contained tasks; the dataset contains 33 pull requests from two repositories.Related pull requests are clustered so examples can provide context for a target task.Source: D SWE-Features Details · p. 20
  • For sleep-time compute, the agent explores the repository using related pull requests before receiving the target task; the baseline receives the related pull request only at test time.Source: D SWE-Features Details · pp. 20–21
  • The test-time budget is controlled by fixing the maximum number of chaining steps, with the prompt instructing the agent to finish within that limit.Source: D SWE-Features Details · p. 21
  • Explicitly appending the current and total step counts to tool returns significantly improves agent performance, especially in low-budget settings.The authors use a fixed maximum number of steps to control test-time budget.Source: D SWE-Features Details · p. 21
  • Evaluation compares the agent’s predicted modified-file set with the ground-truth modified-file set using F1 score.The evaluation extracts files marked modified or new, but scores files with modified status.Source: D SWE-Features Details · p. 21

I Context-Only Baseline

A context-only baseline tests whether Stateful AIME and Stateful GSM-Symbolic questions can be guessed from context alone. Sleep-time compute significantly outperforms that baseline, while Stateful AIME examples are constructed by separating problem statements into context and query.

I Context-Only Baseline

  • Sleep-time compute significantly outperforms a context-only baseline on Stateful AIME and Stateful GSM-Symbolic, indicating that their questions are not trivially predictable from context.The baseline receives only context and must guess the likely question and answer.Source: I Context-Only Baseline · p. 26, Figure 22 · p. 27, Figure 21 · p. 27

J Stateful AIME Construction

  • Stateful AIME uses all but the final statement of each problem as context and the final statement as the query, with manual rearrangement when the question appears earlier.Problems with only one statement have empty context.Source: J Stateful AIME Construction · pp. 27–28
  • When figure LaTeX could reveal answer information, it is removed after ensuring the problem is solvable without the figure.Source: J Stateful AIME Construction · p. 28

K Implementation Details

Sleep-time compute is implemented through function calls that let the model revise its context or end the process, with up to 10 context-revision calls.

K Implementation Details

  • The implementation gives the model access to rethink memory and finish rethinking functions during sleep-time compute.The former replaces the current context with a new string; the latter terminates the process.Source: K Implementation Details · p. 28
  • The rethink memory function replaces the current context with a new string.The model may call this function up to 10 times.Source: K Implementation Details · p. 28
  • The finish rethinking function terminates the sleep-time compute process.Source: K Implementation Details · p. 28

M AIME sleep-time compute scaling results by year

The section presents AIME main-result figures for 2024 and 2025, alongside separate figures on scaling sleep-time compute for each year.

M AIME sleep-time compute scaling results by year

Related papers

Loading 2504.13171v1…