ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality

1 INTRODUCTION

• ReLiveGym evaluates long-lived agents on eight tasks spanning weeks of chronologically replayed real-world streams.The streams include news, social media, and market data.Source: §1 · p. 1
• Tasks vary in timing and reasoning demands, and some provide delayed feedback that enables continuous test-time improvement.Some tasks favor fixed-interval schedules, while others require timely, event-driven reactions.Source: §1 · pp. 1–2
• Trigger effectiveness varies by task and sometimes base model; GPT-family models lead on reasoning-intensive tasks, while Qwen models lead on routine tasks.The authors also find that base-model rankings are fairly consistent across tasks and that trigger mechanisms and continuous-improvement methods can improve performance stability.Source: §1 · p. 2
• Best-of-N trajectory selection can mitigate persistence with a poor initial policy, while memories curated from hindsight feedback improve performance less consistently.The authors identify persistence with a poor policy chosen at the outset as an observed failure mode.Source: §1 · p. 2

2 PROBLEM FORMULATION

• An agent comprises a harness and a base LLM; together they induce a policy mapping interaction history to the next action.The harness defines prompts and tools, initiates turns, and manages context and memory.Source: §2 · p. 2
• A long-lived task fixes an environment, instruction, cost budget, time horizon, and metric, while the world evolves exogenously whether or not the agent watches.The agent cannot observe future environmental changes, and commitment scores depend on their timing.Source: §2 · p. 3
• Solutions may range from fixed workflows to deterministic programs, but the evaluation focuses on ReAct agents and fixed-cadence ReAct agents for transferability across tasks.The general ReAct loop can be transferred by changing its instruction and available tools.Source: §2 · p. 3

3.1 CURATING TASKS AND SIMULATED WORLDS FOR RELIVEGYM TASKS

• The task suite uses timestamped news, market, financial-report, and social-media data, with simulated worlds serving content as of the replay clock.The curation pipeline samples data beyond evaluated models’ known knowledge cutoffs to minimize contamination and is released for evaluation on newer environments.Source: §3.1 · p. 3

3.2 RELIVEGYM TASKS AND CAPABILITY AXES

• Tasks range from fixed daily schedules to sharply time-sensitive alerts or delayed claims, and from deterministic rules to uncertain reasoning across sources.Breakout news detection rewards alerts before a market price move; resolution detection penalizes claims made before a controversy is settled; popularity prediction and breakout news detection provide hindsight feedback.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 EVALUATING AGENTS ON RELIVEGYM

• At each simulated time, agents see only information already available in the replay; the clock advances when they wait, and generation latency is not modeled.The simplification reflects that agents act sparsely over simulated periods of several weeks.Source: §3.3 · p. 4
• Runs specify the task, metric, evaluation-window end, and hard cost budget; agents receive daily spending updates and are scored by comparing timed commitments with ground truth.A run terminates if the budget is exhausted.Source: §3.3 · p. 4

4 EXPERIMENT SETUP

• The experiments ask how base LLM and harness designs affect task performance, and what failure modes arise and how harness design and continuous improvement address them.Source: §4 · p. 4

4.1 TRIGGER MECHANISMS

• Three trigger mechanisms give agents different ways to schedule wakes, and all can schedule them at any timestamp in the evaluation horizon.The sleep harness lets the LLM choose a duration; watcher programs can return control to the LLM; cron jobs run bounded episodes on scheduled triggers.Source: §4.1 · p. 5

4.2 LEARNING FROM HINDSIGHT FEEDBACK

• Hindsight feedback is tested through daily outcome memories and Best-of-N + Prune, which accepts commitments from the best-performing of three agents and prunes on days 2 and 7.Outcome memories add past commitments, settled outcomes, and statistics to agent context.Source: §4.2 · p. 5
• Periodic harness optimization improves performance only with strong optimizer models, such as GPT-5.6-sol, which exhaust the budget.Source: §4.2 · p. 5

4.3 MODEL AND HARNESS DETAILS

• Eight cost-efficient base LLMs were evaluated; task harnesses used pinned OpenCode with Playwright or customized ReAct with a 60,000-token context limit.Stronger models were skipped when cheaper models exhausted $20 per run; effects of other harness axes such as context length were left for future work.Source: §4.3 · pp. 5–6

5.1 MODEL AND TRIGGERING MECHANISM EFFECTS ON PERFORMANCE

• GPT-5.6-luna or terra lead on reasoning-intensive tasks, while Qwen-3.7-plus leads on routine tasks.Trigger effectiveness also varies by task and sometimes by model.Source: §5.1 · p. 6
• On breakout news detection, watcher and cron beat sleep by about five coverage-F1 points for GPT models, but triggers are indistinguishable for Qwen; forecast portfolio favors sleep over cron.On browser-use tasks, sleep can skip entire days; cited weaker-model routine-task cases reach 100% under cron for daily digest and watcher for price check.Source: §5.1 · p. 6, Figure 5 · p. 7
• Seed-to-seed variation is the largest score component on every task; model dominates resolution detection (44%), forecast portfolio (21%), and web broker ops (15%).Model×trigger interaction is significant for breakout news detection (19%) and popularity prediction (38%); trigger main effects also appear on four tasks.Source: §5.1 · p. 7, Figure 6 · p. 7

5.2 ANALYSIS OF FAILURE MODES

• Agents can deviate late: one sleep-harness run operated normally for a week, then slept for three weeks, while a cron run delayed notifications.Source: §5.2 · pp. 7–8
• Early choices can persist: Qwen-3.7-plus popularity prediction scored F1 5.5 at 12-hour triggers versus 38 at six-hour intervals; cron runs can persist with no-ops.With sleep, weekly median intervals changed by less than one hour.Source: §5.2 · p. 8

5.3 UTILIZING TEST-TIME FEEDBACK

• Best-of-N + Prune improved every evaluated popularity-prediction cell, halved sleep-trigger seed variance, and raised average cost 1.61×.Examples: GPT-5.6-luna with sleep rose 43.3 → 49.2, and Qwen-3.7-plus with sleep rose 23.9 → 32.1.Source: §5.3 · p. 8, Table 1 · p. 8
• Memory changes were directionally positive in five of six popularity-prediction cells, except GPT-5.6-luna with sleep, but were less consistent than Best-of-N sampling.On breakout news detection, memory was directionally positive under sleep and watcher but negative under cron.Source: §5.3 · p. 8
• Continuous harness updates worked only with strong reflector models, whose cost exhausted the budget early in this setup.Source: §5.3 · p. 8

• Prior work spans long-horizon tasks, dynamic environments, real-time reasoning, and monitoring triggers; SentinelBench compares waiting mechanisms using success, reaction time, and resource use.Source: §6 · pp. 8–9
• FutureSim replays news chronologically over 90 days at a daily cadence; ReLiveGym shares replay but evaluates finer-grained trigger decisions across forecasting, detection, and routine automation.Source: §6 · p. 9
• Related continuous-improvement work studies feedback-based reflection and experience reuse, as well as optimizing prompts, skills, harnesses, or entire agent programs.Source: §6 · p. 9

7 CONCLUSION

• Best-of-N selection with pruning mitigates early failures left by reflection, while hindsight-curated memories improve performance less consistently.Across the experiments, base models and trigger mechanisms affect performance through standalone effects or interactions.Source: §7 · p. 9
• The study reports three to six runs per configuration and one to three episodes per task, aggregating comparisons rather than ranking individual model-trigger pairs.More seeds and replayed episodes can support cell-level benchmarking without changing the tasks or metrics.Source: §7 · p. 9
• Harness evaluation covers only triggering mechanisms and test-time learning, with three trigger mechanisms and two learning approaches tested.Context management is identified as an additional design axis for future evaluation.Source: §7 · p. 9

AI USE STATEMENT

• AI coding agents helped implement the simulated-time environment, data pipelines, agent loop, triggering mechanisms, and continuous-improvement algorithms; human authors reviewed all code.The environment included pipelines for batch-processing raw data such as Common Crawl news.Source: AI USE STATEMENT · p. 10
• AI assistance was used to create initial agent instructions, synthetic trades, and web portals, with all artifacts reviewed and revised by the human authors.Source: AI USE STATEMENT · p. 10
• AI also supported early brainstorming and related-work identification, later analysis of experimental records, writing polish, and vector-graphic generation.Source: AI USE STATEMENT · p. 10

Popularity Prediction

• Breakout detection recommends up to 10 posts predicted to reach 50 comments within 24 hours, scoring popular-post recommendations by time-weighted F1.First-recommendation credit declines with latency and reaches zero after six hours; recall is normalized against a cap-limited oracle, and precision is popular recommendations divided by settled recommendations.Source: Popularity Prediction · p. 16
• The daily digest task delivers 10 popular posts from the previous day, with on-time accuracy based on the first 10 distinct accepted ids.A scored day without a digest receives zero.Source: Popularity Prediction · p. 16
• The price-check task reports three assets hourly for 14 days, scoring a symbol-hour correct when its price is within five basis points of the cross-venue median.The replay includes nine provider incidents across the three exchanges.Source: Popularity Prediction · p. 16
• Web filing tracking scores each of 63 filings at the first NYSE open after acceptance, requiring a timely, accurate browser-entered record.Records include filing identifiers and acceptance time, plus fiscal period, revenue, and diluted EPS for periodic reports.Source: Popularity Prediction · p. 16
• Broker operations checks 31 duty instances: protective stops within two hours of fills and funding instructions within four hours of margin calls.On-time accuracy is the proportion of duties met by their deadlines.Source: Popularity Prediction · p. 16
• GPT-5.6 terra reaches the score ceiling under any trigger, while other models reach it only under the right trigger; triggers lift and steady below-frontier models.Figure 7 reports mean score ×100 over seeds, with whiskers showing one standard deviation.Source: Figure 7 · p. 17
• Every run’s LLM spending is computed from fixed per-million-token rates, with cached input charged at the prompt-cache read rate.Source: Table 3 · p. 17

A TRIGGERING MECHANISM DETAILS

• The sleep harness gives the agent a sleep tool that fast-forwards environment time to the specified time.Source: A TRIGGERING MECHANISM DETAILS · p. 17
• The watcher harness lets the agent author and run a Python watcher program using workspace tools and the envkit library.The initialized workspace includes instructions and example sleep and task-condition watcher scripts.Source: A TRIGGERING MECHANISM DETAILS · p. 17
• In the watcher ReAct loop, simulated time advances only through envkit.wait, while envkit.handover can return a program argument to the tool output.A loop that never waits is terminated after 200,000 executed lines of agent-authored code.Source: A TRIGGERING MECHANISM DETAILS · p. 17
• The cron harness uses an author-initialized, read-only baseline schedule, while agents may add recurring or one-time triggers.The baseline’s minimum frequency is task-specific.Source: A TRIGGERING MECHANISM DETAILS · p. 17
• Table 4 reports primary metrics by model and timing mode as means and standard deviations over three to six seeds.It also reports mean LLM spend when no seed exhausted its budget, or the mean share of the simulated window survived across all seeds.Source: Table 4 · p. 18
• Table 5 covers online harness updates for breakout news detection with a GPT-5.6-luna actor using the sleep harness across three seeds.Its metric is coverage F1 ×100.Source: Table 5 · p. 18

B ONLINE HARNESS OPTIMIZATION

• The optimization test used GPT-5.6-luna as actor over one four-week episode split into nine three-day spans, with two proposer backbones and an unchanged-harness comparison.At each cut, the proposer saw the trajectory and hindsight available so far, but no future data, held-out validation set, or scorer.Source: B ONLINE HARNESS OPTIMIZATION · p. 18
• GPT-5.6-sol’s updated harness achieved 13.3 coverage F1 versus 11.2 for the static harness, while the GPT-5.6-luna proposer fell to 7.3.The GPT-5.6-luna proposer performed below the no-update baseline.Source: B ONLINE HARNESS OPTIMIZATION · pp. 18–19
• GPT-5.6-sol added stateless LLM review, recalibrated per-market thresholds, bounded sleep duration, and included settled outcomes in prompts.Its notes attributed missed breakpoints to cadence, direction, or reviewer strictness and adjusted thresholds.Source: B ONLINE HARNESS OPTIMIZATION · p. 18
• GPT-5.6-luna replaced judgment with brittle rules based on catalyst terms, direction-specific words, speculative language, identity terms, price checks, and previously seen breakpoint headlines.The authors report that rules distilled from a handful of past events did not transfer to the next three days.Source: B ONLINE HARNESS OPTIMIZATION · p. 19

C SEED-TO-SEED SPREAD BY TRIGGER MECHANISM

• Seed-to-seed spread changed with trigger mechanism on popularity prediction, market price check, and two web tasks, but stayed within a few points on both detection tasks.Table 6 reports pooled within-cell standard deviation of per-seed scores, scaled by 100, among the five strongest models per task.Source: C SEED-TO-SEED SPREAD BY TRIGGER… · p. 19, Table 6 · p. 19

• Common Crawl CC-NEWS articles from March to July 2026 are used inside the simulator and not redistributed.The collection scripts, identifiers, and derived labels are released instead of raw third-party content.Source: D DATA LICENSES · p. 19
• Market metadata and hourly prices come from Polymarket public APIs, while Reddit submissions and comment trees come from the Arctic Shift public archive for non-commercial research.The Reddit content is not used for model training.Source: D DATA LICENSES · p. 19
• The EDGAR inputs are public structured records—ticker-to-CIK mappings, submissions indexes, and XBRL company facts—not filing documents or exhibits.The passage identifies these U.S. government records as public domain.Source: D DATA LICENSES · p. 19