PaperlayerView exact revision on arXiv

Source-linked AI summary

LOLPlay: Benchmarking Real-Time Visual Agents in League of Legends

Zhaojian Yu

arXiv:2611.03685v1cs.AIcs.CV

TL;DR

Visual agents must act on scenes that can change while they reason, raising the question of whether they can accomplish sustained goals through timely interaction. LOLPlay benchmarks this in live League of Legends, where GPT-6 Astra records the top task and subgoal success rates, while the results also show limits to gains from additional reasoning effort.

  • Problem

    Visual agents must turn scene understanding into timely actions that accomplish sustained goals even as the environment changes during reasoning.

    Source: §1 · p. 2

  • Method

    LOLPlay evaluates visual agents in live League of Legends through 36 champion-specific tasks and executable checks for full outcomes and subgoals.

    Source: §1 · pp. 2–3

  • Results

    GPT-6 Astra has the highest task success rate at 25.0% at low and medium effort, and the highest subgoal success rate at 40.3% at medium effort.

    Source: §1 · p. 3, §5 · p. 13

  • Takeaways & Limitations

    The findings motivate agents that coordinate reasoning with timely observation and action while satisfying multiple requirements.

    Source: §5 · p. 13

    The 36 tasks cover one objective per champion rather than the full diversity of MOBA play, and passive progress in some requirements limits interpretation of scores.

    Source: §5 · p. 13

Abstract

from arXiv · show

Visual agents need to coordinate reasoning and action while their environment evolves during inference. We introduce LOLPlay, a benchmark in native \emph{League of Legends} with 36 champion-specific tasks spanning resource management, combat, cooperation, and objectives. Agents interact through screenshots and mouse--keyboard actions; automated initialization and 82 executable subgoal checks distinguish partial progress from completion. Across 324 task evaluations of three models at three reasoning-effort settings, GPT-6 Astra achieves the highest task success rate of 25.0\% at low and medium effort, and the highest subgoal success rate of 40.3\% at medium effort. However, high effort improves neither metric over low effort for any model. For Astra, longer requests leave fewer interaction rounds, suggesting a cost to additional reasoning in real-time control. Trajectories reveal incomplete subgoal composition and actions executing after the observed scene changes, motivating agents that combine visual reasoning with timely action.

1 Introduction

LOLPlay tests whether visual agents can turn scene understanding into timely actions and sustain goals while gameplay continues. It measures both full-task completion and partial progress, finding low overall success and difficulty coordinating multiple requirements as the scene changes.

1 Introduction

  • LOLPlay is a benchmark for real-time visual agents in live League of Legends, where gameplay continues during reasoning and execution.It addresses whether agents can combine timely interaction with multiple task requirements.Source: §1 · p. 2
  • Its 36 champion-specific tasks span farming, development, survival, combat, cooperation, and map objectives, making sustained multi-skill control measurable beyond match wins.The benchmark includes 82 executable subgoal checks to distinguish complete success from partial progress.Source: §1 · p. 3
  • GPT-6 Astra has the highest task success rate at 25.0% at low and medium effort, and the highest subgoal success rate at 40.3% at medium effort.Across 324 task runs, increasing reasoning effort does not consistently improve completion; trajectories also show actions executing after the observed scene changes.Source: §1 · p. 3
  • LOLPlay combines an uninterrupted native MOBA with explicit task and subgoal evaluation, enabling analysis of both goal completion and where partial progress stops.Its contributions also include an empirical study of performance across reasoning configurations, execution budgets, and observation–action mismatch.Source: §1 · p. 3

2 LOLPlay Benchmark and Evaluation

The benchmark specifies tasks as multi-requirement objectives, then evaluates agents acting from screenshots while the game continues. Automated match preparation and live-state and event checks verify both full task completion and individual subgoals.

2.1 Task Definition

  • Each task combines an instruction, starting configuration, time budget, required subgoals, and evaluator; all executable requirements, including applicable timing and ordering checks, must be met for task success.The 36 champion-specific tasks cover six overlapping families and have budgets of 5–20 minutes.Source: §2.1 · pp. 3–4, Table 2 · p. 4

2.2 Agent Interface

  • Agents receive screenshots, instructions, and recent interaction history, then issue mouse–keyboard actions without a separate list of game-state data.The game keeps running during response generation and action execution, so the scene may change before an action is carried out.Source: §2.2 · p. 4, Table 3 · p. 4

2.3 Environment Setup

  • Automated setup prepares a fresh match from the prescribed configuration and validates its initial conditions before starting the task’s execution budget.The representative task scripts illustrate how live-state and event evidence support evaluation.Source: §2.3 · pp. 4, 6, Table 4 · p. 5, Table 5 · p. 6

2.4 Evaluation

  • Dedicated evaluators use Live Client Data to verify task outcomes, including state, timing, event actors, and ordering; separate subgoal results identify partial completion.Two experienced players completed all 36 tasks under the evaluated conditions, establishing feasibility rather than population-level human performance.Source: §2.4 · pp. 6–7

3 Experiments and Analysis

Across 324 runs, the experiments measure full-task completion alongside partial subgoal progress and examine how effort and execution budgets relate to outcomes. Partial progress often falls short of task success, and higher reasoning effort does not consistently improve completion.

3.1 Experimental Setup

  • The evaluation covers three models at three effort settings, with each configuration tested on all 36 tasks for 324 runs and 17,221 completed responses.Episodes use task-specific time budgets and a 100-response cap; one response may contain multiple actions, and in-flight requests or actions are not forcibly interrupted.Source: §3.1 · p. 7

3.2 Overall Results

  • GPT-6 Astra (medium) achieves the highest subgoal success rate (40.3%), while low and medium effort both complete nine tasks; the table defines task and subgoal rates under a common evaluator.Across failed runs, 108/284 satisfy some required subgoals without completing the task.Source: §3.2 · p. 7, Table 6 · p. 8

3.3 Subgoal Analysis

  • Development tasks average 55.5% subgoal success but 23.5% task success; sustain tasks average 27.0% subgoal success with no successful episode.Task failure can follow from an unmet subgoal or violated timing requirement, even when other subgoals succeed; development checks pass more often than combat checks, including 0/45 personal-kill and 0/27 allied-assist checks.Source: §3.3 · pp. 8–9

3.4 Latency and Execution Budgets

  • For Astra, mean request latency rises from 7.47 to 8.39 to 16.67 seconds as mean responses per episode fall from 43.9 to 37.8 to 27.1 across effort settings.DeepSeek-v4.1-Flash (low) is faster than Astra (low), averaging 2.81 versus 7.47 seconds per request, yet has lower task and subgoal success rates: 5.6% and 22.2% versus 25.0% and 37.7%.Source: §3.4 · p. 9
  • Across configurations, 129 runs end at the time budget and 75 at the response cap; every capped run ends before its wall allowance.Of DeepSeek’s 55 capped low/medium runs, 32 achieve no subgoal; only 5/93 positive-attainment checks pass, while 23 achieved checks concern avoidance or state preservation.Source: §3.4 · pp. 9–10

3.5 Reasoning Effort

  • High effort improves neither task nor subgoal success over low effort for any model; Astra’s medium effort raises subgoal success to 40.3% from 37.7% without increasing completed tasks.The effort comparisons are descriptive and do not isolate a causal effect of delay; Sol’s latency is nonmonotonic across settings.Source: §3.5 · p. 10

3.6 Case Study

  • A successful Cho’Gath run satisfies the level 6, R rank 1, and maximum-health checks after 420.8 seconds, within a 540-second budget.The episode includes 33 responses and two deaths; evaluator checks establish the final state, not every attempted cast or the source of health growth.Source: §3.6 · p. 10
  • Warwick’s 144.641-second request outlasts the Voidgrub encounter; only the personal-kill check passes, while the required turret remains intact.The logged click at (810,465) lands in an AFK dialog in the next frame, but the record does not establish when the dialog appeared or which inputs it intercepted; these cases do not estimate error frequencies.Source: §3.6 · pp. 12–13

4 Related Work

Prior work evaluates computer-use, game-playing, and real-time interaction, while LOLPlay focuses on champion-specific objectives in a continuously changing native MOBA. Its evaluation distinguishes partial achievements from completion of the requested objective.

Computer-use agents and evaluation

  • LOLPlay connects visual grounding and multi-step interaction to goals whose execution unfolds in a continuously changing MOBA, complementing prior evaluation of web interaction, executable mobile and desktop tasks, and screenshot-based GUI grounding.Source: §4 · p. 13

Real-time interaction and latency

  • LOLPlay complements temporal analyses with task and subgoal outcomes in a native MOBA, checking whether partial achievements satisfy objectives rather than establishing latency as a new phenomenon.Related work includes VideoGameBench, GameWorld [19], Realtime-Bench [25], and ROAR [28].Source: §4 · p. 13

General-purpose game agents

  • LOLPlay evaluates general-purpose agents through champion-specific goals, exposing which requirements are achieved and which combinations remain incomplete.This complements work on general game-playing capabilities and visual-action policies, including GameBench [7], BALROG [20], SIMA [23], and NitroGen [17].Source: §4 · p. 13

Game environments and specialized agents

  • LOLPlay joins component-level evaluation with uninterrupted native MOBA interaction, but its task scores do not imply strength against trained esports agents.Prior game benchmarks include the Arcade Learning Environment [3], SC2LE [26], and Crafter [12]; specialized MOBA agents include OpenAI Five [18] and AlphaStar.Source: §4 · p. 13

5 Conclusion and Discussion

LOLPlay evaluates real-time visual agents across 36 champion-specific MOBA tasks, with task and subgoal outcomes revealing different aspects of progress. The results and scope boundaries motivate coordinating reasoning with timely observation and action.

Conclusion

  • The highest task success rate is 25.0%, while GPT-6 Astra (medium) reaches the highest subgoal success rate without completing more tasks than low effort.Across failed runs, 108 achieve partial progress; high effort combines longer requests with fewer completed tasks.Source: §5 · p. 13

Limitations

  • The 36 tasks cover one objective per champion rather than the full diversity of MOBA play, and passive progress makes task success essential alongside subgoal success.These boundaries limit interpretation of the scores.Source: §5 · p. 13
  • The intended setting is custom matches with bots, and desktop input plus unrestricted Python require a trusted environment because uncontrolled deployment could enable unwanted gameplay automation.Game assets belong to Riot Games.Source: §5 · p. 14

A Task and Evaluation Details

This appendix supplements the task definitions, agent interface, environment setup, and evaluation in Section 2.

A Task and Evaluation Details

A.1 Complete Task List

The appendix details the benchmark’s task objectives, subgoal checks, action interface, and stopping outcomes.

A.1 Complete Task List

  • The 82 subgoal checks count required items, ability ranks, enemy victims, and turrets separately, with ordering checks added when required.Tasks have one primary reporting family but may involve several skills; context validity and global timing are evaluated separately.Source: §A.1 · p. 18, Table 7 · p. 18

A.2 Action Parameters and Termination

  • Actions use absolute screenshot-pixel coordinates, execute sequentially after a complete response is received and parsed, and gameplay continues throughout.The interface provides 13 mouse–keyboard action types and three control markers; screenshots cover the primary display.Source: §A.2 · p. 18, Table 8 · p. 19
  • Stopping signals, time limits, and turn limits do not determine evaluator success; Table 9 reports stopping reasons separately from task outcomes.The reported counts describe the 324-run cohort, with each block partitioning the same records independently.Source: Table 9 · p. 19, §A.2 · p. 19

A.3 Setup and Evaluation APIs

The appendix details how LOLPlay initializes matches and validates task outcomes, then specifies the screenshot-to-action interface used across configurations.

A.3 Setup and Evaluation APIs

  • The LCU configures the game client, while Live Client Data supplies timestamped state and events for evaluation rather than agent actions.Evaluation applies task-start, deadline, and ordering conditions where required.Source: §A.3 · pp. 19–20

A.4 Success Checks and Timing

  • Success checks use client data and task-specific rules; creep score is evaluated in ten-point bins, a behavior observed in the evaluated client.Checks assess outcomes such as target destruction, resource attributes, ability ranks, and kill participation, not the quality of intervening actions.Source: §A.4 · p. 20, Table 10 · p. 20

A.5 Agent Configuration

  • All nine configurations share a screenshot-to-action interface and prompt template, varying only the model and reasoning-effort setting.Requests include the task instruction and screenshot, with recent valid turns retained up to three; the prompt supplies the action schema.Source: §A.5 · pp. 20–21

B.1 Per-Task Results

Per-task outcomes vary across configurations: 12 of the 36 tasks succeed in at least one configuration, while pooled checks show partial progress beyond task completion.

B.1 Per-Task Results

  • 12 of 36 tasks succeed in at least one configuration, as reported by the per-task diagnostic matrix.The reported subgoal success rate weights each task equally.Source: §B.1 · p. 22
  • 243/738 pooled subgoal checks succeed, a rate of 32.9%.Each of the nine configurations contributes the same 82 checks.Source: §B.1 · p. 22

B.2 Subgoal Breakdown

The subgoal breakdown separates requirement types and shows why pooled subgoal achievements do not necessarily amount to complete task success.

B.2 Subgoal Breakdown

  • 85.2% of state-preservation checks succeed, compared with 45.3% of avoidance/absence checks and 25.4% of positive-attainment checks.These pooled counts cover 738 checks across all 36 tasks and nine configurations; the figure reports achieved/eligible counts by requirement type.Source: §B.2 · p. 22, Table 11 · p. 24
  • 99 of 243 achieved checks (40.7%) concern avoiding violations or preserving conditions, which alone cannot complete tasks also requiring an item or objective.The categories distinguish acquisition or event completion from avoidance and preservation requirements.Source: §B.2 · p. 22
  • The two breakdowns distinguish individual requirements from completing their conjunction within a task, without changing success criteria or removing achieved subgoals.Outcome checks do not attribute results to particular actions.Source: §B.2 · p. 24

B.3 Response Latency

Response latency statistics are pooled across configurations and include service and network overhead, rather than isolating internal reasoning time.

B.3 Response Latency

  • Latency includes service, network, retry, and parsing time, so it does not measure internal reasoning time alone.Requests without a completed response are excluded.Source: §B.3 · p. 25
  • Responses and actions are counted separately because one response can contain several actions.The latency statistics use completed responses.Source: §B.3 · p. 25
  • Across 17,221 completed requests, mean, median, and 95th-percentile latency are 6.536, 5.594, and 14.031 seconds.The statistics pool latency across the nine configurations.Source: §B.3 · p. 25

B.4 Response-Cap Analysis

Among DeepSeek-v4.1-Flash episodes that reach the response cap, low- and medium-effort runs use only part of their wall-time allowances. Capped episodes can still show limited task progress, and some end before tasks requiring play through the deadline are complete.

B.4 Response-Cap Analysis

  • 64.0% and 63.7% are the median shares of wall-time allowances used by low- and medium-effort episodes ending at DeepSeek-v4.1-Flash’s 100-response cap.The table covers capped runs; its subgoal breakdown indicates whether decisions also satisfy task requirements.Source: §B.4 · p. 25, Table 12 · p. 25
  • In task 024, DeepSeek (low) issues 104 actions but ends at level 1 with zero deaths, so its 50% subgoal success reflects the death limit alone.The nine-minute task requires reaching level 5 with at most one death; the run ends after 382.8 seconds of its 540-second budget.Source: §B.4 · p. 25
  • Six low-effort and five medium-effort capped episodes end before the required duration for tasks that require play through the deadline.They receive an incompleteduration outcome; these cases differ from tasks permitting early completion, and do not establish outcomes for uncapped continuations.Source: §B.4 · p. 25

Related papers

Loading 2611.03685v1…