PaperlayerView exact revision on arXiv

Source-linked AI summary

DirCast: Live Esports Directing and Commentary with a Single Streaming Video-Language Model

Zixian Gao, Lixin Xiu, Penghao Luo, Xiangcheng Cao, Ziqi Cai, Zhen Li, Erwin Wu, Kaipeng Zhang

arXiv:2610.31734v1cs.CV

TL;DR

Live esports production requires a director to choose shots and a caster to commentate in real time, while many matches remain fixed-camera and uncommentated. DirCast jointly directs and comments from raw player feeds, winning more comparisons than streaming commentators and outperforming rule-based and zero-shot directors.

  • Problem

    Live esports broadcasts require coordinated directing and commentary, while many matches are streamed with a fixed camera and no commentary.

    Source: §1 · p. 2

  • Method

    DirCast is a single streaming video language model that selects shots with a hold-or-switch director and comments on its chosen shot.

    Source: §1 · pp. 2–3

  • Results

    DirCast outperforms rule-based and zero-shot directors including GPT-5, while winning 29.21 percent of comparisons against GPT-4o versus 19.38 percent for the strongest streaming commentator.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    Program-only context bounds per-second cost for streams of any length while enabling real-time operation on a single GPU.

    Source: §1 · p. 2

Abstract

from arXiv · show

A live esports broadcast needs a director who chooses which camera goes on air and a caster who talks over that choice, every second and without knowing what comes next. Streaming video language models have learned to commentate, but only over a broadcast that a human director has already produced. We study live broadcast production, in which a model receives the raw camera of every player and must both direct and commentate from the past alone. We present DirCast, a single streaming video language model that does both. Every second it scans all cameras at low resolution, decides with a hold-or-switch director whether and where to cut, and reads the chosen camera at high resolution before it speaks. Its context keeps what went on air and what was said and lets the other views expire, which bounds the cost of a stream of any length. We build a dataset of 27 professional Dota 2 matches with a camera for every hero, the human director's choices and the casters' commentary. DirCast directs better than rule-based and zero-shot directors including GPT-5, and its commentary wins 29.2 percent of comparisons against GPT-4o under the Proact-VL protocol, against 19.4 percent for the strongest of eleven streaming commentators that watch the human broadcast.

1 Introduction

DirCast tackles live esports production by jointly choosing the on-air camera and commentating from past observations. It combines a streaming director and commentator, trained and evaluated on matches with all player cameras and human broadcast labels.

1 Introduction

  • Existing streaming commentators [42] [43] consume human-produced broadcasts, leaving raw-feed shot selection outside their task.Live production couples choosing among ten simultaneous scenes with describing the changing shot using only past observations.Source: §1 · p. 2
  • Every second, DirCast scans ten low-resolution feeds, chooses whether and where to cut, then processes the chosen feed at high resolution before generating commentary.Its context retains aired views and spoken words while low-resolution feeds expire, bounding per-second cost and supporting real-time operation on one GPU.Source: §1 · pp. 2–3
  • Perturbed program rollouts train DirCast to recover from its own directing errors, while event-grounded targets treat cuts to kill participants as correct.At inference, cut and speech thresholds follow the model's running rates toward human crew rates without test-data tuning.Source: §1 · p. 2
  • The dataset contains 27 professional Dota 2 matches with per-hero cameras, human director choices, and caster commentary.The task requires producing all three forms of information from raw camera feeds.Source: §1 · pp. 2–3
  • DirCast wins 29.21 percent of comparisons against GPT-4o versus 19.38 percent for the strongest streaming commentator, and directs better than rule-based and zero-shot directors including GPT-5.All baselines are given the finished human broadcast, while DirCast directs its own.Source: §1 · p. 3

2 Related Work

Prior work studies streaming video understanding, automated commentary, and camera selection, but these lines generally assume a preselected broadcast or handle directing separately. DirCast combines causal shot selection with commentary and preserves the available camera views for joint learning and evaluation.

Streaming video language models

  • Streaming video-language models and live commentators process incoming video, but assume a preselected stream; DirCast produces that stream and maintains memory of aired views and commentary.Related work manages long-term context with memory banks, compression, retrieval, attention, eviction, and compact frame representations.Source: §2 · p. 3

Commentary for sports and games

  • Sports and game commentary datasets typically use a single produced broadcast or game data without video, whereas this dataset preserves all available camera views for joint shot-selection and commentary evaluation.Prior examples span soccer, racing, and esports datasets for League of Legends and Dota 2.Source: §2 · p. 3

Automatic directing and view selection

  • Unlike offline camera and view-selection systems, DirCast directs causally each second and couples each choice with commentary.Its perturbed program rollouts also expose the director to states induced by its own errors, paralleling work on exposure bias and expert-corrected learning.Source: §2 · p. 3

3 Dataset

The dataset addresses the lack of live-production data containing both the cameras available to a director and the broadcast choices and commentary. It uses Dota 2 replays to render and align ten hero views with professional broadcasts, then splits matches into training, calibration, and test sets.

3 Dataset

  • A produced broadcast records only one camera, so it loses the alternative views needed to learn directing alongside the chosen shot and caster words.The dataset is intended to preserve every camera the director could have chosen.Source: §3 · p. 4
  • The dataset renders ten hero-following cameras from each of 27 professional Dota 2 replays and labels each second with the matching on-air camera, broadcast state, and commentary.Commentary is transcribed with word timestamps and cleaned for hero and player names; the replay also supplies hero states and timed game events.Source: §3 · p. 4
  • The match-level split uses 20 training, 3 calibration, and 4 test matches, with 2,096 training clips and 496 test clips.Training contains 21 hours of play and testing 5 hours; human directors cut 7.9 times per minute on average, while casters speak in about three quarters of seconds.Source: §3 · p. 4

4 Method

DirCast uses one causal video language model to direct and commentate from the past, generating commentary after its own directing choice. Its streaming design combines coarse-to-fine camera processing, bounded memory, game-state cues, and training and decision rules for live use.

4.1 One stream for directing and commentary

  • Each second, one causal stream scans ten low-resolution cameras, selects a shot, reads that feed at high resolution, and generates commentary over it.The model follows the order of a human crew: the director scans all cameras, then the commentator examines the shot that went on air.Source: §4 · p. 4, §4.1 · p. 4, Figure 2 · p. 5

4.2 Hold or switch

  • Because directors usually hold shots, DirCast separates whether to cut from where to cut, with a gate for switching and a pointer for choosing another camera.The pointer scores cameras using their content and game state; the final distribution defaults to the current camera unless the gate opens.Source: §4.2 · p. 5

4.3 The program is the memory

  • Low-resolution camera views expire after W seconds, while aired views, events, and commentary remain, keeping per-second cache cost constant on streams longer than a training clip.On streams longer than a training clip, entries older than a clip are also removed from the cache.Source: §4.3 · p. 5

4.4 Grounding the game

  • Camera state tokens and event text supply game information, and each event is tied to the camera that shows it, such as flagging the killer’s camera after a kill.The state token includes the hero’s health, recent damage, and nearby enemies.Source: §4.4 · p. 5

4.5 Learning to direct from human broadcasts

  • Training combines commentary, speaking, and directing objectives; on-policy rollouts expose the model to drifting shots, while kill-anchored targets also accept killer or victim cameras and request a cut when neither is on air.During rollouts, the input shot can drift from the human choice for a few seconds while targets remain human, teaching recovery from a wrong shot.Source: §4.5 · p. 6

4.6 Streaming decisions at a human rhythm

  • Adaptive speaking and cutting thresholds respond to recent decisions relative to human rates, lowering the speaking threshold after under-speaking and raising the cut threshold after over-cutting.A new stream’s running speech rate starts at zero, and an utterance ends only when its end probability is high, yielding complete phrases rather than fragments.Source: §4.6 · p. 6

5 Experiments

The experiments evaluate live directing and commentary on unseen Dota 2 matches, with DirCast jointly outperforming comparison systems on key directing and commentary measures. Ablations and longer-stream tests examine the joint design, its components, and bounded-memory streaming.

5.1 Setup

  • The test split contains four unseen matches; directors are compared with three rules and eight zero-shot VLMs, while commentary follows Proact-VL evaluation against eleven streaming commentators.The streaming commentators watch the finished human broadcast; DirCast directs its own program. GPT-5 judges the commentary metrics because GPT-5.1 is unavailable.Source: §5.1 · pp. 6–7

5.2 Main Results

  • Top-1 reaches 26.95 against 25.43 for the action rule, while DirCast wins 29.21 percent of commentary comparisons against GPT-4o versus 19.38 percent for LiveCC-7B-Base.DirCast also covers 89.1 percent of kills, more than any baseline and the human director.Source: Table 1 · p. 7, §5.2 · p. 7, Table 2 · p. 8

5.3 Ablation Studies

  • The joint model reaches Top-1 27.00 versus 25.17 for separate role-specific models, wins 29.21 percent versus 24.36 percent, and halves memory to 18.9 GB.On the human program, the joint model also wins 27.84 percent versus 25.67 percent.Source: Table 3 · p. 8, §5.3 · p. 8
  • Removing the hold-or-switch pointer drops Top-1 from 27.00 to 21.65; removing state tokens or program rollouts drops it to 22.59 and 22.90, respectively.Without program rollouts, commentary CC falls from 29.21 to 28.35; without state tokens, CC remains 29.21.Source: §5.3 · pp. 8–9, Table 4 · p. 9

5.4 Further Analysis

  • A 36-second horizon holds cache length at 6.9k tokens and compute at 0.29 seconds per video second on one GPU, while Top-1 rises from 23.7 to 25.1.The horizon leaves the reported Table 1 and 2 results unchanged because test clips are 36 seconds long.Source: Table 5 · p. 9, §5.4 · p. 9
  • With 20 training matches, Top-1 reaches 27.00 and Any-vis. 41.80; doubling training data from 10 to 20 matches gives gains of 6.36 and 5.56, respectively.The test split is fixed and the training subsets are nested, so the passage attributes the rising curve to added training variety rather than an easier evaluation.Source: Figure 3 · p. 9, §5.4 · p. 9
  • In four illustrated off-air kills, DirCast cuts to and names the fight while the human broadcast stays on an uninvolved hero and misses each kill.The panels show the cut gate against its threshold and the human broadcast; after cutting, the threshold rises and DirCast holds the shot rather than flickering.Source: Figure 4 · p. 10, §5.4 · p. 10

6 Conclusion

Live broadcast production asks a model to select the on-air shot and commentate using only past observations. DirCast unifies these tasks, and its reported results support the feasibility of real-time esports production with one model.

6 Conclusion

  • DirCast addresses live broadcast production by selecting the on-air shot and commentating from past observations.Source: §6 · p. 10
  • Its hold-or-switch director selects the shot, commentary follows that choice, and program-only context bounds long-stream cost while supporting real-time operation on one GPU.Source: §6 · p. 10
  • DirCast outperforms rule-based and zero-shot directors including GPT-5, shows more kills on air than human crews, and wins more comparisons than eleven streaming commentators watching human broadcasts.Source: §6 · p. 10
  • Together, the results establish the feasibility of unified, real-time esports production with a single model.Source: §6 · p. 10

A Dataset Construction

The dataset-construction section covers director data, commentary, and the cleaning applied to both.

A Dataset Construction

A.1 Director Data

The director dataset combines broadcast-derived labels with synchronized hero-camera feeds rendered from match replays. It covers 27 professional Dota 2 matches and records broadcast state and visible heroes alongside per-second director labels.

A.1 Director Data

  • 27 professional Dota 2 matches provide the source data, combining official broadcast video with replay files containing complete game state.Source: §A.1 · p. 14
  • Broadcast and replay alignment uses OCR of the HUD game clock, pauses are subtracted from replay ticks, and hero-name matching filters stretches to the correct game.The alignment is validated by checking whether heroes visible together in a broadcast frame are close together in the replay; all 27 matches pass at zero time offset.Source: §A.1 · p. 14
  • Ten synchronized spectator clients render one Hero Chase camera per player, with training clips sampled at one frame per second and resized to 288 × 288.Source: §A.1 · p. 14
  • Per-second director labels identify the selected hero when visible, otherwise the visible hero nearest the frame centre, with hysteresis and carry-over rules for unclear switches.Each second also receives a broadcast state and a set of all visible heroes, supporting any-of-visible evaluation.Source: §A.1 · p. 14

A.2 Commentary

Commentary targets are aligned to the director labels by time-stamping broadcast speech, transcribing it, and polishing the text under constraints. The polished commentary serves as both the fine-tuning target and the test-time reference.

A.2 Commentary

  • Broadcast audio is aligned to the director labels, transcribed with word-level timestamps, and assigned to the second in which each word starts.Of 2,096 training clips, 1,984 have broadcast audio; about 8% of transcribed words have ASR confidence below 0.5.Source: §A.2 · p. 15
  • A language model polishes merged ASR sentences to correct names and remove fillers while preserving the casters’ words, order, and meaning.Unfinished sentences remain unfinished, and the rewrite may not add actors, numbers, or outcomes.Source: §A.2 · p. 15
  • A rewrite is accepted only if its word count is 70%–140% of the ASR sentence and at least half its words appear in the ASR; otherwise the ASR text is retained with fillers removed.Polishing changes the text of 38% of seconds with speech and removes 2% of words, mostly fillers and repetitions.Source: §A.2 · p. 15
  • Polished commentary trains DirCast and two fine-tuned baselines and serves as the test-time reference for every model under the Proact-VL protocol.Source: §A.2 · p. 15

A.3 Cleaning

The cleaning pipeline restricts director training and evaluation to live gameplay with usable labels, verifies carried-over labels, and separates matches across splits.

A.3 Cleaning

  • 87% of clip-seconds remain after removing replays, production shots, pauses, and seconds without broadcast coverage.Source: §A.3 · p. 15
  • HUD-only labels without the selected hero on screen are dropped because only 52% agree with hero positions in the replay.The remaining labels form the training mask.Source: §A.3 · p. 15
  • 62% of 15,951 carried-over seconds are confirmed by a vision–language model, and rejected seconds are removed from the training mask.On a manually audited sample, model verdicts agree with the human audit in 43 of 45 cases.Source: §A.3 · p. 15
  • The strict evaluation mask keeps 69% of live test seconds with labels observed on screen in that same second.It excludes hysteresis and carry-over labels.Source: §A.3 · p. 15
  • Commentary is cleaned by the polishing and guard, while windows without broadcast audio retain director labels but provide no commentary targets.Source: §A.3 · p. 16
  • Matches are split whole into 20 training, 3 validation, and 4 test matches, preventing games, broadcasts, or caster utterances from crossing between training and test.Validation matches calibrate the base thresholds of the decoding rules.Source: §A.3 · p. 16

A.4 Statistics

The dataset comprises aligned ten-feed footage and polished commentary across training, validation, and test matches, with overlapping clips used for statistics.

A.4 Statistics

  • 15.2 hours of unique footage are packaged as 2,912 overlapping clips of 36 s with an 18 s stride.The footage includes 13.1 hours of live broadcast and 10.3 hours with verified labels, including 7.9 hours in the strict subset.Source: §A.4 · p. 16, Table 6 · p. 16
  • Casters speak in 64% of training clip-seconds and 72% of test clip-seconds.Speech is measured as the share of clip-seconds with polished commentary.Source: §A.4 · p. 16, Table 6 · p. 16
  • About 80% per-second accuracy is reached by a persistence baseline that repeats the previous label, so switch-second accuracy is reported separately.Source: §A.4 · p. 16

B Additional qualitative examples

Additional test-match examples show DirCast directing attention to off-air kills and commentating on the resulting shots.

B Additional qualitative examples

  • Twenty additional off-air kills are shown and named across test matches and random seeds, spanning pickoffs and crowded team fights.Each example is a kill for which the human crew shows an uninvolved hero.Source: B Additional qualitative examples · p. 16, Figure 5 · p. 17, Figure 6 · p. 18
  • Each panel places DirCast’s program and spoken words above the cut-gate plot and human broadcast, with the kill second shaded.The top row has one on-air thumbnail per second; the middle row marks cuts and holds against the gate threshold.Source: B Additional qualitative examples · p. 16
  • DirCast cuts toward a fight around the kill, often shifts from victim to killer, and names a fight hero within about two seconds.It holds the new shot as the threshold rises; the human crew shows an uninvolved hero and shows the kill late or not at all in every panel.Source: B Additional qualitative examples · p. 16

Related papers

Loading 2610.31734v1…