MOBA-VL: Event-Localized Multi-Turn Reinforcement Learning for Real-Time MOBA Commentary

• Real-time MOBA commentary must be continuous and fluent, timely, and accurate in identifying and describing in-game events.Source: §1 · p. 2
• Existing streaming VLMs address fluency and timing but still struggle to describe key events, while telemetry records event times and involved players.Source: §1 · p. 2
• MOBA-VL combines professional-commentary supervised fine-tuning with event-localized multi-turn reinforcement learning using NLI models to locate and score event-related turns.Source: §1 · p. 2
• MOBACast provides 860 professional matches across three MOBA games, and MOBA-VL uses a window-based inference strategy that resets the cache and retains the most recent 4 seconds.Source: §1 · p. 2

• Streaming VLM work targets timestamped speech alignment, long-video context, response timing, and multi-turn reinforcement learning for timing and correctness.Source: §2 · p. 2
• Prior credit-assignment methods redistribute episode returns or estimate segment- and step-level rewards; MOBA-VL uses NLI to locate events across short turns and adjust token-level advantages.Source: §2 · pp. 2–3
• MOBACast provides word-level timestamped commentary and videos across three MOBA games, with a pipeline aligning words to one-second turns and preserving removed speech as silence.Source: §2 · p. 3, Figure 2 · p. 3

3.1 DATASET CONSTRUCTION

• MOBACast collects 860 professional matches across three games, transcribes commentary with word-level timestamps, corrects names, filters sentences for gameplay relevance, and preserves removed speech as silence.SFT uses 8-second windows with 4-second overlap; RL uses 32-second windows containing key events.Source: §3.1 · pp. 3–4
• MOBACast-Bench uses tournaments excluded from training and contains 6 full matches and 94 event-dense clips across three games.Source: §3.1 · p. 4

3.2 TRAINING

• Training fine-tunes Qwen3.5-9B on multi-turn video-commentary windows, then applies GRPO with mean NLI event support as reward and penalties for excess length and repetition.The length and repetition penalties use λL = 0.5 and λD = 0.6.Source: §3.2 · p. 4
• A separate NLI model selects the turn with the largest support increase as an event anchor, falling back to telemetry time when no mention is detected; Gaussian or uniform weights distribute credit around it.Centering token credit makes the adjustment zero-mean, preserving the rollout's mean advantage while leaving scalar reward unchanged.Source: §3.2 · p. 5

3.3 STREAMING INFERENCE AND SILENCE TOKEN

• For streaming, MOBA-VL reuses cache within a growing window and rebuilds it every 32 seconds from the most recent 4 seconds of video and commentary; silence is modeled as a token.Evaluation sets the silence-token bias to −∞ so commentary remains continuous.Source: §3.3 · p. 5

4.1 METRICS

• Overall averages Content Coverage, Fluency, and Density Agreement per game, then averages those game scores across LoL, Dota 2, and Honor of Kings.The metrics assess gameplay-point coverage, textual fluency, and speaking-density agreement; full matches use 300-second windows and clips use their original units.Source: §4.1 · p. 5

4.2 RESULTS

• MOBA-VL achieves the best Overall score on full matches (63.25) and clips (63.45), ranking first in every game.It also achieves the highest Density Agreement in five of six game settings.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
• On full-match LoL, MOBA-VL scores 58.0 in Content Coverage and 88.8 in Fluency, while StreamingVLM scores 29.8 and 83.4, respectively.Nearly all models receive lower Fluency scores on full matches than clips; longer windows can expose repetition and abrupt transitions.Source: §4.2 · p. 6
• MOBA-VL wins more than half of pairwise comparisons against all four reference models in both full-match and clip settings.The protocol prioritizes semantic alignment with human commentary; real-time baselines lose to most references despite generally higher Fluency.Source: §4.2 · p. 6
• Despite higher Overall scores, MOBA-VL trails GPT-5.6-Luna and DeepSeek-V4.1-Flash in Content Coverage on several games.The authors suggest its 9B backbone may limit captured gameplay details, leaving coverage to improve under real-time constraints.Source: §4.2 · p. 6

4.3 CREDIT-ASSIGNMENT ABLATIONS

• Uniform event-localized credit raises Recall from 34.5 to 42.1, versus 38.6 for sequence-level RL, with similar Precision and Fluency.The telemetry anchor reaches only 38.8 Recall; uniform weighting also outperforms Gaussian weighting on Recall and Fluency.Source: §4.3 · p. 7, Table 4 · p. 8

4.4 STREAMING INFERENCE LATENCY

• MOBA-VL’s median and P95 turn latencies are 559 and 812 ms, below the one-second budget and StreamingVLM’s 790 and 1,234 ms.It runs at an RTF of 0.60 with 20.5 GiB peak memory; removing cross-turn reuse raises median latency to 6.0 s, while removing periodic reset exhausts memory after 38.3 minutes.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 APPLICATION

• With NitroGen, MOBA-VL obtains the lowest open-loop action MSE and the highest closed-loop mean rubric score, 45.50 versus 37.17 and 39.33.The closed-loop test controls Daji against bots for the first 130 seconds after task-specific fine-tuning.Source: §4.5 · p. 9

5 CONCLUSION

• 42.1 event recall, up from 34.5 over SFT, while maintaining precision through event-localized credit assignment.NLI models identify which turns describe each game event and redistribute credit accordingly.Source: §5 · p. 9
• MOBA-VL achieves the highest overall scores on MOBACast-Bench, outperforming proprietary VLMs and existing real-time commentary models.Source: §5 · p. 9
• MOBA-VL streams a full match in real time on a single GPU with constant memory.Source: §5 · p. 9

A TRAINING AND EVALUATION DETAILS

• The SFT model trains Qwen3.5-9B for one epoch on 412,858 eight-second windows with a four-second stride; loss falls from 3.995 at update 100 to 1.288 at update 51,600.The final checkpoint is update 51,608 and initializes RL; all parameters except the vision encoder are updated.Source: A TRAINING AND EVALUATION DETAILS · p. 13, Figure 5 · p. 13
• The RL training set contains 1,349 32-second windows—954 LoL, 311 Dota 2, and 84 HoK—with 1–3 core events; all RL variants initialize from the same SFT checkpoint.RL uses GRPO with LoRA adapters in non-visual linear layers; reported RL results use the update-100 checkpoint.Source: A TRAINING AND EVALUATION DETAILS · p. 13, Table 8 · p. 14
• Exploratory RL runs beyond 100 updates showed repeated paraphrases of the same event, consistent with reward hacking that the exact four-gram repetition penalty may miss.All reported RL comparisons use checkpoints from update 100.Source: A TRAINING AND EVALUATION DETAILS · p. 13
• Token weights use a Gaussian kernel or uniform window around event anchors, while the bounded adjustment centers token credit and scales corrections to limit their size.Zero-reward-variance groups and zero-sequence-advantage rollouts receive no correction; the mean advantage and scalar reward remain unchanged.Source: A TRAINING AND EVALUATION DETAILS · p. 15
• The NLI scorer and locator use ModernBERT-large cross-encoders; the scorer is trained on human commentary, while the locator is further trained on SFT-model commentary.The human-commentary corpus contains 82,881 premise–hypothesis pairs; localization excludes examples with fewer than three spoken seconds or a target in a silent second.Source: A TRAINING AND EVALUATION DETAILS · p. 15, Table 9 · p. 16
• The event locator is trained on 3,197 SFT-generated event examples, with the target set to the earliest second by which an event has been described.Source: A TRAINING AND EVALUATION DETAILS · p. 15

C BASELINE INFERENCE SETTINGS

• MOBA-VL samples one frame and generates one turn per second, growing context for 32 seconds before retaining the last four frame–commentary turns; baseline context policies differ.StreamingVLM uses a 16-second sliding window, while LiveCC resets every 75 seconds without retaining earlier frames or commentary.Source: C BASELINE INFERENCE SETTINGS · p. 16, Table 10 · p. 17

B STREAMING CHATML TEMPLATE

• The ChatML template supplies an empty <think> block before generation; finalized historical text omits it, and the runtime appends the final newline.Source: Figure 7 · p. 17

E EVALUATION DETAILS

• Evaluation uses 94 original clips and 41 non-overlapping 300-second full-match units; units are averaged within videos and decoding seeds, then videos within games, with equal weighting across the three games in final Overall.CC and Fluency use valid integer scores from 1 to 5, averaged over three judgments per unit. DA uses smoothed word-count sequences and Lin’s concordance correlation coefficient, computed per unit and averaged with equal unit weights within each video.Source: E EVALUATION DETAILS · pp. 17–19
• Figures 8–10 report turn latency, real-time factor, peak GPU memory, and per-turn processing components; the no-reset configuration covers only 38.3 minutes.Figure 8 marks a one-second latency budget, while Figure 9 marks RTF 1.Source: Figure 8 · p. 18, Figure 9 · p. 18, Figure 10 · p. 19
• Event claims are extracted by majority vote and matched to annotated kills or completed objectives using event-specific identity checks and a 30-second time tolerance.Recall uses one-to-one matching, while precision evaluates claims independently and excludes qualifying recaps; scores are pooled within clip subsets before averaging.Source: E EVALUATION DETAILS · p. 20

F JUDGE PROMPTS

• The appendix reproduces the executed judge prompts, including substituted message templates; model names are omitted, and pairwise comparison uses no separate system message.The prompt blocks cover content coverage, fluency, pairwise comparison, and event extraction.Source: F JUDGE PROMPTS · p. 20

F.1 CONTENT COVERAGE

• Content coverage judges identify important gameplay points in the reference, then assess whether the candidate conveys them, allowing paraphrases but not crediting generic excitement for specific actions or outcomes.Scores range from 1–5, with null permitted when the reference has no usable gameplay content.Source: §F.1 · p. 21

F.2 FLUENCY

• Fluency judges assess whether commentary sounds natural aloud, prioritizing understandable, connected, varied thoughts and live-casting momentum rather than grammar or factual correctness.Short calls, fragments, and self-corrections can be natural; monotonous event logs and mechanical repetition can limit scores.Source: §F.2 · pp. 21–22

F.3 PAIRWISE COMPARISON

• Pairwise judges select the commentary better aligned with the human reference by semantic alignment, using stylistic consistency only when semantic alignment is insufficient.The comparison prompt presents two commentaries and the human commentary.Source: §F.3 · p. 23

F.4 EVENT EXTRACTION

• Event extraction treats the model as a parser of asserted kills and objective takes, not a truth judge, and resolves commentary claims against telemetry using game-specific prompts.The extraction instructions specify eligible event types and fields; the evaluation procedure uses repeated extractions, majority retention, and matching rules.Source: E EVALUATION DETAILS · p. 20, §F.4 · p. 23
• The prompts map garbled names and aliases to roster entries, but require nulls instead of guesses and credit killers only when commentary explicitly attributes the kill.They also distinguish live descriptions of a death from predictions and treat escape descriptions as survival outcomes.Source: §F.4 · pp. 25–26
• Objective labels are game-specific: Dota prompts cover towers, barracks, Roshan, and Tormentor, while the LoL-mapped prompt covers dragon and baron.Only actual takes count; the LoL-mapped prompt excludes turrets, inhibitors or crystals, and the base.Source: §F.4 · pp. 25–27
• The prompts extract only commentary-asserted hero kills and objective takes, recording line, verbatim span, and named participants or team where applicable.They instruct the parser to record what commentary asserts rather than judge whether it is true.Source: §F.4 · pp. 25–26

G LOCATOR QUALITY

• The telemetry-timestamp gap supports locating event mentions in commentary rather than assuming they coincide with event timestamps.Source: G LOCATOR QUALITY · p. 27
• 56.7% Exact agreement for the specialized locator and 57.9% with the scorer, versus 47.9% for the baseline; telemetry timestamps reach 6.4%.The telemetry figure is from a 360-event diagnostic subset; learned variants use 363 events without detection gating.Source: Table 11 · p. 27, G LOCATOR QUALITY · p. 27

H GAME AGENT DETAILS

• Appendix H provides experimental details and additional analyses for the gameplay experiments in Section 4.5.Source: H GAME AGENT DETAILS · p. 27

H.1 ENVIRONMENT AND DATA COLLECTION

• The GUI-based evaluation framework sends game screenshots from the Windows client to a remote server and returns predicted actions for execution.Matches run through Tencent Mobile Game Assistant, with low-latency communication between client and server.Source: §H.1 · p. 27
• Approximately 1,000 minutes of human gameplay demonstrations are collected by recording screenshots and mouse actions every 50 ms.Source: §H.1 · pp. 27–28
• The expanded dataset contains more than 2,000 minutes of training data, combining demonstrations with IDM-labeled livestreams cleaned using visual recognition.The livestreams comprise approximately 24 hours, and the player-controlled hero in all collected data is Daji.Source: §H.1 · p. 28

H.2 AGENT ARCHITECTURE

• The asynchronous dual-system architecture couples MOBA-VL with NitroGen to address the inference overhead of autoregressive VLM decoding under real-time latency requirements.Source: §H.2 · p. 28
• Eight learnable MOBA-VL tokens provide latent guidance, concatenated with NitroGen visual tokens to condition its diffusion transformer for action generation.The diffusion transformer is trained with flow matching.Source: §H.2 · p. 28
• NitroGen predicts actions at every interaction step, while MOBA-VL updates latent guidance once every six fast-system inference calls.During training, pretrained VLM weights are frozen and other parameters are optimized.Source: §H.2 · p. 28

H.3 EVALUATION PROTOCOLS

• Open-loop evaluation compares predicted actions with ground-truth labels on held-out test samples from recorded human and Internet gameplay videos.The models are trained on the train split and evaluated on the test split using mean squared error.Source: §H.3 · p. 28
• MOBA-VL + NitroGen has lower MSE at all 16 open-loop prediction steps, extending gains across the action horizon.Figure 11 reports open-loop MSE at each step within the 16-step action horizon.Source: §H.3 · p. 28, Figure 11 · p. 29
• Closed-loop evaluation fine-tunes models on about 80 minutes of Daji gameplay, then scores their first 130 seconds against in-game bots.GPT-6 Astra via its API scores each recorded video after deployment on the real-time gameplay platform.Source: §H.3 · p. 28
• The closed-loop rubric assesses equipment purchases, skill upgrades, control smoothness, movement strategy, attack timing, and teammate support.Open-loop and closed-loop results are reported in Tables 6 and 7, respectively.Source: §H.3 · p. 29

H.4 LATENT ANALYSIS

• 0.06322 for MOBA-VL versus 0.02011 for Qwen3.5-9B in mean cross-image cosine distance, indicating greater input-dependent variation for MOBA-VL.The analysis compares corresponding latent vectors at eight token positions across ten image pairs from five test images.Source: §H.4 · p. 29, Figure 12 · p. 30
• Figure 12 shows pairwise cosine distances for five test images and token-wise mean distances across ten image pairs, with larger distances indicating greater input variation.The analysis is performed separately with MOBA-VL and Qwen3.5-9B backbones.Source: §H.4 · p. 29, Figure 12 · p. 30