Content Based Video Narration of Gameplay with Vision Language Models

Mathew Varghese

Source-linked AI summary

Content Based Video Narration of Gameplay with Vision Language Models

Mathew Varghese

arXiv:2608.14016v1Aug 14, 2026cs.CVcs.AIcs.GR

TL;DR

Gameplay commentary is largely unavailable outside professional esports, limiting spoken, esports-style narration for arbitrary gameplay recordings. This paper presents a training-free vision-language and text-to-speech system whose temporal mosaics reduce image payloads by 9x and whose qualitative case study demonstrates coarse grounding, register transfer, and narrative continuity.

Problem
Gameplay video outside professional esports is largely silent or contains only game audio, while continuous narration remains poorly served for accessibility.
Method
The paper presents a training-free system combining temporal mosaic packing, context-conditioned prompting, and duration-conditioned speech alignment for arbitrary gameplay video.
Results
The system reduces per-minute image payloads by 9x and produces coarse grounding, register transfer, and narrative continuity in a qualitative strategy-game case study.
Takeaways & Limitations
The released implementation provides a reproducible baseline for spoken gameplay narration without engine telemetry or task-specific training.
Takeaways & Limitations
The proposed region-of-interest fix requires per-title HUD coordinates configured in a YAML file, introducing a limited game-specific dependency.

Abstract

from arXiv · show

Live game commentary is scarce: it exists for professional esports broadcasts and almost nowhere else. We present a content-based video narration system that produces spoken, esports-style commentary for arbitrary gameplay recordings using a general-purpose vision-language model (VLM) and a text-to-speech back end, with no game-specific instrumentation, no engine telemetry, and no task-specific training. Three mechanisms carry the system. Temporal mosaic packing arranges nine uniformly sampled frames into a single 3x3 image, letting an image-native VLM reason about motion while consuming one image payload per segment instead of nine. Context-conditioned prompting replays the K most recent narrations as assistant-role history, suppressing the repetition that dominates per-segment captioning of static scenes. Duration-conditioned generation and elastic alignment constrain narration length in the prompt, then time-scale or symmetrically pad the synthesized audio so each utterance fills its segment slot exactly, giving frame-accurate muxing without a forced aligner. The implementation supports either cloud TTS or a 6-bit quantized 4B-parameter on-device TTS model on Apple silicon, making the speech stage fully local. We report a qualitative case study on real-time strategy footage, a cost model showing the mosaic reduces per-minute image payloads by 9x, and a candid account of observed failure modes - hallucinated game state, resolution loss from mosaicking, and prosody artifacts from time-scaling. We release the system as a reproducible baseline, with an evaluation protocol for the quantitative study a full version will report.

I. INTRODUCTION · II. RELATED WORK

The paper presents a reproducible, training-free system for spoken gameplay narration that trades some accuracy for cross-game generality by using pixels rather than game-specific telemetry. Its three mechanisms—temporal mosaic packing, context-conditioned prompting, and duration-conditioned generation with elastic alignment—address cost, repetition, and exact audio-slot occupancy, with cloud or local speech back ends.

I. INTRODUCTION: Professional esports commentary makes gameplay watchable for nonplayers, but it is almost entirely absent from amateur uploads, training footage, and replays.Human commentary does not scale to the volume of gameplay video produced.
I. INTRODUCTION: Prior automated commentary systems depend on title-specific engine state, replay files, or telemetry APIs, limiting their generality despite accurate grounding.They map structured state to language using templates or trained generators.
I. INTRODUCTION: General-purpose VLMs consume gameplay pixels and produce commentary across genres without game-specific code, but may misread rules, counters, and plausible details.The tradeoff buys generality at the cost of accuracy.
I. INTRODUCTION: The naive approach of captioning and speaking every frame fails on cost, temporality, and speech-slot control, motivating three inexpensive corrective mechanisms.The paper adopts each fix as a contribution.
I. INTRODUCTION: Nine uniformly sampled frames are packed into one 3×3 mosaic image, carrying temporal evidence for a segment at one image’s cost.Duration and word budgets are prompted, while time-scaling or symmetric padding closes the remaining audio-length gap.
I. INTRODUCTION: Context-conditioned prompting replays recent narrations as assistant-role history to suppress repetition in static scenes.The supplied contribution description identifies this as one of the system’s three mechanisms.
I. INTRODUCTION: The speech stage supports cloud TTS or a 6-bit quantized 4B-parameter on-device TTS model running locally on consumer Apple silicon.Elastic alignment yields exact slot occupancy without forced alignment, supporting offline use and footage users do not want to upload.
I. INTRODUCTION: The released baseline includes a cost model, a qualitative strategy-game case study, documented failure modes, and a pre-registered protocol for future quantitative evaluation.The pipeline is described as complete, reproducible, and training-free, with cloud and on-device speech back ends.

A. Video captioning and dense event description … E. Accessibility

The paper situates gameplay narration at the intersection of dense video captioning, sports commentary, vision–language video understanding, local speech synthesis, and accessibility. Its distinctive focus is continuous, speakable narration generated without privileged structured game state, with practical attention to visual and audio constraints.

A. Video captioning and dense event description: Gameplay narration extends dense video captioning by requiring continuous audio coverage and speech that fits a fixed time budget.The system addresses the latter constraint with elastic alignment.
B. Sports and esports commentary generation: Sports and esports commentary systems commonly assume privileged access to structured game or event state, unlike this vision-only setting.Prior work includes SoccerNet-Caption, GOAL, racing-game commentary, and commentary for game-playing agents.
C. Vision–language models for video: Instruction-tuned VLMs and sampled-frame representations make general video description practical without task-specific training.The paper adopts image-grid frame packing following IG-VLM and evaluates it in a production-shaped setting.
C. Vision–language models for video: The image-grid representation causes resolution loss on 1080p frames, which the authors identify as the pipeline’s dominant accuracy limitation.This limitation is reported from practical experience and discussed in Section VII.
D. Speech synthesis: A 6-bit quantized 4B TTS model running through an Apple-silicon inference runtime enables local speech at acceptable commentary quality with no per-request cost.The speech stage is treated as a systems component rather than the paper’s primary research focus.
E. Accessibility: Continuous gameplay narration is structurally analogous to audio description for blind and low-vision audiences, an established practice in film and television.The paper notes that games remain poorly served and revisits this connection in Section VIII.

III. SYSTEM · A. Overview and notation · B. Temporal mosaic packing

The system partitions gameplay video into fixed-duration segments, creates one narration and audio clip per segment, and replaces the original audio track. Temporal mosaic packing supplies nine time-ordered frames per segment in one image, reducing image requests while sacrificing fine spatial detail.

A. Overview and notation: Each T-second segment receives exactly one narration utterance and synthesized audio clip, whose duration is forced to target τ before replacing the video’s audio track.The output audio is the concatenation of the segment clips.
A. Overview and notation: Frames are sampled at 9/T frames per second, so every segment contributes nine frames and T controls the trade-off between temporal resolution and request count.The sampling rate is coupled deliberately to the 3 × 3 mosaic capacity.
B. Temporal mosaic packing: For each segment, a centered, boundary-clamped window of consecutive sampled frames is composited onto a 3 × 3 raster lattice whose reading order matches time.The mosaic is JPEG-encoded, base64-embedded, and sent as one image.
B. Temporal mosaic packing: 9× reduction: mosaic packing consumes R_mosaic = 60/T images per minute instead of per-frame captioning’s R_frame = 540/T at the same temporal resolution.The reduction is independent of T and addresses hosted VLM pricing, rate limits, and daily request caps.
B. Temporal mosaic packing: At the default T = 9 s, mosaic packing uses 6.7 images per minute rather than 60, enabling an hour of footage within the cited daily cap.The passage frames this as the difference between narrating an hour of footage and not.
B. Temporal mosaic packing: Nine tiles expose motion and escalation that a single frame cannot, but aggressive API downsampling destroys fine detail such as counters, health bars, minimap icons, and small text.The composite of nine 1080p tiles is 5760 × 3240; large overlay text remains comparatively useful.

C. Context-conditioned prompting · D. Duration-conditioned generation

Context-conditioned prompting uses recent narrations to suppress repetition and create continuity in visually autocorrelated gameplay. Duration-conditioned prompts constrain each utterance’s length and style, while additional safeguards reduce timing, immersion, and refusal failures.

C. Context-conditioned prompting: Adjacent gameplay segments often look nearly identical, causing stateless captioners to produce repetitive text that is especially unsuitable for speech.This problem is pronounced during base-building phases and positional standoffs.
C. Context-conditioned prompting: The system inserts the K most recent narrations as assistant-role history and explicitly directs the model to maintain context and avoid repetition.K = 5 by default; larger K increases prompt tokens linearly and shows diminishing returns in observation.
C. Context-conditioned prompting: Recent narration history also supports continuity, prompting descriptions of escalation or resolution instead of treating each segment as a fresh scene.This helps the output sound like commentary rather than a sequence of captions.
D. Duration-conditioned generation: Each utterance is targeted to τ seconds, with the default τ = 2T and τ = T yielding exactly real-time narration.The 2T setting reflects continuous commentary over the video rather than sparse narration.
D. Duration-conditioned generation: The prompt injects the target duration through template substitution alongside a hard word budget and a style directive.The template requests a brief overview suitable for realtime esports commentary.
D. Duration-conditioned generation: The < 40 words budget limits synthesis duration, while suppressing screenshot references and specifying an esports register addresses observed immersion and style failures.The prompt focuses on key units, combat, buildings, and resources without naming the game.
D. Duration-conditioned generation: A token limit provides a second generation cap, and refusal-shaped responses trigger retries for up to three attempts.The system identifies a leading apology as refusal-shaped because a spoken refusal would fail the narration artifact.

E. Speech synthesis with interchangeable back ends

Speech synthesis uses a configuration-selected two-function interface with either hosted high-definition TTS or a fully local, quantized on-device implementation. The local path removes network, API-key, and per-request requirements for narration audio, though the VLM remains hosted.

Interchangeable back ends: The TTS stage exposes a two-function interface, generate_tts(text, path), with cloud and on-device implementations selected by configuration.This separates speech synthesis from the rest of the narration pipeline while allowing backend substitution.
Cloud back end: Cloud TTS uses the hosted tts-1-hd model with selectable voices, offering highest quality and lowest engineering cost but charging per character and requiring narration-text uploads.The cloud backend prioritizes quality and implementation simplicity at the cost of usage fees and text transmission.
On-device back end: The on-device backend runs a 4B-parameter Voxtral TTS model quantized to 6 bits on Apple silicon through MLX, exposing 19 voices across several languages.It generates 24 kHz float samples, concatenates streaming chunks, and encodes the result to MP3.
Deployment implications: The local speech path requires no API key, network, or per-request cost, shifting the narration-volume-dependent speech stage locally while the VLM still requires a hosted model.This expands deployment options and removes a privacy objection for unreleased or personal footage on the text-to-audio portion of the pipeline.

F. Elastic temporal alignment

Elastic temporal alignment forces every synthesized narration clip to exactly fill its segment slot, enabling deterministic concatenation and stream-copy muxing without forced alignment or video re-encoding. Symmetric padding improves reaction timing, while excessive compression can create rushed, chipmunked delivery.

F. Elastic temporal alignment: Each synthesized clip is forced after the fact to exactly τ because identical word counts can produce different durations across voices and phonetic content.A 35-word narration may render to 8 or 13 seconds.
F. Elastic temporal alignment: Exact-τ clips place utterance i at offset iτ, allowing stream-copy muxing with no forced aligner, drift accumulation, or video re-encoding.Time-scaling and silence padding produce clips with precisely matched durations.
F. Elastic temporal alignment: Symmetric padding centers speech within each slot, making onset follow the segment boundary rather than pre-empt it.This timing is intended to sound like a caster reacting to what just happened.
F. Elastic temporal alignment: Compression ratios above roughly 1.3 can produce audibly rushed, chipmunked delivery when narration exceeds its word budget.The passage identifies tightening the word budget as preferable to tolerating this artifact.

G. Algorithm · IV. IMPLEMENTATION · V. QUALITATIVE CASE STUDY

The implementation exposes a reproducible, model-substitutable pipeline, while a qualitative RTS case study shows coarse entity grounding, commentary-style register, and history-driven continuity alongside hallucinated state and prompt-domain limitations.

IV. IMPLEMENTATION: The Python implementation uses FFmpeg for frame extraction and muxing, Pillow for mosaics, and pydub for audio time-scaling, padding, and concatenation.These components cover the core media-processing stages.
IV. IMPLEMENTATION: All configuration parameters are exposed in one YAML file, making each run fully described by that file and the input video.This supports reproducibility across runs.
IV. IMPLEMENTATION: The default implementation uses GPT-4o for image-based narration, tts-1-hd with the nova voice for cloud speech, and a 6-bit quantized 4B-parameter Voxtral TTS checkpoint on device.An earlier gpt-4-vision-preview iteration retained the same pipeline apart from the vision model.
IV. IMPLEMENTATION: The three mechanisms are model-substitutable because they require only image input, chat-style role-tagged history, and prompt text for duration conditioning, plus file-output TTS.Specific models are reported because they are needed to reproduce the qualitative results.
IV. IMPLEMENTATION: Frame extraction is idempotent and transcripts are timestamp-archived, reducing repeated decoding costs and enabling comparison of prompt iterations.Frames are skipped when already present, while prior narrations are archived rather than overwritten.
V. QUALITATIVE CASE STUDY: The output transferred the requested commentary register, and prior-utterance history produced an assault-to-tension arc rather than independent scene descriptions, although lexical repetition remained.The observed repetition indicates K = 5 attenuated but did not eliminate the problem.
V. QUALITATIVE CASE STUDY: The case study also exposed evidence and prompt limitations: mosaic downsampling made resource digits unreadable, while an FPS prompt was ignored for description but could not reliably enforce game-specific vocabulary.The source resource bar showed 1125 wood, 657 food, 400 gold, 894 stone, and 35/70 population, yet the narration asserted resources were holding or steady.

A. Cost

At T = 9 s, the pipeline uses 6.7 VLM requests and 6.7 TTS utterances per minute, versus 60 VLM images for per-frame captioning, while on-device speech makes marginal TTS cost zero. Its monetary cost therefore consists exactly of VLM requests.

Cost: 6.7 VLM requests and 6.7 TTS utterances are issued per minute at T = 9 s.These rates apply to the described pipeline configuration.
Cost: 60 VLM images per minute are required by per-frame captioning at identical temporal resolution.The mosaic-based pipeline therefore uses fewer VLM image requests at the same temporal resolution.
Cost: With the on-device speech back end, the TTS component of marginal cost is zero.Under this configuration, the pipeline’s monetary cost is exactly the VLM requests.

VI. PROPOSED EVALUATION PROTOCOL · VII. LIMITATIONS AND FAILURE MODES

The paper presents a preregistered evaluation protocol rather than human-subject results, and identifies limitations spanning grounding, visual resolution, audio, segmentation, latency, voice design, prosody, and absent quantitative validation. Proposed remedies include region-of-interest decomposition, multimodal inputs, event-driven segmentation, and improved duration control.

VI. PROPOSED EVALUATION PROTOCOL: The study reports a protocol, not human-subject evaluation, using 30 gameplay clips across RTS, FPS, and racing genres.The corpus contains 10 clips per genre, each 60–120 seconds.
VI. PROPOSED EVALUATION PROTOCOL: The planned comparison tests the full system against no history, per-frame captioning, human commentary, and original game audio.Per-frame captioning matches temporal resolution while requiring 9× the request cost; human commentary is a ceiling and original audio a floor.
VI. PROPOSED EVALUATION PROTOCOL: Evaluation combines 5-point subjective ratings with factual grounding, repetition, and alignment measures.Participants rate naturalness, informativeness, excitement/appropriateness of register, and synchronization; grounding is annotated against the video by two independent annotators.
VI. PROPOSED EVALUATION PROTOCOL: A separate small-n accessibility study evaluates the system as audio description for blind and low-vision participants.This study distinguishes accessibility from entertainment because informativeness and excitement are weighted differently.
VII. LIMITATIONS AND FAILURE MODES: Confident hallucinations can make unverifiable game-state assertions indistinguishable from grounded information, requiring hedging or verified-value grounding.This limitation is especially consequential when viewers may act on commentary for coaching, analysis, or accessibility.
VII. LIMITATIONS AND FAILURE MODES: Mosaicking downscales nine 1080p tiles enough to destroy HUD text and small sprites, directly trading visual detail for the 9× cost saving.The proposed fix is region-of-interest decomposition: low-resolution temporal playfield mosaics plus native-resolution HUD crops processed with OCR and minimap analysis.
VII. LIMITATIONS AND FAILURE MODES: The system also omits audio, uses fixed segmentation, operates offline, and produces a single voice without broadcast-style interplay.Proposed directions include audio integration, event-driven segmentation, streaming generation with incremental muxing, and an untried multi-voice extension.
VII. LIMITATIONS AND FAILURE MODES: Time-scaling beyond roughly 1.3× creates audible prosody artifacts, and the paper has no quantitative evaluation of its repetition-suppression or register-transfer claims.Tighter word budgets or duration-aware regeneration are preferred to post-hoc compression.

VIII. ETHICAL CONSIDERATIONS

The system’s ethical use depends on prioritizing accurate accessibility support while acknowledging risks to human labour, provenance, and gameplay-content rights. Synthetic commentary should be disclosed, and cloud processing creates exposure that on-device speech only partly reduces.

Accessibility: Audio description for blind and low-vision players is the strongest use case, but it requires accuracy over excitement because hallucinations become safety-relevant defects.This reverses the priorities of the current prompt.
Labour: Near-term deployment is most plausible for amateur uploads, replays, and archives without commentary, not professional broadcasts where human casters’ quality and social role are not substitutable.The authors bound but do not dismiss the labour concern.
Provenance: Generated commentary should be labelled because confident narration over real footage can misrepresent match events, while the system lacks watermarking or disclosure mechanisms.The provenance risk arises from synthetic narration presented over authentic gameplay footage.
Content and rights: Gameplay footage carries rights held by both players and publishers, and cloud processing transmits frames and text to third-party services.On-device speech reduces, but does not eliminate, this exposure.

IX. CONCLUSION

The paper presents a training-free system for spoken, esports-style narration of arbitrary gameplay using a general vision–language model and interchangeable speech back ends, including one that runs entirely on-device. Its temporal mosaics, context-conditioned prompting, and duration-conditioned elastic alignment enable efficient motion reasoning, continuous narration, and exact speech-video fit, while strategy-game footage reveals confident hallucination as the central unresolved problem.

System contribution: The system narrates arbitrary gameplay video with spoken, esports-style commentary using a general vision–language model and an interchangeable speech back end.One speech back end runs entirely on-device.
Core mechanisms: Temporal mosaic packing provides motion evidence at a ninth of the image cost, while context-conditioned prompting converts independent captions into a continuous narrative.These are two of the three mechanisms carrying the system.
Core mechanisms: Duration-conditioned generation with elastic alignment makes speech fit video exactly without a forced aligner.The mechanism controls narration duration and aligns the resulting speech to the video slot.
Case study and limitation: A qualitative strategy-game case study shows correct coarse entity grounding, successful register transfer, and emergent continuity, alongside confident hallucination of discarded game state.The paper identifies hallucinated game state as the central unsolved problem.