PaperlayerView exact revision on arXiv

Source-linked AI summary

MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration

Thomas Hayes, Songyang Zhang, Xi Yin, Guan Pang, Sasha Sheng, Harry Yang, Songwei Ge, Qiyuan Hu, Devi Parikh

arXiv:2204.08058v3cs.CVcs.AI

TL;DR

Video-text research faces data-collection and spatiotemporal-modeling challenges, while existing datasets trade scene complexity against rich interactions. MUGEN builds a controllable video-audio-text dataset from modified CoinRun and benchmarks cross-modal retrieval and generation, with a human-evaluation protocol for generated outputs.

  • Problem

    Video-text research lags image-text work because collecting data and modeling spatiotemporal information are challenging, while existing datasets offer either complex scenes or limited interactions.

    Source: §1 · p. 1

  • Method

    MUGEN uses modified CoinRun and RL gameplay to produce 375K 3.2-second clips with human text, synchronized audio, semantic maps, and templated auto-text.

    Source: §1 · p. 2

  • Results

    MUGEN provides a closed-world dataset with rich annotations for multimodal tasks, and the paper benchmarks retrieval and generation across modality pairs.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    Its constrained visuals and diverse interactions support multimodal research without requiring web-scale data or massive compute.

    Source: §1 · p. 3

    Initial experiments show unstable training with a learnable text encoder.

    Source: §5.2 · p. 11

Abstract

from arXiv · show

Multimodal video-audio-text understanding and generation can benefit from datasets that are narrow but rich. The narrowness allows bite-sized challenges that the research community can make progress on. The richness ensures we are making progress along the core challenges. To this end, we present a large-scale video-audio-text dataset MUGEN, collected using the open-sourced platform game CoinRun [11]. We made substantial modifications to make the game richer by introducing audio and enabling new interactions. We trained RL agents with different objectives to navigate the game and interact with 13 objects and characters. This allows us to automatically extract a large collection of diverse videos and associated audio. We sample 375K video clips (3.2s each) and collect text descriptions from human annotators. Each video has additional annotations that are extracted automatically from the game engine, such as accurate semantic maps for each frame and templated textual descriptions. Altogether, MUGEN can help progress research in many tasks in multimodal understanding and generation. We benchmark representative approaches on tasks involving video-audio-text retrieval and generation. Our dataset and code are released at: https://mugen-org.github.io/.

1 Introduction

MUGEN addresses the difficulty of video-text research with a controllable video-audio-text dataset that combines diverse motion and interactions with rich annotations. The paper also benchmarks cross-modal retrieval and generation and presents a human-evaluation protocol for generated outputs.

1 Introduction

  • Video-text research lags image-text work amid data-collection and spatiotemporal-modeling challenges, while existing datasets trade complex real-world scenes for limited interactions.Source: §1 · pp. 1–2
  • MUGEN modifies CoinRun into a closed-world video-audio-text dataset with added audio, adjusted cameras, and new interactions between its protagonist and other characters and objects.Source: §1 · p. 2
  • 375K 3.2-second clips are sampled from 233K procedurally generated gameplay videos recorded by 14 RL agents with different objectives.Source: §1 · pp. 2–3
  • Each clip pairs human descriptions with synchronized audio, frame-level semantic maps, and templated auto-text, providing multiple annotations for multimodal research.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • MUGEN combines visually simple scenes with diverse motion and interactions, while its annotations and modifiable game engine support varied tasks without web-scale data or massive compute.Source: §1 · p. 3
  • The paper benchmarks retrieval and generation across modality pairs and evaluates generation quality and faithfulness against ground-truth videos through human judgments.Source: §1 · pp. 3–4

2 Related Work

Related work contrasts open-world datasets, which capture complex scenes, with constrained datasets that often lack meaningful multi-entity interactions. It also reviews video-text and audio-related tasks and the limited resources for studying text-to-video generation.

Multimodal Datasets

  • Open-world image datasets such as MSCOCO, ConceptualCaptions, and WIT support image-text research, while closed-world CLEVR reduces visual complexity for reasoning studies.Source: §2 · p. 4
  • Open-world video datasets capture complex events, while constrained sets such as BMNIST and KTH lack meaningful multi-entity interactions; MUGEN simplifies scenes while retaining complex interactions.Source: §2 · p. 4
  • Existing datasets use human-written or speech-derived text, whereas MUGEN also supports templated descriptions for videos of arbitrary lengths.Source: §2 · p. 4
  • Video-audio pairs in open-world datasets may be poorly aligned, while MUGEN synchronizes audio with Mugen’s actions, supporting study of audio generation from video or text.Source: §2 · p. 5

Multimodal Understanding and Generation

  • Text-to-video generation remains relatively understudied because feasible datasets are scarce, and limited motion in BMNIST, KTH, and UCF-101 [48] restricts text-description diversity.Source: §2 · p. 5
  • The paper explores video-audio-text retrieval and generation across modality pairs and conducts evaluations including human evaluation.Source: §2 · p. 5

3 MUGEN Dataset

MUGEN adapts CoinRun into a closed-world dataset with constrained scenes but varied motions and interactions. It pairs large collections of short clips and longer gameplay videos with human text, audio, semantic maps, and automatically generated descriptions.

3 MUGEN Dataset

  • CoinRun modifications add audio, slower physics, camera adjustments, and interactions, with Mugen, 10 monsters, coins, gems, and two world themes.Mugen has 16 actions; monsters have differing action vocabularies.Source: §3 · p. 6
  • 14 RL agents with different reward functions record procedurally generated gameplay using unique map seeds, with no duplicate videos verified.The collection prioritizes video diversity rather than agent performance.Source: §3 · p. 6
  • Manual descriptions cover 375,368 clips, while template-based auto-text derives descriptions from game-engine metadata.The video set is split into 96-frame clips for human annotation.Source: §3 · p. 7
  • Auto-text is structured around uniquely described actions and interactions, and automatic video-text generation can support arbitrarily long data without costly manual annotation.The paper points to text-conditioned long video generation, video grounding, and dense video captioning as possible tasks.Source: §3 · p. 7
  • 375K 3.2s clips pair with 379K manual descriptions, alongside 233K gameplay videos ranging from 3.2s to 21s.Each clip or longer video also has audio, semantic maps, and auto-text.Source: §3 · pp. 7–8
  • MUGEN is several orders of magnitude larger than closed-world datasets, less visually diverse than some open-world datasets, and customizable through its released game engine.Its clips include diverse entities and interactions; most contain 2–4 characters and objects, 2–4 actions, and 2–3 interactions.Source: §3 · p. 8

4 Video-Audio-Text Retrieval and Generation

The paper describes retrieval and generation between pairs of video, audio, and text modalities. Retrieval uses modality-specific or joint models, while generation uses a shared token-based framework.

4.1 Video-Audio-Text Retrieval.

  • Retrieval maps each modality into a joint embedding space and scores cross-modal pairs with temperature-scaled cosine similarity.The paper trains separate video-audio, video-text, and audio-text models, as well as a joint model using all three losses.Source: §4.1 · p. 9
  • At inference, retrieval ranks candidates by cross-modal similarity, and two query modalities can be combined by summing their similarity scores.The combined scores can come from independently trained models or the joint model.Source: §4.1 · p. 9

4.2 Video-Audio-Text Generation

  • Cross-modal generation uses discrete codebooks for each modality and a decoder-only transformer to generate tokens from one modality to another.Video uses a 3D VQ-VAE, audio a 1D VQ-VAE, and text a tokenizer learned from manual descriptions.Source: §4.2 · p. 10
  • During generation, the transformer predicts target-modality tokens autoregressively from source tokens, then decodes audio or video tokens through the corresponding VQ-VAE.Text tokens are reconstructed as sentences from the vocabulary.Source: §4.2 · p. 10

5 Experiments

The experiments evaluate retrieval and generation across video, audio, and text, including automatic metrics and human judgments of quality and faithfulness. Results show task-dependent performance: retrieval benefits from combining modalities, while generation outcomes vary with modality alignment and tokenization settings.

5.1 Video-Audio-Text Retrieval

  • The retrieval task used recall at ranks 1, 5, and 10, and compared two-modality ensembles with joint models for paired queries.Video inputs were generally 256×256×32, with 32 frames sampled evenly from the 96-frame clips to reduce computation.Source: §5.1 · pp. 10–11
  • Ensembles of separately trained models consistently outperformed single-modality retrieval, while joint embeddings typically fell between the separate models.Video-to-audio and audio-to-video performed best among single-modality tasks; text-to-audio and audio-to-text performed worst, consistent with denser video-audio than audio-text alignment.Source: §5.1 · p. 11

5.2 Video-Audio-Text Generation

  • Human evaluation compared generated samples with ground truth for quality and input alignment, using five judges per comparison and majority votes.The evaluation sampled 512 test examples and omitted T2A and A2T because judging audio-text alignment was not straightforward.Source: §5.2 · p. 13
  • Higher video frame rates improved V2T generation, which had 42.6% of samples preferred over ground truth for faithfulness and 31.25% for realism.A2T performed worse than V2T; smaller audio compression ratios, with more tokens, improved A2T. The table compares all generation tasks.Source: Table 3 · p. 12, §5.2 · p. 13
  • Higher frame rates improved T2V automatic metrics, down-sampled training sets outperformed full-data models, and auto-text underperformed manual text.Combining auto-text with manual text did not noticeably improve performance; human judgments also favored faithfulness over quality, with trends similar to automatic metrics.Source: Table 3 · p. 12, §5.2 · p. 14
  • For A2V, smaller compression ratios improved automatic quality and faithfulness; longer audio sequences improved FAD for T2A and V2A.R.Sim. did not necessarily follow the audio-length trend, and human judgments showed higher faithfulness than quality for these tasks.Source: Table 3 · p. 12, §5.2 · p. 14
  • V2T was the easiest generation task, while the other three showed higher faithfulness than quality in human evaluation.The authors suggest improved VQ-VAE reconstruction could potentially raise quality and note that judges can reasonably ignore generation quality when evaluating faithfulness.Source: §5.2 · p. 14

6 Conclusion

The conclusion presents MUGEN as a closed-world, large-scale multimodal dataset built on an enhanced version of CoinRun. It combines rich annotations with cross-modal retrieval and generation tasks, and introduces calibrated human evaluation for comparing generative models.

6 Conclusion

  • MUGEN combines video, human and templated text, frame-level pixel-accurate semantic maps, and audio in a closed-world dataset based on enhanced CoinRun.The paper states these resources support multimodal research without web-scale data or massive compute.Source: §6 · p. 14
  • The paper explores retrieval and generation between every pair of modalities and calibrates human evaluation against ground-truth samples.The authors say this calibration makes comparison and progress tracking easier.Source: §6 · p. 14
  • The dataset, modified game engine, training code, models, and human-evaluation interfaces are released through the MUGEN project site.The passage provides the project URL; it is omitted here to keep the summary focused on the paper's conclusion.Source: §6 · p. 14

A.1 Full list of modifications

The game modifications add audio and interactions while simplifying and stabilizing the visual environment. They also adjust gameplay and camera behavior to support diverse, more observable events.

A.1 Full list of modifications

  • The game adds background music and action-triggered sound effects for Mugen’s core actions.Two songs correspond to the space and snow themes; eight sound effects cover actions such as walking, jumping, collecting coins, and dying.Source: §A.1 · p. 20
  • The visual world is simplified to one protagonist and two themes, while slower physics and increased camera zoom make gameplay easier to observe.The camera follows Mugen horizontally but not vertically, and the map height is reduced for a stable vertical view.Source: §A.1 · p. 20
  • Mugen gains the ability to jump on and kill some monsters, alongside death and kill animations, a gem, and a protective power-up shield.The shield appears when Mugen collects a gem and disappears when Mugen collects a coin.Source: §A.1 · p. 20
  • Additional monsters and hopping animations are added, and monster movement speeds and abilities are adjusted so they are not identical.Before-and-after examples are available in a supplementary video.Source: §A.1 · p. 20

A.2 Manual Text

Annotators describe short video clips through an interface with playback and volume controls. Submission rules and post-processing filter low-quality descriptions.

A.2 Manual Text

  • Annotators describe 3.2-second clips in one or two sentences and refer to each character by its specific name.The interface allows annotators to adjust video playback and volume.Source: §A.2 · p. 20
  • Descriptions must mention Mugen, the clip must be played, and each description must exceed three words before submission.The team also sampled descriptions for manual quality checks, blocked poor-quality annotators, and removed their existing annotations.Source: §A.2 · p. 20
  • After filtering 18,449 descriptions from blocked annotators and 4,778 short descriptions, the dataset retained 378,902 descriptions for 375,368 clips.A very small portion of clips have more than one description; descriptions of no more than 20 characters were removed.Source: §A.2 · p. 21

A.3 Auto-Text

MUGEN’s engine metadata drives templated descriptions by grouping poses into segments, describing each segment, and joining phrases into video-level text.

A.3 Auto-Text

  • Frames with the same pose are grouped into segments, and nearby same-pose segments interrupted by a few other-pose frames are merged.The merge counts the number of segments, helping represent repeated actions such as jumping.Source: §A.3 · p. 21
  • Each segment gets a template-based phrase using pose-specific details such as jump height, repetitions, horizontal movement, enemies crossed, landing surface, and kills.For example, a jump can be described as going up to the right over a snail to a platform.Source: §A.3 · pp. 21–22
  • The short phrases are joined with “and,” prefixed with “Mugen,” and segments shorter than five frames are filtered out.A resulting description can combine walking, jumping to a ladder, and being killed by a frog.Source: §A.3 · p. 22

B Dataset Statistics

MUGEN’s entity positions and interaction times are distributed across the game frame, with patterns reflecting platform placement, character behavior, and level termination.

B Dataset Statistics

  • Entities primarily occupy horizontal bands corresponding to platform heights, while Mugen remains centered on the x-axis.The position distributions use each entity asset’s center in each frame of the 3.2-second clips.Source: B Dataset Statistics · pp. 22–23
  • Snail, worm, and face monsters tend to appear nearer the frame center, while other monsters less often do because Mugen avoids them.The authors attribute the former pattern to Mugen’s incentive to approach and jump on monsters it can kill.Source: B Dataset Statistics · p. 23
  • Gem and coin appear less often in the center because they disappear when Mugen collects them; other entities besides Mugen occur slightly more often on the right due to map-generation bias.Source: B Dataset Statistics · p. 23
  • Interactions are roughly uniformly timed except “killed by monster,” which occurs only at video ends because Mugen’s death ends the level.Temporal distributions span the 96 frames of each video.Source: B Dataset Statistics · pp. 22–23

C Human Evaluation User Interface

The human-evaluation interface separates quality judgments from faithfulness judgments across text, video, and audio generation tasks.

C Human Evaluation User Interface

Loading 2204.08058v3…