PaperlayerView exact revision on arXiv

Source-linked AI summary

The GenAI4IDN Benchmark 3.0 - a Public Tool to Assess Generative AI Tools for the Design of Interactive Digital Narratives

Hartmut Koenitz, Jonathan Barbara, Mirjam Palosaari Eladhari

arXiv:2610.05633v1cs.AI

TL;DR

IDN design lacks a shared benchmark suited to contextual, expert-judged, multimodal, and interactive work. GenAI4IDN 3.0 offers a public, community-extensible platform with automated ratings and new fact-checking and bias tasks. Results indicate maturing but uneven capabilities, with interactive structure remaining difficult and new failure modes emerging.

  • Problem

    IDN outputs have plural, contextual quality and span modalities and interaction, so they cannot be adequately assessed with closed-ended, single-score benchmarks.

    Source: §2.2 · p. 4

  • Method

    The paper develops a public web benchmark with automated evaluation extended to complex media and dedicated fact-checking and bias tasks and raters.

    Source: §1 · p. 2, §6 · p. 19

  • Results

    The field is maturing but uneven: frontier models score consistently high across most design tasks, while genuine interactive structure remains challenging and runaway thinking and overzealous safety filters have emerged.

    Source: §6 · p. 19

  • Takeaways & Limitations

    The platform lets the community run benchmarks on demand, add models, and propose tasks to keep results and coverage current.

    Source: §1 · p. 2, §6 · p. 18

    The fact-checking and bias approach lacks cross-validation by multiple independent raters and remains exploratory, culturally situated case-study evidence rather than evidence of general rater reliability.

    Source: §4.3 · p. 18

Abstract

from arXiv · show

This paper presents GENAI4IDN Benchmark 3.0, the third iteration of an evaluation framework to assess Generative AI tools for creating Interactive Digital Narratives (IDNs). Moving beyond manual testing, this iteration introduces AI-assisted evaluation through a publicly accessible web application (https://genai4idn.com), enabling the community to run benchmarks on demand, add new models, and propose new tasks. The revised evaluation framework is "blinded" to avoid model-bias, and can handle complex media such as music, videos, and full IDNs that previously required human raters. A significant addition - responding to concerns raised during ICIDS 2025 - is the addition of fact-checking and bias detection with dedicated tasks and rubrics, validated by human raters with lived experience in the depicted contexts. Findings from a diverse range of models report on maturing creative capabilities while observing runaway thinking and overzealous safety filters as limitations. Fact-checking reliably caught subtle historical inaccuracies, anachronisms, and fabricated claims while the bias rater consistently exposed structural assumptions, tropes, and marginalized group erasures.

1 Introduction

GenAI4IDN addresses the lack of a shared way to assess what generative AI can do for interactive digital narrative design. Its third iteration makes the framework publicly accessible, expands automated evaluation, and adds dedicated checks for factual accuracy and bias.

1 Introduction

  • The framework addresses the missing shared yardstick for assessing GenAI tools in IDN design, building on work with professional narrative designers and later updates to tools and prompting methods.Prior iterations introduced iterative and contextualized prompting and AI-assisted evaluation.Source: §1 · pp. 1–2
  • The third iteration turns manually executed procedures into a public web benchmark where the community can run tests on demand, add models, and propose tasks.It also makes benchmark results publicly accessible and updates the roster of tools.Source: §1 · p. 2
  • The benchmark extends automated rating to music, videos, and games, and adds fact-checking and bias tasks in response to concerns raised by the IDN community at ICIDS 2025.The new tasks and raters are included in standard benchmark runs.Source: §1 · p. 2

2 Related Work and Background

Research on IDN design has shifted toward human–AI co-creation, while evaluation remains difficult because outputs are voluminous, multimodal, interactive, and judged contextually. This framework addresses that gap and extends evaluation to factual accuracy and bias.

2.1 Related Work

  • AI-assisted evaluation combines human-expert and LLM judgments to assess GenAI tools for IDN design at a scale that professional raters’ time alone would not allow.The field has moved from systems producing complete narratives toward co-creative and mixed-initiative approaches involving human authors [46, 34, 13].Source: §2.1 · p. 3

2.2 Benchmarking

  • IDN design lacks ground-truth answers and requires contextual expert judgment across multimodal, interactive outputs, making closed-ended, single-score benchmarks insufficient.Related critiques identify construct-validity problems and overreliance on text-based, one-time tests.Source: §2.2 · pp. 3–4

2.3 Bias

  • Bias in IDN design can narrow the stories told and cause representational harm, yet stereotyped storyworlds may also be deliberately used to foreground or subvert stereotypes.The framework therefore seeks to detect and describe bias rather than issue a single biased/unbiased verdict.Source: §2.3 · p. 4
  • The framework evaluates factual accuracy through accuracy, groundedness, and consistency, and bias through fairness, representation, and sensitivity.Culturally situated tasks can be assessed jointly for factual groundedness and bias.Source: §2.3 · pp. 4–5

3 A Benchmarking Web Application for 2026

The section describes a shift from brittle manual procedures to a self-contained benchmarking platform, alongside expanded tasks and automated evaluation. It details the platform’s architecture, new task coverage, and methods for rating multimodal outputs and playable IDNs.

3 A Benchmarking Web Application for 2026

  • GenAI4IDN 3.0 replaces time-consuming, brittle manual procedures with a self-contained platform; the authors describe supervised coding as the human role in its implementation.The platform uses an API-based approach.Source: §3 · p. 5

3.1 Architecture and Evaluators

  • Sonnet 5 serves as the canonical evaluator, while the model, task, and rater lists are modifiable and results appear alongside generated artifacts.Changing Sonnet 5’s reasoning level produced a combined delta of +.27, which the authors considered insufficient to justify five to seven times slower results and higher costs.Source: §3.1 · pp. 5–6

3.2 New and Updated Models

  • The model list expanded, while manual imports accommodate image and audio outputs where APIs are rare and CSV uploads allow human-generated content to be benchmarked.API keys are stored separately and protected by the web app’s authentication and authorization system.Source: Import Generated Content · p. 6, §3.2 · p. 6, API keys · p. 6

3.3 Tasks

  • Three new tasks add historically accurate narrative framing and foundations for historical narratives in the Global North and Global South; T15.2 now requests a browser game instead of a Unity project.The change addresses technical and licensing issues with making Unity projects playable online; the browser-game prompt uses supplied assets and more precise instructions.Source: §3.3 · p. 8

3.4 Blinded Raters, Fact-Checking, and Bias Detection

  • Blinded raters have model traces removed; two models rate video and music with same-model or same-family outputs excluded, while lived-experience participants reviewed country, decade, and identity combinations.The case studies covered Sweden, Germany, Malta, Colombia, Ghana, Nigeria, and Thailand, using convenience sampling.Source: §3.4 · p. 9
  • The fact checker rates accuracy, groundedness, and internal consistency, using web search and thinking to catch unsupported or mistimed claims and a median of three runs to manage variance.The bias prompt attends particularly to stereotypical representation and rates fairness, representation, and sensitivity on a 1–5 Likert scale.Source: §3.4 · p. 10

3.5 Automatic Player

  • An automatic player runs IDNs in isolated, engine-appropriate environments, observes them through vision and source-derived mechanics, and supplies screenshots and transcripts for text-model raters.Its playability verdict ranges from played-to-ending to not-executed and hardcaps the Production score, making that score observed rather than inferred.Source: §3.5 · p. 12

4 Results and Observations

The benchmark’s results place Claude Opus 5 at the top of the overall and per-task rankings, while detailed observations show both useful ratings and important failure modes. The section also reports that the fact- and bias-checking case studies are exploratory rather than evidence of general rater reliability.

4.1 Results

  • Claude Opus 5 leads the overall leaderboard at 12.04 and leads the per-task average list, while Claude Opus 4.8 and Fable 5 score at least 4.0 in six tasks.The overall leaderboard pools mean Inspiration, Training, and Production scores for models tested on at least eight tasks.Source: §4.1 · pp. 12–13

4.2 Detailed observations

  • Fact-checking caught a 1560 Istanbul barracks-location error and a term anachronism, while German case studies exposed historical misattribution and overgeneralization.The German rater identified a claim about East German male partners that applied before 1961, not in the 1980s, and housing generalization true only for part of the population.Source: Task-based observations · p. 16, Germany · p. 17
  • T05 scaffolding remained a challenge: all four raters found limited agency, and the generated storylines tended toward happy endings.The result aligned with the prior year’s findings.Source: Task-based observations · p. 16
  • The Malta case exposed an AI fact-checker’s confusion between 1998 and 1996, while Nigerian human and automated evaluations agreed on transparency about verifiable information and evidence gaps.The Malta bias rating was judged accurate by the human rater.Source: Malta · p. 16, Nigeria (Igbo) · p. 16
  • Swedish ratings matched the human rater’s recollection while also flagging immigrant women’s absence and period tropes; Thai review found errors, omissions, and male-centric framing.The Thai review also identified mislabelling of masculine-presenting female communities and reduced representation of non-binary identity.Source: Germany · p. 17, Thailand · p. 17
  • Fact-checking and bias detection can help authors distinguish grounded material from stereotypes, which may also serve deliberate creative purposes.For the same subtasks, ChatGPT 5’s more event-focused output scored above DeepSeek’s more trope-reliant output, a comparison the human rater considered appropriate.Source: General observations · pp. 17–18

4.3 Limitations

  • The fact- and bias-checking approach has not been cross-validated by multiple independent raters, so its case studies do not establish general rater reliability.Source: §4.3 · p. 18

5 Addressed Limitations and Ethical Concerns

The benchmark addresses manual handholding by automating configured runs, while using blinding and a bias rater to mitigate concerns about rating bias. Variation can be controlled, but unfair use of artists’ work for training remains unresolved.

5 Addressed Limitations and Ethical Concerns

  • The website executes all tasks autonomously once a run is configured, addressing handholding in content generation procedures.Earlier work identified handholding AI as a limitation.Source: §5 · p. 18
  • Raters are blinded by scrubbing identifying marks from generated content, and a bias rater assesses the content.The authors describe these measures as mitigating concerns about dataset and rater bias.Source: §5 · p. 18
  • API temperature settings control output variation; when unavailable for thinking, the authors use the median of three iterations.Source: §5 · p. 18
  • The unfair use of artists’ work to train AI models without representation and compensation remains unresolved.Source: §5 · p. 18

6 Conclusion

GenAI4IDN 3.0 reports strong but uneven creative capabilities alongside challenges with interactive structure and new failure modes. It moves the benchmark to a public application and expands automated evaluation to additional media, factual accuracy, and bias.

6 Conclusion

  • The benchmark shifts to a public web application for on-demand runs, model additions, and proposed tasks, with automated rating extended to music, video, and IDN outputs.Dedicated tasks, raters, and rubrics now cover factual accuracy and bias.Source: §6 · pp. 18–19
  • Frontier models score consistently high across most design tasks, but genuine interactive structure remains challenging and runaway thinking and overzealous safety filters have emerged.Source: §6 · p. 19
  • Fact-checking and bias raters validated against human raters with lived experience are reliable enough to support authors working outside their own background.Source: §6 · p. 19
  • Future work includes refining the automated player, making bias detection more fine-grained, and developing harder challenges as frontier models improve.The authors invite the community to help keep the benchmark’s models, tasks, and raters current.Source: §6 · p. 19

A Appendix

The appendix captions identify model generations for T15.2 and T16.

A Appendix

Loading 2610.05633v1…