PaperlayerView exact revision on arXiv

Source-linked AI summary

The Unwritten Benchmark: A New Challenge for Multimodal Machine Learning in Abstract Perceptual Reasoning

Garima Arya Yadav, Nilay Yilmaz, Yezhou Yang

arXiv:2608.14558v1cs.AIcs.CV

TL;DR

Existing benchmarks largely test recognition of explicit content, leaving abstract inference from unseen dynamic processes underexplored. The Unwritten Benchmark evaluates acousto-kinematic word inference from pen-scratch audio and hand-motion video without visible ink, finding that state-of-the-art multimodal models categorically fail while humans perform substantially better.

  • Problem

    Existing benchmarks rarely measure whether models can infer unseen abstract outcomes from dynamic, generative processes by synthesizing temporal multimodal cues.

  • Method

    The Unwritten Benchmark tests word inference from synchronized pen-scratch audio and hand-motion video across three handwriting styles, with no visible ink trace.

  • Results

    State-of-the-art MLLMs categorically fail, with GPT-4o scoring 8.85% on Muted Video for the synthetic set.

  • Takeaways & Limitations

    The benchmark provides a concrete tool for measuring progress toward models that understand physical and causal processes rather than merely recognize explicit content.

  • Takeaways & Limitations

    Models often process video as isolated frames, losing the continuous pen-tip micro-kinematics needed to reconstruct letterforms and infer words.

Abstract

from arXiv · show

Current multimodal models have demonstrated remarkable proficiency in recognizing static visual and auditory content. However, their capacity for abstract perceptual reasoning, inferring unseen information from dynamic, generative processes, remains a critical and underexplored frontier. In this paper, we introduce The Unwritten Benchmark, a new challenge designed to probe this abstract perceptual and cognitive ability. We define the core task as acousto-kinematic word inference: models must decipher words, across 3 different writing styles, being written solely from the audio of pen scratches and the video of hand movements, without any visible ink trace. Our evaluation results reveal a profound gap between human and machine performance: while human participants achieve high ordered letter accuracy (over 80%), leading Multimodal Machine Learning Models, including GPT-4o and Gemini 2.5-Pro, struggle significantly, failing to surpass 10%. Furthermore, we identify a paradoxical fusion effect in the models, where providing both modalities often degrades performance rather than improving it. This finding indicates a fundamental breakdown in their ability to synthesize complementary perceptual cues for this cognitive task. These findings highlight significant limitations in both cross-modal causal reasoning and the understanding of the micro-kinematics essential for such cognitive and intuitive perceptual reasoning.

1. Introduction

Existing multimodal benchmarks mainly test recognition and description of explicit static features, whereas abstract perceptual reasoning requires inferring an unseen outcome from dynamic generative processes. The Unwritten Benchmark addresses this gap with acousto-kinematic word inference from pen-motion video and scratching audio without visible ink, a task that challenges current SOTA MLLMs.

  • Motivation: Existing multimodal models excel at correlating explicit, static features across sensory inputs in tasks such as VQA and image captioning.This paradigm primarily evaluates recognition, description, and linking of content already present in the data.
  • Motivation: Abstract perceptual reasoning requires inferring an unseen outcome from a dynamic generative process by synthesizing temporal cues while reasoning about causality, movement, and intent.The paper characterizes this ability as the human “perception of becoming.”
  • Problem: Existing benchmarks fail to isolate this process-oriented perceptual-cognitive skill, leaving a significant blind spot in modern AI.They measure recognition of explicit content rather than inference of an abstract concept from the physical process creating it.
  • Contribution: The Unwritten Benchmark tests acousto-kinematic word inference by asking models to decipher words written without visible ink from hand-motion video and pen-scratch audio.The benchmark uses synchronized audio and video recordings of three handwriting styles and shifts evaluation from static recognition to dynamic inferential reasoning.
  • Contribution: The task is cognitively intuitive for humans but profoundly difficult for current state-of-the-art Multimodal Large Language Models.The benchmark presents synchronized audio and video as the available perceptual information while withholding the ink trace.

2. Related Work

Prior work benchmarks handwriting mainly as static images or time-sequenced pen trajectories, while multimodal datasets and fusion methods address broader audio-visual recognition tasks. These foundations leave the paper’s dynamic, cross-modal handwriting-inference setting distinct from established benchmarks.

  • Handwriting Recognition Benchmarks and Limitations: Offline handwriting benchmarks use static written-text images for optical character recognition, writer identification, and word spotting.Examples include IAM, NIST, and EMNIST.
  • Handwriting Recognition Benchmarks and Limitations: Online handwriting datasets instead record pen trajectories over time, including coordinates, timing, and pen-up/pen-down events.IAM-OnDB and UNIPEN represent writing as a temporal process rather than only as an image.
  • Multimodal Action and Sound Recognition: Multimodal action and sound benchmarks pair visual and auditory streams for activity, sound-event, and spatial audio-visual recognition.Examples span Kinetics, EPIC-Kitchens, AudioSet, VGG-Sound, Ego4D, CREMA-D, VGG-Sound Source, and STARSS23.
  • Multimodal Fusion Strategies and Challenges: Multimodal fusion methods range from feature-level early fusion and decision-level late fusion to intermediate cross-modal or mid-fusion interactions.Cross-modal approaches can use attention mechanisms or co-training at intermediate layers.

3. Dataset and Methodology

The Unwritten Benchmark uses synchronized, ink-free audio-video handwriting recordings to build a scalable multimodal word-inference dataset from live-recorded letter primitives. Its modular synthesis pipeline supports multiple styles, modalities, large word sets, and extension across languages and writing systems.

  • Dataset foundation: The dataset records synchronized audio and video of handwriting without visible ink, targeting abstract reasoning from perceptual cues.The benchmark is designed to test abstract reasoning in AI using multimodal handwriting recordings.
  • Dataset foundation: Three participants supplied live-recorded capital and lowercase letter samples in three distinct, consistent handwriting styles based on standard American and British stroke patterns.The source patterns came from elementary-school alphabet tracing worksheets.
  • Dataset configuration: The benchmark represents three writing styles—Standard, Cursive, and Retrace—and three modalities: Audio, Muted Video, and Audio+Video.These style and modality sets define the benchmark’s experimental dimensions.
  • Word synthesis: Words are generated by sequentially concatenating corresponding live-recorded letter clips, preserving their original style-specific kinematic and auditory properties.The resulting benchmark test set contains 10,491 files.
  • Extensibility: The modular pipeline can generate millions of English samples, reuse Latin-alphabet clips for languages such as French, Spanish, and German, and extend to new scripts by adding character primitives.This design makes the benchmark extensible and language-agnostic.

4. Experimental Setup

The evaluation tested four leading multimodal models zero-shot on a 300-file corpus, using human participants as a practical upper bound. Because full-word accuracy was near-zero, Ordered Letter Accuracy was adopted to measure partial, position-specific understanding.

  • Models: Four leading MLLMs—GPT-4o, Gemini 2.5 Pro, Gemini 2.5 Flash, and Qwen2.5-Omni-7B—were evaluated in a zero-shot setting.The setup targeted intrinsic, non-task-specific reasoning.
  • Implementation: GPT-4o lacked Audio+Video evaluation because its API does not accept premerged AV files, while Qwen2.5-Omni-7B used fallback frame rates from 30 fps to 10 fps.The fallback addressed occasional failures processing longer videos at 30 fps.
  • Prompting: A length constraint populated n with each sample’s correct word length after open-ended prompting caused mode collapse toward common short words.Examples of default responses included “the” and “cat”.
  • Evaluation Metric: Ordered Letter Accuracy measures the percentage of predicted characters matching ground-truth characters at the same positions, providing a granular alternative to binary full-word accuracy.For “cat” versus “bot”, OLA is 1/3 ≈0.33.
  • Evaluation Corpus: 300 total files were randomly sampled from generated word lists with balanced distributions across writing styles and word lengths.Human and AI participants inferred the written English word or letter from multimodal cues.
  • Human Baseline: 20 adult native or fluent English speakers were evaluated on aligned Audio, Muted Video, and Audio+Video trials to establish a human performance gold standard.The study was hosted on Amazon Mechanical Turk; each participant completed 15 trials.

5. Results and Analysis

Current multimodal models perform poorly on acousto-kinematic word inference, while humans achieve high accuracy, especially from muted video. Model fusion often fails or harms performance, reflecting distinct low-level failure modes and trends that persist on live-recorded data.

  • Overall Performance: Below 10%, current state-of-the-art models show categorical failure across modalities; Gemini 2.5-Pro peaks at 10.04% on audio-only input.The reported peak remains below 11% on every single task.
  • Human Baseline: 80.78% on Muted Video and 77.01% on Audio+Video, humans substantially outperform models and demonstrate that video provides rich, solvable information.Humans’ performance contrasts with models’ inability to exploit kinematic information in video.
  • Multimodal Fusion: 3.95% Audio+Video accuracy for Qwen2.5-Omni is below its audio-only 7.83% and video-only 5.03%, while Gemini gains no meaningful benefit from fusion.Gemini 2.5 Pro and 2.5 Flash score 9.07% and 8.44% on Audio+Video, respectively.
  • Multimodal Fusion: 80.78% on Muted Video contrasts with humans’ 19.47% audio-only accuracy, while ambiguous Cursive performance drops from 65.57% to 57.22% when audio is added.The results indicate context-dependent human modality use that is absent from the models.
  • Handwriting Styles: 84.56% and 93.94% Muted Video accuracy on Standard and Retrace styles falls to 65.57% on Cursive for humans, whereas models remain universally low but peak on Retrace.Gemini 2.5 Pro’s audio-only score rises from 10.04% on Standard to 11.69% on Retrace, suggesting reliance on repetitive low-level signals.
  • Validation Study: 8.85% versus 8.80% for GPT-4o Muted Video and 8.44% versus 8.06% for Gemini 2.5 Flash Audio+Video show that live-recorded validation preserves the main benchmark’s trends.The live set also retains Qwen2.5-Omni’s Audio-to-Audio+Video drop from 6.50% to 3.76%, paralleling the synthetic decline from 7.83% to 3.95%.

6. Discussion

The benchmark exposes systemic architectural and conceptual limitations in current multimodal models, rather than merely a data-domain problem. Their failures reflect weak temporal-motion representations and fusion mechanisms that do not model the shared causal event across modalities, despite the task being heuristically solvable.

  • State-of-the-art MLLMs consistently and profoundly fail on The Unwritten Benchmark, indicating systemic limitations beyond the data domain.The discussion characterizes the limitation as architectural and conceptual.
  • Models appear to process video as a bag of frames or sparse keyframes instead of continuously representing pen-tip flow, velocity, and acceleration.The paper argues that future architectures must move beyond static-frame analysis toward spatio-temporal understanding.
  • The paradoxical fusion effect reveals brittle mechanisms that treat pen-scratch audio and pen-movement video as conflicting rather than complementary manifestations of one pen stroke.The modalities share an underlying causal event, but models fail to use one to disambiguate the other.
  • The task is not computationally impossible: a MediaPipe landmark-extraction pipeline combined with CTC could plausibly map motion trajectories to character sequences.The proposed heuristic pipeline extracts pen or finger coordinates before modeling character sequences with Connectionist Temporal Classification.

7. Conclusion

The Unwritten Benchmark evaluates abstract perceptual reasoning in multimodal AI and exposes a profound gap between current models and human intuition. It offers a concrete tool for measuring progress toward systems that understand physical and causal processes.

  • Benchmark contribution: The Unwritten Benchmark is a novel dataset and task for evaluating abstract perceptual reasoning in multimodal AI.The paper introduces it as a concrete benchmark for a critical, currently unsolved problem.
  • Key finding: State-of-the-art MLLMs categorically fail at the task, revealing a profound gap between their capabilities and human intuition.The evaluation also identifies a paradoxical fusion effect that challenges common assumptions in multimodal learning.
  • Research direction: The benchmark is intended to catalyze a shift from recognizing the world as it is toward understanding the physical and causal processes that shape it.The authors position this direction as essential for building more robust, general, and truly intelligent systems.

Supplementary Material · A. Appendix · A.1. Dataset Creation: Detailed Methodology

The synthetic WORD DATASET was created in two stages: compiling common English words and programmatically synthesizing multimodal samples from live-recorded letter primitives.

  • A.1. Dataset Creation: Detailed Methodology: The dataset creation process began by generating a clean list of common English words.
  • A.1. Dataset Creation: Detailed Methodology: The second stage programmatically synthesized multimodal samples from live-recorded letter primitives.

A.1.1. Common Word List Generation

The benchmark uses a high-quality common-word list generated from frequent English words and refined through a strict filtering pipeline. The process draws on wordfreq-based lists and applies length, character, stoplist, and per-style coverage rules.

  • Common Word List Generation: The word list began with high-frequency common English words and was subjected to strict filtering to protect benchmark integrity.The initial list was designed to provide a clean, reliable vocabulary for evaluation.
  • Common Word List Generation: The seed vocabulary came from wordfreq’s top English word lists, which are based on large text corpora.The procedure can also be reproduced with similar open-source frequency lists, provided the filtering pipeline is retained.
  • Common Word List Generation: The sequential filters retained 2–5-character words containing only lowercase ASCII letters while removing non-alphabetic forms and applying per-style coverage exclusions.The stated exclusions include digits, hyphens, apostrophes, and other non-alphabetic characters.

A.1.2. Word Synthesis Pipeline … A.5. Ethics and Reproducibility Statements

The appendix describes how multimodal word samples were synthesized and quality-checked, how humans and models were evaluated, and how prompting was refined to avoid degenerate outputs. It also documents handwriting guidance, participant protections, and reproducibility provisions.

  • A.1.2. Word Synthesis Pipeline: Words were synthesized by mapping each word to constituent letter clips and concatenating them into a final multimodal sample.Styles were not mixed, and synthesis proceeded only when all three modalities existed for every letter.
  • A.1.2. Word Synthesis Pipeline: Letter clips were concatenated back-to-back using lossless ffmpeg video stitching and raw bytewise audio concatenation.No artificial spacing was introduced between letters, and synthesized samples were manually checked for synchronization, artifacts, corruption, and modality misalignment.
  • A.2. Human Evaluation: Additional Details: Human evaluation established a performance ceiling and collected qualitative evidence about how participants solved the task.Participants frequently mimicked writing motions with their fingers, suggesting active embodied simulation during perception.
  • A.3. Prompting Strategy: Prompt engineering added the correct word-length constraint after open-ended prompts caused mode collapse toward common, short words.The constraint restricted output length without providing semantic information about the answer.
  • A.3. Prompting Strategy: Final audio and video prompts required one lowercase English word of length n, included a style note, and instructed models to guess when unsure.The prompts were finalized through iterative engineering intended to reduce ambiguity and common failure modes.
  • V2: Initial Open-Ended Prompt with Answer length limit - Discarded: Discarded prompt variants produced apologies, defaulted to short high-frequency words, or retained substantial mode collapse despite style information.The inclusion of {n} was identified as the most critical factor for eliciting meaningful responses.
  • A.4. Handwriting Style Guidance: Participants used elementary handwriting worksheets with stroke-order diagrams and dotted outlines to guide American Standard and British Cursive data collection.The appendix also documents American Standard and British Capital and Small cursive stroke patterns.
  • A.5. Ethics and Reproducibility Statements: All data came from consenting, informed adults, was fully anonymized, contained no sensitive content, and was publicly released with model versions and complete prompts.These provisions support the stated ethics and reproducibility goals.
Loading 2608.14558v1…