PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Evaluation of Russian-language Architectures

Artem Chervyakov, Ulyana Isaeva, Anton Emelyanov, Artem Safin, Maria Tikhonova, Alexander Kharitonov, Yulia Lyakh, Petr Surovtsev, Denis Shevelev, Vildan Saburov, Vasily Konovalov, Elisei Rykov, Ivan Sviridov, Amina Miftakhova, Ilseyar Alimova, Alexander Panchenko, Alexander Kapitanov, Alena Fenogenova

arXiv:2511.15552v3cs.CLcs.AIcs.CV

TL;DR

Russian-specific benchmarks had focused on text, while multimodal benchmarks were largely English-centric. MERA Multi introduces an 18-task Russian evaluation framework spanning text, image, audio, and video. Its leaderboard results favor broader modality coverage under the benchmark’s scoring metrics.

  • Problem

    Russian-specific benchmarks focused on text, leaving a gap in multimodal evaluation attentive to Russian linguistic and cultural context.

    Source: §1 · p. 1

  • Method

    MERA Multi combines 18 tasks across text, image, audio, and video with a unified ability taxonomy, standardized prompts, and complementary evaluation components.

    Source: §1 · p. 1, §3.3 · p. 4, §3.3.2 · p. 4

  • Results

    Broader modality coverage leads to a higher Total Score under the leaderboard metrics; Qwen3-Omni-30B-A3B-Instruct leads with a Total Score of 0.5.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The benchmark indicates that audio and video understanding are less represented in models and datasets than image understanding, as reflected in lower scores on benchmark tasks.

    Source: §5 · p. 8

    The 18-task benchmark may underrepresent abilities or domains, so performance on it may not predict performance on specialized tasks or domains.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are currently at the center of research attention, showing rapid progress in scale and capabilities, yet their intelligence, limitations, and risks remain insufficiently understood. To address these issues, particularly in the context of the Russian language, where no multimodal benchmarks currently exist, we introduce MERA Multi, an open multimodal evaluation framework for Russian-spoken architectures. The benchmark is instruction-based and encompasses default text, image, audio, and video modalities, comprising 18 newly constructed evaluation tasks for both general-purpose models and modality-specific architectures (imageto-text, video-to-text, and audio-to-text). Our contributions include: (i) a universal taxonomy of multimodal abilities; (ii) 18 datasets created entirely from scratch with attention to Russian cultural and linguistic specificity, unified prompts, and metrics; (iii) baseline results for both closed-source and open-source models; (iv) a methodology for preventing benchmark leakage, including watermarking for private sets. While our current focus is on Russian, the proposed benchmark provides a replicable methodology for constructing multimodal benchmarks in typologically diverse languages, particularly within the Slavic language family.

1 Introduction

Russian-specific benchmarks had remained text-only, leaving multimodal evaluation unaddressed. MERA Multi introduces an 18-task, culturally and linguistically grounded Russian benchmark and presents a methodology intended to extend to other languages.

1 Introduction

  • MERA Multi is introduced as the first Russian multimodal benchmark, with 18 tasks spanning default text, image, audio, and video under a unified taxonomy.Source: §1 · p. 1
  • The authors position the benchmark methodology as a blueprint for Slavic and other morphologically rich languages where direct adaptation of English-centric benchmarks is inadequate.Source: §1 · p. 1
  • The datasets incorporate Russian cultural and linguistic specificities and use unified prompts and metrics.Source: §1 · pp. 1–2
  • The work provides baseline results for open- and closed-source models and a leakage analysis and watermarking strategy for private evaluation datasets.Source: §1 · p. 2
  • A standardized codebase, automated-scoring submission platform, and public leaderboard support reproducible evaluation.Source: §1 · p. 2

2 Related Work

Prior work established text-based and multimodal evaluation benchmarks, but Russian-specific benchmarks remained text-only and existing multimodal benchmarks were largely English-centric. MERA Multi extends Russian instruction-following assessment to multiple modalities.

2 Related Work

  • English text benchmarks progressed from NLU standards such as GLUE and SuperGLUE to instruction- and reasoning-focused benchmarks such as BIG-bench and HELM, while Russian resources remained text-based.Source: §2 · p. 2
  • Multimodal benchmarks span general-purpose, visual, audio, and video evaluation, but most existing benchmarks are English-centric.Examples include MultiBench (Liang et al., 2021), MMBench (Liu et al., 2024e), SUPERB (Yang et al., 2021), and Video-MME (Fu et al., 2025).Source: §2 · p. 2
  • Russian instruction-based evaluation had remained text-only, and the authors report no existing multimodal benchmark for Russian; MERA Multi addresses this gap with unified, culturally adapted evaluation across modalities.Source: §2 · pp. 2–3

3 Overview of MERA Multi Benchmark

MERA Multi organizes Russian multimodal evaluation around a unified skill taxonomy and a shared methodology for tasks spanning text, image, audio, and video.

3.1 Benchmark General Structure

  • The benchmark covers 11 image, 4 audio, and 3 video datasets, using multiple-choice and short-answer formats and combining 7 public tasks with 11 private tasks.Private tasks were built from scratch to reflect Russian cultural nuances, cover underexplored skills, and mitigate contamination.Source: §3.1 · p. 3

3.2 Skill Taxonomy

  • Its extendable taxonomy synthesizes multimodal skills across knowledge, perception, and reasoning, while mapping each task to combinations of abilities.A task may jointly test visual perception, OCR, and reasoning.Source: §3.2 · pp. 3–4

3.3.1 Evaluation Framework Implementation

  • The evaluation framework extends lm-eval for generative multimodal inputs and combines exact string matching with semantic judging, using task-specific metrics where needed.ExactMatch assesses correctness and format adherence; JudgeScore assesses semantic agreement.Source: §3.3.1 · p. 4

3.3.2 Prompt Structure

  • Each dataset uses 10 prompt variants assigned uniformly across samples; statistically significant score shifts across prompts support the multi-prompt design against evaluation bias.Prompt layouts vary surface form and may include or omit a reasoning request.Source: §3.3.2 · pp. 4–5

3.3.3 Judge Scoring

  • The RuModern-BERT judge achieves F1 score 0.96 on a held-out test set and 99.6% agreement with ExactMatch on identical answers.It is trained on human-annotated outputs from Russian-language instruction-based benchmarks.Source: §3.3.3 · p. 5

3.3.4 Evaluation Pipeline

  • Equal-per-modality weighting separates attempted-task quality from benchmark coverage, and their product enables joint ranking without imputing zeros for missing modalities.The quality score remains invariant when tasks are added or removed, while coverage reflects breadth.Source: §3.3.4 · p. 5

3.3.5 Submission

  • Submissions are automatically scored and kept private unless publication is requested, after which experts verify reproducibility before leaderboard posting.Users can choose which modality-specific or multi leaderboard receives an approved result.Source: §3.3.5 · p. 6

3.4 Data Protection

  • Private-set safeguards combine modality-specific watermarking with multimodal leakage detection; watermarking changes Judge Score by less than 5% with 95% probability, usually less than 1%.MSMIA compares model loss patterns on original examples and semantically perturbed neighbors to estimate whether samples were in training data.Source: §3.4 · p. 6, §3.4.1 · p. 6, §3.4.2 · pp. 6–7

4 Baselines

The baseline evaluation spans publicly available multimodal models and GPT 4.1, alongside crowd and expert human baselines across MERA Multi tasks.

4.1 Model Baselines

  • The study evaluates over 50 public multimodal models ranging from 1B to 110B parameters and includes proprietary GPT 4.1 for comparison.Models are evaluated under the same environments and procedures.Source: §4.1 · p. 7

4.2 Human Baselines

  • Human baselines aggregate crowd annotator results, with expert baselines for tasks requiring additional domain expertise.Honeypot checks and post-hoc filtering screen annotation quality; the collection platform provides anonymity, fair compensation, and ethical compliance.Source: §4.2 · p. 7

5 Results

Qwen3-Omni leads the overall leaderboard, while results also show modality-specific differences and a gap between semantic correctness and exact-format compliance. The reported takeaways note uneven modality representation and robustness to missing task scores.

5 Results

  • 0.5 Total Score puts Qwen3-Omni-30B-A3B-Instruct first, with 0.563 Attempted Score and 0.889 Coverage; GPT 4.1 scores 0.159 with 0.333 Coverage.The leaderboard favors broader coverage, though Qwen2.5-VL-72B-Instruct and Qwen2-VL-72B-Instruct reach 0.302 and 0.254 Total Scores, respectively.Source: §5 · p. 7, Table 6 · p. 8
  • 0.474 Audio Total Score for Qwen2.5-Omni-7B exceeds ultravox specialists’ 0.311, while Qwen2-VL-72B-Instruct leads Video Total Score at 0.625.The passage reports that video-specialist LLaVA-NeXT-Video models show poorer metrics.Source: §5 · p. 7
  • Judge Score (JS) exceeds EM across models, indicating semantically correct answers can still violate the requested output format.When JS is below EM, an answer may be extractable even though the full generation is misleading.Source: §5 · pp. 7–8
  • The reported takeaway is that image understanding is relatively mature, while audio and video are underrepresented in models and datasets and score lower on benchmark tasks.Omni-models partially bridge the gap between modalities.Source: §5 · p. 8
  • The authors report that overall metrics are robust to missing task scores and multiple modalities.The passage frames this as a benchmark takeaway, without supplying a numerical robustness estimate here.Source: §5 · p. 8

Conclusion

The conclusion presents MERA Multi as a Russian multimodal evaluation framework spanning 18 tasks and four modalities, with reproducibility and automated evaluation support. It also envisions expanding the benchmark and using its methodology for culturally aware benchmarks in other languages.

Conclusion

  • MERA Multi is a framework for transparent, culturally grounded Russian multimodal evaluation, covering 18 tasks across default text, image, audio, and video.It systematizes multimodal abilities with a proposed taxonomy and evaluates tasks using methodologically verified prompts and metrics.Source: Conclusion · p. 8
  • A standardized codebase and submission platform provide reproducibility, automated evaluation and scoring, and open leaderboards.The conclusion describes the codebase as standardized and the platform as offering automated evaluation.Source: Conclusion · p. 8
  • The authors plan to add scenarios and encourage community contributions, envisioning a methodological foundation for culturally aware multimodal benchmarks across non-English languages such as Slavic.They describe MERA Multi as a collaborative initiative promoting transparent evaluation practices.Source: Conclusion · p. 8

Limitations

The authors identify limits in benchmark coverage and reproducibility: the task set may omit important abilities or domains, and scores can vary with the inference stack. They request hardware and library details in submissions.

Limitations

  • The 18-task benchmark may underrepresent abilities or domains that matter for particular applications.The authors caution that benchmark excellence does not rule out poor performance on a specialized domain or task.Source: Limitations · p. 9
  • Even with fixed prompts and decoding settings, scores can vary with hardware and software choices, including GPU, libraries, precision, and batching.The authors note that some components are nondeterministic.Source: Limitations · p. 9
  • For reproducibility, the authors ask public submissions to use the same parameters and specify the GPUs and library versions used.The requested details include hardware and software versions.Source: Limitations · p. 9

Ethical Statement

The section describes the scope, goals, design constraints, construction, and validation of Russian-language audio, video, and visual datasets, including potential data-leakage limitations. It also notes that evaluating model bias was outside the benchmark’s scope.

Ethical Statement

  • The benchmark evaluates multimodal semantic understanding but does not explicitly test model bias toward underrepresented minorities, leaving this important direction for future work.The authors state that annotators were adequately compensated; researchers are encouraged to follow ethical practices and undergo regular ethical review, and writing tools were used only for English-language editing under ACL policies.Source: Ethical Statement · p. 9

A.1 AQUARIA

  • AQUARIA evaluates Russian audio-language models across nine question types and the perception, knowledge, and reasoning skills needed to analyze speech, non-verbal sounds, and music.Its scenarios combine dialogue, background sounds, and music; recordings used professional studio equipment, and some music tracks were generated with models.Source: §A.1 · p. 15

A.2 CommonVideoQA

  • CommonVideoQA is a public Russian-language video question-answering dataset testing comprehension, detail recognition, knowledge, action order, counting, and temporal localization without requiring audio understanding.Its clips come from open repositories, including EPIC-KITCHENS and Kinetics, which should be considered when interpreting evaluations.Source: §A.2 · pp. 15–16

A.3 LabTabVQA

  • LabTabVQA evaluates Russian-language interpretation of medical tables in images, including their structure, content, formatting, and data relationships, with a methodology extensible beyond medicine.The dataset uses 339 selected images from real telemedicine consultations; AI-generated questions were validated by assessors, but its generation methodology may bias tasks toward the GPT-o4 model family.Source: §A.3 · pp. 16–17

A.4 RealVideoQA

  • RealVideoQA is a closed Russian-language video question-answering dataset testing comprehension, detail recognition, knowledge, action sequences, counting, and temporal intervals using clips absent from public sources.The crowdsourced clips do not require audio comprehension, a limitation to consider when interpreting evaluation results.Source: §A.4 · pp. 17–18

A.5 RealVQA

  • RealVQA tests Russian visual question answering that requires image-grounded reasoning, including responses to distractor questions that probe resistance to hallucination.Its everyday-image questions target causal and object relationships without requiring expert knowledge; trick questions about absent objects make up 10% of the dataset.Source: §A.5 · pp. 18–19

A.6 ruCLEVR

  • RuCLEVR adapts CLEVR for Russian with automatically generated 3D-object scenes and templated question families that test controlled visual reasoning about attributes, comparisons, existence, counting, and integer comparison.The dataset was created from scratch to reduce bias and focus evaluation on reasoning about object attributes and relationships rather than pattern recognition.Source: §A.6 · p. 19
  • ruCLEVR questions and answers were generated from templates for automatically created 3D scenes, translated from English into Russian, filtered for linguistic acceptability, and checked for duplicates.The scenes varied object attributes and configurations; color-replacement augmentation was also applied to validation images, with Russian questions checked for grammatical correctness.Source: §A.6 · p. 20

A.7 ruCommonVQA

  • ruCommonVQA uses images from public datasets, which may create indirect training-data leakage, alongside a second component of consented, crowdsourced images required to be unique and unavailable online.The dataset covers simple and complex questions about real-world photos and abstract illustrations; simple questions rely on visual perception, while complex questions require reasoning.Source: §A.7 · pp. 20–21

A.8 ruEnvAQA

  • ruEnvAQA evaluates Russian-language understanding of music and non-verbal audio using translated and partly modified questions from English datasets, excluding speech-understanding questions.The original recordings were retained with length trimming; questions span literal audio-event perception, audio-scene inference, and reasoning with general knowledge.Source: §A.8 · pp. 21–22
  • ruEnvAQA questions underwent overlapping crowd validation, audio trimming to 5–20 seconds, and Dawid–Skene aggregation; only questions with one selected valid answer were retained.Annotators excluded questions with multiple correct options, and the longest audio fragment in each retained group was selected.Source: §A.8 · p. 23

A.9 ruHHH-Image

This section describes multimodal evaluation datasets spanning ethical VQA, math and science, spoken-language understanding, audio dialogue, and visual commonsense. They cover dialogue and educational contexts, with task-specific answer formats and constraints.

A.9 ruHHH-Image

  • ruHHH-Image evaluates Russian text-and-image responses by asking models to choose the more ethical or safe answer, extending HHH and ruHHH with three new ethical categories.The six categories are Empathetic, Etiquette, Open/Impartiality, Harmless, Helpful, and Honest; the dataset balances questions across them. It identifies problematic responses but is not suitable for evaluating global or multicultural ethics.Source: §A.9 · pp. 23–24

A.10 ruHHH-Video

  • ruHHH-Video adapts ruHHH-Image’s methodology to video and is presented as the first Russian-specific dataset for evaluating ethical reasoning in videos.Source: §A.10 · p. 24

A.11 ruMathVQA

  • ruMathVQA is a multimodal dataset of school math problems presented as images with annotated questions that elicit unambiguous answers, assessing whether systems can interpret the problems and answer them.It does not assess the solution process or require deriving reasoning; answers are short numbers or formulas, so Accuracy is used.Source: §A.11 · pp. 24–25

A.12 ruNaturalScienceVQA

  • NaturalScienceQA adapts ScienceQA (Lu et al., 2022) into image-dependent school science questions across physics, biology, chemistry, and natural history.Questions combine an image and context with answer choices, and may use relatively low-resolution images to probe visual extraction from poor-quality inputs.Source: §A.12 · p. 25

A.13 ruSLUn

  • RuSLUn evaluates Russian audio models’ end-to-end intent detection and slot filling, aligning recordings directly with semantic annotations without an intermediate ASR step.Its localization reflects Russian usage scenarios, vocabulary, and contexts for voice assistants and speech-driven services.Source: §A.13 · p. 27

A.14 ruTiE-Audio

  • ruTiE-Audio tests whether audio-capable chat models can sustain coherent dialogue across changing topics by choosing among four text answers using prior context.Its challenges include long dialogue histories, Russian cultural and linguistic knowledge, and conversational skills such as irony, jokes, reasoning, and reacting to situational audio prompts.Source: §A.14 · p. 27
  • ruTiE-Audio tests broad conversational abilities, but its context-linked questions must be administered one at a time and in fixed order to prevent answer cueing.Experts wrote scripts and made previously unpublished recordings; background noise came from public datasets and custom studio and field recordings.Source: §A.14 · p. 28

A.15 ruTiE-Image

  • ruTiE-Image tests context-sensitive dialogue across linguistic, cultural, reasoning, and social-cognitive skills using text-and-picture turns.It comprises three dialogues simulating 500 user requests each, with four answer options per task; questions must be presented one at a time in order because later questions may cue earlier answers.Source: §A.15 · pp. 28–29

A.16 SchoolScienceVQA

  • SchoolScienceVQA assesses multimodal reasoning through image-and-text multiple-choice questions across scientific subjects, with explanations of correct answers.Developed from scratch for Russian cultural and educational context, it adapts ScienceQA to the Russian school curriculum and uses novel images made through photography, illustration, computer graphics, or neural generation.Source: §A.16 · pp. 29–30

A.17 UniScienceVQA

  • UniScienceVQA tests image-based expert knowledge across fundamental, social, and applied sciences and cultural and professional fields using short answers or multiple-choice formats.It does not assess reasoning processes or require explanations; Accuracy is the metric, and each of its 25 subdomains contains 200–400 expert-created tasks.Source: §A.17 · p. 30

A.18 WEIRD

  • WEIRD evaluates whether image-capable models detect violations of commonsense by matching visual information with plausibility judgments, using Accuracy.It extends WHOOPS! through iterative synthetic generation of violation categories, descriptions, and images, followed by manual filtering.Source: §A.18 · p. 31

B Skill taxonomy

This section presents a taxonomy of multimodal skills, organizing them under Perception, Reasoning, and Knowledge and detailing reasoning abilities across five taxonomy groups. It also describes a method for detecting whether a model was trained on benchmark data.

B.1 Perception

  • Perception turns image, audio, and video inputs into task-relevant evidence through recognition, coarse localization, and cross-modal alignment.It includes single-instance and cross-instance perception, as well as correspondences between language and media.Source: §B.1 · pp. 31, 33

B.1.1 Fine-grained single-instance perception

  • Single-instance perception covers salient objects or events, including coarse object positions, object and event recognition, motion, body poses, facial expressions, and gestures.Object localization predicts relative image positions, unlike media grounding, which uses coordinates; the listed skills apply to different modality combinations.Source: §B.1.1 · p. 33

B.1.2 Fine-grained cross-instance perception

  • Cross-instance perception covers separating overlapping entities, attributing speech to speakers, recognizing spatial and temporal relations, patterns, and interactions among entities.Its interaction skills include object-object, human-object, and human-human interactions.Source: §B.1.2 · p. 33

B.1.3 Cross-modal alignment

  • Cross-modal alignment links language to media through text extraction or media grounding, separating transcription from localization in image, audio, and video.Text extraction includes OCR, speech and lyrics recognition; media grounding maps descriptions to visual coordinates or temporal spans.Source: §B.1.3 · p. 34

B.2 Knowledge

  • Knowledge covers retrieving and applying stored factual and conceptual information, distinguishing everyday commonsense from broadly accessible subject knowledge and specialized expert knowledge.The taxonomy also places safety- and ethics-aligned judgment in this group as a simplification to limit the number of major categories.Source: §B.2 · pp. 34–35

B.3 Reasoning

  • Reasoning derives conclusions from perceptual evidence and prior knowledge through uncertain, multi-step, relational, causal, quantitative, counterfactual, and holistic inference.Holistic judgments such as topic, style, provenance, or media characteristics count when they integrate multiple cues rather than identify one localized entity.Source: §B.3 · p. 35

B.3.1 Inductive reasoning

  • Inductive reasoning includes holistic attributes such as topic, style, scene, generated-content status, media characteristics, and emotion, alongside entity-grounded physical properties and object function.Coarse attributes characterize a sample without fine localization or multi-entity comparison; object attributes concern entities, including their properties, functions, and identity cues.Source: §B.3.1 · pp. 35, 37
  • Inductive reasoning includes recognizing identity-related cues and affective states from facial expressions or vocal affect.Source: §B.3.1 · p. 37

B.3.2 Deductive reasoning

  • Deductive reasoning covers constraint-driven inference, detecting incongruity or humor, analogical mapping, and general multi-hop or constraint-based deduction.Source: §B.3.2 · p. 37

B.3.3 Abductive reasoning

  • Abductive reasoning includes forming explanations consistent with observations, reasoning about hypothetical interventions, and identifying causes and effects over time.Source: §B.3.3 · p. 37

B.3.4 Quantitative reasoning

  • Quantitative reasoning includes counting entities in a still image, counting events and objects over time, and solving mathematical problems grounded in visual or multimodal contexts.Static and temporal counting complement pattern-recognition skills focused on repeated structures.Source: §B.3.4 · p. 37

B.3.5 Other reasoning

  • Other reasoning skills include evaluating evidence under ambiguity, maintaining correctness under counterfactual interventions, decomposing problems into steps, and comparing instances or times.Source: §B.3.5 · p. 37

C.1 Setup.

  • The leakage experiment estimates whether a target model was trained on a multimodal example by comparing a base model with a version fine-tuned on benchmark data.The leaked version is created with SFT-LoRA on a benchmark subset for the selected modality.Source: C Data leakage details · p. 37, §C.1 · p. 37

C.2 Neighbor Generation and Feature Extraction.

  • For each example, the method creates 24 text neighbors using masking, deletion, duplication, and random-word swapping, leaves the modality data unchanged, and extracts text embeddings.Each of the four perturbation techniques is applied six times; embeddings are extracted for the original text and its neighbors using a fixed encoder.Source: §C.2 · pp. 37–38

C.3 Detector Training

  • The detector learns from 48 labeled training triplets per original example to estimate whether model features come from a model trained on the target data.The features are loss and embedding differences, labeled according to whether they came from the base model or its benchmark-fine-tuned version.Source: §C.3 · p. 38

C.4 Inference and Evaluation

C.4 estimates whether a test model has seen a data point by aggregating detector probabilities across its neighbors and then across dataset samples. It reports AUC-ROC for binary leaked-versus-clean classification.

C.4 Inference and Evaluation

  • A data point’s leakage score averages the detector’s probability across all K neighbors; the dataset score averages those scores across samples.The test model’s loss and embedding differences are computed before the detector is applied.Source: §C.4 · p. 38
  • AUC-ROC is reported for binary leaked-versus-clean classification of evaluated image, audio, and video MLLMs.The cited tables list performance metrics by modality, but their supplied captions do not give score values.Source: §C.4 · p. 38, Table 9 · p. 38
  • The Origin Model trains MSMIA, while the Test Model supplies losses used to test whether a sample appeared in its training data.Source: §C.4 · p. 38

D.1 Data collection

The appendix describes how the judge dataset is collected and trained, selects an encoder-based judge, and deploys it for benchmark scoring.

D.1 Data collection

  • The judge dataset combines benchmark question, gold-answer, and prediction triplets labeled for semantic correctness, with source-dataset-separated splits and synthetic augmentations.Only items with unanimous annotator agreement are used; Table 12 summarizes dataset sizes and task-format distributions.Source: §D.1 · p. 38, Table 12 · p. 39

D.2 Training models

  • The judge is trained as a fully fine-tuned encoder classifier using cross-entropy, class weights, mixed precision, and development-set F1 early stopping.Table 13 summarizes the training configuration; optimization is constrained to one A100 80GB GPU.Source: §D.2 · p. 39, Table 13 · p. 39

D.3 Model selection

  • Encoder models were more suitable than zero-shot decoder baselines, providing deterministic class probabilities and practical advantages in size, speed, and long-context handling.Unconstrained decoder generations may fail to produce valid classifications, making scores ambiguous; Table 14 summarizes model comparisons.Source: §D.3 · p. 39

D.4 Model deployment

  • The selected judge is served with vLLM on one A100 40GB GPU and exposed through a score(q, ref, pred) → {0,1} API.The trained model weights are publicly provided on HuggingFace Hub.Source: §D.4 · p. 39

D.5 Model analysis

The analysis reports the judge’s agreement with human labels and describes prompt construction through fixed and task-specific blocks.

D.5 Model analysis

  • Judge errors primarily involve complex LaTeX formatting or multi-step reasoning, where a binary classifier without rationale generation struggles with ambiguous justifications and symbolic intricacies.Source: §D.5 · p. 40

E.1 Prompts formulations

  • Prompt formulations use 13 blocks, with fixed formulations and task-specific task-description and task-context blocks.Five prompts include reasoning blocks, while one “zero prompt” omits answer formatting; block combinations are assembled by script.Source: §E.1 · p. 40

E.2 Statistical analysis

The analysis uses an OLS specification to test prompt-formulation effects on evaluation metrics. Prompt sensitivity varies by dataset, supporting the use of multiple formulations rather than assuming one prompt works uniformly.

E.2 Statistical analysis

  • The OLS analysis tests each of ten prompt formulations against zero effect, treating p-values below 0.05 as statistically significant.The specification accounts for model and inference engine, with frames for video and domain for selected datasets.Source: §E.2 · p. 41
  • RuSLUn, ruTiE-Audio, and ruTiE-Image are omitted from the prompt-effect figure because of strict scoring or dataset design that confounds prompt effects.For the two ruTiE datasets, answers to later questions appear in earlier question text, making prompt effects difficult to separate reliably.Source: §E.2 · p. 41
  • 12 of 16 datasets show prompt sensitivity, supporting uniform prompt distribution and averaging metrics across formulations rather than relying on one prompt.No single formulation dominates; a prompt that helps on one dataset can hurt on another.Source: §E.2 · p. 43
  • Prompt effects range from metric shifts of 0.1–0.2 to near zero, with magnitude depending on the dataset.Some near-zero effects are still statistically significant when variance is low.Source: §E.2 · p. 43
  • Prompts requesting chain-of-thought rationales do not always lower scores, while some models fail to follow the prescribed answer format.The authors report that the LLM-as-judge can cope with reasoning-formatted responses.Source: §E.2 · p. 43

F.1 Model Baselines Details

Baseline evaluations show different strengths and weaknesses across image, audio, and video tasks. The paper also describes how human-baseline responses were collected, filtered, and supplemented with expert annotations for expert tasks.

F.1 Model Baselines Details

  • The baseline results cover image, audio, and video models in Tables 18–21, with task-specific metrics including Exact Match and Judge Score.The listed model baselines appear in Tables 15–17.Source: §F.1 · p. 43, Table 18 · p. 46, Table 19 · p. 46
  • Image models are strongest on natural-image semantics and basic spatial reasoning, but weaker on OCR, diagram and table parsing, and structured compositional reasoning.On math tasks, Exact Match lags Judge Score, indicating partially correct solutions that miss the required final answer.Source: §F.1 · p. 43
  • Audio models show relative strength in acoustic-scene understanding and temporal or comparative environmental-audio reasoning, with weaknesses in speech recognition, diarization, and answer-format fidelity.The authors suggest ASR robustness, speaker attribution, and constrained decoding or answer templates as improvement targets.Source: §F.1 · p. 43
  • Video models handle scene and object recognition and short events relatively well, but struggle with temporal localization, action sequences, counting, causal reasoning, and social context.The authors identify temporal grounding and answer-format control as practical improvement targets.Source: §F.1 · p. 43

F.2 Human Baseline Details

  • Human baselines use aggregate annotator responses after filtering workers with repeated errors and disagreement with the majority; expert tasks also receive expert annotations.Expert responses are aggregated with overlap of three; the experts had not participated in dataset creation, and the paper notes a limited pool of niche-domain experts.Source: §F.2 · pp. 43–44
Loading 2511.15552v3…