A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation

Hodong Lee; Sanghee Park; Dohoon Ryu; Jungwhan Kim; Junyeob Kim; Soyoon Kim; Geewook Kim

Source-linked AI summary

A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation

Hodong Lee, Sanghee Park, Dohoon Ryu, Jungwhan Kim, Junyeob Kim, Soyoon Kim, Geewook Kim

arXiv:2609.01315v1Sep 1, 2026cs.AI

TL;DR

Omni-modal evaluation remains fragmented across modality-specific toolchains with incompatible configurations and difficult cross-run comparisons. OmniEvaluator composes existing engines and frameworks through a shared schema, reproducible artifacts, a dashboard, and a CPU-capable verifier. The system was used in developing HyperCLOVA X 8B Omni and is publicly available, while the verifier reaches 85.0 accuracy and matches or exceeds the cited cost-efficient proprietary judges.

Problem
Existing evaluation tools cover individual modalities but lack a consistent setup for evaluating one omni-modal model across text, image, video, and audio.
Method
OmniEvaluator connects existing inference engines and evaluation frameworks through a unified intermediate schema, records complete run artifacts, and provides integrated visualization and verification.
Results
OmniEvaluator was used in developing HyperCLOVA X 8B Omni; its verifier reaches 85.0 accuracy, matching or exceeding GPT-5.4-mini 82.8 and Claude-Haiku-4.5 84.3.
Takeaways & Limitations
The system provides one workflow for cross-modal evaluation, reproducibility, dashboard comparison, federated inference, and lower-cost semantic verification.
Takeaways & Limitations
The verifier judges only textual triples and returns a binary verdict, so it cannot assess aspects beyond that representation.

Abstract

from arXiv · show

Building an omni-modal foundation model means evaluating it across text, image, video, and audio. Excellent evaluation toolkits exist for each modality, but their inference engines, prompt conventions, and metric implementations are mutually incompatible, so practitioners end up maintaining separate environments for every toolchain and still struggle to compare results across them. OmniEvaluator grew out of this need in our own model development: rather than reimplementing benchmarks, it connects existing inference engines and curated evaluation libraries at a higher level, exposing four inference backends, four evaluation frameworks, and over a thousand benchmarks through a single interface. Every run is recorded as an artifact capturing the full configuration for exact reproduction, and results flow into a shared dashboard for cross-model comparison. A federated mode shares GPU inference servers across concurrent evaluations, and a built-in verifier, small enough to run on CPU, keeps its score stable across engines and prompts where rule-based scoring fluctuates under configuration mismatch, matching cost-efficient commercial LLM judges without their recurring API cost. The system, demo video, and dashboard are publicly available. (https://github.com/naver-ai/omni-evaluator)

1 Introduction

OmniEvaluator addresses fragmented omni-modal evaluation by composing existing engines and frameworks through a shared schema, while recording reproducible artifacts and consolidating results for comparison.

Motivation: Omni-modal models require evaluation across text, image, video, and audio, but existing toolchains use separate prompts, preprocessing, scoring code, interfaces, and environments.These differences leave difficult-to-set-up modalities unmeasured and make scores difficult to compare across papers or runs.
System contribution: The schema reduces direct integration from N × M pairwise connections to N + M thin adapters, enabling modular combinations of engines and frameworks.Each engine and framework translates once to the common representation, after which any engine can pair with any framework.
Reproducibility: Every evaluation run produces a provenance-rich artifact containing configuration and outputs, supporting exact reproduction and inspection of decisions affecting scores.Recorded fields include prompts, generation parameters, model revisions, benchmark versions, metric settings, raw predictions, postprocessed results, and final scores.
System contribution: A unified intermediate schema connects four inference backends and four evaluation frameworks to over a thousand benchmarks across text, image, video, and audio.Any supported backend can run any framework’s benchmarks, while federated evaluation shares inference servers across concurrent evaluations.
Analysis and comparison: An integrated dashboard gathers per-modality results, exposes coverage gaps, and supports cross-model and cross-checkpoint comparison for model selection.The dashboard is part of the system’s shared workflow for comparing evaluation results.
Verification: A CPU-capable built-in verifier judges semantic correctness and varies less across engines and prompts than rule-based scoring under configuration mismatch.Its normalized score reduces reliance on costly API judges while preserving a common scoring view.
Availability: OmniEvaluator was used during development of HyperCLOVA X 8B Omni and is publicly available with a live demo, demo video, and evaluation dashboard.The paper provides the system and associated demonstration resources for public use.

2 Related Work

Prior evaluation tools provide strong modality-specific coverage, but omni-modal models still require combining several systems whose prompts, parsing, and scoring conventions differ. OmniEvaluator addresses this gap with a shared schema, reproducible artifacts, and a verifier whose score remains steadier across configurations.

Gap addressed by this work: Evaluation tools have matured within individual modalities, but no established setup evaluates one model across all supported modalities consistently.This is the central gap motivating a unified omni-modal evaluation system.
Text evaluation: Text evaluation frameworks broaden coverage and centralize measurement, while Evalverse unifies several text frameworks but does not extend beyond text.These systems establish precedents for holistic and centralized evaluation within text.
Vision evaluation: Vision-language toolkits cover many models and benchmarks, but prompts, answer parsing, and scoring still differ across frameworks.Benchmark standardization does not eliminate framework-level execution differences.
Multimodal and audio evaluation: Audio and multimodal tools each cover parts of the modality space, so evaluating an omni-modal model still requires combining several toolkits.Examples include dedicated audio toolkits alongside broader multimodal evaluation systems and benchmarks.
Verification: Existing verifiers decide whether a prediction matches a reference, whereas LLM-as-a-judge often rates response quality without requiring a reference answer.OmniEvaluator incorporates verification as infrastructure for evaluation rather than as a standalone reasoning or reward application.
Gap addressed by this work: OmniEvaluator responds to score fragility by making heterogeneous evaluators interoperable, recording reproducible artifacts, and stabilizing verifier scores across engine or prompt configurations.The related-work discussion identifies prompt, example, answer-order, and cross-framework changes as sources of substantial evaluation variation.

3 OmniEvaluator

OmniEvaluator unifies heterogeneous evaluation components through a shared intermediate schema and records each run as a self-contained reproducibility artifact.

OmniEvaluator centers prompt templates, generation settings, model revisions, benchmark versions, and metric parameters in one inspectable configuration.
Composable architecture: Its four independent stages—data iteration, inference, postprocessing, and metric computation—exchange structured records through a unified intermediate schema.Records contain benchmark samples, raw predictions, postprocessed outputs, and metric scores.
Composable architecture: Thin adapters connect inference engines and evaluation frameworks to the common representation, reducing integration cost from O(N ×M) to O(N +M).
Reproducible specification: Every run emits an artifact containing configuration, intermediate outputs, and final scores, enabling exact reruns and inspection of score-generating decisions.The dashboard automatically ingests these artifacts for cross-experiment comparison.
Evaluation modes: Local CLI and remote server modes both produce provenance-rich artifacts and stream completed results to the dashboard.Remote evaluation uses a persistent server and requires no local installation.

4 Supported Features

OmniEvaluator combines an integrated comparison dashboard, a CPU-capable verifier, and federated inference to make heterogeneous omni-modal evaluation more comparable and efficient.

Integrated dashboard: The dashboard integrates results across experiments and modalities, overlays models on selected benchmark axes, and exposes untested modality coverage.It also supports side-by-side sample inspection and reports a mean verifier score on a common [0, 100] scale.
Verifier: Up to 88 points of variation can arise when identical model outputs are scored under different prompt and metric configurations.Prompt templates constrain output shape, while metric implementations are written for that shape.
Verifier: OmniEval Verifier reads a question, reference, and prediction, then returns a rationale and binary correctness verdict without external API calls.It is distributed as an 8-bit Q8 GGUF model and runs on CPU-only machines via llama.cpp.
Verifier: The verifier score remains comparatively stable under mismatched engines and prompts, while native metrics can move by 40 to 88 points on GQA and POPE.Under benchmark-specific prompts, the verifier and native metric largely agree.
Verifier: 85.0 accuracy on the human-verified held-out split matches or exceeds GPT-5.4-mini at 82.8 and Claude-Haiku-4.5 at 84.3.The comparison uses identical text-only configurations.
Federated evaluation: 1.3–2.8× wall-time speedup over conventional per-process evaluation results from pooled GPU utilization and continuous in-flight batching.The largest gains occur on image benchmarks.

5 Conclusion

OmniEvaluator packages existing engines and frameworks into one reproducible workflow, with artifacts feeding a shared dashboard and verifier scores supporting cross-configuration comparison.

Existing engines and frameworks compose through one schema, every run produces a reproducible artifact, and results flow into one dashboard.
Cross-framework comparisons reveal that identically named benchmarks diverge across engines and prompts, whereas verifier scores move far less on the same predictions.
The system, verifier, live demo, dashboard, and demo video are publicly released.

Limitations

The system's verifier and evaluator scope impose important boundaries on what OmniEvaluator can guarantee. It inherits upstream framework bugs, handles only textual correctness, and targets omni-modal understanding rather than multimodal generation.

Upstream evaluator bugs can flow into OmniEvaluator's reported scores.Version-pinned artifacts and cross-engine comparisons make regressions and disagreements visible, but do not prevent inherited bugs.
The verifier returns binary judgments for textual triples, so it cannot assess visual grounding, audio quality, or generation quality.Tasks requiring these criteria should use their corresponding native metrics.
The paper covers omni-modal understanding with text predictions, not multimodal outputs such as image or speech generation.

A Intermediate Schema

OmniEvaluator uses one Record representation for every benchmark, while allowing modality-specific data fields. This shared shape lets engines and framework adapters exchange standardized predictions and postprocessed outputs.

Every benchmark is represented by the same Record object regardless of modality.
The output field contains a modality-specific sub-output, including raw and postprocessed predictions for text-producing benchmarks.
A top-level reasoning_content field stores separate thinking traces when models emit them.Which fields carry data varies by benchmark, but the Record shape remains consistent.

B Installation and Server Launch

OmniEvaluator is installed by cloning the repository with submodules and installing the package in editable mode. An administrator then launches a persistent evaluation server with a specified port and log directory.

Installation begins with git clone --recursive to retrieve the repository and its submodules.
The persistent evaluation server is started with launch_server.py, a port, and a log directory.
The package is installed from the repository with pip install -e .

C Verifier: Data and Training

OmniEval Verifier is trained on diverse, human-verified examples spanning four modalities and produces rationale-plus-binary correctness judgments. Its evaluation addresses pathological rule-based scores and verification cost while achieving strong held-out accuracy.

Data: 16 evaluated models across four modalities and 155 benchmarks provide training predictions for the verifier.Gold labels use a multi-teacher API pipeline, balanced positive and negative classes, and limited rule-based augmentation.
Data: The held-out test split contains n=1,566 balanced examples over modality-task cells and is fully human-verified.Dispute samples receive priority during test-set construction.
Training: The verifier reads a question, reference, and prediction, then produces a rationale and binary verdict.The training mixture spans four modalities, from short exact-match responses to long reasoning traces.
Verification behavior: Rule-based corpus metrics can be distorted by pathological samples, such as LibriSpeech refusals with per-sample WER 266.6.An offtranscript hallucination produces WER 175.0, while the verifier marks such predictions incorrect.
Verification cost: 50×5 API-verification passes cost approximately $12,000 for GPT-5.4-mini and $70,000 for Claude-Opus-4.8.The OmniEval Verifier runs at near-zero marginal cost.
Verification accuracy: 85.0 accuracy on the human-verified test split exceeds base Qwen3-0.6B's 56.1 and matches or exceeds the listed cost-efficient proprietary judges.GPT-5.4-mini scores 82.8 and Claude-Haiku-4.5 scores 84.3.

D Supported Benchmarks and Full Results

OmniEvaluator reports benchmark results across four modalities and maintains full results on a shared dashboard. The evaluation tables also document how prompts and scoring choices affect benchmark outcomes.

Results cover text, image, video, and audio benchmarks spanning knowledge, instruction following, math, VQA, documents, video, speech, sound, and music.
Table 9 shows that benchmark-specific and uniform prompts can change answer formats and scores across engines and benchmarks.It compares one Qwen2.5-Omni-7B prediction with native and verifier scores; Table 3 reports all five benchmarks.
Table 10 reports per-modality benchmark results using accuracy, OCRBench, WER, and BLEU metrics.OCRBench is reported on a 0–1,000 scale; ASR/AST use WER and BLEU.