Source-linked AI summary
OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lenses
Guangzheng Hu, Ziyue Jiang, Weixu Qiao, Lixin Zhang, Jianye Kang, Yuru Wu, Rong Bao, Niantong Li, Wei Wang, Ziyi Cheng, Xinfa Zhu, HangRui Hu, Ting He, Bing Zhao, Lin Qu, Hu Wei, Jin Xu
TL;DR
Existing OmniJudge benchmarks provide limited evidence about fine-grained understanding because positive-heavy data and conflated failure modes can mask capability gaps. D3-Omni constructs a balanced, decoupled, and dynamic benchmark from verified positive seeds and controlled perturbations. Across T2I, T2V, and TTS, judges show modality-related weaknesses, stronger confirmation than violation detection, and substantial decision coupling, while balanced diagnostics reveal limitations hidden by aggregate accuracy.
Problem
OmniJudge reliability for fine-grained multimodal evaluation remains unclear because existing benchmarks overrepresent positive examples and conflate distinct failure modes.
Method
D3-Omni uses 53 orthogonal binary dimensions across T2I, T2V, and TTS, deriving dimension-isolated negatives from verified fully positive seeds through controlled prompt rewriting and atomic perturbations.
Results
Balanced evaluation finds modality-related dimensions near a capability floor, stronger confirmation of satisfied requirements than detection of violations, and judge predictions that collapse distinct dimensions into coupled decisions.
Takeaways & Limitations
Aggregate judge accuracy can conceal systematic blind spots that balanced, decoupled diagnostics expose across score regimes, polarities, modalities, and dimensions.
Takeaways & Limitations
Mixed average accuracy and low dimension coupling can be misleading when a judge emits near-constant predictions, so coupling must be interpreted with per-dimension accuracy.
Abstract
from arXiv · showhide
Multimodal understanding models that can jointly judge text-to-image (T2I), text-to-video (T2V) and text-to-speech (TTS) generation are increasingly used as "OmniJudges" for evaluation and automatic annotation. How reliably they understand what they score remains unclear, since existing benchmarks and training data tend to overemphasize positive examples and to conflate distinct failure modes, so a judge may score well without recognizing failures while its capability gaps stay hidden. Motivated by this, we introduce D3-Omni, a balanced and decoupled benchmark for diagnosing fine-grained multimodal understanding, covering 53 orthogonal binary dimensions (17/22/14) and 10,671 samples (3,526/1,998/5,147) across the three tasks. Rather than re-generating outputs, which may leak information across dimensions, we fix verified fully positive seeds and derive negatives through controlled prompt rewriting and atomic, dimension-isolating perturbations. The resulting D3 design is Dual-balanced, which helps alleviate negative-sample scarcity and per-dimension label imbalance; Decoupled, so that each error is attributable to a single capability; and Dynamic, steering construction toward under-represented regions of the label distribution as generative models improve.The suite reaches near 1:1 per-dimension parity and a uniform distribution over all total-score levels. Under this balanced view, even strong OmniJudges tend to struggle on modality-related dimensions, to confirm satisfied requirements far more reliably than they detect violated ones, and to treat nominally distinct attributes as largely a single decision, suggesting that aggregate accuracy may hide systematic blind spots that a balanced and decoupled lens can help expose and, in turn, address.
1 Introduction
D3-Omni addresses the underdeveloped evaluation of OmniJudges by balancing fine-grained labels, isolating failure dimensions, and enabling scalable benchmark updates. It reframes evaluation from aggregate scoring toward diagnosing what judges understand and miss.
- Existing OmniJudge evaluation remains less developed than generation evaluation despite expanding use as annotators, preference judges, reward models, and distillation teachers.
- Imbalanced benchmarks can yield high judge accuracy while leaving failure detection and fine-grained capability gaps untested.Positive outputs dominate real-world data, whereas judging requires detecting occasional errors and subtle mismatches.
- Manual construction of balanced, decoupled omni-modal benchmarks is costly and quickly becomes less discriminative as Omni-LLM capabilities evolve.The proposed direction shifts human effort toward targeted verification and quality control while using automatic construction and consistency checking.
- D3-Omni diagnoses fine-grained multimodal understanding across T2I, T2V, and TTS rather than relying on broad modality coverage or aggregate preference accuracy.
- 53 orthogonal binary dimensions and 10,671 samples support near 1:1 per-dimension parity and a uniform distribution over total-score levels.The dimensions comprise 17 T2I, 22 T2V, and 14 TTS dimensions.
2 Related Work
Prior work evaluates generated-media quality and uses automated judges, but generally treats evaluators as trusted black boxes. Existing judge benchmarks also suffer from skewed fine-grained labels, allowing majority-class behavior to inflate aggregate accuracy.
- Generation benchmarks cover modality-specific properties such as compositional alignment, motion quality, temporal consistency, naturalness, intelligibility, and speaker similarity.
- Existing media evaluators are often treated as trusted black boxes, leaving their fine-grained perceptual, alignment, and instruction-following understanding untested.
- LLM-as-a-judge systems provide scalable comparison, scalar scoring, and rubric-based critique, while generative-media judges and reward models remain largely modality-specific.
- Judge benchmarks reveal biases such as position and verbosity bias, weak factual sensitivity, and persistent limitations in multimodal evaluation.
- Skewed fine-grained labels let judges exploit class priors, so even a majority-class baseline can achieve high accuracy while aggregate metrics miss capability gaps.
3 Benchmark Dimension Design
D3-Omni decomposes judging into atomic, orthogonal prompt-related and modality-related requirements, enabling failures to be localized to language grounding or signal perception. Its taxonomy spans 53 binary dimensions across T2I, T2V, and TTS.
- Each benchmark instance pairs a prompt and generated artifact with a binary label vector, and each dimension asks whether one specific requirement is satisfied.The total score counts satisfied requirements.
- Prompt-related dimensions assess semantic alignment between prompt and output, whereas modality-related dimensions assess intrinsic signal fidelity, coherence, plausibility, and artifacts.
- Orthogonal atomic dimensions allow one requirement to be flipped through a controlled perturbation without altering other labels, localizing failures rather than confounding correlated attributes.
- T2I: 17 T2I dimensions cover prompt-related composition and perceptual properties including material realism, edge clarity, and anatomical coherence.
- T2V: 22 T2V dimensions cover prompt-related content and alignment plus modality-related temporal, sharpness, rhythm, audio-quality, and synchronization properties.
- TTS: 14 TTS dimensions separately evaluate textual and speaker characteristics alongside voice clarity, noise, volume stability, and spectral integrity.
4 Balanced Benchmark Construction
D3-Construction creates dimension-isolated negatives from verified fully positive seeds, then balances score levels and per-dimension labels while dynamically targeting under-represented regions. Its coupling matrix checks whether judges preserve or spuriously collapse distinctions.
- Decoupling: Re-synthesizing outputs from perturbed prompts can cause cross-dimensional leakage, so construction fixes a positive seed and changes one factor at a time.
- Decoupling: A fully positive seed satisfies every rubric dimension, providing the verified basis for generating controlled single-dimension negatives.
- Decoupling: Prompt-side flips use minimal, isolated, structurally coherent rewrites, while modality-side flips apply human-verified atomic operators to the generated artifact with the prompt fixed.
- Decoupling: The dimension-coupling matrix compares Pearson correlations among ground-truth and predicted YES/NO indicators to expose judge-generated spurious coupling.
- Dual-Balancing: Dual balance requires uniform representation across total-score segments and parity across each dimension, because either constraint alone can leave systematic imbalance.
- Dual-Balancing: Exact dual balance is realizable with |B| = M(Dτ + 1) whenever M(Dτ + 1) is even, and monitored quantities can be driven below any prescribed threshold.
5 Experiments
Across T2I, T2V, and TTS, aggregate accuracy masks score-dependent collapses, polarity bias, and modality-specific blind spots. Balanced diagnostics show that judges confirm satisfied requirements more reliably than they localize violated ones, especially in speech.
- Overall accuracy: 72.4%–78.6% on T2I and 69.5%–76.7% on T2V form tight macro-accuracy bands that conceal different fine-grained weaknesses.TTS accuracy compresses to 58.7%–65.7% and reorders the leading judge families.
- Score-segment behavior: Every judge forms a U-shaped score profile, with substantial accuracy loss on mixed-quality samples where each dimension must be decided independently.On T2I, GPT-5.2 drops from 84.9% at s=0 and 83.4% at s=D to about 66% in the middle; TTS is hardest, reaching only 65.7% for its best judge.
- Calibration: −3.67 average hi−lo on T2I shows endpoint asymmetry, while temporal-task troughs shift toward mostly violated samples and deepen substantially.GPT-Audio-1.5 rises from 46.3% on all-violated speech to 98.5% on all-satisfied speech, approximating a monotonic ramp.
- Yes/No perfect match: 58.3% versus 0.44%: GPT-Audio-1.5 confirms all-satisfied speech far more often than it catches all-violated dimensions.Across TTS judges, No perfect-match averages only 0.06%–2.15%, demonstrating severe defect-localization failure.
- Yes/No perfect match: 44.92% on T2I, 23.31% on T2V, and 2.67% on TTS are the mean high-score No perfect-match rates, despite only one or two remaining defects.The high-score rebound therefore reflects confirmation of satisfied dimensions rather than reliable localization of defects.
- Perfect-match diagnosis: 84.9% per-dimension accuracy but only 4.06% fully correct for GPT-5.2 on all-violated T2I samples reveals the combinatorial gap between average accuracy and perfect matching.Per-dimension accuracy can remain reassuring when every dimension is defective, while all-or-nothing correctness collapses.
6 Conclusion
D3-Omni provides a balanced, decoupled, and dynamic instrument for diagnosing OmniJudge weaknesses across T2I, T2V, and TTS. Its evaluations reveal modality-specific floors, entangled decisions, class-prior dependence, and limited cross-modal generalization.
- 6 Conclusion: D3-Omni covers T2I, T2V, and TTS with 53 near-orthogonal binary dimensions over 10,671 samples and uniform total-score levels.Its construction combines Dual-balanced sampling, orthogonal one-dimension perturbations, and Dynamic targeting of sparse label regions.
- 6 Conclusion: Modality-internal perception remains a universal floor, with Dm macro accuracies near 50% and constant prediction on TTS Audio Clarity.The reported TTS Audio Clarity signature is exactly 0.500x.
- 6 Conclusion: Judges collapse distinct attributes into modality-specific clusters and may retain approximately 70% macro accuracy despite dozens of entangled T2V dimension pairs.This indicates that aggregate accuracy can coexist with highly compressed decision behavior.
- 6 Conclusion: Every judge confirms satisfied content more reliably than violated content, with TTS No perfect-match accuracy at most 12.24% across judges.On high-score TTS segments it reaches only 2.7%, compared with 44.9% on T2I.
- 6 Conclusion: No OmniJudge generalizes across modalities, as different model families lead the TTS and visual-task rankings.The reported example is Gemini-3.1-Pro at 78.56% on T2I and 60.70% on TTS, while Qwen-3.5-Omni-Plus leads TTS.
- 6 Conclusion: D3-Omni links each diagnosed weakness to a corresponding construction or training-data prescription, while inviting broader seed pools and operator libraries.The proposed extensions target each model’s weakest diagnostic cells and support collective benchmark growth.
A Additional Details
The appendix contains additional details.
- A Additional Details: Additional details can be placed in the appendix.
A.1 Proof of Lemma 1
The proof constructs a benchmark satisfying score uniformity and dimension parity by assigning cyclic flip sets to verified positive seeds. It then shows that both constraints are jointly compatible.
- A.1 Proof of Lemma 1: Each sample is represented by a flip set T, with label vector y_d = 1[d ∉ T] and score s = D − |T|.
- A.1 Proof of Lemma 1: The construction requires exactly M samples at every score s from 0 through D, corresponding to M flip sets of size D − s.
- A.1 Proof of Lemma 1: Consecutive cyclic arcs of prescribed lengths tile the dimension cycle, and composite flip operators realize each target flip set from a verified full seed.
- A.1 Proof of Lemma 1: When M(D + 1) is even, every dimension is flipped and retained exactly M(D + 1)/2 times, so score uniformity and dimension parity both hold.The two constraints therefore produce the same total positive count and are mutually compatible.
A.2 Full Evaluation Dimension Tables
The evaluation taxonomy enumerates atomic binary requirements for T2I, T2V, and TTS. Each task separates prompt-related alignment from modality-related intrinsic quality.
- A.2 Full Evaluation Dimension Tables: The taxonomy defines atomic binary requirements as the dimensions of the decoupled evaluation framework.
- A.2 Full Evaluation Dimension Tables: T2I uses 17 dimensions divided between prompt alignment and intrinsic image quality.
- A.2 Full Evaluation Dimension Tables: T2V uses 22 dimensions for prompt alignment and intrinsic video and audio quality, while TTS uses 14 dimensions for instruction or text alignment and intrinsic audio quality.
A.3 Modality-Side Atomic Operators
The modality-side atomic operators create isolated negative examples by flipping one modality-related dimension while preserving all others. They cover task-specific artifacts in T2I, T2V, and TTS, with dimensional isolation validated on held-out data.
- Design: Each modality-side operator flips a single dimension while leaving every other dimension intact.The operators were validated on a held-out audit set to confirm dimensional isolation.
- T2V: T2V uses six operators targeting spatial warping, blur, frame instability, retiming, audio noise, and audio–visual desynchronization.
- T2I: T2I uses three operators for greasy-texture distortion, Gaussian blur, and anatomical deformation.
- TTS: TTS uses four operators for muffled voice, reverberation, volume instability, and frequency-specific squeal or muddiness.
A.4 Evaluation Prompt Templates
The evaluation templates present each generated image or video with its generation prompt and require ordered binary judgments across fine-grained content, quality, and modality dimensions. Every judge receives task-specific prompts without modification apart from substituted inputs.
- A.4 Evaluation Prompt Templates: The same English per-dimension prompt templates are sent to every corresponding-task judge, with only task inputs substituted at evaluation time.TTS additionally substitutes the speaker instruction and spoken text.
- A.4.1 Text-to-Image Judging Prompt (17 dimensions): T2I outputs must be a 17-element JSON array containing only ordered Yes or No answers after thorough image examination.
- A.4.1 Text-to-Image Judging Prompt (17 dimensions): T2I judges compare an image with its generation prompt across 17 dimensions spanning composition, color, lighting, emotion, subject attributes, text, framing, material, clarity, and anatomy.
- A.4.1 Text-to-Image Judging Prompt (17 dimensions): The T2I template includes fine-grained checks for emotional expression, basic attributes, appearance, behavior, negative prompts, spatial relations, and scene type.
- A.4.1 Text-to-Image Judging Prompt (17 dimensions): It also evaluates style, text accuracy and typesetting, shot requirements, material detail, edge clarity, and object anatomy.
- A.4.2 Text-to-Video Judging Prompt (22 dimensions): T2V judges compare a video with its prompt across 22 dimensions covering subjects, actions, scenes, spatial relations, framing, cinematography, lighting, color, style, and audio.
- A.4.2 Text-to-Video Judging Prompt (22 dimensions): The T2V template separately checks completeness, appearance, subject–camera relations, shot parameters, lighting, color tone, style, and atmosphere.
- A.4.2 Text-to-Video Judging Prompt (22 dimensions): Its remaining dimensions assess audio content, style, parameters, video reality, focus, continuity, rhythm, audio quality, and audio–video synchronization, returned as ordered Yes or No answers.
A.5 Polarity-Resolved Judging Diagnostics
Polarity-resolved diagnostics separate judges’ confirmation of satisfied requirements from detection of violated ones and reveal that similar aggregate behavior can arise from distinct decision strategies. They also expose polarity-specific coupling among nominally separate dimensions.
- Polarity decomposition: The diagnostics split each per-dimension quantity by ground-truth polarity into Yes accuracy and No accuracy.The English runs are used, while Chinese counterparts are qualitatively identical.
- Per-dimension Yes/No accuracy: Yes accuracy measures recall on ground-truth-Yes dimensions, whereas No accuracy measures specificity on ground-truth-No dimensions.
- Per-dimension Yes/No accuracy: T2V Video Focal reaches Yes 0.99/No 0.02 and TTS Volume Stability reaches Yes 0.99/No 0.03, indicating near-constant Yes prediction rather than random guessing.
- Modality-specific limitation: Qwen-3.6-27B, lacking an audio channel, stays near chance on T2V audio prompt-alignment dimensions but defaults to Yes on intrinsic quality and synchronization.For the latter dimensions, Yes accuracy is 0.66 and No accuracy is ∼0.26.
- Polarity-conditioned coupling: Polarity-conditioned coupling is mostly confirmation-side on T2I, heavy on both sides for T2V, and balanced on TTS.The reported pair counts are T2I 31 versus 4, T2V 61 versus 77, and TTS 22 versus 22.
- Polarity-conditioned coupling: Polarity splitting reveals couplings hidden by mixed correlations, including Gemini-3-Flash’s TTS Audio-Clarity-with-Volume-Stability correlation of 0.71 on the Yes side versus 0.25 mixed.