Source-linked AI summary
UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture
Shuo Cao, Jiayang Li, Xiaohui Li, Yuandong Pu, Kaiwen Zhu, Yuanting Gao, Siqi Luo, Yi Xin, Qi Qin, Yu Zhou, Xiangyu Chen, Wenlong Zhang, Bin Fu, Yu Qiao, Yihao Liu
TL;DR
MLLMs have progressed on semantic vision tasks but remain limited in perceptual judgments involving aesthetics, quality, structure, and texture. UniPercept introduces a hierarchical benchmark and trains a unified model with domain-adaptive pre-training and task-aligned reinforcement learning. The model generalizes across rating and question-answering tasks, outperforms existing MLLMs, and supports perceptually aligned text-to-image optimization.
Problem
Perceptual-level image understanding remains underexplored and unstable in MLLMs, especially for fine-grained judgments of aesthetics, quality, structure, and texture.
Method
UniPercept-Bench uses a hierarchical taxonomy and complementary VR/VQA tasks, while UniPercept is trained with Domain-Adaptive Pre-Training and Task-Aligned RL.
Results
UniPercept consistently outperforms generalized and specialized MLLMs across VR and VQA, with strong in-domain performance and cross-domain generalization.
Takeaways & Limitations
UniPercept provides a unified perceptual baseline and can serve as a plug-and-play reward model for perceptually aligned text-to-image post-training.
Takeaways & Limitations
Existing perceptual datasets often cover only a single format or aspect, motivating UniPercept-Bench’s broader multidimensional evaluation.
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have achieved remarkable progress in visual understanding tasks such as visual grounding, segmentation, and captioning. However, their ability to perceive perceptual-level image features remains limited. In this work, we present UniPercept-Bench, a unified framework for perceptual-level image understanding across three key domains: Aesthetics, Quality, Structure and Texture. We establish a hierarchical definition system and construct large-scale datasets to evaluate perceptual-level image understanding. Based on this foundation, we develop a strong baseline UniPercept trained via Domain-Adaptive Pre-Training and Task-Aligned RL, enabling robust generalization across both Visual Rating (VR) and Visual Question Answering (VQA) tasks. UniPercept outperforms existing MLLMs on perceptual-level image understanding and can serve as a plug-and-play reward model for text-to-image generation. This work defines Perceptual-Level Image Understanding in the era of MLLMs and, through the introduction of a comprehensive benchmark together with a strong baseline, provides a solid foundation for advancing perceptual-level multimodal image understanding.
1. Introduction
Existing MLLMs excel at semantic understanding but remain limited on subtle, subjective perceptual judgments about aesthetics, quality, structure, and texture. UniPercept addresses this gap with a unified taxonomy, benchmark, and model spanning these domains and both rating and question-answering tasks.
- Motivation: Semantic-level MLLM capabilities have advanced rapidly, while perceptual-level understanding of fine-grained appearance remains substantially underexplored.Perceptual tasks assess attributes such as aesthetic harmony, degradation severity, structural regularity, and surface texture, which differ from semantic interpretation.
- Motivation: Perceptual cues matter for content creation, image enhancement, and generative-model alignment, yet current MLLMs often produce unstable or inconsistent judgments.The paper motivates a unified framework to define, evaluate, and improve perceptual understanding in closer alignment with human judgments.
- UniPercept-Bench: UniPercept-Bench organizes perceptual understanding through a Domain–Category–Criterion hierarchy covering aesthetics, quality, structure, and texture.The framework is intended to define perceptual scope, visual aspects, and fine-grained criteria for evaluation.
- UniPercept: UniPercept combines Domain-Adaptive Pre-Training with task-aligned reinforcement learning and generalizes across VR and VQA in IAA, IQA, and ISTA.The reported framework is designed as a unified baseline for perceptual understanding across the three domains.
- Scope: UniPercept-Bench targets a less explored assessment dimension by systematically defining Image Structure and Texture Assessment alongside established aesthetics and quality assessment.The paper describes ISTA as receiving little prior systematic definition or evaluation.
- UniPercept-Bench: The benchmark supports complementary Visual Rating and Visual Question Answering tasks for unified, fine-grained evaluation across perceptual domains.Existing assessment resources often emphasize only numerical scoring or question answering, whereas this benchmark combines both formats.
4. UniPercept
UniPercept combines domain-adaptive pre-training with task-aligned reinforcement learning to support perceptual image understanding across visual rating and visual question answering. Its rewards align discrete VQA correctness with continuous VR consistency.
- 4.1. Domain-Adaptive Pre-Training: UniPercept uses approximately 800K pre-training samples spanning IAA, IQA, and ISTA to build perceptual-level image understanding.The data include text-based QA, structured ISTA reasoning pairs, and visual-rating data linking perceptual attributes to quantitative scores.
- 4.2. Task-Aligned RL for VR & VQA: Task-aligned reinforcement learning applies task-specific rewards through GRPO for both Visual Question Answering and Visual Rating.VQA uses binary correctness rewards, whereas VR uses an adaptive Gaussian soft reward based on deviation from ground-truth scores.
- 4.2. Task-Aligned RL for VR & VQA: The adaptive Gaussian soft reward evaluates VR predictions continuously, producing smoother gradients and avoiding threshold-induced discontinuities.Predicted and ground-truth scores are mapped to [0, 100], with σ0 controlling base smoothing and α controlling adaptive smoothing.
- 4.2. Task-Aligned RL for VR & VQA: Token As Score derives VR ratings from predicted token distributions before incorporating the task-specific reward into the GRPO objective.This formulation aligns model behavior with continuous perceptual consistency for VR and discrete correctness for VQA.
5. Experiments
Experiments evaluate UniPercept against proprietary, open-source, and specialized models across perceptual visual rating and VQA tasks. UniPercept consistently outperforms generalized and specialized models, while its rewards improve text-to-image generation and provide complementary evaluation metrics.
- 5.2.1. Visual Rating: General-purpose MLLMs generally trail specialized models on Visual Rating because direct numerical generation produces hallucinated or unstable predictions.Specialized methods use structured strategies such as Token As Score to stabilize regression on continuous scales.
- 5.2.2. Visual Question Answering: Generalized models reach up to 68.28%, 72.15%, and 81.13% accuracy on IAA, IQA, and ISTA VQA, but average only 51.62%, 51.45%, and 60.56%.ISTA is somewhat easier because its objective geometric, structural, and material properties align better with visual-text pretraining than subjective IAA and IQA judgments.
- 5.2.2. Visual Question Answering: VQA performance drops below 50% on fine-grained categories such as Overall Gestalt, Material Representation, and Geometric Composition.Holistic categories including Composition & Design and Theme & Communication generally exceed 60%, while Level Prediction averages around 36% on IAA and 45% on IQA.
- 5.3.1. Performance: UniPercept consistently outperforms generalized and specialized models across VR and VQA in IAA, IQA, and ISTA, including cross-domain generalization.The comparison covers UniPercept-Bench visual rating and VQA results reported in Tables 1–4 and Figure 7.
- 5.3.2. UniPercept Reward: Combining IAA, IQA, and ISTA rewards into UniPercept Reward (All) achieves the best overall text-to-image generation performance.The individual rewards emphasize distinct perceptual facets, while their combination integrates complementary cues; qualitative results show improved fidelity, perceptual quality, and human preference.
- 5.3.3. UniPercept as a Metric: UniPercept evaluates generated images across complementary IAA, IQA, and ISTA dimensions, with current models showing stronger IQA than IAA or ISTA.The evaluation uses DPG and GenEval benchmarks, while dataset analysis identifies Unsplash and Blip3o-60K as strongest overall across the three domains.
6. Conclusion
UniPercept frames perceptual-level image understanding as a unified family spanning aesthetics, quality, and structure–texture, complementing semantic recognition with human-aligned visual judgments. The framework supports unified evaluation, controllable perceptual improvement, dataset curation, and downstream reward modeling.
- UniPercept-Bench and UniPercept provide a unified benchmark and baseline for perceptual understanding, with applications including plug-and-play reward modeling for post-training.
- Perceptual-level understanding concerns how images look and feel, including beauty, fidelity, distortion, structural coherence, and textural realism rather than categorical recognition.
- UniPercept unifies Image Aesthetics Assessment, Image Quality Assessment, and Image Structure and Texture Assessment into a complementary perceptual hierarchy.IAA captures holistic appeal, IQA captures fidelity and distortion, and ISTA captures fine-grained structure and texture realism.
- The three perceptual dimensions are complementary but nonredundant: high quality does not imply high aesthetics, and rich texture does not imply high quality or aesthetics.
- Unified perceptual signals support shared representations, targeted generation or editing, perceptual dataset curation, and evaluation of generative or restoration models.
8. Details of ISTA
ISTA operationalizes structure–texture richness through hierarchical annotations covering morphology, materials, geometry, semantics, and style. Its deterministic scoring combines component-level perceptual factors into interpretable image-level ratings clipped to the 0–100 range.
- Definition of ISTA-10K: Structural Annotation organizes each component across Physical Structure, Material Representation, Geometric Composition, Semantic Perception, and Style Type.
- Definition of ISTA-10K: ISTA-10K rates structure–texture richness from texture complexity, material and geometric diversity, and the richness and organization of structural components.
- Definition of ISTA-10K: Texture intensity assigns increasing weights to weak, medium, and strong morphology groups, with higher weights representing irregular, high-frequency, or structurally complex patterns.The three groups are ordered by increasing structural complexity.
- Definition of ISTA-10K: Component scores combine physical structure, material representation, geometric composition, and semantic perception terms, excluding unavailable entries.
- Definition of ISTA-10K: The resulting formulation is deterministic and interpretable while aligning structure–texture richness with UniPercept-Bench’s hierarchical annotation schema.
9. Details of UniPercept-Bench
UniPercept-Bench evaluates perceptual image understanding through a Domain–Category–Criterion taxonomy and two complementary task formats. It covers IAA, IQA, and ISTA with both continuous ratings and question answering, enabling fine-grained cross-domain assessment.
- Benchmark Definition: UniPercept-Bench organizes perceptual attributes through a three-level Domain–Category–Criterion taxonomy.
- Benchmark Tasks: The benchmark evaluates IAA, IQA, and ISTA using both Visual Rating and Visual Question Answering formats.
- Benchmark Scope: UniPercept-Bench provides finer-grained categories, supports rating and textual formats, uses scalable human–MLLM-assisted annotation, and covers all three perceptual domains.
- Benchmark Scope: The three domains are largely independent: high IQA can coexist with poor IAA, while strong aesthetics can coincide with weaker structural or textural assessment.
10. Further Discussion on UniPercept
UniPercept is trained with domain-adaptive pre-training and task-aligned reinforcement learning across three perceptual domains and two task formats. Ablations show that perceptual pre-training, adaptive soft rewards, and multi-domain training materially support performance across VR and VQA.
- Training Strategy: UniPercept combines approximately 800K IAA, IQA, and ISTA samples during Domain-Adaptive Pre-Training and mixes VR and VQA data across all three domains during Task-Aligned RL.
- Evaluation: UniPercept’s training and evaluation cover both VR and VQA dimensions, with ablation results reported in the benchmark’s VR and VQA tables.
- Training Strategy: Removing Domain-Adaptive Pre-Training causes substantial drops on both Visual Rating and Visual Question Answering.The ablation indicates that large, domain-relevant data are needed to provide fundamental perceptual capabilities.
- Reward Design: Adaptive Gaussian Soft Reward consistently outperforms threshold-based reward and also improves VQA performance, despite rating rewards being applied to VR.The results suggest correlations between the VR and VQA tasks, while the soft reward better captures score deviation.
- Multi-Task and Multi-Domain Training: Training on a single perceptual domain performs well on that domain but falls short overall, whereas mixed-domain training achieves stronger overall results and can surpass single-domain models on some tasks.
11. More Examples of UniPercept-Bench
Figure 11 provides additional examples from UniPercept-Bench.
- Figure 11 presents additional UniPercept-Bench examples.
- The examples extend the benchmark’s visual materials beyond the previously shown instances.
- The figure serves as an example gallery for UniPercept-Bench.
12. UniPercept-Constructed Image Profiles
UniPercept performs comprehensive perceptual-level image analysis across aesthetics, image quality, and structure-and-texture assessment. Its outputs combine visual ratings with fine-grained, multidimensional image profiles and example question-answering analyses.
- UniPercept provides accurate visual-rating evaluations across the IAA, IQA, and ISTA dimensions.
- The model generates fine-grained, multidimensional analytical outputs that form a detailed profile for each image.
- The profile examples examine composition, design, emotion, viewer response, and technical execution.
- Emotion and viewer-response quality is assessed using low, medium, and high answer levels.
- Hierarchical emphasis is evaluated through questions about the most visually prominent element.
A. Simulate natural petal surfaces
The examples operationalize perceptual analysis through questions and criteria spanning composition, distortions, geometry, materials, semantics, and texture.
- The examples use question-answer templates to assess visual elements, composition, material properties, and image quality.
- Distortion analysis covers type, severity, and location, with examples including blur, compression, saturation changes, and overexposure.
- Additional examples include assessments of spatial clarity, lighting effects, texture visibility, and distortion severity.
- Structural analysis represents geometric composition through two-dimensional contours such as squares, hexagons, circles, and pentagons.
- Texture and material analysis distinguishes surface morphology, material classes, glossiness, stylistic classification, and material identification.
Image Aesthetics Assessment (IAA)
The image-aesthetics assessment describes an extremely poor image across composition, technical execution, and comprehensive evaluation.
- Composition & Design: The composition is described as nonexistent, with a blurred, indistinct subject and randomly scattered elements.
- Technical Execution: Technical execution is characterized by catastrophic errors, extreme blurriness, invisible details, and nonexistent focus.
- Comprehensive Evaluation: The comprehensive evaluation identifies the image as suffering from extreme blurriness.
Image Quality Assessment (IQA)
The IQA examples span catastrophic blur, flawless clarity, and detailed scene descriptions, illustrating how image quality is assessed through distortion and visual fidelity.
- Quality conditions: The benchmark’s quality examples distinguish severe blur from flawless clarity, covering both degradation and high-fidelity image conditions.The blurred garden scene is described as catastrophically out of focus, while the historic-building image is judged flawless and distortion-free.
- Distortion analysis: The blurred example localizes degradation across the entire scene, including the pergola, lanterns, and foliage.Its uniform loss of sharpness prevents meaningful evaluation of other distortion types.
- Structural representation: Scene decomposition complements quality judgments by representing components through structure, material, geometry, and arrangement.The examples encode elements such as pergolas, foliage, houses, roofs, and temple structures using structured descriptors.
- Perceptual context: The examples also show that perceptual assessment can preserve aesthetic and compositional information alongside technical quality judgments.Descriptions of historical architecture and jungle scenes discuss visual impact, balance, color contrast, focal points, and thematic communication.
- Quality analysis: The high-quality historic-building example combines sharp focus, accurate color, balanced composition, and strong exposure with no visible flaws.The passage characterizes the photograph as a perfect, high-quality image.