PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Adversarial and Collaborative Learning for AI-Generated Image Quality Assessment

Baoliang Chen, Qing Lin, Sijie Mai

arXiv:2608.24372v1cs.CV

TL;DR

AIGIQA needs to assess perceptual fidelity and prompt alignment while accounting for their competitive and cooperative interactions, which human ratings reveal are not independent. The paper introduces a Dual-Gated Mixture-of-Experts framework with adversarial and collaborative paths and task-aware routing. Across multiple benchmarks, it reports state-of-the-art performance and interpretable, human-consistent interaction patterns.

  • Problem

    AIGIQA models often treat perceptual fidelity and prompt alignment as independent despite human ratings showing interdependent competitive and cooperative interactions.

  • Method

    The paper introduces a Dual-Gated Mixture-of-Experts framework that dynamically routes features through collaborative and adversarial reasoning paths using task-aware prompts.

  • Results

    Across multiple AIGIQA benchmarks, the approach achieves state-of-the-art performance and reveals strong complementarity between its adversarial and collaborative learning paths.

  • Takeaways & Limitations

    The findings support modeling AI-generated image quality beyond single-factor scoring through adaptive reinforcement and disentanglement of perception-alignment information.

Abstract

from arXiv · show

AI-generated image quality assessment (AIGIQA) requires jointly reasoning about perceptual fidelity and prompt alignment, two quality dimensions that are often treated as independent in existing AIGIQA models. However, by re-examining human ratings, we uncover a previously overlooked phenomenon: the two dimensions are interdependent and exhibit both competitive and cooperative interactions during human rating. This observation suggests that a unified model should neither collapse the two dimensions nor rigidly separate them, but rather adaptively negotiate their interplay. Motivated by this insight, we introduce an interaction-aware learning framework that models perception-alignment relations through adversarial and collaborative inference pathways. Instead of designing a rigid dual-branch architecture, our method employs a gated interaction module that dynamically routes features according to the inferred relationship between the two dimensions. Task-aware prompts further modulate the gating behaviour, enabling the model to switch between competition and cooperation when necessary. Experiments across multiple AIGIQA benchmarks demonstrate that our approach not only achieves state-of-the-art accuracy but also yields interpretable interaction patterns, offering a more faithful approximation of human judgment. The codes are available at https://github.com/LQAMEI/ACL-IQA.

I. INTRODUCTION

AIGIQA must assess both perceptual realism and prompt alignment because generated-image distortions are semantic and the two dimensions can interact rather than remain independent. The paper therefore proposes adversarial and collaborative learning paths to model these relationships jointly.

  • Motivation: AIGIQA evaluates perceptual quality and adherence to textual prompts, unlike traditional IQA focused mainly on physical image degradations.Generated images may contain surreal textures, implausible geometries, and image-text mismatches.
  • Motivation: Existing AIGIQA methods model image-text correspondence, but the interaction between perception and alignment remains underexplored.Prior approaches include dual encoders and CLIP-based alignment modeling.
  • Human-rating insight: Human ratings show that perceptual quality and prompt alignment can behave independently, conflict, or reinforce each other.This motivates asking whether the dimensions should be estimated adversarially, collaboratively, or through both modes.
  • Approach: The proposed framework jointly models adversarial and collaborative interactions to reason about perceptual fidelity and prompt alignment.Collaborative learning captures complementary information, while adversarial learning decouples interfering information and suppresses bias.
  • Approach: DG-MoE dynamically routes expert features through collaborative or adversarial paths within a unified transformer architecture.The module uses task-aware gating to select experts specialized for the two reasoning modes.
  • Results: Experiments across multiple AIGIQA benchmarks report superior performance and strong complementarity between the two learning paths.The contribution is positioned as evidence that both interaction modes are useful for quality assessment.

B. Mixture of Expert

The methodology uses mixture-of-experts routing to select input-dependent expert features and combines multimodal quality assessment with collaborative or adversarial text instructions. Images are represented through patches and a resized global view, while templates specify the quality dimension and learning strategy.

  • Mixture-of-Experts: Mixture-of-Experts uses multiple expert networks and a gating mechanism to select the most relevant experts for each input.The gating strategy influences expert selection, model efficiency, performance, and generalizability.
  • Model overview: The model takes an AIGI and its corresponding prompt as multimodal inputs to predict perception and alignment quality scores.Its framework contains multimodal inputs, text and image encoders, and learning objectives.
  • Image representation: Images are segmented into 224 × 224 patches and combined with a resized image before weighted fusion for final prediction.Using both representations addresses the risk that patch segmentation may split a single object across patches.
  • Text templates: Quality templates separately guide perception-quality and alignment-quality assessment.Tper describes image quality, while Taln describes how well the image matches the prompt.
  • Text templates: Collaborative and adversarial templates instruct the model to use complementary information or exclude specified considerations during evaluation.Tcol and Tadv respectively encode collaborative and adversarial learning instructions.

B. Text and Image Encoders

The encoders extract instruction-guided text features and collaboratively or adversarially learned image features. A dual-gated MoE block routes visual features to specialized experts using prompt-guided gating, while difference convolutions provide fine-grained details complementary to transformer features.

  • The text encoder derives collaborative, adversarial, perception, and alignment features from four corresponding text templates.
  • The image encoder extracts collaboratively and adversarially learned features representing decomposed image components for dual-path prediction.
  • The dual-gated MoE block uses specialized experts and learning-instruction features to guide collaborative or adversarial image-feature routing.
  • The dual gating network predicts expert weights from gating features and selects or weights expert outputs through softmax and top-k routing.The gating networks share the same design, and learnable weights project gating features to scalar expert-output weights.
  • Difference-convolution experts capture fine-grained, high-frequency details that complement the transformer’s low-frequency information.The expert set includes vanilla convolution and five difference-convolution types; 1 × 1 convolutions reduce and restore channel dimensions for efficiency.

C. Collaborative and Adversarial Learning

The framework forms perception- and alignment-related similarity matrices from image and text features, using collaborative fusion and adversarial gradient reversal. These matrices are converted into quality scores and trained with task-specific supervision and expert-balancing objectives.

  • Image and text quality features are compared with cosine similarity to produce four types of similarity matrices for perception and alignment assessment.
  • Gradient reversal is used in the adversarial pathway to ablate specific quality information through negative gradient backpropagation.
  • A trainable weighting scheme fuses collaborative and adversarial similarity information into final perception and alignment estimates.
  • Similarity matrices contain one row per image patch and five columns corresponding to the five quality levels defined by the perception and alignment prompts.
  • Softmax converts similarity values along the quality-level dimension into patch-level quality probabilities before supervision losses are applied.The probability denotes the j-th patch’s likelihood under the i-th quality level.
  • The total perception loss combines ground-truth perception error, collaborative and adversarial alignment losses, and an expert-utilization balancing loss.The balancing loss encourages all experts to have equal importance.

1) Datasets:

ACL-IQA is evaluated across five AIGIQA benchmarks using multimodal quality prediction, with experiments spanning early-stage and massive contemporary datasets.

  • Datasets: The evaluation covers AGIQA-1K, AGIQA-3K, AIGCIQA2023, AGIQA-20K, and EvalMi-50K.AGIQA-1K contains 1,080 images generated from 180 prompts, with MOS annotations for perception and text-image correspondence.
  • Implementation: ACL-IQA uses CLIP ViT-B/32 and retains a single-gated MoE for single-dimension evaluations.Training lasts 200 epochs on early-stage datasets and 50 epochs on AGIQA-20K and EvalMi-50K.
  • Evaluation: The model is denoted ACL-IQA and is evaluated using SRCC and PLCC.The implementation uses Adam with an initial learning rate of 5×10−6, weight decay of 1×10−3, and top-3 expert selection.
  • Results: On AGIQA-20K, ACL-IQA establishes state-of-the-art performance against MA-AGIQA and Q-Align.On EvalMi-50K, it surpasses FGA-BLIP2, Qwen2.5-VL (8B), and Llama3.2-Vision (11B) across perception and alignment.
  • Results: The results attribute performance to balancing global encoder modeling with local convolutional expert features and preserving beneficial perception-alignment interplay.

2) Generalization Ability Comparison:

ACL-IQA generalizes across datasets, domain shifts, extreme interaction scenarios, and qualitative comparisons, while ablations support the complementary roles of its components.

  • Cross-dataset transfer: ACL-IQA achieves the strongest cross-dataset performance, including SRCCs of 0.761 and 0.676 when transferring from AIGCIQA2023 to AGIQA-3K.The reverse transfer reaches 0.760 and 0.632, with reported improvements over IPCE in both directions.
  • Domain-shift evaluation: Transferring from AGIQA3K to EvalMi-50K raises perception SRCC from 0.2914 to 0.6057 against MA-AGIQA.ACL-IQA also outperforms IPCE across perception and alignment under this domain shift.
  • Ablation studies: The MoE improves over standard transformer blocks, while the dual-path mechanism itself improves over the ablated baseline.
  • Ablation studies: The full model performs best across datasets because adversarial and collaborative mechanisms capture complementary quality representations.Removing the learning instruction lowers perception SRCC on AGIQA-3K and alignment SRCC on AIGCIQA2023.
  • Interpretability: Path-specific attention and expert selection distinguish features learned for perception, alignment, adversarial reasoning, and collaborative reasoning.Adversarial similarity is significantly lower than collaborative similarity, indicating limited quality information transfer from the other dimension.
  • Extreme scenarios: ACL-IQA maintains strong consistency with human ratings and outperforms IPCE in both fully collaborative and fully adversarial scenarios.
  • Qualitative comparison: In gMAD comparisons, ACL-IQA differentiates paired-image quality more consistently with MOS than IPCE across perception and alignment cases.

V. CONCLUSION

The paper argues that perceptual fidelity and prompt alignment interact both collaboratively and adversarially, and models this interaction through dynamically routed complementary paths.

  • Conclusion: Human judgments show that perceptual fidelity and prompt alignment are not independent attributes but exhibit collaborative and adversarial interactions.
  • Conclusion: ACL-IQA uses a Dual-Gated Mixture-of-Experts framework to route features through complementary reasoning paths.The framework adaptively balances reinforcement and disentanglement.
  • Conclusion: Experiments report state-of-the-art performance together with more interpretable and human-consistent predictions.

A. OVERVIEW

The supplementary analyses examine scalability, domain shift, prompt-level interaction patterns, and gradient conflicts to support ACL-IQA’s interaction-aware design.

  • Extended benchmarks: ACL-IQA is evaluated on EvalMuse-40K, GenAIBench, and RichHF-18K for fine-grained alignment and compositional artifacts.The supplementary comparison reports competitive and computationally efficient evaluation from a model of approximately 192M parameters.
  • Domain shift: Modern generators occupy wider semantic and visual manifolds than early-stage datasets, yet ACL-IQA bridges this domain discrepancy in cross-dataset transfer.
  • Network-level analysis: Gradient cosine similarities contain substantial positive and negative regions across benchmarks, directly evidencing both synergistic learning and optimization conflict.This supports using gradient reversal for adversarial decoupling and dual-gated MoE routing.

D. VISUALIZATION OF SPECIALIZED CONVOLUTIONAL EXPERTS

The framework embeds six specialized convolution experts within a Vision Transformer to capture complementary global, structural, and geometric cues in AI-generated images. Additional analyses test statistical significance and prompt-template robustness.

  • Specialized convolutional experts: Six convolution experts—Vanilla, Central Difference, Angular Difference, Radial Difference, Horizontal Difference, and Vertical Difference—are embedded in the Vision Transformer’s deep latent space.The operators target fine-grained details in AI-generated images.
  • Specialized convolutional experts: Vanilla convolution anchors global context and salient objects, whereas difference convolutions activate around structural boundaries, misaligned edges, and geometric distortions.Because Vision Transformer tokens have global receptive fields, these responses complement broad contextual information with localized structural sensitivity.
  • Statistical significance analysis: Ten-fold cross-validation compares ACL-IQA with IPCE on AGIQA-3K and AIGCIQA2023 using 95% confidence intervals and paired t-tests.The evaluation uses identical data splits across the ten folds.
  • Statistical significance analysis: p-values below 0.01 across all evaluations indicate statistically significant improvements for ACL-IQA over IPCE.The reported analysis attributes the gains to the decoupled interaction-aware mechanism rather than random data variation.
  • Prompt sensitivity analysis: The Default prompt template achieves the highest overall balance and correlation for perception and alignment while gating remains generally robust across four language formulations.The variants are Default, Concise, Verbose, and Synonym, evaluated by retraining on EvalMi-50K for 10 epochs.

F. CROSS-TASK FEATURE DECOUPLING AND IMPLICIT LOCALIZATION

Cross-task evaluations show that collaborative features retain information across perception and alignment, while adversarial features suppress interfering representations on decoupled tasks. Qualitative analyses further connect adaptive routing with regime-specific reasoning and implicit localization of artifacts and misalignment.

  • Cross-task feature decoupling: The collaborative feature Fcol retains mutual information and predicts both primary and secondary quality dimensions accurately, whereas Fadv approaches zero or negative correlation on the secondary task.This pattern supports targeted decoupling of interfering representations through the adversarial pathway.
  • Specialized feature routing: The convolution experts provide pixel-level and patch-level feature maps, with the top three dynamically gated experts fused into the final map.The expert set includes Vanilla, Central Difference, Angular Difference, Radial Difference, Horizontal Difference, and Vertical Difference Convolutions.
  • Evaluation context: The statistical-significance analysis uses 10-fold cross-validation, paired t-tests, and a p-value threshold below 0.01 for ACL-IQA versus IPCE.These evaluations are summarized in Table E.1.
  • Evaluation context: The prompt-template comparison reports SRCC and PLCC for Default, Concise, Verbose, and Synonym templates trained on EvalMi-50K.The table organizes performance by prompt formulation and correlation metric.
  • Adaptive interaction regimes: ACL-IQA dynamically shifts routing weights between collaborative and adversarial paths across collaborative, complex-constraint, and stylistically degraded regimes.Collaborative routing benefits mutually reinforcing visual and semantic cues, while adversarial routing mitigates bias from conflating semantic execution with visual quality.
  • Cross-task feature decoupling: Table F.1 evaluates cross-task decoupling for a model trained on perceptual quality, with Fcol denoting the collaborative feature.The table compares collaborative-feature behavior with isolated feature predictions.
  • Cross-task feature decoupling: Table F.2 reports joint and isolated predictions for alignment quality, using downward arrows to indicate performance decreases relative to the collaborative path.The table explicitly compares joint performance with decoupled feature predictions.
  • Implicit anomaly localization: Grad-CAM localizes structural flaws and text-image misalignment without pixel-level mask supervision, with heatmaps overlapping human-annotated masks.Examples include deformed hands, misspelled rendered text such as Panera Bread logos, and prompt-specified missing objects.
Loading 2608.24372v1…