PaperlayerView exact revision on arXiv

Source-linked AI summary

Panoptic Pairwise Distortion Graph

Muhammad Kamran Janjua, Abdul Wahab, Bahador Rashidi

arXiv:2604.11004v1cs.CVcs.AIcs.LG

TL;DR

Existing image-assessment methods emphasize global analysis, leaving region-level pairwise distortion understanding insufficiently represented. The paper introduces Distortion Graphs, PANDASET, PANDABENCH, and PANDA, and reports that current MLLMs struggle on region-wise assessment while training or DG prompting improves it.

  • Problem

    Global image analysis and existing MLLM designs provide limited fine-grained, region-level pairwise distortion understanding.

  • Method

    The paper represents image pairs as region-grounded Distortion Graphs and develops PANDASET, PANDABENCH, and PANDA for learning and evaluating them.

  • Results

    PANDA achieves the best performance across all three benchmark settings and four tasks, while training on PANDASET or prompting with DG improves region-wise assessment.

  • Takeaways & Limitations

    Structured, region-first representations support compact pairwise assessment and can complement MLLMs with region-wise distortion analysis.

  • Takeaways & Limitations

    The evaluation notes that passing only cropped regions can make methods blind because region sizes vary, and some outputs cannot be scored reliably.

Abstract

from arXiv · show

In this work, we introduce a new perspective on comparative image assessment by representing an image pair as a structured composition of its regions. In contrast, existing methods focus on whole image analysis, while implicitly relying on region-level understanding. We extend the intra-image notion of a scene graph to inter-image, and propose a novel task of Distortion Graph (DG). DG treats paired images as a structured topology grounded in regions, and represents dense degradation information such as distortion type, severity, comparison and quality score in a compact interpretable graph structure. To realize the task of learning a distortion graph, we contribute (i) a region-level dataset, PandaSet, (ii) a benchmark suite, PandaBench, with varying region-level difficulty, and (iii) an efficient architecture, Panda, to generate distortion graphs. We demonstrate that PandaBench poses a significant challenge for state-of-the-art multimodal large language models (MLLMs) as they fail to understand region-level degradations even when fed with explicit region cues. We show that training on PandaSet or prompting with DG elicits region-wise distortion understanding, opening a new direction for fine-grained, structured pairwise image assessment.

1 INTRODUCTION

The paper argues that global, top-down image analysis limits fine-grained distortion understanding and introduces a region-grounded graph representation for pairwise assessment. It supports this task with PANDASET, PANDABENCH, and PANDA, while showing that current MLLMs struggle with region-level reasoning.

  • Global top-down analysis limits fine-grained understanding and restricts distortion VQA, ranking, description, and quality scoring to image-level outputs.
  • Distortion Graph represents paired images through region nodes, comparative inter-region edges, distortion attributes, severity, and region-level quality scores.
  • PANDASET contains over 500K image pairs spanning 15 distortions and four severity levels, with region-associated quality scores.
  • PANDABENCH evaluates region-level distortion reasoning across increasing difficulty, while current MLLMs often use templates and face context-length limits.
  • Frontier MLLMs have stronger instruction-following abilities but perform not much better than random chance on the benchmark.

2 RELATED WORK

Prior distortion and region-understanding methods address complementary parts of the problem, but do not jointly provide comparative, region-first, densely annotated distortion analysis. The paper positions its dataset and benchmark as addressing this combined gap.

  • Existing distortion MLLM work develops datasets, benchmarks, and methods for distortion identification, VQA, quality scores, and descriptive analysis.
  • Region-understanding methods use visual markers, region-of-interest features, or token markers, while Seagull focuses on descriptive distortion analysis for single images.
  • Prior work does not simultaneously provide comparative reasoning, region-first design, and dense region-level annotations across distortion type, severity, quality, and pairwise comparison.

3 DISTORTION GRAPH

Distortion Graph represents paired images as region-grounded structures, with cross-image distortion relations and region attributes. PANDA predicts these graph elements through a multi-head architecture that combines image and region features.

  • Graph definition: A Distortion Graph contains region nodes, cross-image distortion edges, optional within-image scene edges, and region-level attributes.Distortion edges connect anchor-image regions to target-image regions; scene information is optional because it is orthogonal to DG semantics.
  • Graph properties: Matched regions are indexed across the anchor and target images, and distortion relations are ordered from anchor to target.The graph requires valid cross-image edges and interprets its comparative relation set relative to the target image.
  • PANDA architecture: PANDA takes an image pair and predicts each region’s comparative relation, distortion type, severity, and quality score using multiple DETR-like prediction heads.Categorical cross-entropy trains relation and distortion-type heads, while L1 loss trains score regression.
  • Feature processing: The architecture uses pretrained image features, panoptic region masks, and learnable convolutions to support variable numbers of regions with minimal compute.Region masks are combined with image features, while a learnable 1 × 1 convolution adapts pretrained features.
  • Degradation decoding: Cross-attention lets each region attend to the other image’s features, producing a representation of how that region compares with its counterpart.The resulting representation is pooled spatially and passed to four three-layer MLP prediction heads.
  • Benchmark context: PANDABENCH compares the proposed region-first task with prior distortion benchmarks, whose distinguishing gap is the lack of simultaneous region-first and comparative design.The benchmark summary frames PANDABENCH around this combined evaluation setting.

4 DATASET & BENCHMARK

PandaSet provides region-level paired distortion data, while PandaBench evaluates region-wise understanding across progressively mixed distortion settings. The dataset encodes diverse distortions, severities, quality scores, and comparative relationships for structured pairwise analysis.

  • PandaSet: PandaSet combines PSG and Seagull-100w to address the lack of region-level pairwise comparative distortion data.PSG contributes scene and panoptic region information, while Seagull-100w contributes real distortions simulated through ISP parameters and region-wise segmentation maps.
  • PandaSet: PandaSet is built from 2,200 unique images split into 2,000 training, 50 validation, and 150 test images, with up to 112 and an average of 18 regions per image.The images span diverse indoor and outdoor scenes, lighting conditions, and camera angles.
  • PandaBench: Across PandaBench, evaluation reports accuracy, precision, recall, and F1 for distortion, comparison, and severity, plus SRCC/PLCC for quality scores.Higher values are better; the supplied table captions identify Easy and Hard result tables for MLLM comparisons.
  • PandaSet: PandaSet contains over 500K image pairs with 15 distortion conditions and four severity levels, each region carrying a quality score.The dataset covers sensor-induced, equipment-failure, and weather distortions; clean regions are included as a severity category.
  • PandaSet: Distortion Graph comparative relationships classify region-pair quality differences as same, slightly better or worse, or significantly better or worse using TOPIQ score thresholds.Differences below 0.1 are labeled same; differences in ±[0.1, 0.3) are slight, and differences above 0.3 are significant.
  • PandaBench: PandaBench samples 300 pairs each for Easy, Medium, and Hard splits with increasing region-level difficulty from uniform to mixed distortions and severities.Easy pairs use one distortion type per image, Medium mixes distortions in one image, and Hard mixes them in both images.

5 EXPERIMENTS

Experiments show that current MLLMs struggle with region-wise distortion assessment, while PANDA and DG-based prompting improve structured distortion understanding.

  • MLLM Evaluation: Several open-source and closed-source MLLMs fail to reliably perform comparative analysis at the region level.Open-source models can classify distortion type and severity but still cannot perform region-wise comparative tasks; frontier models perform better.
  • Benchmark Results: PANDA achieves the best performance across Easy, Medium, and Hard settings and all four PANDABENCH tasks.DepictQA lags significantly behind PANDA and frequently hallucinates or omits regions because of context limitations.
  • Benchmark Results: Performance declines consistently from Easy to Hard, but PANDA exhibits the smallest drop in the Hard setting.The trend highlights the difficulty of fine-grained distortion understanding under complex degradations.
  • Distortion and Severity: Several models perform worse than random on severity classification in the Hard setting.Closed-source models outperform open-source counterparts on Easy distortion classification, but the gap narrows as difficulty increases.
  • Comparison and Quality: Open-source methods struggle with region-wise comparison and quality-score prediction, while several MLLMs approach chance under Hard evaluation.These results expose failure modes that simpler evaluation benchmarks may hide.
  • Showcase Application: DG in chain-of-thought prompting elicits emergent region-wise distortion understanding in GPT-5 Mini.GPT-5 Mini often overrides DG when pixel evidence conflicts with the predicted graph, but tends to trust DG when contradictory visual evidence is weak.

6 CONCLUSION

The paper presents Distortion Graph as a compact, interpretable representation for region-grounded pairwise image assessment. It concludes that DG, supported by PANDASET, PANDABENCH, and PANDA, improves fine-grained distortion reasoning and may generalize across comparative assessment settings.

  • Conclusion: PANDABENCH evaluates region-level distortion understanding, while PANDASET training or DG prompting substantially improves region-wise assessment.The conclusion reports clear gaps in current open-source models and identifies region-first representations as a direction for further work.
  • Conclusion: Distortion Graph represents pairwise perceptual differences with object-anchored nodes, attributes, and explicit comparative relations.Its edges localize where and how paired inputs diverge, unlike holistic embeddings or scalar quality scores.
  • Conclusion: DG is presented as a general formalism that can extend from image distortions to paired-video, forgery-detection, medical-imaging, and image-processor assessment.The paper positions this as a step toward a unifying comparative formalism.

B ABLATION STUDIES

The ablations examine PANDA’s architecture, evaluation granularity, query cost, prompting behavior, and DG’s extension to whole-image ranking.

  • Architecture Ablations: Decoder ablations show that DINOv2 features alone are insufficient for learning distortion relationships and attributes.The decoder enables each region to retrieve complementary information from its paired image.
  • Architecture Ablations: PANDA maintains a balance among model size, performance, and efficiency across backbone and Transformer-block design choices.The design-choice ablation compares backbone feature extractors and Transformer blocks.
  • Whole Image vs. Region-Wise: Region-wise reasoning is substantially harder than whole-image reasoning for MLLMs because of missing region-wise design and rigidity from fixed distortion settings.PANDABENCH exposes this granularity-dependent performance gap.
  • Cost Analysis: PANDA is significantly cheaper in computational cost and parameters than the compared methods for an image pair with 14 regions.Costs are reported on one NVIDIA V100 GPU with batch size 1 for open-source methods.
  • Prompting and MLLM Analysis: Prompt templates differ across open-source and closed-source MLLMs, with required tags and region descriptions supplied according to each method.Closed-source prompts provide image-pair regions with descriptions and bounding boxes; open-source prompts use method-specific tags and formats.
  • Generality of DG Representation: DG scores yield 78.83% ranking accuracy on KADID10k, while DG predicates yield 76.90%.On TID2013, PANDA reaches 77.8% with DG scores and 78.4% with DG predicates, outperforming listed MLLM baselines.

C.1 METHODS CONSIDERED, NOT REPORTED

Several considered open-source models were not reported because their outputs were unreliable on PANDABENCH. Their failures included inconsistent comparative reasoning, omitted regions, instruction-following problems, and insensitive distortion predictions.

  • The paper therefore omitted Q-Instruct, Co-Instruct, and Janus-Pro-7B from the main tables because their outputs were unreliable.Their behaviors are documented separately with examples and discussion.
  • Q-Instruct was excluded because stacking two images did not reliably preserve comparative behavior when their order changed.It was designed for single-image distortion analysis rather than comparative assessment.
  • Co-Instruct struggled with PANDABENCH’s multi-instruction setting, often omitting regions or repeating one region until exhausting the output budget.It also failed to generalize beyond its supervised fine-tuning regime.
  • Janus-Pro-7B predicted nearly identical distortion and severity labels for most regions on the Medium split.This behavior indicates insensitivity to degradation in the inputs and made its performance uninformative.

C.2 OPEN-SOURCE METHODS CONSIDERED, REPORTED

Reported open-source methods were evaluated with region-wise prompting because variable region counts and limited output lengths hindered all-region generation. Performance was generally limited, especially on PANDABENCH’s Hard split.

  • The open-source evaluation included Q-Insight, Q-SiT, Gemma 3-27B, Seagull, and DepictQA under prompt type (a).Gemma 3-27B was the only method listed that was not distortion-specific.
  • Open-source methods were queried separately for each region in each image pair because their maximum generated-token limits could not cover variable region counts.This required a separate forward pass for every region pair.
  • Performance was generally limited, particularly on the Hard split, indicating weak region-wise image understanding for distortion analysis.The limitation persisted across the evaluated open-source methods.
  • Seagull’s region-first design did not prevent difficulties with instruction following and comparative assessment.Its region-wise formulation therefore did not ensure successful pairwise evaluation.

C.3 CLOSED-SOURCE METHODS CONSIDERED, REPORTED

Closed-source frontier MLLMs were evaluated across PANDABENCH’s progressively harder Easy, Medium, and Hard splits. Their setup was less constrained by instruction following and output length, with 8192 tokens sufficient for all regions.

  • Four closed-source frontier MLLMs were evaluated: GPT-5 Nano, GPT-5 Mini, GPT-4o, and Gemini 2.5 Pro.
  • Closed-source models exhibited stronger instruction-following abilities than open-source models and were not constrained by output length.The authors found that 8192 tokens sufficed to cover all regions in PANDABENCH images.
  • PANDABENCH difficulty increases from Easy to Hard through progressively more region-specific degradation patterns.Easy uses one uniform distortion with varying severity, while Medium and Hard introduce increasingly varied regional distortions.
  • Table 9 reports standard deviation for the accuracy metric across all four PANDABENCH tasks over three independent runs.Standard deviations for precision, recall, and F1 are omitted but reported to follow similar trends.
  • In the Hard split, each region can have a different degradation, such as noise affecting the ground in one image but not its pair.

E HYPERPARAMETER SENSITIVITY ANALYSIS

The hyperparameter analysis searched λ values by cross-validation and used baseline comparisons for early stopping. PANDA’s selected configuration was trained jointly over all regions of each image pair.

  • Each λ was selected from {0.01, 0.1, 1.0, 5.0} using cross-validation.Baseline values with every λ set to 1.0 guided early stopping, with full training reserved for combinations that improved on baseline.
  • The final objective used λ1 = 0.1, λ2 = 1.0, λ3 = 0.1, and λ4 = 1.0, with each coefficient shared by its anchor and target heads.
  • PANDA was trained for 30 epochs with batch size 6 on 8× NVIDIA V100 32GB GPUs.
  • PANDA processes all regions for an image pair simultaneously.This design contrasts with region-by-region querying used for constrained open-source methods.
  • The analysis reports experiment outcomes in Figure 16, marking significantly worse runs in gray and the top five settings with colored × marks.

F.1 LIMITATIONS & FUTURE WORK

The paper identifies limitations in PANDA’s baseline design and PANDASET’s construction, while outlining reproducibility resources and broader future applications for DG.

  • Limitations: PANDA is intentionally a minimal baseline that leaves room for improvement, particularly in handling complex regions.The authors propose generating DG as an intermediate reasoning step before final region-wise distortion answers.
  • Dataset construction: PANDASET combines natural scenes and real-world ISP distortions with synthetic distortions to enable deterministic scores, comparative labels, and controllable benchmark difficulty.The synthetic design supports Easy-to-Hard difficulty variation in PANDABENCH.
  • Dataset construction: Comparative relations in PANDASET may inherit perceptual biases from the underlying TOPIQ image-quality model.The authors note that a similarly large real-world dataset with human-annotated region-level comparative relations would require substantial annotation effort.
  • Future work: DG is proposed as a general formalism that could extend beyond distortion analysis to broader comparative tasks in vision and multimodal settings.The paper positions PANDASET and PANDABENCH as foundations for region-grounded comparative quality assessment.
  • Reproducibility: The work provides reproducibility details covering architecture, experiments, hyperparameters, compute requirements, and planned public release of code, models, dataset, and benchmark.PANDASET is built from PSG and Seagull-100w, which provide region-level segmentation and scene information.
Loading 2604.11004v1…