Source-linked AI summary
Fine-Grained Anomaly Perception in Wild UGC-Enhanced Images: A Comprehensive Dataset and Difference-Fusion Framework
Yan Zhong, Gefei Chen, Qiufang Ma, Zhen Wang, Zhiwei Fan, Lei Shi, Tingting Jiang
TL;DR
Existing IQA and VAD methods do not adequately capture localized enhancement-induced anomalies in real-world UGC, especially when references are non-ideal or missing. The paper defines UEAP, introduces the UEAP-4k paired-image benchmark, and proposes DFAP-UGC with LADTP; experiments report stronger performance than adapted detectors.
Problem
Existing IQA and VAD methods do not adequately support paired local detection, categorization, and severity assessment of enhancement-induced UGC anomalies.
Method
The paper defines UEAP, builds UEAP-4k, and proposes DFAP-UGC with difference fusion, dense spatial queries, regional verification, quality-aware ranking, and LADTP training.
Results
DFAP-UGC achieves stronger performance than adapted detectors, and ablations validate the contribution of its main training and ranking designs.
Takeaways & Limitations
UEAP-4k and DFAP-UGC establish a benchmark and tailored framework for paired-image perception of local UGC enhancement failures.
Abstract
from arXiv · showhide
Image enhancement and restoration have become standard back-end operations on short-video and social media platforms to boost UGC visual experience. Yet these processes inevitably introduce visual anomalies--especially in faces, texts, and textures--that directly undermine perceptual fidelity and viewer trust. While existing IQA methods perform well on classic distortions, they target holistic quality assessment and fail to capture the specific, localized anomalies caused by enhancement algorithms in real-world UGC. To bridge this gap, we formally define a new task-quality Anomaly Perception for UGC image Enhancement (UEAP), and contribute the first UEAP benchmark dataset, named UEAP-4k, curated from the real business scenarios. It provides fine-grained annotations for anomaly categories, localization and severity levels. Furthermore, we propose a Difference-Fusion Anomaly Perception Method (DFAP-UGC) for wild UGC-enhanced images, which leverages explicit problem-reference difference fusion with dense spatial querying, regional verification, and quality-aware ranking, enabling robust anomaly identification in challenging scenarios. To handle the inherent coupling of subtasks in this new task, we propose a Locality-Aware Dynamic Task Prioritization (LADTP) training strategy that enables effective end-to-end learning and eliminates multi-stage overhead. Extensive experiments show that our method outperforms baselines adapted from classical approaches for this task, validating the value of this dataset and the superior of DFAP-UGC for robust UGC-enhanced image anomaly perception. Code and data will be public.
1 Introduction
UGC enhancement can introduce localized anomalies that existing global IQA and single-image VAD do not adequately characterize. The paper defines UEAP, builds the UEAP-4k benchmark, and proposes DFAP-UGC with LADTP for paired-image anomaly perception.
- UGC enhancement introduces local content, privacy, color, and texture anomalies that can undermine user experience and risk complaints or brand damage.
- IQA generally lacks fine-grained localization, anomaly classification, and regional severity assessment, while its full-reference setting assumes unavailable high-quality references.
- VAD analyzes deviations in a single test image and therefore does not compare post-enhancement images with non-ideal pre-enhancement references to separate new artifacts from original defects.
- UEAP is a paired-image task that detects local enhancement failures and outputs anomaly bounding boxes, categories, and severity scores without requiring ideal references.
- UEAP-4k is the first benchmark for this task, containing paired pre-/post-enhancement images with annotations for localization, category, and severity.The dataset is drawn from public video-content sources and uses multiple annotators with label reconciliation.
- DFAP-UGC fuses problem, reference, and difference features, then uses dense spatial queries, regional verification, and quality-aware ranking for subtle local artifacts.
- LADTP dynamically prioritizes subtasks using localization quality and task difficulty, suppressing classification and scoring weights when bounding-box regression is poor.It uses GIoU loss as a real-time localization measure and exponential moving-average smoothing for stable training.
2 Related Works
Related work includes IQA methods for perceptual quality and VAD methods for deviations from normal visual distributions. UEAP instead evaluates paired pre-/post-enhancement images for localized, categorized, and severity-rated enhancement failures.
- Image Quality Assessment: IQA studies perceptual image quality in full-, reduced-, and no-reference settings, with no-reference methods suited to images lacking pristine references.
- Visual Anomaly Detection: VAD methods detect deviations using reconstruction errors, feature distributions, memory retrieval, or teacher-student feature discrepancies.
- Visual Anomaly Detection: Unlike single-image VAD, UEAP compares pre-enhancement UGC images with enhanced counterparts and requires localization, category prediction, and severity estimation.
- UEAP-4k provides the first real-world benchmark for this paired anomaly-perception setting, alongside the tailored DFAP-UGC framework.
3 Task Definition and Datasets
UEAP defines anomaly perception as identifying newly introduced, instance-level quality failures by comparing pre- and post-enhancement UGC images. UEAP-4k operationalizes this task with paired data, fine-grained annotations, and unified labels covering anomaly location, category, and severity.
- Task Definition: UEAP compares a non-ideal original UGC image with its enhanced counterpart to identify newly introduced local quality failures.Unlike full-reference IQA, it produces instance-level abnormal changes rather than a global quality score.
- Task Definition: Each anomaly instance is represented by a bounding box, category, and severity score ranging from −5 to −1.The three categories are portrait effect, text effect, and texture effect anomaly; lower scores indicate more severe degradation.
- Dataset Construction: UEAP-4k uses paired pre- and post-enhancement images with human annotations for anomaly localization, category, and severity.The dataset was constructed from public sources and a simulated short-video processing pipeline involving compression, encoding, decoding, and enhancement.
- Dataset Construction: Three annotators independently marked only target-image regions that became worse than the reference and tightly covered affected areas.They were instructed to avoid differences caused only by normal enhancement and used a consistent desktop annotation interface.
- Label Unification: Greedy label unification clusters same-category boxes using IoU > 0.5, then averages coordinates and severity scores.The procedure groups records by sample and category before assigning the majority textual label.
- Dataset Statistics: UEAP-4k contains 4,222 paired samples and 13,662 anomaly boxes, with portrait, texture, and text anomalies comprising 58.0%, 23.1%, and 18.9% of instances.Samples contain 3.24 boxes on average, with counts ranging from 1 to 33.
- Dataset Statistics: The dataset has a long-tailed box distribution: 75% of samples contain at most 4 boxes and 90% contain at most 6.A small number of hard cases contain many local artifacts.
4 Methodology
DFAP-UGC performs paired anomaly perception by fusing problem, reference, and difference features for dense local prediction, regional verification, and quality-aware ranking. LADTP coordinates the coupled objectives by adapting task weights to training difficulty and localization quality.
- Difference-Fusion Detection: DFAP-UGC fuses problem, reference, and absolute-difference features into 784 dense tokens for local anomaly detection.The main path uses Stage 3 features and preserves a 28 × 28 detection grid before Transformer processing.
- Dense Predictions: Each dense query predicts anomaly categories, normalized bounding boxes, and severity scores, with severity kept separate from detection confidence.The three foreground categories are represented by class logits, while severity is predicted on its own scale.
- Task-Aligned Assignment: Dense task-aligned assignment selects the top-5 in-box queries per ground-truth anomaly using classification probability and IoU.When no query center lies inside a box, the nearest query to the box center is selected, producing multiple positives per anomaly.
- Regional Verification: A regional verification branch samples 3 × 3 RoI features from higher-resolution Stage 2 representations and uses hard negatives to suppress salient non-anomalous regions.The regional classifier combines dense query features, pooled regional features, and predicted box coordinates.
- Quality-Aware Ranking: A dense-IoU quality branch estimates localization reliability for candidate ranking without directly reshaping the localization head.Its targets use the maximum class-specific IoU above 0.3; inference combines main, regional, and quality probabilities before class-wise NMS.
- LADTP Training: LADTP balances classification, localization, severity, regional, and quality losses according to epoch-level difficulty and localization quality.It prevents secondary objectives from dominating while localization is unreliable, then increases their effective weights as localization improves.
5 Experiments
Experiments on UEAP-4k compare DFAP-UGC with adapted detectors and analyze component ablations. DFAP-UGC performs best overall, while qualitative cases and ablations support the value of paired difference fusion and its associated designs.
- Experimental Settings: UEAP-4k evaluation uses 4,222 paired samples, including 844 untouched test images with 2,729 ground-truth anomaly boxes.The test protocol counts a true positive only when category and IoU >= 0.5 are both correct.
- Comparison with Adapted Baselines: DFAP-UGC achieves 0.4328 AP50, 0.2038 mAP, and 0.8106 R50, outperforming the strongest Faster R-CNN baseline on AP50 and mAP.The reported improvements are 5.70 points in AP50 and 3.18 points in mAP, with substantially improved recall.
- Comparison with Adapted Baselines: YOLOv12n and YOLOv1 variants underperform in AP and recall, while Faster R-CNN variants remain competitive in severity MAE.The comparison attributes the one-stage detectors’ weaker AP and recall to difficulty handling small, fine-grained enhancement artifacts.
- Ablation Study: Full DFAP-UGC tops AP50, mAP, and R50, ranks second in MAE, and provides the best overall balance across the ablation variants.Removing dynamic weighting reduces mAP, R50, and MAE; removing TAL causes the largest AP loss, and single-image input lowers AP.
- Ablation Study: All ablations contribute positively, but none match the full model’s mAP and recall; removing paired reference and difference features lowers AP.The ablation findings support the contribution of the training, assignment, fusion, regional, regression, and quality-ranking designs.
- Inference Case Study: Qualitative cases show DFAP-UGC predictions are more consistent with annotated local anomalies than adapted detectors, especially for subtle distortions requiring reference comparison.The cases also show fewer missed or noisy detections in UGC scenes.
6 Conclusion
The paper introduces UEAP and the UEAP-4k paired-image benchmark for local failures caused by UGC enhancement, then proposes DFAP-UGC for this task. Experiments show stronger performance than adapted detectors, while ablations validate the main training and ranking designs.
- Conclusion: UEAP is a paired-image anomaly perception task for identifying local failures introduced by UGC enhancement.The task is accompanied by the UEAP-4k benchmark and the DFAP-UGC framework.
- Conclusion: DFAP-UGC combines problem-reference difference fusion, dense spatial queries, regional verification, quality-aware ranking, and locality-aware dynamic task prioritization.These components are presented as the framework’s principal design elements.
- Conclusion: Experiments show DFAP-UGC achieves stronger performance than the adapted detectors, and ablations validate the contribution of its main training and ranking designs.The conclusion reports both comparative performance and component-level validation.