Source-linked AI summary
TGSR: Text-Grounded Diffusion for Faithful Image Super-Resolution
Zhiming Hu, Ran Zhang, Angela Ye, Raghav Goyal, Faraz Ali, Tristan Ty Aumentado-Armstrong, Samrudhdhi B. Rangrej, Konstantinos G. Derpanis, Iqbal Mohomed, Alex Levinshtein
TL;DR
Diffusion-based SR can generate text that is absent from the ground truth, altering scene meaning. TGSR selectively detects and suppresses likely hallucinated text during generation. It improves text fidelity across multiple backbones without degrading perceptual quality, with OCR precision increasing by up to 17.68 points.
Problem
Diffusion-based SR can generate text absent from the ground truth, and such hallucinations can alter scene meaning.
Source: §1 · p. 1Method
TGSR uses intermediate diffusion features to detect likely hallucinated text and steer unreliable regions toward an upsampled LR reference at test time.
Source: §1 · p. 1Results
Across multiple diffusion-based SR backbones, TGSR improves text fidelity without degrading perceptual quality, with OCR precision increasing by 17.68 points.
Source: §5.2 · p. 6, §6 · p. 7Takeaways & Limitations
The results highlight the importance of selective control in generative super-resolution.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Diffusion models have achieved remarkable success in image super-resolution (SR) by leveraging strong generative priors to enhance perceptual quality. However, for textual content, these priors often produce text hallucinations: non-existent or distorted text, which alter scene semantics and compromise model reliability. While recent methods focus on enhancing text generation, they do so indiscriminately, leaving hallucinations fundamentally unresolved. We argue the core problem is a lack of selective control. Therefore, we reformulate text SR as a selective generation problem: text is resolved only when reliable generation is expected, while other scene content is fully super-resolved. We introduce TGSR (Text-Grounded Super-Resolution), a test-time framework that progressively identifies and suppresses likely hallucinated text regions during generation. To enable this, we train dedicated detectors on a large-scale dataset curated from diverse diffusion-based SR outputs. TGSR consistently improves text fidelity across multiple backbones, increasing recognition precision by up to 17.68 points without degrading perceptual quality, underscoring the value of selective control in generative SR.
1 Introduction
Diffusion-based SR can hallucinate text when visual evidence is unreliable, altering scene meaning. TGSR selectively suppresses likely hallucinations at test time while preserving other image content, and experiments report improved text fidelity across backbones without degrading perceptual quality.
1 Introduction
- Text hallucinations can arise when generative priors override weak LR evidence or when text falls outside the prior distribution, altering scene meaning.Such errors can undermine reliability in applications where textual fidelity matters and may mislead downstream automated systems.Source: §1 · p. 1
- TGSR reframes text SR as selective generation, using intermediate diffusion features to identify unreliable text and steer those regions toward an upsampled LR reference at test time.It adds a lightweight detection head to a frozen backbone and leaves other regions available for super-resolution.Source: §1 · p. 1
- SA-Text-HTD provides over 3.2M cropped text instances with hallucination labels for training and evaluating detection in super-resolved images.The labels are generated by comparing MLLM-recognized text with ground-truth transcriptions.Source: §1 · p. 2
- TGSR significantly improves text fidelity while preserving overall perceptual quality across multiple diffusion-based SR backbones.The results emphasize the value of selective control in generative super-resolution.Source: §1 · p. 2
2 Related Work
Prior scene-text SR methods focus on cropped text reconstruction, while diffusion SR generally refines images uniformly. Text-aware SR adds recognition conditioning but typically requires backbone finetuning and does not explicitly account for text-recovery reliability.
Scene Text Image Super-Resolution (STISR)
- Scene Text Image Super-Resolution targets visual quality and OCR accuracy through localized character reconstruction, often using recognition guidance or generative priors.The paper distinguishes its focus on reliability-aware correction within full-image generative SR.Source: §2 · p. 2
Diffusion-Based Image Super-Resolution
- Most diffusion-based SR approaches apply generative refinement uniformly, which can hallucinate characters where underlying signals are ambiguous.The paper instead uses hallucinated-text detection to provide a reliability signal for selective correction.Source: §2 · p. 2
Text-Aware Super-Resolution
- Text-aware SR methods condition diffusion on recognized text but require backbone finetuning and restore detected text indiscriminately, potentially amplifying hallucinations in unreliable regions.The paper proposes reliability-aware restoration rather than uniform text enhancement.Source: §2 · p. 2
3 Hallucinated Text Detection Dataset
SA-Text-HTD is curated by degrading ground-truth images, super-resolving them with multiple diffusion models, and labeling cropped text against ground-truth transcriptions. It adds fine-grained hallucination labels absent from existing text-aware SR and text-spotting datasets.
3.1 Data Curation
- The curation pipeline creates LR inputs under three degradation setups and runs four diffusion-based SR models to generate outputs for text-region detection.The setups use Real-ESRGAN-based degradations adopted in prior work.Source: §3.1 · pp. 2–3, Figure 2 · p. 3
- Text crops are recognized with Qwen2.5-VL-72B and labeled normal when the recognized text matches the corresponding ground-truth transcription, otherwise hallucinated.The crops use ground-truth text bounding boxes.Source: Automatic Hallucination Label Generation · p. 3
- Qwen2.5-VL-72B labels align with PaddleOCRv5 on 94.32% and Gemma4-31B on 97.12% of samples in the reported comparison.On disputed PaddleOCRv5 crops, Qwen2.5-VL-72B correctly recognizes 85.3% versus PaddleOCRv5’s 14.6%.Source: Automatic Hallucination Label Generation · p. 3
3.2 Dataset Analysis
- Existing text-aware SR and text-spotting datasets provide text annotations but lack explicit hallucination labels; SA-Text-HTD adds fine-grained labels and expands SA-Text by a factor of 12.The expansion comes from four SR models and three degradation levels.Source: §3.2 · p. 3, Table 1 · p. 3
4 Methods
TGSR detects unreliable text during diffusion SR and uses test-time restoration guidance to selectively correct those regions. Its detector reuses the frozen SR model’s features, and its guidance updates the clean latent without changing model parameters.
4.1 Hallucinated Text Detection
- The detector reuses multi-scale features from the SR model’s frozen U-Net, training only a lightweight text-spotting head for localization and hallucination classification.A dedicated head uses character-level features to predict instance-level hallucination scores; focal loss addresses class imbalance.Source: §4.1 · p. 4
4.2 Mitigating Text Hallucinations
- At each denoising step, the detector creates a binary mask of hallucinated regions, and a masked latent-space MSE steers those regions toward an encoded non-generative reference.MUSIQ and CLIPIQA terms on the decoded image stabilize global perceptual quality and discourage degradation outside the mask.Source: §4.2 · p. 5
- After computing the loss, TGSR takes one gradient step on the clean latent without updating model parameters, then uses the updated latent in the reverse diffusion process.When no hallucinated text is detected, perceptual terms remain active and the gradient step is damped by ρ < 1.Source: §4.2 · p. 5
5 Evaluation
Across diffusion-based SR backbones, TGSR improves text recognition while maintaining image quality. The evaluation tests its detector and guidance on real-world and synthetically degraded text images, and ablations examine guidance choices.
5.1 Experimental Setting
- Evaluation uses Real-Text and SA-Text-Test for guidance, and SA-Text-HTD-trained detectors are tested on Real-Text-HTD, including unseen SR backbones.Real-Text has 847 real-world HR-LR pairs; SA-Text-Test has 1,000 HR images with three Real-ESRGAN degradation levels. Guidance is evaluated with full- and no-reference image-quality and text-recognition metrics.Source: §5.1 · p. 5
5.2 Main Results
- OCR-F1 rises 5.66 points (47.23 to 52.89) with TeReDiff +TGSR, alongside +4.87 MUSIQ and +0.054 CLIPIQA; OCR precision gains 17.68 points.The authors report comparable recall and FR-IQA metrics, with similar text-recognition trends across SUPIR and PiSA-SR; SA-Text-Test gains extend across all three degradation levels.Source: §5.2 · pp. 5–6, Table 2 · p. 6
- On Real-Text, TGSR suppresses text hallucinations and preserves background details compared with other diffusion-based SR models.Source: Figure 4 · p. 6
- On Real-Text-HTD, the detector more than doubles baseline precision (60.1–65.3 vs. 24.2–28.2) and generalizes to SR backbones unseen during training.The deployed detector operating point is τ=0.3, and the study prioritizes precision because false positives can blur legible text.Source: §5.2 · p. 6, Table 3 · p. 7
- TGSR improves precision by +24.7 points on small text and more than doubles it on extremely small text (15.0→36.5), with a modest recall cost.Hallucinations concentrate in small text, where low-resolution evidence is weakest.Source: §5.2 · p. 7, Table 4 · p. 7
5.3 Ablation Studies
- Latent-space guidance improves OCR-P by 2.3 points and OCR-F1 by 0.4 points over RGB guidance at comparable NR-IQA; adding MUSIQ and CLIPIQA yields the best OCR-F1 (52.89) and NR-IQA trade-off.The combined perceptual terms improve MUSIQ by +5.9 and CLIPIQA by +0.066 over the MSE-only variant.Source: §5.3 · p. 7, Table 5 · p. 7
6 Conclusion
TGSR treats text super-resolution as selective generation, suppressing likely hallucinated text while fully super-resolving other scene content. It improves text fidelity across multiple diffusion-based SR backbones without degrading perceptual quality.
6 Conclusion
- TGSR selectively suppresses likely hallucinated text during generation while allowing other scene elements to be fully super-resolved.The approach is framed as reliability-aware selective generation.Source: §6 · p. 7
- Experiments show improved text fidelity across multiple diffusion-based SR backbones without degrading perceptual quality.Source: §6 · p. 7
- The reliability-aware approach may extend to identity-sensitive visual elements such as faces, license plates, barcodes, and QR codes.Source: §6 · p. 7
TGSR: Text-Grounded Diffusion for Faithful
The supplement provides dataset, experimental-setup, and additional-results details, alongside further qualitative examples.
TGSR: Text-Grounded Diffusion for Faithful
- The supplement adds details about the hallucinated text detection dataset, including its statistics.Source: A Overview · p. 10
- It describes the training and inference setup used in the experiments.Source: A Overview · p. 10
- It presents additional quantitative results for test-time adaptation and the detector, plus more qualitative examples.Source: A Overview · p. 10
B.1 Statistics on the Hallucinated Text Instances
The supplement reports how degradation and SR-model choices shape hallucination statistics, and documents the detector’s losses, implementation, and computing setup.
B.1 Statistics on the Hallucinated Text Instances
- 76.71% of the 271,226 PASD text regions were labelled hallucinated under its matched Real-ESRGAN degradation setting, versus 14.03% normal regions under StableSR’s setting.The passage attributes the discrepancy to the PASD checkpoint being trained on its own degradation setting.Source: §B.1 · p. 10
- Varying degradation pipelines and SR models produces diverse hallucination patterns and increases the instances available for robust detector training and evaluation.Source: §B.1 · p. 10
C.1 Loss Functions for Hallucinated Text Detector
- The detector combines encoder supervision for coarse boxes with decoder supervision for fine text polygons and recognition outputs.The described losses include classification, box regression, generalized IoU, polygon regression, character recognition, and hallucination classification.Source: §C.1 · p. 10
C.2 Additional Implementation Details
- The detector is trained at timestep 0 with AdamW, a 1 × 10^-4 learning rate, batch size 128, and 100K iterations.Source: §C.2 · p. 10
- For each backbone, the diffusion U-Net is frozen and only the text-spotting head is trained on its multi-scale features.SUPIR uses an identity-initialized 1×1 adapter; guidance is applied at every sampling step for TeReDiff and SUPIR, but once before decoding for PiSA-SR.Source: §C.2 · p. 11
C.3 Computing Infrastructure
- TeReDiff and SUPIR experiments used an eight-H100-GPU node, while PiSA-SR used one RTX 6000 Ada GPU; inference latency was measured on one GPU.Source: §C.3 · p. 11
C.4 Evaluation Metric Details
TGSR improves recognition under increasingly difficult degradation while preserving image-quality metrics; the supplement also compares TGSR-lite and explains the evaluation metrics.
C.4 Evaluation Metric Details
- OCR-P is the fraction of predicted instances that are correct, and OCR-R is the fraction of ground-truth instances correctly recovered; OCR-F1 is their harmonic mean.Precision is emphasized because suppressing hallucinations can remove incorrect predictions, while recall reflects the cost to legible text.Source: §C.4 · p. 11
D.1 Results on Real-Text with TGSR-lite
- TGSR-lite has higher OCR-F1 than TGSR on PiSA-SR (51.10 vs. 50.98) and SUPIR (43.02 vs. 42.58), but falls below the base model on MUSIQ and CLIPIQA across all three backbones.The passage reports that TGSR exceeds the base model on both no-reference metrics, while both variants remain marginally below it on QAlign.Source: §D.1 · p. 12
D.2 Results on SA-Text-Test
- Under level-3 degradation, TeReDiff OCR-P rises from 27.73 to 49.12, while TGSR and TGSR-lite achieve the highest OCR-F1 at every degradation level.The passage reports preserved image-quality metrics; TGSR-lite remains below the base model on both NR-IQA metrics in every setting.Source: §D.2 · p. 12
D.3 Additional Ablation Studies
The ablations show how guidance objectives, perceptual-loss weights, and empty-mask damping trade off recognition and image quality, while the full TGSR objective improves quality metrics at added inference cost.
D.3 Additional Ablation Studies
- Latent-space guidance is adopted because it improves OCR-P by 2.33 points and OCR-F1 by 0.40 points over RGB guidance under MSE alone.With perceptual terms added, RGB-space MUSIQ guidance lowers OCR-P by 6.59 points and OCR-F1 by 1.92 points.Source: §D.3 · p. 12
- Perceptual-loss weights above the defaults degrade recognition precision, as the quality metrics reward sharp plausible detail that resembles hallucination.At λm=10−2, OCR-P falls from 70.91 to 64.20; at λc=10−2, it falls from 69.66 to 54.12.Source: §D.3 · p. 12
- At ρ=0.3, empty-mask damping attains the table’s best OCR-F1 while recovering 4.44 MUSIQ points and 0.047 CLIPIQA points between ρ=0 and ρ=1.Increasing ρ improves perceptual quality but reduces full-reference fidelity; at ρ=1, PSNR falls from 25.15 to 24.83 and LPIPS rises from 0.284 to 0.291.Source: §D.3 · p. 13
- TGSR-lite adds 16–35% runtime for TeReDiff and SUPIR and 96% for PiSA-SR, with at most 0.4 GB memory overhead.For PiSA-SR, about half the overhead comes from reference-related processing and half from the detection pass.Source: §D.3 · p. 13
- TGSR adds at most 24% runtime and 12% memory while improving no-reference image quality and OCR performance on all three backbones.Compared with TGSR-lite, OCR-F1 stays within 0.5 in every block, while TGSR improves NR-IQA metrics, including QAlign.Source: §D.3 · p. 13
E.1 Detection Performance on Real-Text
On Real-Text, TGSR raises recognition precision and OCR-F1 across three backbones while preserving perceptual quality. Detector training on more model outputs and degradations favors precision, with a recall trade-off on both evaluation settings.
E.1 Detection Performance on Real-Text
- On Real-Text-HTD, including more SR model outputs improves hallucination-detection precision, and more degradations improve it further at the cost of recall.The authors use AllModel-AllDeg with a 0.3 confidence threshold for TeReDiff test-time adaptation.Source: §E.1 · pp. 13–14
- Across all three backbones on Real-Text, TGSR raises OCR-P and OCR-F1 while preserving perceptual quality.The table compares base SR models with their test-time adapted versions, including TGSR-lite.Source: Table 8 · p. 14
E.2 Detection Performance on SA-Text-Test
- On SA-Text-Test, more model outputs and degradations similarly benefit detection precision, while false hallucination predictions can suppress legible text and harm readability and recall.AllModel-AllDeg achieves the highest detection precision in the table.Source: §E.2 · p. 14
F Additional Qualitative Examples
The examples show TGSR suppressing hallucinated text while retaining plausible structures and the perceptual quality of legible text and backgrounds; TGSR-lite has comparable visual quality. The accompanying results indicate gains in recognition across degradation levels.
F Additional Qualitative Examples
- When inputs lack sufficient visual cues, diffusion SR often hallucinates text, whereas TGSR suppresses those hallucinations while preserving plausible structures.The qualitative examples cover Real-Text and SA-Text-Test.Source: F Additional Qualitative Examples · p. 14, Figure 7 · p. 20, Figure 8 · p. 21
- TGSR-lite maintains visual quality comparable to TGSR while offering efficiency gains.Figure 9 compares the variants on Real-Text and SA-Text-Test.Source: F Additional Qualitative Examples · p. 14, Figure 9 · p. 22
- The dataset examples contrast readable text such as “TEAM” with small or large non-readable text where SR outputs show more hallucination.Other readable examples are omitted because their SR outputs are nearly identical and non-hallucinatory.Source: Figure 5 · p. 15
- Producing faithful text super-resolution becomes increasingly difficult across the dataset’s higher degradation levels.The passage describes three levels of degradation in SA-Text-Test.Source: Figure 6 · p. 16
- Across three backbones and degradation levels, TGSR raises OCR-P and OCR-F1 while preserving perceptual quality, with precision gains growing as degradation severity increases.The results cover all three degradation levels on SA-Text-Test.Source: Table 9 · p. 17