TOLA: Text-aware One-Step Latent Adaptation for Diffusion-based Text Image Super-Resolution

• Blind TSR must restore fine visual structures and character-level semantics because a missing, merged, or hallucinated stroke can change transcription despite visual sharpness.Source: Introduction · p. 1
• Multi-step diffusion adds computation and latency, while repeatedly injected incorrect text priors may reinforce image-text predictions and amplify recognition errors.Source: Introduction · p. 1
• TOLA constructs confidence-weighted TransOCR–MoM conditioning once, suppresses uncertain OCR predictions, and uses latent residual correction to restore missing or distorted stroke details without another reverse trajectory.Source: Introduction · p. 2
• TOLA achieves state-of-the-art results across reported metrics on CTR-TSR-Test ×4 and RealCE-200, outperforming 200-step DiffTSR by 4.2137 dB in PSNR.It also reduces FID by 12.2271 and inference time from 9.9548s to 0.1319s, a 75.5× speedup.Source: Introduction · p. 2

• TSR methods model strokes, sequences, layout, deformation, and location, using recognition, style, or structure priors, but cues from degraded inputs remain unreliable.Source: Related Work · p. 2
• Diffusion TSR conditions denoising on text-related cues, but methods including DiffTSR, PRISM, DualTSR, and TeReDiff retain iterative refinement [Zhang et al., 2024; Min et al., 2026].Source: Related Work · p. 2
• Generic one-step diffusion restoration shortens trajectories but targets generic SR, whereas TOLA adapts pretrained DiffTSR to one IDM evaluation with one-time confidence-weighted conditioning and latent correction [Song et al., 2021; Yue et al., 2023; Wang et al., 2024b; Tai et al., 2026; Dong et al., 2025; Xu et al., 2026].Source: Related Work · pp. 2–3

Overview

• The overview forms one condition from confidence-weighted text tokens and LR-image latents, makes one IDM noise prediction, corrects the clean latent with LRC, and decodes without TDM or iterative reverse updates.Source: Overview · p. 3

Text-Aware One-Step Adaptation

• Text-aware one-step adaptation evaluates the VAE encoder/decoder, TransOCR, MoM, IDM, and LRC once each, omitting TDM; DiffTSR evaluates IDM and TDM 200 times each.Confidence-weighted embeddings condition the single IDM prediction, and LRC corrects local clean-latent errors after noise-to-clean conversion.Source: Text-Aware One-Step Adaptation · p. 3

Training Objective

• Training uses pixel, perceptual, and recognition objectives, with OCR gradients updating LoRA and LRC; no latent ℓ1, noise-prediction, distillation, or teacher loss is used.Source: Training Objective · p. 4

Experimental Setup

• Evaluation uses all 8,089 CTR-TSR-Test ×4 pairs under a unified protocol, with non-diffusion methods retrained on the same CTR-TSR training split and paired comparisons using identical per-image noise.RealCE-200 uses matched official 13 mm and 52 mm crops and is excluded from training, validation, and checkpoint selection.Source: Experimental Setup · pp. 4–5

Main Results

• TOLA surpasses DiffTSR in all nine CTR-TSR-Test metrics, gaining 4.2137 dB PSNR and 0.0756 TransOCR ACC while reducing FID by 12.2271.PaddleOCR ACC/NED rise from 0.3970/0.5873 to 0.4626/0.6585.Source: Main Results · p. 5
• Across five selected CTR-TSR-Test examples and varied RT50 degradations, comparisons show more coherent strokes and closer character shapes and layouts.The examples cover Chinese, English, and numeric text; RT50 conditions include blur, illumination, viewpoint, resolution, and background variation.Source: Main Results · p. 5, Figure 3 · p. 6, Figure 4 · p. 6
• On RealCE-200, TOLA leads all six metrics, exceeding DiffTSR by 1.5953 dB PSNR, 4.50 points TransOCR ACC, and 8.00 points PaddleOCR ACC.The benchmark contains 200 pairs and is excluded from training and checkpoint selection.Source: Main Results · p. 5, Table 2 · p. 5
• TOLA uses 297.497 G MACs and 0.1319 seconds per image, versus DiffTSR’s 59,213.970 G MACs and 9.9548 seconds; its saved adapted state has 2.46M parameters.Timing is synchronized batch-1 inference on 100 fixed CTR images using one NVIDIA RTX PRO 6000, excluding loading; VAEs and text modules are excluded from restoration-module parameters.Source: Main Results · p. 5, Table 3 · p. 7, Figure 5 · p. 7

Ablations and Diagnostic Analyses

• LoRA drives one-step adaptation, while LRC adds 0.3847 dB PSNR and reduces LPIPS/DISTS by 0.0121/0.0017; a parameter-matched plain correction loses 0.8249 dB PSNR.LRC's ACC change is not significant (p = 0.584), and IDM-only LoRA remains closer to Full than VAE-only LoRA.Source: Ablations and Diagnostic Analyses · p. 5, Table 4 · p. 8
• Confidence weighting improves over uniform confidence, and diagnostics show TOLA outperforming DiffTSR across confidence bins and text categories while reducing substitutions, deletions, and insertions.Confidence weighting adds 0.1908 dB PSNR and 0.0134/0.0138 TransOCR ACC/NED, while reducing LPIPS/DISTS/FID by 0.0213/0.0160/4.2766; correct-versus-incorrect prior exact-match differences identify recognition errors as the main limitation.Source: Ablations and Diagnostic Analyses · pp. 6–7, Figure 7 · p. 7
• Rank 4 and t = 999 perform best among tested settings; controls support the one-step route, while paired tests find no significant LRC effect on ACC or NED.Across three draws, maximum standard deviations are 0.0019 dB PSNR and 0.0012 ACC; paired intervals exclude zero for LRC effects on PSNR, SSIM, LPIPS, and DISTS.Source: Ablations and Diagnostic Analyses · pp. 6–8

Conclusion

• TOLA improves all nine CTR-TSR-Test metrics over DiffTSR while reducing IDM/TDM evaluations from 200/200 to 1/0 and latency from 9.9548 to 0.1319 seconds per image.The optimized and saved state contains 2.46M parameters, and inference avoids another iterative restoration trajectory.Source: Conclusion · p. 8
• TOLA constructs a confidence-aware TransOCR–MoM condition once, adapts IDM and the VAE encoder with rank-4 LoRA, and corrects clean-latent residual errors with LRRB without invoking TDM.Component ablations and independent-OCR evaluation support the complementary roles of LoRA, LRRB, and confidence-aware conditioning.Source: Conclusion · p. 8
• Results on RealCE-200 and RT50 demonstrate generalization across paired camera observations, Chinese, English, and numeric text, and varied real capture conditions.Source: Conclusion · p. 8

Appendices

• The supplement adds implementation and evaluation details, system boundaries, one-step route modules, saved adaptation state, datasets, metrics, controlled variants, and extended diagnostics.It also includes additional ablations, inference controls, uncertainty analysis, and qualitative comparisons.Source: Appendices · p. 12

A.1 One-Step Inference Route

• TransOCR is evaluated once; confidence-weighted token embeddings and scaled image latents enter frozen MoM, while IDM uses only MoM’s text condition as cross-attention context.Source: §A.1 · p. 12
• IDM receives zt and zLR and predicts noise once, from which the initial clean latent is reconstructed using the checkpoint’s epsilon-prediction formulation.Source: §A.1 · p. 12
• After clean-latent reconstruction, LRC corrects the LR-to-clean residual and the frozen decoder produces one super-resolved image without a reverse chain or multi-sample averaging.Source: §A.1 · p. 12

B.1 LoRA Placement and Frozen Components

• Rank-4 LoRA adapters with α = 4 and zero dropout are inserted into eligible IDM and VAE encoder layers, while the decoder and base weights remain frozen.The first IDM input convolution and decoder-side latent projection are also unadapted; TransOCR, MoM, and the loaded but unused TDM decoder remain frozen.Source: §B.1 · p. 12
• Table S3 reports the optimized and saved parameter counts.Source: Table S3 · p. 12

B.2 Latent Residual Correction

• LRC takes the concatenated LR latent and LR-to-clean residual through input and output projections around an RRDB correction branch.The branch uses three residual dense blocks, each containing five densely connected 3×3 convolutions; its output projection is zero initialized so it initially leaves the clean latent unchanged.Source: §B.2 · p. 12
• The default LRC has 0.180323M parameters, while small and large variants have 0.045235M and 0.360323M, respectively.A parameter-matched plain correction branch has 0.180093M parameters.Source: §B.2 · p. 12

C Training Protocol

• The supplement reports the computing environment, and Table S1, Table S2, and Table S4 identify system boundaries, TOLA’s inference boundary, and the primary training configuration.Table S2 states that TransOCR, MoM, and IDM execute once and TDM does not execute.Source: C Training Protocol · p. 12, Table S1 · p. 13, Table S2 · p. 13
• Training uses a local deterministic 98%/2% split of CTR preprocessing images and online Real-ESRGAN or BSRGAN degradations at scales 1, 2, or 4.The objective combines L1, LPIPS-VGG, and OCR losses; the OCR recognizer remains differentiable to the image input, and no latent L1, noise-prediction, score-distillation, VSD/CSD, or teacher-regularization loss is used.Source: C Training Protocol · p. 12

C.1 Hyperparameter Search and Selection

• The final architecture, objective, OCR-loss weight, and training timestep are selected using validation performance, with validation LPIPS used for most candidate checkpoints.The no-LPIPS objective uses validation PSNR.Source: §C.1 · p. 13, Table S5 · p. 14

D.1 Dataset Inventory

• The primary benchmark is the complete 8,089-image CTR-TSR-Test ×4 set, with methods matched to the same references and evaluated on a 128×512 output canvas.Source: §D.1 · p. 13

D.2 Motivation and Coverage

• CTR-TSR-Test primarily reflects synthetic degradations, while real-image datasets vary in language coverage, capture conditions, annotation format, and reference availability.RealCE-200 and RT50 broaden evaluated conditions through paired real multi-focal observations and balanced Chinese, English, and numeric text collected through varied routes.Source: §D.2 · p. 13

D.3 RealCE-200 Construction

• RealCE-200 comprises 200 filtered, deduplicated pairs from matched 13 mm and 52 mm crops in the RealCE validation split, without geometric registration or manual relabeling.RealCE is not used for training, validation, or checkpoint selection.Source: §D.3 · p. 13, Table S6 · p. 14

D.4 RT50 Construction

• RT50 contains 50 real text inputs: 17 Chinese, 17 English, and 16 numeric samples, partitioned into 10 mild, 20 medium, and 20 hard cases.Selection was finalized before model inference without consulting restoration or OCR outputs.Source: §D.4 · p. 14

D.5 Included Dataset Records

• The planned public release includes the evaluated RealCE-200 pairs and RT50 images with manifests, labels or transcriptions, provenance, and verification metadata.Source: §D.5 · p. 14

D.6 Metric Implementations

• PSNR and SSIM are computed on luminance without border cropping, while evaluation LPIPS uses AlexNet and training LPIPS uses VGG.DISTS and FID use PyIQA; FID is computed once over the complete output and reference directories.Source: §D.6 · p. 14
• Recognition uses normalized TransOCR predictions for exact-match ACC and averaged per-image normalized edit-distance similarity, with PaddleOCR as an independent cross-check.PaddleOCR is not used for conditioning, training, validation, or checkpoint selection.Source: §D.6 · p. 14

E.1 Additional RealCE Comparison

• Figure S1 provides additional paired RealCE-200 examples under the unified evaluation protocol.Source: §E.1 · p. 14

E.2 BTL Cross-Protocol Comparison

• Five BTL-test ×4 examples compare TOLA with DiffTSR, TeReDiff, and PRISM across Chinese text, English words, and numeric sequences.BTL-test combines real and rendered HQ sources with BSRGAN/Real-ESRGAN degradations; the PRISM row is reproduced from the original paper.Source: §E.2 · p. 14

• Filename-paired comparisons use identical per-image Gaussian noise, with 5,000 paired bootstrap resamples for effect intervals and exact paired McNemar tests for ACC.Table S7 defines the architecture and conditioning controls.Source: F Controlled Variant Definitions · p. 14, Table S7 · p. 15
• Figure S1 compares five paired RealCE-200 examples across the LR input, restoration methods, TOLA, and the HR reference.Source: Figure S1 · p. 15
• Figure S2 compares five BTL-test ×4 examples across the LR input, DiffTSR, TeReDiff, PRISM, TOLA, and ground truth.BTL-test combines real and rendered HQ sources with BSRGAN/Real-ESRGAN degradations; the PRISM row is reproduced from the original paper.Source: Figure S2 · p. 15
• Table S8's image-quality reference configuration uses rank 4, a 0.180M LRC, OCR weight 0.02, and t = 999.Source: Table S8 · p. 16
• Main comparisons, ablations, and controls use one restored output per image; only the stochastic-stability analysis uses three inference runs and reports mean and standard deviation.Bootstrap resampling does not generate additional restored outputs.Source: F Controlled Variant Definitions · p. 16

G.1 Objective and Hyperparameter Diagnostics

• Rank 4 and the default 0.180M LRC outperform smaller and larger variants overall; removing ℓ1 most reduces PSNR, while removing LPIPS most degrades perceptual and distributional metrics.OCR weight 0.02 outperforms 0 and 0.05, and t = 999 gives the strongest overall training result among tested timesteps.Source: §G.1 · p. 16, Table S9 · p. 16

G.3 Inference Controls and Stochastic Stability

• Although t = 949 slightly improves PSNR and SSIM, it worsens LPIPS, DISTS, and FID; t = 999 provides the best overall balance, and repeated inference is low-sensitivity to sampled noise.Zero noise and disabling VAE LoRA substantially reduce reconstruction quality; input controls rule out copying or a frozen-VAE round trip as explanations for the gains.Source: §G.3 · p. 16, Table S11 · p. 17

G.4 Paired Uncertainty

• Paired intervals exclude zero for LRC effects on PSNR, SSIM, LPIPS, and DISTS, but not NED; ACC has no significant paired McNemar difference.Predicted conditioning significantly improves ACC over null context, while the ground-truth-prior gap indicates remaining headroom from prior errors.Source: §G.4 · p. 16, Table S12 · p. 17, Figure S3 · p. 18

G.5 Exact Recognition Diagnostics

• TOLA improves ACC and NED across all four confidence intervals and text categories, while reducing character edits from 42.34 to 32.55 per 100 characters.Substitutions, deletions, and insertions all decrease; the large correct-versus-wrong-prior gap identifies prior recognition as the primary remaining source of exact-transcription error.Source: §G.5 · p. 17, Table S14 · p. 17, Table S15 · p. 17

G.2 Prior-Stratified Performance

• The correct-versus-wrong predicted-prior performance gap identifies prior recognition as the primary remaining source of exact-transcription error.Table S10 stratifies image and recognition metrics by whether the predicted TransOCR prior matches ground truth.Source: §G.2 · p. 16, Table S10 · p. 16, §G.5 · p. 17