PaperlayerView exact revision on arXiv

Source-linked AI summary

ClearText-Video: A Large-Scale Text-Centric Video Dataset Bridging Video Restoration and Scene-Text Enhancement

Jinlong Li, Jiaming Ding, Dingfu Lu, Malcolm Hsiu, Chuang Ke, Kangning Yang, Bochen Guan, Lan Fu, Jie Cai, Huiming Sun, Zibo Meng

arXiv:2608.28784v1cs.CV

TL;DR

Existing benchmarks provide limited evidence about whether restored text-centric videos preserve the textual evidence needed for multimodal reasoning. CTVid addresses this with paired HQ/DQ/RQ videos and unified restoration–VideoQA evaluation, showing that visual improvement is not necessarily text-faithful or reasoning-beneficial.

  • Problem

    Existing benchmarks provide limited evidence about whether restoration preserves text quality and how resolution variation affects text-centric video reasoning.

  • Method

    CTVid pairs the same text-rich videos across HQ, DQ, and RQ regimes and evaluates restoration alongside spatial and temporal VideoQA.

  • Results

    Restoration is not monotonic: videos can become sharper while altering character strokes or spatial cues, leaving MLLMs less accurate than on degraded inputs.

  • Takeaways & Limitations

    Text-centric restoration should be evaluated with both visual or text-fidelity metrics and downstream reasoning metrics.

  • Takeaways & Limitations

    Controlled bicubic downsampling and motion blur do not cover artifacts such as exposure changes, rolling shutter, sensor noise, ISP effects, or complex compression.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have recently made strong progress in visual--linguistic understanding. However, their performance on text-centric video reasoning remains highly sensitive to input quality. Real-world user-provided videos often contain motion blur, compression artifacts, noise, and low-resolution text, which impair reliable text reading and downstream reasoning. Whether MLLMs can robustly read and reason about real-world scene text under diverse quality conditions remains a fundamental open question. We introduce ClearText-Video (CTVid), a large-scale, scene-text-aware benchmark for studying text-centric video understanding under controlled quality variation. CTVid contains 4,639 real-world text-rich egocentric videos, 550K+ frames, 1.6M human-verified scene-text annotations, and 220K+ spatial/temporal question--answer pairs in Chinese and English. For each high-quality video, CTVid provides content-matched Degraded-Quality and Restored-Quality variants, supporting two task families: Text-Centric Video Restoration and Multi-Quality VideoQA. We evaluate 18 representative restoration methods and 16 state-of-the-art MLLMs on CTVid. The results show that visual enhancement does not guarantee textual fidelity or downstream reasoning gains: blur is more damaging than low resolution, restored videos can alter the textual evidence used by MLLMs, and OCR-only pipelines remain far below direct multimodal reasoning. CTVid exposes the gap between video restoration and text-grounded understanding, providing a rigorous foundation for restoration-aware, quality-robust text-centric video systems.

1 Introduction

Existing TextVQA benchmarks largely overlook resolution variation, leaving open whether MLLMs can reliably read and reason about text in real-world videos across quality levels. ClearText-Video addresses this gap with a bilingual, scene-text-aware benchmark and a unified evaluation framework spanning restoration and multi-quality video question answering.

  • Benchmark: ClearText-Video enables controlled analysis of input video quality through High-Quality, Degraded-Quality, and Restored-Quality regimes.Each content instance is evaluated across a quality-controlled video triplet to analyze changes in textual evidence available to MLLMs.
  • Motivation: Existing TextVQA benchmarks largely overlook resolution variation and multiple quality levels.This leaves quality-varying text-centric video understanding insufficiently examined.
  • Benchmark: ClearText-Video contains 4.6K+ real-world text-centric high-quality videos, 550K+ frames, and 1.6M annotated text instances.The benchmark supports bilingual Chinese/English egocentric question answering with rich scene-text annotations.
  • Evaluation: CTVid benchmarks MLLMs in multilingual, quality-varying, text-rich settings and reveals persistent performance gaps.The framework evaluates textual fidelity and reasoning robustness under visual degradation, from low-level restoration to high-level reasoning.
  • Task Design: The benchmark defines Text-Centric Video Restoration and Multi-Quality Video Question Answering as two task families.Restoration includes image and video super-resolution and video deblurring, while VideoQA covers spatial and temporal understanding.

2 Related Work

Prior work spans real-world video restoration, multimodal language models for video understanding, and text-aware VQA benchmarks. Existing methods address degraded video quality and text-grounded reasoning, while image-based benchmarks lack temporal video reasoning.

  • Real-world Video Restoration: Real-world video restoration recovers high-quality footage degraded by imperfect capture, motion blur, or compression.
  • Real-world Video Restoration: Recent diffusion-based video restoration methods use optical-flow propagation, motion-aware objectives, streaming restoration, staged optimization, or large-scale video-diffusion priors.
  • Multimodal Large Language Models: MLLMs support text VQA, document understanding, and text recognition, and newer systems extend these capabilities to dynamic video content.
  • Text-Aware VQA Benchmarks: TextVQA, ST-VQA, and ESTVQA ground image questions in scene text, but their image-based settings do not capture temporal reasoning in video.
  • Text-Aware VQA Benchmarks: Multilingual resources such as MTVQA and EgoTextVQA broaden language coverage, while video benchmarks require reading and reasoning over on-screen text.

3 ClearText-Video Dataset

ClearText-Video is a manually collected, multilingual benchmark for high-quality, text-rich video, with explicit text regions and frame-level captions supporting text-aware restoration evaluation. The section presents its construction process, annotation schema, dataset statistics, and unified evaluation protocol.

  • Video Collection: ClearText-Video targets multilingual, high-quality, text-rich video using manually collected text-centric footage rather than repurposed benchmarks.Videos are captured with a Sony A7R V and a variable-focal-length 28–70 mm f/3.5–5.6 FE lens, supporting multiple zoom levels and perspectives.
  • Dataset Comparison: ClearText-Video combines high-resolution video, explicit text regions, and frame-level captions for text-aware restoration evaluation.Compared restoration datasets in the first two task categories lack the annotations needed for text-aware restoration evaluation, while ClearText-Video offers higher resolution than the compared datasets.
  • Dataset Statistics: The dataset statistics summarize text carrier types, scene categories, captions, and frame-level text annotations through visualizations.The statistics include word-frequency visualizations for video captions and text annotations.

4 ClearText-Video Benchmark

ClearText-Video supports text-centric video restoration and high-level understanding through temporally aligned quality variants and multi-quality VideoQA. Its benchmark evaluates restoration with image- and text-centric metrics and assesses spatial scene-text understanding across multiple input-quality conditions.

  • Benchmark design: CTVid provides paired, temporally aligned videos for image super-resolution, video super-resolution, and video deblurring, alongside a multi-quality VideoQA benchmark.The VideoQA benchmark is constructed from High-Quality, Degraded-Quality, and Restored-Quality video variants.
  • Text-centric restoration tasks: Text-centric image super-resolution reconstructs frames from 4× bicubic downsampled inputs, with captions enabling semantic-aware or language-guided restoration.Text-centric video super-resolution similarly reconstructs high-resolution videos from low-resolution inputs and can use the middle-frame caption when semantic information is needed.
  • Text-centric VideoQA tasks: Spatial VideoQA uses questions grounded in scene text within specific bounding-box regions and offers multiple-choice, true/false, and fill-in-the-blank formats.Rule-based spatial partitioning isolates meaningful text groups, after which annotators screen and refine generated questions to reduce hallucinations.
  • Evaluation protocol: Restoration evaluation combines full-reference and no-reference image-quality metrics with PaddleOCR-based text detection and recognition measures.Image-quality metrics include PSNR, SSIM, LPIPS, DISTS, FID, NIQE, MANIQA, MUSIQ, and CLIPIQA; text metrics include Pdet, Rdet, F1det, Accrec, and NEDrec.
  • Evaluation protocol: VideoQA is evaluated with accuracy (Acc), uncertainty-aware accuracy (UAcc), overconfidence ratio (OC), and abstention rate (Abs).These metrics assess correctness, confidence-linked reliability, overconfidence, and the fraction of queries for which the model withholds an answer.
  • Evaluation protocol: Spatial-understanding VideoQA compares HQ, DQ-Low_res, DQ-Blur, RQ-DOVE, RQ-MIMO, and RQ-S3DIFF input-quality conditions.The listed conditions represent high quality, low-resolution and blur degradations, video-based super-resolution or blur restoration, and image-based super-resolution.

5 Experiments

Experiments benchmark restoration methods and MLLMs on CTVid under multiple quality conditions. Results show distinct strengths across video super-resolution, spatial VideoQA, and uncertainty-aware reliability.

  • Text-Centric Video Super-Resolution: DOVE achieves the best video-based Rdet and F1det scores, 63.36% and 74.53%, and the highest video-based Accrec, 44.72%.These results come from benchmarking six state-of-the-art video super-resolution models on CTVid.
  • Multi-Quality VideoQA: Gemini-2.5-pro obtains the best spatial VideoQA accuracy on HQ, DQ-Low_res, and RQ-S3DIFF inputs, with scores of 71.67%, 65.00%, and 70.00%.The benchmark evaluates 16 MLLMs, including proprietary and open-source models, across quality conditions.
  • Multi-Quality VideoQA: 58.43%, 50.72%, 49.79%, 53.21%, 52.89%, and 51.31% are Qwen2.5-VL-7B-SFT’s best open-source accuracies across the six quality conditions.The tuned model improves over Qwen2.5-VL-7B by 6.31–10.11 percentage points, while lower UAcc shows accuracy and uncertainty-aware reliability can diverge.
  • Multi-Quality VideoQA: Claude-Sonnet-4.6 consistently ranks first across all five temporal VideoQA conditions evaluated with uniformly 2× downsampled inputs.The protocol minimizes temporal subsampling where possible to preserve continuity across sequences.

6 Discussion and Limitations

CTVid treats restoration and video understanding as coupled rather than independent, testing whether restoration preserves textual evidence for downstream reasoning. Its results motivate joint visual/text-fidelity and reasoning evaluation, while controlled degradations keep content and annotations fixed across quality regimes.

  • Restoration and understanding are coupled: CTVid couples restoration with VideoQA to test whether visually improving degraded text videos preserves the textual evidence required for downstream reasoning.This diagnostic cannot be isolated by prior single-quality TextVQA benchmarks.
  • Restoration and understanding are coupled: Restoration quality is not monotonic with reasoning accuracy because sharper videos can alter character strokes, introduce plausible-looking text, or shift spatial cues.These changes can make MLLMs less accurate than they are on degraded inputs.
  • Restoration and understanding are coupled: CTVid supports evaluating restoration with both visual/text-fidelity metrics and downstream reasoning metrics.The recommendation follows from restored videos potentially changing the textual evidence used by MLLMs.
  • Controlled degradations and data release: Controlled bicubic downsampling and motion blur keep underlying content, text annotations, and QA pairs fixed across HQ, DQ, and RQ comparisons.This design supports controlled quality comparisons and data release.

7 Conclusion

CTVid is introduced as a large-scale, scene-text-aware video QA benchmark for examining how input video quality affects text-centric multimodal reasoning. Its controlled quality protocol evaluates identical text-rich videos across High-Quality, Degraded-Quality, and Restored-Quality regimes, alongside a unified evaluation suite.

  • 7 Conclusion: CTVid is designed to examine how input video quality affects text-centric multimodal reasoning.The benchmark is described as the first large-scale, scene-text-aware video QA benchmark for this purpose.
  • 7 Conclusion: The controlled quality protocol evaluates the same underlying text-rich videos under High-Quality, Degraded-Quality, and Restored-Quality regimes.This design enables analysis of how degradation and restoration affect model reasoning.
  • 7 Conclusion: CTVid defines a unified evaluation suite alongside its dataset construction.The passage presents this suite as part of the benchmark’s broader contribution beyond dataset construction.

A ClearText-Video Dataset · A.1 Diversity of Text-Centric Video Scenes · A.2 Test Set Details

ClearText-Video spans multilingual, multi-domain, and structurally diverse text-centric video scenes, supporting realistic text and video-language evaluation. Its 312-video test set provides matched HQ, LR, and blurry conditions with balanced language coverage and varied sources and lighting.

  • A.1 Diversity of Text-Centric Video Scenes: ClearText-Video covers transportation hubs, urban driving scenes, commercial areas, and public facilities rather than only canonical street-view scenes.Examples include airports, highways, roadside signs, storefronts, billboards, parks, and parking lots.
  • A.1 Diversity of Text-Centric Video Scenes: The benchmark includes English and Chinese text with layouts ranging from large outdoor billboards and vehicle liveries to small, densely packed information boards.Text appears across varying viewpoints, distances, and background clutter, with instances localized by fine-grained bounding boxes over time.
  • A.1 Diversity of Text-Centric Video Scenes: This diversity supports video text spotting, text reading, and higher-level video-language understanding in realistic, text-rich environments.The benchmark exposes models to a broad spectrum of real-world uses of text in videos.
  • A.2 Test Set Details: 312 HQ videos form the test set, split equally between English and Chinese scenes after prioritizing clear illumination, visible text regions, and limited global motion blur.Three-quarters of the test set comes from offline recordings, while the remaining quarter is collected from online sources.
  • A.2 Test Set Details: Bicubic downscaling (×4) produces LR versions, while local motion blur produces blurry versions of the HQ videos.These transformations create controlled quality variants for evaluation.
  • A.2 Test Set Details: Matched HQ, LR, and blurry video sets contain the same number of samples and identical content and question distributions.This design enables controlled evaluations across three quality conditions.
  • A.2 Test Set Details: Offline videos constitute 74% of the test set, while online videos constitute 26%; both source groups are evenly split between English and Chinese clips.The test set mixes artificial and natural lighting, with the online portion almost entirely captured under artificial lighting.

A.3 Release, Licensing, and Dataset Comparison … B.4 Details of MLLMs

The supplementary sections define CTVid’s release and benchmark distinctions, detail spatial and temporal QA generation, and summarize restoration and MLLM evaluation setups. They also describe representative image/video restoration and deblurring baselines and the model compositions used for spatial and temporal VideoQA.

  • A.3 Release, Licensing, and Dataset Comparison: CTVid will release splits, verified text annotations, captions, trajectories, QA pairs, generation scripts, evaluation code, prompts, and baseline outputs under research-oriented licensing constraints.Self-captured clips are distributed under a research license, while restricted online clips are represented through permitted source IDs or URLs, ranges, annotations, and preprocessing scripts.
  • A.3 Release, Licensing, and Dataset Comparison: CTVid jointly provides large-scale QA, dense frame-level text boxes and transcripts, bilingual scenes, paired HQ/DQ/RQ videos, and restoration-aware evaluation.HQ/DQ/RQ denotes content-matched high-quality, degraded-quality, and restored-quality inputs; restoration-aware evaluation measures downstream reasoning under restoration intervention.
  • A.4 Question Answering Generation Details: Spatial VideoQA uses multiple-choice, true/false, and fill-in-the-blank questions grounded in specified text bounding-box regions and validated for spatial consistency.AnswerChecker verifies box references, logical spatial constraints, and answer formatting, while distractors combine OCR-level, semantic, and visual confusion.
  • A.4 Question Answering Generation Details: Temporal VideoQA derives questions from text trajectories using frame-wise coordinates, areas, speeds, accelerations, visibility, localization, motion, scale, and boundary interactions.Deterministic statistical thresholds generate structured questions about appearance patterns, region traversal, motion direction, size trends, and frame-edge contact.
  • B Additional Experimental Details: The experimental setup evaluates restoration baselines alongside MLLMs in spatial and temporal VideoQA experiments and qualitative analyses.The setup covers image/video super-resolution and deblurring models as low-level restoration baselines.
  • B.1 Details of Image Super-Resolution Models: Image restoration baselines include Real-ESRGAN, OSEDiff, and S3Diff, which model complex real degradations or use degradation-guided diffusion and low-rank adaptation.OSEDiff initializes from the low-quality image and targets single-step reconstruction, while Real-ESRGAN simulates noise, blur, compression, and ringing through high-order degradation.
  • B.2 Details of Video Super-Resolution Models: Video restoration baselines include RealBasicVSR, RealViFormer, MGLD-VSR, and DOVE, addressing temporal propagation, channel-based fusion, motion-guided diffusion, and efficient single-step training.RealViFormer uses CAF and ICA modules, MGLD-VSR uses motion-guided latent diffusion, and DOVE reports up to a 28× speedup over methods such as MGLD-VSR.
  • B.3 Details of Deblurring Models: Deblurring baselines comprise MIMO-UNet+, NAFNet, Restormer, ShiftNet, and ShiftNet(GoPro), using multi-scale inputs, lightweight attention, transformer restoration, or spatiotemporal feature shifts.ShiftNet configurations learn inter-frame motion from DVD or GoPro sequences without explicit optical flow.

B.5 Additional Diagnostic Analyses

The diagnostic analyses attribute failures to both restoration artifacts and MLLM errors, show that blur harms spatial QA more than low resolution, and find OCR-only reasoning insufficient. Restoration can improve direct visual reasoning without improving OCR-mediated performance.

  • Failure-mode analysis: Failure analysis separates restoration-side artifacts, including hallucinated or distorted character strokes, from MLLM-side localization, reasoning, and temporal aggregation errors.These two failure sources explain why visually sharper restored videos do not always improve downstream QA.
  • Degradation analysis: 3.14 points: low resolution reduces spatial QA accuracy from HQ, while 6.05 points: blur reduces accuracy when averaged over 16 MLLMs.Both degradations reduce accuracy on the same videos and question–answer pairs, but blur is more harmful.
  • OCR-mediated baseline: OCR+LLM remains far below direct Qwen2.5-VL-7B, showing that OCR-only processing is insufficient for this benchmark.PaddleOCR extracts text from each quality version, and Qwen2.5-7B answers using only the extracted text without image or video input.
  • OCR-mediated baseline: RQ improves Qwen2.5-VL-7B over DQ but does not improve OCR+LLM, indicating different effects on direct visual reasoning and OCR-mediated processing.Restored outputs may remain difficult for OCR or contain text distortions despite benefiting direct visual reasoning.

B.6 Visualization for Text-Centric Video Restoration

Visualization examples show that restoration methods can improve apparent clarity while altering or corrupting scene text. Video-based methods generally preserve text more accurately than image-only methods, yet compact Chinese characters remain challenging.

  • Video deblurring: Stripformer produces relatively clear deblurred outputs but changes the “COSTCO” sign’s color and misreads “Jingu” as “Jimgu”.MIMO-UNet+, Restormer, and RVRT provide only limited deblurring in the shown examples.
  • Image super-resolution: Image super-resolution handles larger English text but can corrupt words, rendering “South” as “Sauth” and “Ride Share” as “Paln Sharo”.All evaluated image super-resolution methods struggle with dense Chinese characters and smaller text, sometimes generating illegible or unrelated text.
  • Video super-resolution: Video super-resolution methods appear to preserve text more accurately, although Upscale-A-Video changes letters and DOVE slightly deforms “COSTCO”.Examples include “Commarcial” for “Commercial” and “Lθading” for “Loading”.
  • Implications: Restoration methods can introduce text distortions beyond the degraded input, potentially harming downstream performance.Video-based methods often benefit from temporal information but still struggle with compact Chinese characters.

B.7 Spatial VideoQA Visualization under Multi-Quality Videos

The section visualizes text-centric spatial VideoQA across multiple video-quality conditions using localized scene-text questions. The examples show that models struggle with fine-grained text details and spatial grounding, while restoration does not consistently improve spatial reasoning.

  • Qualitative visualization: The visualizations use dynamic clips with localized scene-text regions and spatial questions about reading, identifying, or completing text within bounding boxes.Examples include full street names, outdoor signs, and missing-character completion.
  • Qualitative visualization: The qualitative results clarify the quantitative trends in Table 4 and expose the fine-grained difficulty of spatial text understanding in realistic videos.
  • Failure patterns: Models often fail on missing tokens, incorrect ordering, visually similar answer options, and incomplete fill-in-the-blank answers despite roughly recognizing the target text.
  • Quality effects: When moving from HQ to degraded videos, several models show weaker spatial grounding and appear to rely excessively on partial OCR cues.
  • Quality effects: Restored videos do not always improve spatial reasoning, even when the text appears perceptually sharper.

B.8 Temporal VideoQA Visualization under Multi-Quality Videos

The temporal VideoQA visualizations evaluate text evolution across five video-quality conditions using 120-frame clips and eight temporal models. They show that ClearText-Video exposes quality-sensitive, example-level failures beyond aggregate metrics, including errors in motion, position, and boundary reasoning.

  • Evaluation setup: Each 120-frame clip poses one temporal question about a text instance under HQ, DQ-Blur, RQ-DOVE, RQ-MIMO, and RQ-S3DIFF conditions.Visualizations pair keyframes with the question, ground-truth answer, and predictions from eight temporal models.
  • Overall finding: The visualizations demonstrate a challenging, quality-aware benchmark and reveal temporal text-reasoning failure patterns not evident from aggregate metrics alone.More detailed example-level analyses appear in the captions of Figs. 16–18.
  • Motion reasoning: Most models misread the consistently leftward motion of “↓363-378Gates” as stationary, indicating reliance on static keyframe cues.Only a few models predict the correct motion direction across the displayed quality conditions.
  • Position reasoning: Many models confuse the dominant bottom-left position of “NO SUPRISES!” with bottom-right or center regions, with some predictions varying by quality condition.The example tests where the text appears for most of the clip, requiring temporal spatial tracking rather than recognition alone.
  • Boundary reasoning: Most models classify “PHARMACY” as fully inside or merely near an edge instead of recognizing that it touches or runs off the frame boundary.Errors are especially evident under degraded and restored inputs, highlighting difficulty with boundary-aware temporal text reasoning.
Loading 2608.28784v1…