PaperlayerView exact revision on arXiv

Source-linked AI summary

TokBench: Evaluating Your Visual Tokenizer before Visual Generation

Junfeng Wu, Dongliang Luo, Weizhi Zhao, Zhihao Xie, Yuanhao Wang, Junyi Li, Xudong Xie, Yuliang Liu, Xiang Bai

arXiv:2505.18142v2cs.CVcs.DB

TL;DR

Compression in visual tokenizers and VAEs loses information, limiting generation fidelity, while text and faces pose especially challenging fine-grained evaluation targets. TokBench uses curated data and OCR and face-recognition measures to assess image and video reconstruction; its experiments find that targeted measures complement conventional metrics, which can mislead on these targets.

  • Problem

    Compression information loss in visual tokenizers and VAEs constrains generation fidelity, and conventional metrics may miss perceptually important text and face errors.

    Source: §1 · p. 2

  • Method

    TokBench curates text- and face-rich image and video data and evaluates text recognition and face identity with OCR and face-recognition models.

    Source: §1 · p. 2

  • Results

    TokBench’s experiments show that targeted text-readability and face-identity measures complement conventional metrics, which can mislead for these reconstruction tasks.

    Source: §4.2 · p. 10, §6 · p. 12

  • Takeaways & Limitations

    Direct assessment of text readability and face identity can mitigate misleading results from conventional metrics when evaluating these targets.

    Source: §6 · p. 12

    The proposed metrics do not evaluate pixel-level probabilities across the whole image and may miss contrast or saturation distortions.

    Source: §5 · p. 11

Abstract

from arXiv · show

In this work, we reveal the limitations of visual tokenizers and VAEs in preserving fine-grained features, and propose a benchmark to evaluate reconstruction performance for two challenging visual contents: text and face. Visual tokenizers and VAEs have significantly advanced visual generation and multimodal modeling by providing more efficient compressed or quantized image representations. However, while helping production models reduce computational burdens, the information loss from image compression fundamentally limits the upper bound of visual generation quality. To evaluate this upper bound, we focus on assessing reconstructed text and facial features since they typically: 1) exist at smaller scales, 2) contain dense and rich textures, 3) are prone to collapse, and 4) are highly sensitive to human vision. We first collect and curate a diverse set of clear text and face images from existing datasets. Unlike approaches using VLM models, we employ established OCR and face recognition models for evaluation, ensuring accuracy while maintaining an exceptionally lightweight assessment process <span style="font-weight: bold; color: rgb(214, 21, 21);">requiring just 2GB memory and 4 minutes</span> to complete. Using our benchmark, we analyze text and face reconstruction quality across various scales for different image tokenizers and VAEs. Our results show modern visual tokenizers still struggle to preserve fine-grained features, especially at smaller scales. We further extend this evaluation framework to video, conducting comprehensive analysis of video tokenizers. Additionally, we demonstrate that traditional metrics fail to accurately reflect reconstruction performance for faces and text, while our proposed metrics serve as an effective complement.

1 Introduction

TokBench targets text and faces because their small-scale, detailed features are perceptually sensitive and conventional metrics can miss important errors. It combines curated data with task-aware recognition measures to assess image and video tokenizers.

1 Introduction

  • TokBench curates text- and face-rich images and videos because compression information loss constrains the upper bound of visual generation fidelity.The selected contents are small-scale, texture-rich, prone to collapse, and sensitive to human vision.Source: §1 · p. 2
  • TokBench evaluates text readability and face identity using OCR and face-recognition models, with frame-by-frame averages for reconstructed videos.Its metrics include T-ACC and T-NED for text and F-Sim for faces.Source: §1 · pp. 2–3
  • Conventional pixel- and feature-based metrics can disagree with human judgments on text and face reconstruction, motivating task-aware evaluation.Pixel metrics may miss semantically important text errors, while feature metrics may lack granularity for facial attributes and character recognition.Source: §1 · pp. 2–3
  • The evaluation pipeline requires only 2GB VRAM and four minutes to complete.The paper describes the assessment as lightweight while using established OCR and face-recognition models.Source: §1 · p. 3
  • The benchmark evaluates image tokenizers and VAEs and extends its assessment to video tokenizers.The authors also report that conventional metrics are inconsistent with human evaluation for text and face reconstruction.Source: §1 · p. 3

2 Related Work

Related work covers visual latent representations, reconstruction metrics, and the datasets used to build TokBench. It motivates scale-aware text and face data alongside task-specific evaluation.

2.1 Visual Tokenizers and VAEs

  • Visual tokenizers span continuous VAEs, discrete VQVAE/VQGAN and lookup-free quantization, while video methods include 3D causal VAEs.The literature also explores semantic supervision, communication between continuous and discrete tokens, and 2D-to-1D token compression.Source: §2.1 · p. 3

2.2 Evaluation of Image Reconstruction

  • Pixel-level metrics compare image values or structure, while feature-level metrics assess semantic or distributional similarity but can underweight small text and faces.The passage notes that pretrained feature metrics require careful interpretation for out-of-distribution samples.Source: §2.2 · pp. 3–4

2.3 Text and Face Datasets

  • TokBench draws text from eight datasets spanning fonts, styles, scales, and backgrounds, and selects WFLW for unconstrained, often multi-face images.Text-in-video datasets and face-containing video-understanding datasets also provide sources for video assessment.Source: §2.3 · p. 4

3 TokBench

TokBench targets reconstruction of text and faces across image scales, extending evaluation to videos. It pairs curated data with specialized measures of text legibility and face identity preservation.

3 TokBench

  • TokBench targets text legibility and face identity across image scales, emphasizing small instances; the benchmark also extends to text- and face-rich videos.Its image data is described as having balanced instance-scale distributions, with particular emphasis on small-scale faces and text.Source: §3 · p. 4, Figure 2 · p. 5

3.1.1 Text Data Curation

  • Text data combines eight scene-text and document datasets, then filters invalid or recognizer-unreadable regions to yield 6,000 images and 76,126 valid instances grouped by character scale.The cleaning procedure removes instances smaller than 5 × 5 pixels or below a relative-scale threshold, and retains text correctly recognized in the original images.Source: §3.1.1 · pp. 5–6

3.1.2 Face Data Curation

  • Face data starts from WFLW images and adds detected faces, producing 17,700 valid targets for similarity evaluation after image and face-scale filtering.The curation retains images with aspect ratios no greater than 2 and detected faces with confidence above 0.5 and scale greater than 0.05.Source: §3.1.2 · p. 7

3.2 Evaluation Protocols

  • GT-cropped text uses OCR for exact-word T-ACC and character-level T-NED, while paired face crops use InsightFace cosine similarity (F-Sim) to assess identity preservation.T-ACC distinguishes uppercase and lowercase; T-NED uses Levenshtein distance for a finer-grained text assessment.Source: §3.2 · p. 7

3.3 Video Data Curation

  • The video extension evaluates text frame by frame on 15,921 frames with 347,468 valid text instances, and faces across 328 curated clips with 81,556 targets.Text recognition is run only on valid cropped regions; face clips are selected for continuous face presence and multiple detected faces.Source: §3.3 · pp. 7–8

4 Experiments

The experiments compare image and video tokenizer reconstructions across resolutions, target scales, and data settings. They find persistent fine-detail gaps, particularly for small text, and show that conventional metrics can miss differences captured by TokBench.

4.1 Evaluation Setting

  • Images are reconstructed at 256, 512, and 1024 against resize baselines; calculating image T-ACC and F-Sim takes 2GB GPU memory and 4 minutes.Video evaluation uses 256 and 480 resolutions; its metrics require 2GB memory and about 30 minutes, reducible to 6 minutes with multi-GPU processing.Source: §4.1 · p. 8

4.2 Main Results

  • At 256 resolution, discrete tokenizers struggle with small text and faces; UniTok leads F16 peers, F8 variants outperform matching F16 models, and SD3.5 and FLUX VAEs score highest.TiTok performs weakest on both targets.Source: §4.2 · p. 9
  • FID and PSNR have limited power to distinguish text and face reconstruction: substantial T-ACC and F-Sim differences can accompany marginal FID gaps and contradictory rankings.Improved text reconstruction typically correlates with better conventional-metric scores, but comparisons also reveal contradictory trends.Source: §4.2 · pp. 9–10

4.3 Detail Evaluation for Text and Face

  • Text performance generally improves with resolution, while continuous-discrete gaps widen; at 1024, FLUX’s VAE reaches resize-comparable T-NED, and small-object gaps grow at higher resolutions.Even resizing may fall short of 100% T-ACC and T-NED because many original text images exceed 1024 pixels.Source: §4.3 · p. 10, Figure 5 · p. 10

4.4 Video Tokenizers and VAEs

  • Video VAEs reconstruct faces closer to the resize upper bound than small text, while all evaluated 4 × 8 × 8 VAEs still show substantial small-text gaps.The authors suggest extensive facial-data exposure during training may explain the closer face results.Source: §4.4 · p. 10, Table 3 · p. 11
  • At identical compression ratios, Step-VAE outperforms 8 × 16 × 8 Cosmos-VAE but remains below Hunyuan-Video and CogVideoX VAEs while achieving 8× compression.Source: §4.4 · p. 11

4.5 Ablation of Training Data

  • Adding 230k text-rich images to ImageNet improves T-ACC and T-NED only marginally relative to architectural enhancements, which matter more in this ablation.Source: §4.5 · p. 11, Table 4 · p. 11

5 Limitation

TokBench evaluates text reconstruction through recognition accuracy, so it does not assess every visual property of the reconstructed image.

  • Text reconstruction quality is judged by text-recognition accuracy rather than pixel-level probabilistic evaluation across the entire image.Source: §5 · p. 11
  • Accurate text can still have contrast or saturation distortions that TokBench’s proposed metrics do not directly capture.Source: §5 · p. 11
  • The authors position TokBench’s metrics as a complement to PSNR and FID, which evaluate pixel-level and feature-statistics properties, respectively.Source: §5 · p. 11

6 Conclusion

TokBench targets text readability and face identity preservation in image and video compression. Its experiments support using these assessments alongside existing metrics to evaluate visual reconstruction quality.

  • TokBench targets text and human faces across wide scale distributions in evaluating visual generative models’ image and video compression quality.Source: §6 · p. 11
  • It directly evaluates text readability and identity preservation using mature toolchains for reconstructed faces and text.Source: §6 · p. 12
  • The authors conclude that these assessments complement existing metrics and help ensure the upper bound of visual generation quality.Source: §6 · p. 12

A Evaluation Setting

The evaluation uses a range of continuous and discrete image tokenizers, including multiple variants of several model families.

  • The continuous-space set includes VA-VAE, SDXL, SD3.5, FLUX, and a 32× downsampling DC-AE with 32 latent dimensions.Source: §A.1 · p. 17
  • Discrete models include baseline VQGAN, LlamaGen and Open-MAGVIT2 F16/F8 variants, VAR-d36, TiTok-L-32, and 12-bit MaskBit.Source: §A.1 · p. 17
  • Open-MAGVIT2 also includes a pretrained F16 model trained on 100M data; OmniTokenizer uses imagenet_k600 and FlexTok uses its DFN-trained version.Source: §A.1 · p. 17

A.2 Difficulty Rating

The available passages specify a minimum valid face scale and a more lenient rating strategy for video evaluation.

  • The minimum valid face scale is approximately 25 ÷ 256 ≈ 0.1.Source: §A.2 · p. 18
  • Video evaluation uses a more lenient rating strategy because resizing the shorter edge yields longer edges around 500 pixels.Source: §A.2 · p. 18

B Detailed Comparison on Face Set

Face reconstruction generally improves with resolution, but results still vary substantially by tokenizer, especially for small-scale faces.

C More Visualization

Qualitative comparisons show reconstruction quality varies by resolution and tokenizer type, with small-scale content remaining challenging.

  • At 256 resolution, most discrete tokenizers fail to reconstruct text and faces accurately, and high-compression DC-AE also performs poorly.Source: C More Visualization · p. 19
  • At 1024 resolution, VAEs and low-compression F8 discrete tokenizers produce satisfactory results, while F16 tokenizers still struggle with small-scale objects.Source: C More Visualization · p. 19
  • F8 Open-MAGVIT2 exhibits noticeable color distortion at 1024 resolution despite the generally satisfactory results for F8 tokenizers.Source: C More Visualization · p. 19

D Ablation Setting

The ablation setting compares F8 and F16 VQGANs trained with and without text-rich image augmentation under matched training conditions.

Loading 2505.18142v2…