Source-linked AI summary
InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generation
Yang Yue, Fangyun Wei, Tianyu He, Jinjing Zhao, Zanlin Ni, Zeyu Liu, Jiayi Guo, Lei Shi, Yue Dong, Li Chen, Ji Li, Gao Huang, Dong Chen
TL;DR
Discrete tokenizers often lose the fine-grained structures needed for readable text and distinctive faces because standard objectives are weakly aligned with these content types. InsightTok adds localized, domain-specific perceptual losses and improves text and face reconstruction at a compact bottleneck, with gains transferring to autoregressive generation. The approach maintains competitive general reconstruction quality while producing clearer text and more faithful facial details.
Problem
Aggressive downsampling and quantization, together with generic tokenizer objectives, make it difficult to preserve readable text and distinctive facial features.
Method
InsightTok trains discrete visual tokenizers with localized, domain-specific perceptual losses on detected text and face regions, combined through weighted aggregation.
Results
InsightTok outperforms existing tokenizers on text and face reconstruction while maintaining competitive general reconstruction quality, and these improvements transfer to autoregressive image generation.
Takeaways & Limitations
Aligning tokenizer supervision with perceptually critical content is a practical approach for improving discrete image generation.
Takeaways & Limitations
The approach targets text and faces rather than all visual content, and current reconstruction and generation support is limited to English text and prompts.
Abstract
from arXiv · showhide
Text and faces are among the most perceptually salient and practically important patterns in visual generation, yet they remain challenging for autoregressive generators built on discrete tokenization. A central bottleneck is the tokenizer: aggressive downsampling and quantization often discard the fine-grained structures needed to preserve readable glyphs and distinctive facial features. We attribute this gap to standard discrete-tokenizer objectives being weakly aligned with text legibility and facial fidelity, as these objectives typically optimize generic reconstruction while compressing diverse content uniformly. To address this, we propose InsightTok, a simple yet effective discrete visual tokenization framework that enhances text and face fidelity through localized, content-aware perceptual losses. With a compact 16k codebook and a 16x downsampling rate, InsightTok significantly outperforms prior tokenizers in text and face reconstruction without compromising general reconstruction quality. These gains consistently transfer to autoregressive image generation in InsightAR, producing images with clearer text and more faithful facial details. Overall, our results highlight the potential of specialized supervision in tokenizer training for advancing discrete image generation.
1 Introduction
InsightTok addresses text and face fidelity failures in discrete visual tokenization by adding localized, content-aware supervision. At a compact 16,384-entry codebook and 16× downsampling, its improvements transfer to autoregressive text-to-image generation while preserving general reconstruction quality.
- Aggressive downsampling and quantization discard fine-grained details, making text and faces prominent tokenizer failure modes.These regions matter because people attend disproportionately to text and faces and are highly sensitive to distortions there.
- Standard pixel and LPIPS objectives optimize generic reconstruction rather than explicitly prioritizing text readability and facial fidelity.Reducing compression through larger codebooks or more tokens adds computational and modeling costs without targeted supervision.
- InsightTok adds localized perceptual losses on detected text and face regions, combining them through weighted aggregation.The weighting targets perceptually critical content while maintaining general-purpose reconstruction.
- At 16× downsampling with a 16,384-entry codebook, InsightTok improves text and face reconstruction while remaining competitive on standard metrics.The method uses the same compact bottleneck while targeting content-specific fidelity.
- The work motivates moving beyond VQGAN-style supervision toward richer, content-aware supervision for discrete representation learning.This frames specialized tokenizer training as a promising direction for discrete image generation.
2 Related Work
Related work establishes discrete tokenization as the interface between images and autoregressive or multimodal sequence modeling. Prior tokenizer and generation methods improve representations or target text and identity, but fine-grained text and face preservation remains challenging.
- Autoregressive image generation: Discrete tokenizers convert images into low-resolution visual token grids that autoregressive Transformers model as conditional next-token distributions.This sequence interface also supports text-to-image synthesis and joint multimodal handling of text and image tokens.
- Discrete tokenizer designs: VQ-VAE introduced the encoder–quantizer–decoder framework, while VQGAN combined reconstruction, perceptual, and adversarial supervision.Later work explores codebook-free quantization, refined assignments, and multi-code schemes.
- Empirical comparison: Figure 2 compares reconstruction quality for InsightTok, LlamaGen, O-MAGVIT2, and IBQ at a 16,384-entry codebook and 16× downsampling.All models are evaluated at 512 × 512 resolution.
- Discrete tokenizer designs: Recent benchmarks indicate that discrete autoencoders still struggle with fine-grained visual information, especially text and faces.
- Text and face generation: Text-generation methods commonly use glyph or layout guidance and OCR losses, whereas face methods leverage identity representations for preservation.These approaches primarily address diffusion-based synthesis or identity conditioning rather than the tokenizer bottleneck.
3 Method
InsightTok augments conventional discrete-tokenizer training with localized, content-aware perceptual supervision for text and faces. It detects corresponding regions in input and reconstructed images, processes them with specialized recognition models, and combines their losses with standard objectives.
- 3.1 Preliminary: Discrete Image Tokenizer: InsightTok combines reconstruction, codebook, perceptual, and adversarial losses in its conventional image-level tokenizer objective.The objective is Limage = Lrec + β · Lcodebook + γ · Lperc + η · LGAN.
- 3.2 Content-Aware Supervision: The framework adds text and face perceptual losses to standard tokenizer training so supervision focuses on perceptually critical regions.These localized losses complement the conventional image-level objective.
- 3.2.1 Text Perceptual Loss: Text regions are detected, cropped from the input and reconstruction, resized to 32 × 128, and compared using a pretrained text-recognition network.Features from five hidden layers are used by default to compute region-level text perceptual loss.
- 3.2.1 Text Perceptual Loss: Area-based text weighting prevents tiny text instances from dominating training because they are harder to reconstruct and produce disproportionately large feature discrepancies.The weighting balances contributions across text regions of different scales.
- 3.2.2 Face Perceptual Loss: Faces are detected through landmarks, aligned to a canonical template with a similarity transform, and extracted from both input and reconstructed images for recognition-based supervision.The alignment reduces variation in pose, scale, and in-plane rotation before face-specific feature comparison.
- Autoregressive Image Generation: InsightAR rasterizes InsightTok’s 16× downsampled token grid and trains a Transformer to predict each token conditioned on prior tokens and the text prompt.At generation time, sampled token sequences are decoded by the InsightTok decoder.
4 Implementation
InsightTok uses a 16× downsampling rate and a 16,384-entry codebook, while InsightAR is trained on large-scale discrete token sequences using a Janus-Pro-style multimodal setup. The implementation evaluates reconstruction across specialized and general metrics and compares generated images with Janus-Pro.
- Tokenizer Configuration: InsightTok uses 16× downsampling, a 16,384-entry codebook, 256-dimensional embeddings, and a 426M-parameter convolutional architecture.Tokenizer training proceeds in stages, beginning with 200k steps of standard objectives and continuing with 40k steps of text and face perceptual losses.
- Autoregressive Generator: InsightAR is trained on approximately 150M 512² images represented by 1,024 tokens, using a 7B-parameter multimodal model and a visual-tokenizer adapter.The training mixture includes LAION, Flux-Reason-6M, Echo-4o, and synthetic text-rendering data.
- Evaluation: Table 1 evaluates specialized reconstruction with T-ACC, T-NED, and F-Sim, and general reconstruction with rFID and PSNR on 512 × 512 images.BPP measures the number of representation bits divided by image spatial resolution.
- Qualitative Comparison: Figure 5 compares Janus-Pro and InsightAR on generated images using the same prompts.Additional visualizations are provided in Appendix G.
5 Experiments
Experiments show that InsightTok improves text and face reconstruction while preserving general reconstruction quality, and that these gains transfer to autoregressive image generation. Ablations indicate that localized perceptual supervision and area-based reweighting are important to balancing specialized fidelity with overall quality.
- Image Reconstruction: InsightTok improves text accuracy by 28.89 percentage points and face similarity by 0.09 over IBQ at the same compression ratio.It also outperforms Emu3.5-IBQ despite its 131k-entry codebook.
- Image Reconstruction: InsightTok remains competitive on general reconstruction, reaching a PSNR over 23.6 without sacrificing non-textual and non-facial regions.
- Autoregressive Text-to-Image Generation: InsightAR achieves the highest MagFace score among autoregressive models using the same number of tokens per image.The evaluation generates crowds containing up to twenty individuals.
- Autoregressive Text-to-Image Generation: InsightAR consistently generates long-form text with higher accuracy, using normalized edit distance against ground-truth text.The evaluation renders paragraphs of up to 300 characters on blank backgrounds.
- Autoregressive Text-to-Image Generation: InsightAR remains comparable to Janus-Pro and other autoregressive models on general multimodal generation while producing clearer text and more faithful facial details.
- Analytic Experiments: Area-based loss weighting prevents specialized perceptual losses from degrading other regions, limiting changes to rFID and PSNR.Without reweighting, text and face losses dominate optimization and worsen general reconstruction quality.
- Analytic Experiments: Applying specialized losses only to the decoder yields marginal gains, suggesting improvements arise from a refined latent representation rather than decoder strength alone.
- Analytic Experiments: InsightTok outperforms OCR-VQGAN because its localized text loss is more sensitive to text patterns than OCR-VQGAN’s global supervision.
6 Conclusion
The paper attributes text- and face-centric failures in discrete image generation to insufficiently targeted tokenizer supervision and proposes localized, domain-specific perceptual losses. Experiments show improved reconstruction and downstream autoregressive synthesis while maintaining competitive general quality.
- InsightTok targets text readability and facial fidelity with localized, domain-specific perceptual losses.
- InsightTok outperforms existing tokenizers on text and face reconstruction while maintaining competitive general reconstruction quality.
- The improvements transfer to autoregressive image generation, producing higher-quality text and face synthesis in quantitative and qualitative evaluations.
- The findings support aligning tokenizer supervision with perceptually critical content as a practical direction for discrete image generation.
A Limitations and Broader Impact
InsightTok’s scope is targeted: it improves reconstruction for text and faces, with English-only text support, while its tokenizer uses EMA codebook updates and random restarts to maintain codebook utilization. Improved facial fidelity also carries dual-use and demographic-bias risks.
- Limitations: InsightTok targets text and face reconstruction rather than reconstruction across all visual content.
- Limitations: Current text reconstruction and generation are limited to English text and English prompts; extending to other languages is left for future work.
- Broader impact and risks: Improved facial fidelity may enable accessibility and creative applications but also creates risks including identity cloning and impersonation.
- Broader impact and risks: Pretrained face-recognition models may propagate demographic biases, motivating more diverse models and demographic performance audits.
- Tokenizer design: InsightTok updates codebook embeddings with EMA and periodically reinitializes rarely used codes to maintain high utilization.
C Additional Analytic Experiments
Additional experiments examine model scaling, supervision components, detector coverage, and comparisons with continuous tokenizers. The results support stronger reconstruction with larger models, targeted supervision, and better region identification, while clarifying that continuous and discrete tokenizers serve different generative interfaces.
- Model scaling: Reconstruction performance improves as model size increases.
- Supervision ablations: Text-aware and face-aware losses improve their corresponding targets, while combining both achieves strong performance with only a slight trade-off.
- Detector coverage: Better region identification leads to stronger reconstruction quality when detector coverage is varied.
- Continuous-tokenizer comparison: Continuous tokenizers provide an approximate upper bound because they typically use higher bottleneck capacity, but they are not directly interchangeable with discrete tokenizers.
D Extra Overhead
InsightTok adds limited computational cost: recognition models contribute less than 2% of total FLOPs, while measured training latency rises by approximately 2% and GPU memory remains nearly unchanged.
- Theoretical cost: Recognition models account for less than 2% of total FLOPs because only small detected regions receive perceptual supervision.
- Empirical cost: Training latency increases from 2056 ms to 2099 ms per iteration, approximately 2%, under the same batch-size-16 setup.
E Implementation Details
InsightTok uses a convolutional VQGAN-style tokenizer with a 16× downsampling factor, a 16,384-entry codebook, and staged training that adds localized text- and face-aware supervision. InsightAR maps these discrete visual codes into a language-model interface for autoregressive image generation.
- Tokenizer architecture: The tokenizer uses five convolutional stages with 16× downsampling, four residual blocks per stage, and a 16,384-entry codebook with 256-dimensional embeddings.
- Training data: Training combines unannotated images for general reconstruction with region-annotated text- and face-containing data.
- Training objective: The objective combines reconstruction, codebook, LPIPS, adversarial, text-aware, and face-aware losses.
- Training procedure: Tokenizer training proceeds through pretraining, text-and-face training, and decoder fine-tuning phases.
- Autoregressive generator: InsightAR uses an MLP adapter to map visual codebook embeddings into a 7B language model’s input space and concatenates image tokens with prompt tokens.
- Controlled comparison: A controlled LlamaGenTok-AR variant uses the LlamaGen tokenizer while following the same training recipe.
F Benchmark and Evaluation Protocol
The evaluation covers specialized reconstruction benchmarks for text and faces, general reconstruction metrics, and targeted generation tests for facial and textual fidelity.
- Compression Rate: BPP measures the bits used to represent an image divided by its spatial resolution.The bit count equals tokens per image multiplied by codebook information capacity.
- Text Reconstruction: TokBench evaluates text reconstruction with T-ACC and T-NED across diverse text images and size categories.Text instances include varied fonts, styles, scales, and backgrounds, with OCR-based comparison to ground-truth transcriptions.
- Face Reconstruction: TokBench measures face reconstruction using cosine similarity between ground-truth and reconstructed face embeddings.The benchmark uses unconstrained images containing multiple faces, including small face instances.
- General Reconstruction: ImageNet-1K validation images provide general reconstruction evaluation with rFID and PSNR.Both metrics are computed on 50k validation images.
- Face Generation: InsightAR face generation is tested with 15 prompts in crowded scenes containing up to twenty people.Small face regions make the setting sensitive to preservation of fine-grained facial details.
- Text Generation: InsightAR text rendering is evaluated on 200 English quotes containing 100 to 300 characters using OCR-based normalized edit distance.The quotes are generated on blank backgrounds, and additional examples are shown in Figure 8.
G Additional Visualizations
Additional qualitative visualizations show InsightAR outputs for general text-to-image generation.
- Figure 9 presents additional qualitative results for text-to-image generation using InsightAR.