Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization

1 INTRODUCTION

• Prior approaches treat visual input as an unsupervised prompt for text generation, constraining models to image-conditioned text comprehension.Adapter-based methods also delegate vision-language alignment to modules with limited trainable parameters.Source: §1 · p. 1
• Emu jointly trains visual and textual tokens using visual-feature regression, but its differing image and text objectives are not conducive to unified multimodal modeling.LaVIT instead applies a unified generative objective to both modalities.Source: §1 · p. 2
• LaVIT represents images as discrete tokens like a foreign language, enabling images and text to be handled under one autoregressive next-token objective.The approach requires no specific architectural modification to the LLM.Source: §1 · pp. 2–3
• The visual tokenizer selects informative patches, merges discarded-patch information onto retained patches, and quantizes the results into discrete codes.The resulting sequence length varies with image content, and the retained tokens carry high-level semantics from similar patches.Source: §1 · p. 2
• LaVIT achieves state-of-the-art zero-shot performance on a wide range of vision-language tasks.It can take any modality combinations as input and perform in-context generation of both images and text.Source: §1 · p. 3

Vision-Language Pre-training

• Early vision-language pretraining used dual encoders with contrastive objectives to learn generic cross-modal aligned representations.Source: §2 · p. 3
• Most LLM-based vision-language methods use adapter-style networks to bridge a pretrained vision encoder and a frozen language model, while visual inputs often remain prompts outside optimization.Examples include Flamingo’s Perceiver Resampler and QFormer-based methods.Source: §2 · p. 3

Vector Quantization in Computer Vision

• Most quantized visual-token methods use fixed sequences of 512 or 1024 tokens, whereas LaVIT’s tokenizer supports dynamic lengths to reduce redundancy and inference computation.Source: §2 · p. 3
• LaVIT’s visual tokenizer selects informative patches and merges discarded-patch information onto retained tokens to preserve image semantics.The tokenizer is trained to maximize semantic reconstruction.Source: Figure 3 · p. 4

3.1 STAGE-1: DYNAMIC VISUAL TOKENIZER

• The dynamic tokenizer selects informative patches, merges dropped patches into retained tokens by feature similarity, and quantizes the resulting variable-length visual features into discrete codes.Its training objective encourages semantic reconstruction while controlling the target mean percentage of retained tokens; a conditional denoising U-Net can decode generated tokens into realistic images.Source: §3.1 · pp. 4–5

3.2 STAGE-2: UNIFIED GENERATIVE MODELING

• LaVIT concatenates image and text tokens in either order and trains them with a shared autoregressive language-modeling objective.For image-conditioned text generation, it uses continuous visual features rather than quantized embeddings to mitigate detail loss in fine-grained tasks such as visual question answering.Source: §3.2 · p. 6

3.3 MODEL PRE-TRAINING

• Stage 1 trains the tokenizer on about 100M LAION-400M images for 50K steps, while keeping the ViT encoder frozen and using a target retention rate of 1/3.The tokenizer stage uses pure image data without captions; its conditional U-Net is initialized from Stable Diffusion v1.5 and finetuned for 20K steps on the same dataset.Source: §3.3 · p. 6
• Stage 2 uses a raw 7B LLaMA and about 93M image-text samples for comprehension, with an additional 100M LAION-Aesthetics pairs for text-to-image synthesis.English RedPajama text is mixed into training to reduce catastrophic forgetting of the LLM’s reasoning capacity.Source: §3.3 · p. 6

4.1 ZERO-SHOT MULTIMODAL UNDERSTANDING

• LaVIT leads both zero-shot understanding benchmarks, covering NoCaps/Flickr30k captioning and four VQA datasets scored with CIDEr and VQA accuracy.The VQA datasets are VQAv2, OKVQA, GQA, and VizWiz.Source: §4.1 · pp. 6–7, Table 1 · p. 7

4.2 ZERO-SHOT MULTIMODAL GENERATION

• LaVIT improves 4.3 FID over Emu and matches Parti with 0.2B versus 2B training image-text pairs.The comparison is on zero-shot text-to-image synthesis on MS-COCO-30K.Source: §4.2 · p. 7, Table 2 · p. 7
• Without fine-tuning, LaVIT accepts text, image+text, and image+image prompts, producing images that reflect prompt semantics and style.The examples include modifying an input image using multiple prompt images.Source: §4.2 · p. 8, Figure 4 · p. 8

4.3 ABLATION STUDY

• Replacing next-visual-token classification with visual-embedding regression severely degrades model performance in the objective ablation.The ablation compares discrete-token classification with regression of the next visual embeddings.Source: §4.3 · p. 8, Table 3 · p. 9
• Dynamic tokenization averages 94 tokens, about 36% of the fixed length, accelerates training by 40%, and reduces inference computation.The comparison concerns dynamic versus fixed-length tokenization.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 QUALITATIVE ANALYSIS

• The tokenizer filters repetitive or trivial background patches, while learned codes group similar patterns and convey interpretable visual semantics.Examples include a code for a skateboard part and another for giraffe texture.Source: §4.4 · p. 9, Figure 5 · p. 9

5 CONCLUSION

• LaVIT uses a dynamic visual tokenizer to represent vision and language as unified discrete tokens under an autoregressive generative learning paradigm.The conclusion presents LaVIT as capable of simultaneously performing multimodal understanding and generation.Source: §5 · p. 9

A THE DETAILS OF IMAGE SYNTHESIS

• Figure 6 details image synthesis from a textual prompt; generation for prompts in other modalities is similar.Source: Figure 6 · p. 14
• MS-COCO text-to-image evaluation uses top-k sampling with a maximum of 300 sampled tokens and a softmax temperature of 1.0.Source: A THE DETAILS OF IMAGE SYNTHESIS · p. 14
• The evaluation sets αcfg to 1.5, generates four samples per prompt, and selects the best using a CLIP-VIT/L image-prompt matching score.Larger αcfg favors prompt alignment while sacrificing some diversity.Source: A THE DETAILS OF IMAGE SYNTHESIS · p. 14

B.1 THE IMPACT OF VISION-LANGUAGE PRE-TRAINING ON LLM

• The evaluation measures common knowledge and reasoning on MMLU in a 5-shot setting; Table 4 reports results across learning rates and data proportions.Table 4’s caption describes the comparison and notes only a slight drop in original common knowledge, but provides no cell values here.Source: §B.1 · p. 14, Table 4 · p. 15
• With LLM learning rate 5e−5 and 1.5e−4 for other parts, LaVIT slightly affected language ability (-1.5 on MMLU) and achieved the best results on multimodal understanding and generation tasks.The authors suggest a 2:1 language-to-vision-language data proportion to maintain learned knowledge.Source: §B.1 · p. 15
• An e−4-level LLM learning rate speeds vision-language task convergence but causes an 8.0 performance drop in common knowledge compared with original LLaMA.The authors suggest an e−5-level LLM learning rate and e−4-level visual learning rate as a balance.Source: §B.1 · p. 15

B.2 MORE ABLATION STUDY

• Causal attention outperforms bidirectional self-attention across all multimodal understanding tasks; the authors conjecture that partial information leakage explains the gap.The comparison is reported in Table 6.Source: §B.2 · p. 15, Table 6 · p. 15
• For image-to-text inputs, continuous visual features achieve better multimodal understanding performance than quantized codebook vectors.The authors use token-merger outputs as LLM inputs and the corresponding discrete codes as next-visual-token supervision; Table 7a reports the ablation.Source: §B.2 · pp. 15–16, Table 7 · p. 16
• Removing the token merger causes a distinct performance drop in multimodal understanding tasks.The merger compresses unselected patches onto retained ones by feature similarity; Table 8 reports this ablation.Source: §B.2 · p. 16, Table 8
• Freezing the LLM while optimizing only the token merger leads to noticeable performance degradation compared with unlocking the LLM during pretraining.The comparison is reported in Table 7b.Source: §B.2 · p. 16, Table 7 · p. 16

C MORE VISUALIZATIONS

• The denoising U-Net reconstructs original images from discrete visual tokens with high semantic and general-structure consistency.Figure 7 shows the pixel-decoding examples.Source: C MORE VISUALIZATIONS · pp. 16–17
• Visual codes represent high-level semantics, with examples including a human arm, car part, bus, and donuts.Figure 8 arranges patches sharing the same discrete code together.Source: C MORE VISUALIZATIONS · p. 17, Figure 8 · p. 17
• LaVIT produces image samples described as high-quality, content-rich, and appearance-diverse, including samples aligned with long, complex prompts.The authors report that Stable Diffusion samples miss subtle details in some complicated prompts.Source: C MORE VISUALIZATIONS · p. 17
• Multimodal in-context image generation reflects input-context semantics and supports diversified editing of the input image.Figure 9 illustrates examples of this generation capability.Source: C MORE VISUALIZATIONS · p. 18, Figure 9 · p. 18
• LaVIT captions image content and answers questions about detailed visual clues in the illustrated understanding examples.The examples are shown in Figure 13.Source: C MORE VISUALIZATIONS · p. 18

D PRE-TRAINING DETAILS

• Table 9 reports LaVIT’s detailed training hyperparameters.The passage supplies no individual hyperparameter values.Source: D PRE-TRAINING DETAILS · p. 18

• LaVIT inherits LLM hallucinations, including nonexistent knowledge and factual errors.Source: E LIMITATIONS · p. 18
• Training only on web-scale image-text pairs with noisy, short descriptions leaves LaVIT insufficient for modeling extensive text-image correspondences.Source: E LIMITATIONS · p. 18
• The authors suggest that more high-quality, long image-text interleaved documents may alleviate the correspondence-modeling limitation.Source: E LIMITATIONS · p. 18