Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision

1 Introduction

• Existing language-pretraining frameworks rely on language context for self-supervision and do not borrow grounding information from the visual world.The paper contrasts this with human language learning that uses multiple modalities, especially vision.Source: §1 · p. 1
• The BERT realization jointly optimizes masked language modeling and voken classification on the same inputs while sharing model weights.Masked language modeling predicts masked tokens; voken classification predicts their corresponding contextually generated vokens.Source: Figure 2 · p. 2
• Visually grounded datasets differ from general-language corpora in scale and distribution, and only about 28% of English Wikipedia tokens are estimated to be grounded.The introduction contrasts about 120M grounded-language tokens with 3,300M in BERT training data and 220B in T5.Source: §1 · pp. 1–2
• A vokenizer trained on small visually grounded datasets contextually maps tokens to images, then generates vokens for large language corpora.The sentence context can help map otherwise non-visually-grounded tokens, such as “angry,” to related images.Source: §1 · pp. 1–2
• Voken-supervised models consistently improve over BERT on GLUE, SQuAD, and SWAG, and the vokens also transfer to RoBERTa.The introduction reports the same phenomenon with RoBERTa.Source: §1 · p. 2

2.1 Vokens: Visualized Tokens

• Vokens are images aligned with text tokens and support a proposed voken-classification pretraining task.Source: §2.1 · p. 2

2.2 The Voken-Classification Task

• Localized per-token language features can feed a linear-softmax classifier that predicts corresponding vokens without changing the model architecture.Source: §2.2 · p. 3

Example: Visually-Supervised BERT

• BERT combines masked-token prediction with voken classification over all tokens, summing the two losses with a ratio λ.Source: Example: Visually-Supervised BERT · p. 3

2.3 Two Challenges in Creating Vokens

• Phrase localization is costly and insufficient; image-caption sentence–image mappings are the most promising source, so the authors propose contextual token-image matching to address both challenges.Source: §2.3 · pp. 3–4
• Grounded captions differ from general corpora in sentence length, vocabulary, n-gram distributions, and scale; English Wiki’s grounding ratio is 27.7%, almost half Visual Genome’s.Source: §2.3 · pp. 3–4

3.1 The Vokenization Process

• Each token receives a relevant image retrieved from a finite image set using its relevance to the token in its sentence context.The resulting vokens can serve as targets for a voken-classification task during language-model training.Source: §3.1 · p. 4

3.2 Contextual Token-Image Matching Model

• A contextual matcher scores token-image pairs using the whole sentence, learning from caption-image pairs by contrasting aligned images with unaligned images.At inference, retrieval uses nearest-neighbor search over the learned feature representations.Source: §3.2 · pp. 4–5

3.3 Revokenization

• Revokenization transfers vokens between tokenizers by assigning each target token the voken of the source token with the greatest span overlap.The overlap is quantified by intersection-over-union of token ranges in the raw sentence.Source: §3.3 · pp. 5–6

4.1 Pre-training Data and Fine-tuning Tasks

• The models are pretrained on English Wikipedia and Wiki103, then fine-tuned on GLUE, SQuAD, and SWAG, with four larger GLUE datasets selected for evaluation.The selected GLUE datasets are SST-2, QNLI, QQP, and MNLI.Source: §4.1 · p. 6

4.2 Implementation Details

• The study excludes Toronto Books Corpus because it is not publicly released, prioritizing reproducibility over matching the original BERT training data.The paper notes that the original BERT also used this corpus.Source: §4.2 · p. 6
• Pretraining removes next-sentence prediction, fixes sequence length at 128, and runs BERTBASE for 200K Wikipedia steps or a reduced six-layer model for 40 Wiki103 epochs.The reduced model is used because it cannot fit the full English Wikipedia dataset.Source: §4.2 · p. 6

4.3 Results

• Voken classification consistently improves downstream task performance and yields large average gains, with the same pattern observed when the vokenizer transfers to RoBERTa.Table 2 reports fine-tuning results across GLUE, SQuAD, and SWAG, with task averages summarizing improvement from voken supervision.Source: Table 2 · p. 7, §4.3 · p. 7

5.2 Token-Level vs. Sentence-Level Approaches

• Token-level approaches outperform sentence-level approaches across retrieval methods and supervision types, with contextual vokens ahead of sentence-label and term-frequency alternatives.Sentence-level retrieval uses sentence labels or propagates them to tokens; term-frequency retrieval ignores context. The comparison uses the same hyperparameters and dataset as BERT12L/768H+Voken-cls.Source: Table 5 · p. 8, §5.2 · p. 8

5.1 Limit of Visually-Grounded Language

• Models trained on MS COCO perform significantly worse than Wiki103-trained models on all downstream tasks, while vision-and-language models resemble BERT with random weight noise.The paper cautions that its BERT comparison is not fully fair because the original BERT also used the unavailable Toronto Books Corpus.Source: §5.1 · pp. 7–8

5.3 Visualization of Vokens

• Voken visualizations show different images selected for different tokens and contextual disambiguation, but also ungrounded tokens and shifted phrase alignments.The authors suggest these misalignments may reflect sentence-image weak supervision without strong token-image annotations.Source: §5.3 · pp. 8–9, Figure 4 · p. 9

Language (Model) Pre-training

• Language pre-training shifted from token-level and sentence-level objectives toward contextual token-level supervision, which informs the paper’s contextual token-level design.The cited approaches include word and sentence representations, followed by contextual encoders such as LSTMs and Transformers.Source: §6 · p. 9

Vision-and-Language Pre-training

• Vision-and-language pre-training builds joint cross-modal representations for multimodal tasks, but the paper reports that these models do not improve pure-language tasks.The paper attributes this limitation to the particularity of grounded language.Source: §6 · p. 9

Visually-Aided Language Learning

• Earlier visually aided language work targets specific tasks or grounded-language representations, whereas this work focuses on visually supervised pre-training for general language understanding.The paper notes that many visual-language representation studies focus on grounded language and may face the dataset discrepancy discussed in Sec. 2.3.Source: §6 · p. 9

7 Conclusion

• Contextual token-image matching vokenizes language corpora, and models supervised by the generated vokens significantly improve over purely self-supervised models on multiple language tasks.The conclusion describes the approach as a way to address challenges in visually grounded language.Source: §7 · p. 9

A.1 Full Implementation Details

• The contextual token-image matching model is trained on MS COCO for 20 epochs using BERT and ResNeXt features, with both backbone models frozen.The token and image features are concatenated last-four-layer BERT outputs and mean-pooled ResNeXt-101-32x8d feature maps; the learned projection modules use two fully connected layers.Source: §A.1 · p. 13
• Vokenization retrieves Visual Genome images at 100K tokens per second on four Titan V100 GPUs, finishing full Wikipedia in eight hours.The retrieved images are not used in MS COCO training, and the voken size is 50,000.Source: §A.1 · p. 13
• Pure-language pre-training uses BERTBASE on English Wikipedia for 200K steps and a reduced six-layer model on Wiki103 for 40 epochs.The training removes next-sentence prediction and uses fixed sequence length 128; pure MLM receives around 10% more steps to account for the voken-classification computation.Source: §A.1 · p. 13
• GLUE, SQuAD, and SWAG fine-tuning use specified task schedules, with implementations and evaluation code borrowed from PyTorch Transformers.The passage specifies three epochs for GLUE, SQuAD v1.1, and SWAG, and two epochs for SQuAD v2.0.Source: §A.1 · p. 14

• The estimate uses MS COCO, and the section presents a randomly sampled list of 2,406 grounded tokens.Source: §A.2 · p. 14
• The sample includes visual entities and settings such as photograph, skyscraper, mound, and courtyard.Source: §A.2 · p. 14
• It also includes actions and descriptors such as skiing, waving, rusty, and nicely.Source: §A.2 · p. 14

A.3 Maximum Inner Product Search of Norm-1 Vectors

• For norm-1 vectors, maximizing inner product is equivalent to nearest-neighbor search.The proof assumes a unique vector attains the maximum inner product.Source: §A.3 · p. 14

A.4 Details of Sentence-level Retrieval in Analysis

• Sentence-level retrieval uses a sentence embedding—the first-token CLS output—as the language representation and assigns the retrieved image as the sentence label.The image is retrieved from the same image set used for vokenization.Source: §A.4 · p. 14

A.5 Details of Token-level Retrieval in Analysis

• Token-level retrieval indexes image-captioning sentences and aggregates VQA and Visual Genome data with MS COCO; the experiment uses temperature γ=0.01.Source: §A.5 · pp. 14–15

A.6 Voken Ablation Studies

• All tested alternatives to standard vokenization score lower than the reference strategy.The alternatives include random vokens, shuffled vokens, and original tokens used as supervision.Source: Table 6 · p. 14, §A.6 · p. 15

A.7 Correlations between Improvements and Grounding Ratio

• Datasets with higher grounding ratios, such as MNLI, show significant gains, while lower-grounding QNLI does not benefit much from visual supervision.The authors say the correlation may indicate that visual supervision helps understanding of visually grounded tokens, and that mapping non-grounded words depends on the data’s original grounding ratio.Source: §A.7 · p. 15