$\left|\,\circlearrowright\,\boxed{\text{BUS}}\,\right|$: A Large and Diverse Multimodal Benchmark for evaluating the ability of Vision-Language Models to understand Rebus Puzzles

1 Introduction

• The paper tests whether VLMs can turn rebus images into word-or-phrase answers, a task requiring image recognition, commonsense knowledge, and multi-step reasoning.Earlier rebus-puzzle work did not provide a diverse benchmark spanning difficulty levels or a model-agnostic solution for open- and closed-source models.Source: §1 · p. 2
• The benchmark contains 1,333 English rebus puzzles with answers and annotated metadata, including hints, difficulty, and whether visual attributes or spelling variations matter.ControlNet-generated backgrounds add visual distraction while preserving the puzzle's core content.Source: §1 · p. 2
• 20–30% relative improvement over description-only and VisProg baselines is reported for Qwen2-VL-7B, with F1 rising from 0.2 to 0.264 in three-shot RebusDescProgICE.For GPT-4o, Word-Level F1 rises from 0.489 in zero-shot normal prompting to 0.512 in three-shot RebusDescProgICE.Source: §1 · p. 2
• RebusDescProgICE combines detailed image descriptions, code-based reasoning, and reasoning-aligned in-context example selection in a model-agnostic framework.The framework uses minimal training and does not modify the VLM's underlying parameters.Source: §1 · p. 2
• The paper also identifies distracting visual backgrounds as a way to increase puzzle difficulty without changing the rebus itself.The benchmark's answers are in English.Source: §1 · p. 2

2 Background

• Code-based reasoning includes VISPROG [14], which generates modular code coordinating vision models and logical operations for visual problems, and PoT [7], which helps LLMs solve math by writing and running code.PAL [11] turns text problems into code, allowing a Python interpreter to handle computation.Source: §2 · p. 2

3.1 Our Annotation Pipeline

• The annotation pipeline comprises three stages for curating rebus puzzles and their metadata.The pipeline is shown in Figure 2.Source: §3.1 · p. 2, Figure 2 · p. 3

3.1.1 Stage 1: Collecting Rebus Puzzles from multiple Internet Sources.

• Stage 1 collects rebus images and ground-truth answers from three internet sources, removes duplicates to retain 722 puzzles, and manually verifies or corrects answers.Four annotators record solving-related metadata, image features, and whether attributes such as color, position, orientation, size, or object count matter.Source: §3.1.1 · p. 2, §3.1.2 · p. 3

3.1.3 Stage 3: Using ControlNet to obtain modified versions of Rebus Puzzles.

• ControlNet converts the 722 collected puzzles into modified images; annotator review retains 611 that preserve the original answer, bringing the dataset to 1,333 puzzles.The images are treated as Canny Edge Maps, with generated images filtered for meaningfulness and answer consistency.Source: §3.1.3 · p. 3

3.2 Dataset Description and Details

• The dataset's 1,333 puzzle images include 611 ControlNet-generated images in a different artistic style, and answers are organized into 18 categories predicted by ChatGPT.The categories span language and expression, knowledge and facts, culture and society, activities and hobbies, and nature and living things.Source: §3.2 · p. 4
• A CLIP-embedding UMAP visualization places ControlNet-generated puzzles semantically far from their originals despite identical answers, illustrating visual diversity.The images are color-coded by artistic style.Source: §3.2 · p. 4, Figure 5 · p. 4

3.3 Proposed approach: RebusDescProgICE

• RebusDescProgICE generates an unstructured image description and structured code-based reasoning, then retrieves training examples using embedding similarity to the test input.A unified embedding represents rebus-puzzle images, and example selection is based on similarity between code-based reasoning components.Source: §3.3 · p. 4

4.2 Baselines

• Baselines include zero-shot prompting, few-shot CoT with puzzle hints, and few-shot prompts using GPT-4o-generated image descriptions, with examples randomly sampled from the holdout set.Source: §4.2 · p. 4

4.1 Experimental Setup

• The evaluation benchmarks GPT-4o, GPT-4o-mini, GPT-4 Turbo, Phi-3.5-Vision, Pixtral-12B, and Qwen2-VL-7B.Open-source experiments used four L40 GPUs with 48GB VRAM each.Source: §4.2 · p. 4, §4.1 · p. 4

4.3 Automated Evaluation Metrics

• Word-level F1 balances token-level precision and recall, while substring accuracy checks whether the prediction appears contiguously within the reference.Substring accuracy accommodates multiple surface realizations when a prediction partially overlaps the reference answer.Source: §4.3 · p. 5

4.4 RebusDescProgICE vs. Baselines

• Qwen2-VL-7B F1 rises from 0.200 to 0.241 in one-shot prompting and reaches 0.264 in three-shot RebusDescProgICE, while closed-source models outperform open-source models overall.On augmented data, RebusDescProgICE also raises Pixtral’s F1 from 0.186 with one-example CoT to 0.201 with two examples.Source: §4.4 · p. 5
• On augmented data, GPT-4o reaches a maximum word-level F1 of 0.402, and RebusDescProgICE raises Pixtral’s F1 from 0.186 to 0.201.The authors attribute the modest scores to layered semantic reasoning compounded by ControlNet-augmented noisy backgrounds; more in-context examples do not uniformly improve performance.Source: §4.4 · p. 5

• Closed-source VLMs remain far ahead of open-source models on rebus puzzle solving.Source: §5 · p. 5
• RebusDescProgICE is robust across settings and especially benefits open-source models that otherwise struggle.Source: §5 · p. 5
• The conclusion identifies integrated VisProg and descriptive grounding—not simply adding examples—as the main performance boost.Source: §5 · p. 5