Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning

• Object correctness matters to people, especially visually impaired users, and improvements on captioning metrics do not necessarily reduce hallucination.Source: §1 · p. 1
• Scaling the same biased data or increasing model size is not expected to eliminate object biases.Source: §1 · pp. 1–2
• It augments captions and corresponding object-label inputs by replacing objects, without extra data or model parameters.Source: §1 · p. 2
• Normalizing object co-occurrence statistics reduces language-prior and visual-feature dependence, curbing frequent-object hallucinations and enabling correct lower-probability predictions.Source: Figure 1 · p. 1, §1 · p. 2
• The method improves CHAIR hallucination scores while boosting baseline image-captioning evaluation metrics.Source: §1 · p. 2
• The approach works with both cross-entropy and REINFORCE losses.Source: §1 · p. 2

• Captioning advanced from attention-based encoder-decoder models to detector features, metric-optimized training, and Transformers.Source: §2 · p. 2
• Prior bias work targets gender with confidence/appearance loss, action preferences with ConceptNet-based causality [46], and VQA spurious correlations through image modification.Source: §2 · p. 2
• Object hallucination can escape standard similarity metrics, motivating CHAIRs for sentence-level and CHAIRi for object-level measurement [33].Source: §2 · p. 2

3. Methods

• The methods aim to reduce object bias by making object co-occurrence statistics more uniform through data augmentation.The authors hypothesize that a more uniform co-occurrence matrix will make models hallucinate less.Source: §3 · p. 2

3.1. A Small Tweak to Any Captioning Model

• Object-label embeddings are concatenated with bounding-box features, enabling augmentation of labels and caption supervision.The captioning model then follows the same training procedure with the extended features.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2. Sentence Simplification

• Sentence simplification removes object-related adjectives before augmentation, avoiding descriptions that may become false when an object is replaced.The simplified captions also provide sentence variation that acts as a regularizer for exploiting language priors.Source: §3.2 · p. 3

3.3. Augmentation of Sentences

• After simplification, augmentation samples a replacement object according to a distribution conditioned on another object in the sentence.The methods differ in how they define that sampling distribution.Source: §3.3 · p. 3

3.3.1 Uniform Sampling

• Uniform sampling gives every object equal probability and disregards dataset statistics.This strategy is motivated by the hypothesis of creating a uniform object-label co-occurrence matrix.Source: §3.3.1 · p. 3

3.3.2 Inverse Multinomial Sampling

• Inverse multinomial sampling favors replacement objects that co-occur infrequently with the paired object in the dataset.The sampling distribution uses the dataset's object co-occurrence matrix.Source: §3.3.2 · p. 3

3.3.3 Updating Co-Occurence Matrix

• Co-occurrence updating tracks and adjusts the matrix during sampling to reduce the risk of creating a new bias toward low-frequency pairs.Tracking co-occurrences during training is also described as diminishing shortcut opportunities and allowing faster convergence to a uniform matrix.Source: §3.3.3 · p. 4

4.1. Dataset and Baseline Models

• Experiments use the Karpathy split of MSCOCO, with 113,287 training images and 5,000 images each for validation and testing.The training images have five captions each.Source: §4.1 · p. 4
• Caption quality is measured with CIDEr, BLEU, METEOR, SPICE, and SPICE-U; hallucination is measured with CHAIRs and CHAIRi, for which lower is better.CHAIRs is sentence-level and CHAIRi is object-level.Source: §4.1 · p. 4
• The evaluated baselines are UpDown, which attends over salient detector regions, and AoA, which adds attention to assess relevance between attention results and queries.Models given object labels as inputs are referred to as UD-L and AoA-L.Source: §4.1 · p. 4

4.2. Implementation Details

• Models are trained for 30 epochs using cross-entropy or REINFORCE, with original or augmented captions randomly selected as ground truth.The best validation CIDEr-D model is kept; inference uses no beam search, and testing uses ground-truth object labels as input.Source: §4.2 · p. 4

4.3. Comparison to State of Art

• Larger models, stronger captioning metrics, and better features do not by themselves reduce CHAIR scores.The authors observe this among state-of-the-art and comparison models.Source: §4.3 · p. 4
• Self-critical training boosts captioning metrics but makes models hallucinate more.The authors report this pattern for several models in Table 1.Source: §4.3 · p. 4
• Concatenating object labels with visual features reduces hallucination and beats state-of-the-art models on CHAIR by around 1 to 4%.The authors report this result for both UpDown and AoA models.Source: §4.3 · p. 5
• The methods' hallucination reduction is bounded at around 2–3%, with the remaining hallucination attributed mostly to noisy visual features.The authors distinguish this visual-feature noise from dataset bias, which their methods target.Source: §4.3 · p. 5
• Co-Occurrence Updating performs best on CHAIR, while Inverse Multinomial sampling performs best on standard captioning metrics.The table reports results on the Karpathy test split; lower CHAIR scores are better.Source: §4.3 · p. 5, Table 1 · p. 5
• Proposed methods maintain or improve informative, unique captions on SPICE-U, with a reported 2% improvement in self-critical training.The authors also report beating some state-of-the-art SPICE-U numbers without training an extra LSTM.Source: §4.3 · pp. 5–6

4.4. What if we have perfect label extractor?

• Ground-truth object labels improve all metrics compared with object-detector labels.The authors note this is expected because the models were trained with ground-truth annotations.Source: §4.4 · p. 6
• With ground-truth labels, the label-versus-augmentation gap is 0.9% and 0.6% for UpDown, and 1.7% and 1.1% for AoA on CHAIRs and CHAIRi, respectively.The authors suggest that more precise labels could raise the proposed methods' performance.Source: §4.4 · p. 6
• Co-Occurrence Updating achieves the best CHAIR scores across models trained with cross-entropy or self-critical loss.The authors connect this result to their hypothesis that making the co-occurrence matrix uniform decreases object bias.Source: §4.4 · p. 6

4.5. Data augmentation effect on the models

• Augmentation makes models rely less on visual features and place more emphasis on object labels.The authors report less visual-feature significance for augmented UD-L variants than for UD-L.Source: §4.5 · p. 6
• With ground-truth labels, the reported difference is 18% for CHAIRs and 17% for CHAIRi.For UD-L, the corresponding detector-to-ground-truth differences are reported as 0.1% and 0.4%.Source: §4.5 · p. 7
• Co-Occurrence Updating exploits object labels the most among the four compared models.The authors report a 15% improvement from UD-L to UD-L+Occ in the label-use comparison.Source: §4.5 · p. 7

4.6. Captioning images with uncommon object pairs

• For object pairs with fewer than 200 co-occurrences, original UD and AoA models show higher object bias than their label-augmented variants.The filtered pairs account for 23.6% of the MSCOCO test set.Source: §4.6 · p. 7
• The authors report that augmentation handles low-frequency pairs with both cross-entropy and self-critical training.With perfect labels, results on these pairs are better than the overall results reported in Table 2.Source: §4.6 · p. 7
• The gap between original models and Co-Occurrence Updating is larger on low-frequency pairs, consistent with the authors' uniform co-occurrence hypothesis.The figure plots CHAIRs and CHAIRi for the selected pairs.Source: §4.6 · p. 7

4.7. Ablation Study

• With sentence simplification, Uniform and Inverse Multinomial results improve by only 0.1%, which the authors say may reflect randomness.The paper says simplification does not seem to have much effect for these two sampling strategies.Source: §4.7 · p. 7
• Sentence simplification has little effect with Uniform and Inverse Multinomial sampling but significantly affects Co-Occurrence Updating.The ablation compares UpDown with and without sentence simplification.Source: §4.7 · p. 7, Table 4 · p. 7
• The authors conjecture that Co-Occurrence Updating selects more object pairs, allowing adjective–replacement-object correlations to emerge.Their example links “cute little” first with broccoli and then with clock; they contrast this with the other sampling strategies.Source: §4.7 · p. 8

4.8. Qualitative Results

• The models both delete hallucinated objects and replace incorrect objects with correct ones in the qualitative examples.Examples include removing predicted surfboard, frisbee, beach umbrella, or kite, and correcting a predicted dog or skateboard to a horse or traffic light.Source: §4.8 · p. 8
• Verb and action predictions remain the same, including sitting, running, and jumping.The authors identify action augmentation as a direction called for by these examples.Source: §4.8 · p. 8
• Even when captions are wrong, the models can identify the correct object but remain constrained by the language models.Source: §4.8 · p. 8

5. Conclusion

• The proposed sentence-augmentation techniques significantly diminish object bias on hallucination metrics without extra data or annotations, larger models, or architectural changes.Source: §5 · p. 8
• The methods decrease dependence on visual features and, by making object co-occurrence statistics more uniform, result in models generalizing better.Source: §5 · p. 8
• The conclusion reports much higher scores with advances in object detectors.Source: §5 · p. 8