PaperlayerView exact revision on arXiv

Source-linked AI summary

ILLUME: Rationalizing Vision-Language Models through Human Interactions

Manuel Brack, Patrick Schramowski, Björn Deiseroth, Kristian Kersting

arXiv:2208.08241v4cs.LGcs.AIcs.CLcs.CVcs.HC

TL;DR

Pretrained language models can bootstrap VLMs, but their rationales may not align with human intent. ILLUME iteratively samples rationales and uses human feedback to fine-tune VLMs; the authors report competitiveness with approaches trained on substantially larger datasets while lowering annotator workload.

  • Problem

    Rationales from VLMs bootstrapped from pretrained language models may not align with human intent, limiting alignment of answers with users’ reasons.

    Source: §1 · p. 1

  • Method

    ILLUME iteratively samples rationales from a VLM and fine-tunes it using human feedback on fitting model-generated explanations.

    Source: §1 · p. 1, §3.3 · p. 3

  • Results

    ILLUME remains competitive with approaches trained on substantially larger datasets while transferring commonsense reasons from language models to downstream VLMs.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The interactive feedback loop can lower annotator workload and align model rationales with users’ rationales.

    Source: §6 · p. 9

    ILLUME yielded no satisfying CLEVR-X results, which the authors attribute to difficulty transferring logical reasoning from language models using few training examples.

    Source: §5 · p. 9

Abstract

from arXiv · show

Bootstrapping from pre-trained language models has been proven to be an efficient approach for building vision-language models (VLM) for tasks such as image captioning or visual question answering. However, outputs of these models rarely align with user's rationales for specific answers. In order to improve this alignment and reinforce commonsense reasons, we propose a tuning paradigm based on human interactions with machine-generated data. Our ILLUME executes the following loop: Given an image-question-answer prompt, the VLM samples multiple candidate rationales, and a human critic provides feedback via preference selection, used for fine-tuning. This loop increases the training data and gradually carves out the VLM's rationalization capabilities that are aligned with human intent. Our exhaustive experiments demonstrate that ILLUME is competitive with standard supervised finetuning while using significantly fewer training data and only requiring minimal feedback.

1. Introduction

ILLUME addresses the misalignment that can pass from pretrained language models to bootstrapped vision-language models. It uses human feedback on model-generated rationales to transfer commonsense reasoning and align rationalizations with human intent.

1. Introduction

  • The loop samples multiple rationales for a VQA input, has an annotator select fitting explanations, then fine-tunes on selected rationales and repeats.It stops when each sample has a fitting reason or no progress is observed.Source: Figure 1 · p. 1
  • ILLUME combines language models’ commonsense capabilities with human interaction on machine-generated data to transfer commonsense knowledge and align VLM rationales with human preferences.The process is intended to achieve both goals in a single tuning process.Source: §1 · p. 1
  • The paper evaluates three VLMs’ commonsense reasoning, transfer of language-model rationalization to vision, iterative tuning with human feedback, and rationale generation for immoral images.VQA is the target formulation, described as encompassing most vision-language tasks.Source: §1 · p. 2
  • The authors report competitive performance on open-ended visual reasoning benchmarks and satisfactory rationales on a socio-moral task.They frame the approach as balancing human feedback against the labor of producing ground-truth data.Source: §1 · p. 2

2. Background

Prior work explores human-in-the-loop tuning, visual commonsense reasoning, and open-ended VQA explanations. ILLUME builds on this landscape by using a pretrained multimodal VLM as a more versatile alternative to task-specific explanation architectures.

2. Background

  • InstructGPT showed that human-in-the-loop language-model tuning can yield outputs people prefer to those of larger conventionally trained models; ILLUME applies minimal critic feedback to multimodal capability transfer.The paper positions its approach as similar in using human feedback on self-generated samples, but for multimodal applications.Source: §2 · p. 2
  • Visual commonsense reasoning datasets include rationale explanations, but one cited task requires selecting answers and explanations from predefined options rather than open-ended generation.The passage identifies Zellers et al. as an example.Source: §2 · p. 2
  • PJ-X generates open-ended VQA explanations and evidence heatmaps, while FM uses textual and visual representations with a pre-existing answering model; these architectures are task-specific.ILLUME instead uses a pretrained multimodal VLM to leverage its underlying language-model capabilities.Source: §2 · p. 2

3. Rationalizing Vision-Language Models

The paper frames VQA rationale generation as open-ended multimodal reasoning and proposes transferring language-model capabilities to VLMs through iterative sampling, human feedback, and fine-tuning. Its method uses self-talk to assess language-model candidates, while ILLUME selects fitting generated explanations and tunes the VLM.

3.1. Problem Statement

  • ILLUME targets alignment of rationales with human intent while transferring language-model capabilities to VLMs, using feedback on generated data instead of human-written ground-truth explanations.The paper motivates VQA as a formulation applicable to most vision-language tasks and makes efficient decoder adjustments to elicit rationalization.Source: §3.1 · p. 2
  • The task is open-ended generation of a textual explanation e from an image, question, and answer tuple, rather than multiple-choice reasoning.Here, “explanation” means an explicitly generated text sequence, not an interpretation of the model’s output.Source: §3.1 · p. 2

3.2. Self-talk Prompting

  • Self-talk prompts a language model to generate clarification context and use it to predict an answer, providing a baseline for selecting language-model candidates for multimodal reasoning.The approach assumes language models that perform well on this task are suited to multimodal reasoning through VLMs.Source: §3.2 · p. 3

3.3. ILLUME: Tuning through Machine Interaction

  • ILLUME iteratively samples explanations from the tuned VLM and uses human feedback to mark fitting explanations before fine-tuning on selected self-generated samples.The unfiltered generated explanations are called “jabber”; the critic labels each as fitting or not fitting.Source: §3.3 · p. 3
  • Sampling combines top-k filtering with temperature sampling, and multiple temperatures produce varied outputs while aiming to retain syntactic and semantic soundness.Prompt engineering selects a suitable explanation prompt; using multiple prompts can increase diversity but substantially increases computing requirements, while the best prompt is sufficient in most cases.Source: §3.3 · p. 3
  • For benchmark evaluation, generated candidates can be scored against human-written references with ROUGE-L, but this requires labor-intensive prior labeling and is limited by metric shortcomings.The paper says this setup evaluates ILLUME on established benchmarks; it also demonstrates the approach on a socio-moral task using real human feedback without ground-truth explanations.Source: §3.3 · p. 3
  • Fine-tuning updates decoder bottleneck adapters and combines answer and explanation language-model losses; additional VQA samples help robustness and explanations while reintroducing prior task knowledge.The explanation set grows across iterations, and the VQA loss is scaled relative to the explanation data.Source: §3.3 · p. 4

4. Experimental Results

The experiments compare language-model and vision-language reasoning, then test whether ILLUME can improve visual rationales through iterative feedback. ILLUME is competitive with ground-truth-trained models on benchmark tasks where the model already has rudimentary capabilities, and improves human-rated rationales on a socio-moral task.

4.1. Experimental Protocol

  • The experiments compare three VLMs and six language-based commonsense benchmarks, alongside open-ended visual reasoning on VQA-X, ACT-X, and CLEVR-X.The authors select MAGMA for ILLUME based on its experiments in self-talk and zero-shot reasoning.Source: §4.1 · p. 4
  • The visual reasoning comparisons use automatic NLG metrics against ground-truth explanations, which can undervalue valid paraphrases and suitable explanations absent from the references.The authors consider the metrics useful for identifying poor performance but caution that comparisons between decently performing models lack significance.Source: §4.1 · p. 5

4.2. Self-talk Prompting

  • Across six commonsense tasks, GPT-based models GPT-J and Luminous outperform BERT and OFA in self-talk reasoning.BERT and OFA barely beat chance on most datasets.Source: §4.2 · p. 5, Table 1 · p. 6

4.3. Zero-Shot Visual Reasoning

  • On VQA-X validation data, OFA and BLIP repeat the answer in 63% and 89% of rationale prompts, respectively, while MAGMA produces decent zero-shot explanations.The authors use MAGMA for subsequent experiments.Source: §4.3 · p. 6, Figure 2 · p. 6

4.4. ILLUME – Benchmark

  • ILLUME’s benchmark performance improves rapidly at first and then converges; on ACT-X, ground-truth tuning shows a validation-score drop while ILLUME continues to improve.The authors suggest that new-sample growth below 5% may indicate performance saturation on datasets without ground-truth references.Source: §4.4 · pp. 6–7, Table 3 · p. 7
  • ILLUME slightly outperforms standard supervised learning on VQA-X and remains competitive with MAGMAfull while using less data and no ground-truth explanations.The authors qualify this result to tasks where the model has rudimentary capabilities through language or multimodal pre-training.Source: §4.4 · p. 7

4.5. ILLUME – Real World

  • On SMID, the study applies ILLUME to images rated below 2.0 for immorality, using 208 training photos and 66 held-out test photos with human feedback.Rationales rated excellent or satisfactory were labeled fitting for tuning.Source: §4.5 · p. 8
  • On the held-out SMID test set, ILLUME improves the quality of human-rated rationales over successive iterations, with fitting explanations outnumbering non-fitting ones after one iteration.Training once on all samples accumulated over two iterations produced higher human-rated quality than the iterative setup, an effect not observed on automatic NLG metrics.Source: §4.5 · p. 8, Table 4 · p. 8

5. Discussion & Limitations

The discussion highlights how ILLUME improves explanations across samples, while noting limits in logical reasoning, automated evaluation, and per-dataset tuning. It also identifies prompt-tuning and logic-based methods as possible directions for addressing these constraints.

Progressive Explanation Alignment

  • The discussion frames qualitative evaluation as a useful complement to NLG metrics for understanding how explanation alignment progresses.NLG metrics provide high-level information on iterative progress, while qualitative evaluation can provide additional insights.Source: §5 · p. 8
  • ILLUME’s VQA-X improvements arise from other self-generated examples rather than ground-truth explanations, and the authors report transfer to morality rationales such as bullying and child abuse.The same generalization capabilities are reported for the morality experiment.Source: §5 · p. 8

Limitations of ILLUME

  • Prompt-tuning is presented as a potential way to reduce computation and avoid tuning a dedicated adapter set for each dataset.Initial experiments optimizing the explanation-prompt embedding without changing parameter weights yielded positive results.Source: §5 · p. 9
  • Automated NLG metrics can score semantically identical explanations as dissimilar and reward explanations with little valid rationale, limiting model comparisons despite their use as method indicators.The authors motivate direct human feedback in training and evaluation.Source: §5 · p. 9

Flaws in Logical Reasoning

  • ILLUME yields no satisfying results on CLEVR-X; the authors attribute this to difficulty inferring logical reasoning from few examples and suggest logic-based methods as a possible alternative.They argue that training and evaluating logical reasoning as pure text generation may be inherently flawed.Source: §5 · p. 9

6. Conclusion

The paper concludes that ILLUME transfers commonsense reasons from language models to vision-language models while remaining competitive with training on substantially larger labeled datasets. It also points to further work on reasoning and describes a self-talk prompting procedure for eliciting clarification context.

6. Conclusion

  • ILLUME transfers commonsense reasons from language models to downstream vision-language models and remains competitive with approaches trained on substantially larger labeled datasets.The authors state that interactive feedback can lower annotator workload and align models to users’ rationales.Source: §6 · p. 9
  • Future work includes investigating capability transfer from increasingly powerful public language models and strengthening transformers’ logical and commonsense reasoning capabilities..Source: §6 · p. 9

A. Natural Language Self-Talk Prompting

  • Self-talk prompting elicits world knowledge by generating clarification context for a question, then using the premise and that context to predict the final answer.For each dataset, question prefixes guide generation of clarification questions and answers.Source: A. Natural Language Self-Talk Prompting · p. 13

B. Visual Reasoning Benchmark Datasets

The benchmark datasets provide explanations for visual question-answering or related visual reasoning tasks, with different origins and annotation processes. The VQA-X test split is constructed from its original validation set because no public test split is available.

B. Visual Reasoning Benchmark Datasets

  • VQA-X adds human-annotated explanations to COCO-based VQA datasets, while ACT-X adds explanations for human activities and is cast as VQA using prompts such as “What is the person doing”.ACT-X is not originally a VQA task, since it provides an activity answer but no question.Source: B. Visual Reasoning Benchmark Datasets · p. 13
  • CLEVR-X provides synthetically generated explanations based on CLEVR scene graphs, using templates with varying wording.The paper describes its ground-truth labels as similar in quality to human-generated ones.Source: B. Visual Reasoning Benchmark Datasets · p. 13
  • Because the VQA-X test split is not publicly available, the authors randomly divide the original validation set into custom validation and test sets..Source: B. Visual Reasoning Benchmark Datasets · p. 13

C. Threshold in Automatized Feedback

The threshold study selects ROUGE-L ≥0.7 as a trade-off between the number of generated training samples and their closeness to ground truth. Its results also indicate that the model can generalize with small sample sizes.

C. Threshold in Automatized Feedback

  • ROUGE-L ≥0.7 is selected because raising the threshold rapidly reduces samples, while explanation scores fall as sample counts decrease; the study uses VQA-X validation results.The authors report that maintaining some variation from ground-truth alignment may support a more robust training process.Source: C. Threshold in Automatized Feedback · p. 13, Table 5 · p. 13
  • The threshold experiment indicates that ILLUME can generalize with small sample sizes, supporting applicability when few fitting explanations are initially available..Source: C. Threshold in Automatized Feedback · p. 13

D. Training on Self-Generated Samples

The section examines how well ILLUME can train on its own fitting explanations and how VQA-task data affects iterative fine-tuning. It reports similar performance to matched ground-truth training, while additional VQA samples help prevent overfitting across iterations.

D. Training on Self-Generated Samples

  • Initial sampling yielded fitting explanations for roughly 4% of VQA-X and 10% of CLEVR-X training examples, but no more than 1% for ACT-X.The ACT-X baseline produced significantly fewer fitting explanations.Source: D. Training on Self-Generated Samples · p. 13
  • Training on 1,207 self-generated explanations performed similarly to training on the same ground-truth explanations, though other random ground-truth subsets scored slightly better.The authors note that the difference between matched self-generated and ground-truth explanations was negligible and might reflect noise.Source: Table 6 · p. 14, D. Training on Self-Generated Samples · p. 14
  • The self-generated-sample experiment compared matched training quantities, using 1,207 fitting model-generated explanations or ground-truth explanations, alongside a random ground-truth subset.The experiment used the same hyperparameters across the three VQA-X training setups and did not add further VQA samples.Source: D. Training on Self-Generated Samples · p. 14

E. ILLUME as Continual Learning Setup

VQA-X ACT-X CLEVR-X

  • After two iterations, ILLUME with additional VQA samples clearly outperformed the alternatives; explanation-only tuning performed worse than zero-shot MAGMA.Without additional VQA samples, the setup barely improved over baseline.Source: VQA-X ACT-X CLEVR-X · p. 15

F. Benchmarking MAGMA for VQA Explanation Generation

The benchmark estimates MAGMA’s performance on VQA explanation generation using ground-truth training explanations. MAGMA is reported as competitive with specialized models, but the comparisons and tuning are limited.

F. Benchmarking MAGMA for VQA Explanation Generation

  • MAGMA was fine-tuned separately on each dataset using ground-truth explanations from the entire training split.The benchmark compares MAGMA with PJ-X and FM, identified as existing VQA explanation-generation models.Source: F. Benchmarking MAGMA for VQA… · p. 15
  • The benchmark used a limited guided hyperparameter search, and its automated-metric results are intended only to provide performance intuition.The authors did not conduct exhaustive grid searches and did not aim to report new state-of-the-art results.Source: F. Benchmarking MAGMA for VQA… · p. 15
  • MAGMA remained competitive with PJ-X and FM across VQA-X, ACT-X, and CLEVR-X, though VQA-X comparisons use different splits.The authors describe the results as estimates rather than new state-of-the-art claims.Source: F. Benchmarking MAGMA for VQA… · p. 15
  • The model-capacity comparison focuses on zero-shot VQA-X explanation generation because MAGMA’s underlying language model only slightly outperforms the base model on language reasoning.Source: F. Benchmarking MAGMA for VQA… · p. 15

G. ILLUME – Extended Metrics

The section reports extended ILLUME results for VQA-X and ACT-X.

G. ILLUME – Extended Metrics

H. Flaws in Logical Reasoning

On CLEVR-X, explanation quality declined over successive ILLUME iterations, and the sampled training subset did not generalize well. The section also reports iterative results for VQA-X and ACT-X.

H. Flaws in Logical Reasoning

  • On CLEVR-X validation, each ILLUME training iteration reduced textual-explanation quality and generated fewer fitting explanations; fine-tuning on the 5–10% self-talk training subset also failed to generalize explanations to the rest of the training set.Source: H. Flaws in Logical Reasoning · p. 16
  • Fine-tuning on the 5–10% self-talk training subset failed to generalize explanations to the rest of CLEVR-X training data.After fine-tuning on the training split, all metrics except CIDEr dropped significantly.Source: H. Flaws in Logical Reasoning · p. 16
  • Table 11 reports ILLUME iterations on VQA-X and ACT-X until validation scores plateau, with test-set scores in its bottom rows.Its delta values compare self-generated-sample training with training on the same amount of ground-truth data.Source: Table 11 · p. 17
Loading 2208.08241v4…