DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback

• DRESS uses critique feedback to assess response strengths and weaknesses, and refinement feedback to provide specific suggestions for improving responses toward the ground-truth reference.Source: §1 · p. 2
• To train on non-differentiable feedback, DRESS generates responses conditioned on critique or refinement using language-modeling loss on the responses.Source: §1 · p. 2
• The paper evaluates helpfulness on open-ended visual question answering, honesty on image captioning, harmlessness on adversarial prompting, and interaction on multi-turn tasks.Source: §1 · p. 2
• DRESS is reported to learn from feedback during multi-turn interactions and refine responses on the fly.Source: §1 · p. 2
• DRESS reports relative improvements over prior SOTA of 9.76% for helpfulness, 11.52% for honesty, and 21.03% for harmlessness alignment.Source: §1 · p. 2
• The authors produce and open-source 63K annotated vision-language NLF samples covering helpfulness, honesty, and harmlessness.Source: §1 · p. 2

• Many LVLMs use frozen LLMs, image-caption pairs, and a projector that maps image features into the LLM embedding space [2].Source: §2 · p. 2
• Large-scale vision-language instruction tuning is used to align LVLMs with human preferences and support instruction following [20] [22] [32] [42] [64] [75].Source: §2 · p. 2
• Feedback for LLM training includes numerical scores, preference rankings, and natural language; DRESS relies on LLMs for NLF because it is harder and more expensive to annotate [1] [4] [79].Source: §2 · p. 2
• Unlike work using preference-ranking data and numerical-score rewards [66], DRESS categorizes NLF as critique or refinement and uses generalized conditional reinforcement learning for alignment and interaction.Source: §2 · p. 2

3.1. Training Recipe for DRESSft

• DRESSft uses image-caption pretraining, visual instruction data, and VLSafe adversarial prompts to prepare the model and feedback-collection data.The recipe retains 161K samples for SFT and uses 3K VLSafe samples for feedback annotation.Source: §3.1 · p. 3

3.2. Gathering Feedback From LLMs

• GPT-4 provides 3H ratings, concise critique NLF, and refinement suggestions; iterative turns use prior responses and refinement feedback to generate new responses.The feedback data distinguishes critique about response strengths and weaknesses from concrete refinement advice toward the ground-truth reference.Source: §3.2 · p. 4

3.3. Harnessing Feedback for Training

• Conditional reinforcement learning trains DRESS to produce responses from scores, critique, refinement, and prior turns, using cross-entropy loss on responses.Scores and critique support alignment learning, while refinement feedback across turns supports learning to incorporate feedback in interaction.Source: §3.3 · pp. 4–5

3.4. Inference

• At inference, DRESS uses the “<excellent> [Nice response.]” prefix to target its best response after training on score verbalizers and critique-conditioned responses.The model is trained to distinguish responses using numerical score verbalizers and critique NLF.Source: §3.4 · p. 5

4. Experiment

• The experiments compare DRESS with six prior LVLMs and cover helpfulness, honesty, harmlessness, multi-turn interaction, fundamental visual question answering, and ablations.The evaluation section also includes qualitative analysis.Source: §4 · p. 5, §4.1 · p. 5

4.3. Image Captioning for Honesty Evaluation

• DRESS significantly reduces hallucination compared with previous instruction-finetuned LVLMs when evaluated on image captioning.The evaluation uses CHAIR metrics to assess mismatches between generated and annotated objects; lower scores indicate less mismatch.Source: Table 3 · p. 6, §4.3 · p. 6

4.4. Robustness to Adversarial Prompting for Harmlessness Evaluation

• On adversarial prompts, DRESS is less readily elicited to produce harmful responses than LVLMs without harmlessness alignment, and its responses can persuade people against causing harm.All instruction-tuned models attain response-relevance scores close to 100, according to the passage.Source: Table 4 · p. 6, §4.4 · p. 7

4.2. Open-ended Visual Question Answering for Helpfulness Evaluation

• DRESS achieves overall better helpfulness scores than previous SOTA LVLMs across three LLaVA-Eval question types and on LLaVA-Bench, with higher Relevance and Level of Detail scores there.The evaluation uses GPT-4 scoring; the supplied passages do not provide numerical table values.Source: §4.2 · p. 6

4.5. Multi-turn Interaction

• In multi-turn interactions, DRESS can learn from provided natural-language feedback and continually refine previous responses, while existing LVLMs cannot take advantage of that feedback; Figure 3 reports results averaged across two feedback providers.The authors attribute this effectiveness to incorporating refinement NLF in the training dataset.Source: Figure 3 · p. 6, §4.5 · p. 7

4.6. Fundamental Capability

• DRESS performs comparably to existing LVLMs on fundamental visual question answering, especially on knowledge-extensive OK-VQA; removing RLAIF regularization degrades performance.The authors say regularization helps preserve knowledge and visual concepts acquired during pretraining.Source: Table 5 · p. 7, §4.6 · p. 7

4.7. Ablation Study

• The ablation study reports a 6.24% relative improvement from RLAIF and finds benefits from critique and refinement NLF, as well as fine-grained helpfulness and honesty feedback.The ablations are evaluated on LLaVA-Eval.Source: Table 6 · p. 8, §4.7 · p. 8

4.8. Case Study

• Case examples contrast DRESS’s refusal and discouragement of harmful actions with prior models’ potentially harmful suggestions, and show more helpful, visually grounded responses and feedback integration.The comparisons include InstructBLIP for helpfulness and LLaVA for visual grounding without hallucination.Source: §4.8 · pp. 8–9

5. Conclusion

• DRESS combines fine-grained helpfulness, honesty, and harmlessness annotations with critique and refinement NLF, using generalized conditional reinforcement learning to improve alignment and multi-turn interaction.The conclusion describes critique and refinement as two categories of feedback.Source: §5 · p. 9

A.1. Vision-Language Modeling

• The paper frames LVLM research around systems that handle varied tasks without specific adaptations, with recent approaches leveraging LLMs and instruction finetuning.The appendix describes common LVLM architectures as connecting image features to LLMs.Source: §A.1 · p. 13

A.2. Interaction.

• The paper treats interaction as dynamically refining an original response from feedback across turns, and proposes concrete ways to improve LVLM interaction ability.Examples mentioned include creative writing and web navigation.Source: §A.2 · p. 13

Model Architecture

• DRESS connects a frozen image encoder and an LLM through a linear projector that maps image-patch embeddings into the LLM embedding space.The model uses EVA-CLIP-Giant and Vicuna-13b-v1.5 for the encoder and LLM, respectively.Source: B. Model Architecture &… · p. 13
• The model fine-tunes the projector and an LLM LoRA module across pretraining, SFT, and RLAIF.LoRA tunes a small set of LLM parameters.Source: B. Model Architecture &… · p. 13

Hyper-parameter Configuration

• Pretraining runs for 20K steps, SFT for one epoch, and RLAIF for three epochs, each with cosine learning-rate decay and 1K warmup steps.The respective peak learning rates are 5e-5 for pretraining and 1e-4 for SFT and RLAIF.Source: B. Model Architecture &… · p. 13

Quality of the LLM-generated Feedback

• For 300 sampled instances, three annotators rated responses and assessed critique and refinement feedback as good or bad, with majority vote deciding feedback appropriateness.The response rating was averaged across the three annotators.Source: C. Quality of the LLM-generated Feedback · p. 13
• GPT-4 ratings had a Spearman rank correlation of 0.93 with human ratings.Source: C. Quality of the LLM-generated Feedback · p. 13
• Annotators’ appropriateness scores for LLM-generated critique and refinement feedback were 0.91 and 0.96, respectively.The authors interpret these metrics as validating strong alignment between GPT-4-generated feedback and human scores.Source: C. Quality of the LLM-generated Feedback · p. 13

D. Quality of the LLM Evaluation

• Human ratings validate the training framework, with human–GPT-4 Spearman correlations of 0.92 on LLaVA Eval and 0.81 on LLaVA Bench.Three annotators rated helpfulness on a 1–10 scale; ratings were averaged.Source: D. Quality of the LLM Evaluation · p. 15
• GPT-4’s harmlessness and general-capacity evaluations achieve precision/recall of 0.91/0.98 and 0.89/0.97, respectively, against human annotations.The human annotations covered sampled sets of 300 responses for each evaluation.Source: D. Quality of the LLM Evaluation · p. 15

E. Examples of Annotated NLF

• Examples illustrate LLM-generated helpfulness, hallucination, and harmlessness annotations, plus multi-turn interactions using refinement feedback.The authors attribute insightful feedback and appropriate scores to dense image annotations and detailed annotation guidance.Source: E. Examples of Annotated NLF · p. 15

F. VLSafe

• The authors use GPT-3.5-Turbo for image-related harmlessness data because GPT-4 mostly refuses adversarial prompts and open-source unaligned LLMs have weaker reasoning and language use.GPT-4 had been extensively fine-tuned to exclude unsafe questions.Source: F. VLSafe · p. 15
• The authors iteratively refine instructions on COCO data and filter samples by failure mode, while withholding prompt-construction details to prevent malicious use.They describe the approach as discrete optimization drawing on textual adversarial-attack research.Source: F. VLSafe · p. 15
• The final VLSafe dataset contains 5,874 samples, split into 4,764 training and 1,110 evaluation samples.The paper points to Figure 10 for two typical examples.Source: F. VLSafe · p. 15

G. Iterative Generation-Annotation

• The iterative generation-annotation process creates long multi-turn interaction data so LVLMs can learn to improve earlier responses using refinement feedback.The process begins with DRESSft generating responses that LLMs score and annotate.Source: G. Iterative Generation-Annotation · pp. 15–16
• For a second turn, samples rated 1 or 2 are selected for possible improvement; others are saved directly to the feedback dataset.New responses are conditioned on the question, prior response, and refinement feedback, then annotated by LLMs.Source: G. Iterative Generation-Annotation · p. 16
• Failed trajectories are retained so LVLMs can learn from critique feedback, while successful trajectories may continue through further selection and refinement.In successful trajectories, samples rated 2 or 3 may be selected for additional improvement; the dataset includes up to four turns.Source: G. Iterative Generation-Annotation · p. 16

H. Qualitative Examples

• Qualitative examples show DRESS aligning with preferences for helpfulness, honesty, and harmlessness and refining earlier responses during interaction.Figure 12 presents examples of on-the-fly response refinement.Source: H. Qualitative Examples · p. 16

I. Discussion

• The authors propose using feedback scores and critique rationales to train an LVLM for web-scale RLAIF on image-caption pairs with partial visual information.A distilled LVLM could provide scores and critique feedback for training; the dataset’s questions can be generated from image-caption pairs.Source: §I · p. 16
• Constructive critique for improvement is described as challenging, with only GPT-4 observed to perform it well [72].The authors suggest training another LVLM as an inference-time critic that learns common failure modes and misaligned patterns.Source: §I · pp. 17–18
• The paper treats natural dialogue as an early multi-turn setting and proposes extending the work to tasks that combine refinement feedback with external tools.Embodied AI and web navigation are named as potential tasks.Source: §I · p. 18

J. Instruction & Prompts to Guide GPT-4

• Appendix J collects the instructions and prompts used for feedback annotation and automatic evaluation.Source: J. Instruction & Prompts to Guide GPT-4 · p. 18

J.1. Honesty Annotation

• Honesty annotation compares a model response with a ground-truth reference, using image consistency and a 1–4 rating to guide concise corrective feedback.The prompt supplies scene descriptions and bounding boxes, and asks for a brief reason, rating, and feedback without mentioning scene descriptions.Source: §J.1 · pp. 18–19

J.2. Helpfulness Annotation

• Helpfulness annotation rates usefulness, relevance, and query adherence from 1 to 4, emphasizing practical image-grounded information and concise improvement feedback.Its rubric ranges from unrelated answers to responses that integrate visual information into a comprehensive, useful answer.Source: §J.2 · p. 19

J.3. Harmlessness Evaluation

• Harmlessness evaluation asks whether a response contains harmful content misaligned with human values and ethics, then requests constructive feedback against the ground truth.The required output uses “Harmfulness” and “Feedback” fields.Source: §J.3 · p. 20

J.5. LLaVA Bench Evaluation

• LLaVA Bench evaluation gives an overall 0–10 score based on helpfulness, relevance, accuracy relative to scene descriptions, and level of detail.The prompt uses one descriptive image sentence as reference and instructs GPT-4 to grade strictly.Source: §J.5 · p. 21
• The LLaVA Bench prompt calls for a low score when a response is unhelpful or contains ungrounded visual elements.Source: §J.5 · p. 21
• LLaVA Bench evaluation presents the scene description, query, model response, and ground-truth response as inputs to the evaluation prompt.Source: §J.5 · pp. 21–22

J.6. VL Safe Evaluation

• VL Safe evaluation frames the evaluator as a helpful assistant assessing a model response to a malicious query.Source: §J.6 · p. 22
• VL Safe evaluation assigns three binary scores to a vision-language model’s response to a malicious query: informativeness, safety, and persuasiveness in discouraging malicious actions.The prompt requests a dictionary-formatted response.Source: §J.6 · p. 22
• The VL Safe prompt requires strict security scoring, assigning 0 as soon as a response contains any unsafe content.Source: §J.6 · p. 22

J.7. VQA Evaluation

• VQA evaluation asks whether a predicted answer has the same meaning as reference answers given the visual question, and requires a direct Yes-or-No judgment.Source: §J.7 · p. 22
• The VQA prompt directs evaluators to compare answers using the task, question, and answers’ context rather than superficial patterns such as word overlap.Source: §J.7 · p. 22