Source-linked AI summary
Contrastive Instruction Tuning
Tianyi Lorena Yan, Fei Wang, James Y. Huang, Wenxuan Zhou, Fan Yin, Aram Galstyan, Wenpeng Yin, Muhao Chen
TL;DR
Instruction-tuned LLMs remain vulnerable to semantically equivalent instructions expressed in different forms, producing inconsistent outputs. The paper introduces COIN, which contrastively aligns representations of equivalent instruction-instance pairs and separates distinct ones, using paraphrased FLAN instructions. On PromptBench, COIN consistently improves robustness across four variation levels, with an average accuracy gain of +2.5% over continual instruction tuning.
Problem
Instruction-tuned LLMs are not robust to semantically equivalent instructions expressed with different forms or language styles, causing performance and response variation on unseen instructions.
Method
COIN uses contrastive learning to align hidden representations of semantically equivalent instruction-instance pairs and differentiate semantically distinct pairs, supported by paraphrased FLAN instructions.
Results
+2.5% average accuracy over continual instruction tuning on PromptBench across character-, word-, sentence-, and semantic-level instruction variations.
Takeaways & Limitations
COIN consistently enhances LLM robustness to semantic-invariant instruction variations across the four evaluated perturbation levels.
Takeaways & Limitations
The contrastive data selection method currently uses paraphrasing for positive instruction augmentation, while broader augmentation methods and larger evaluations remain open.
Abstract
from arXiv · showhide
Instruction tuning has been used as a promising approach to improve the performance of large language models (LLMs) on unseen tasks. However, current LLMs exhibit limited robustness to unseen instructions, generating inconsistent outputs when the same instruction is phrased with slightly varied forms or language styles. This behavior indicates LLMs' lack of robustness to textual variations and generalizability to unseen instructions, potentially leading to trustworthiness issues. Accordingly, we propose Contrastive Instruction Tuning, which maximizes the similarity between the hidden representations of semantically equivalent instruction-instance pairs while minimizing the similarity between semantically different ones. To facilitate this approach, we augment the existing FLAN collection by paraphrasing task instructions. Experiments on the PromptBench benchmark show that CoIN consistently improves LLMs' robustness to unseen instructions with variations across character, word, sentence, and semantic levels by an average of +2.5% in accuracy. Code is available at https://github.com/luka-group/CoIN.
1 Introduction
Instruction-tuned LLMs can respond inconsistently to semantically equivalent instructions expressed differently, limiting robustness to unseen instruction variations. COIN addresses this problem with contrastive instruction tuning and improves robustness across multiple perturbation levels.
- Current instruction-tuned LLMs may produce different, sometimes incorrect, responses when equivalent instructions differ in wording, style, format, or contain minor mistakes.Prior instruction-tuning methods primarily align outputs for individual instruction-input pairs rather than explicitly addressing robustness to instruction variation.
- COIN aligns hidden representations of semantically equivalent but textually different instruction-instance pairs while separating semantically distinct pairs.Positive pairs use perturbed instructions with the same instance, whereas hard negatives pair the same instruction with different inputs and outputs.
- COIN improves accuracy on unseen instructions across character-, word-, sentence-, and semantic-level variations, averaging +2.5% over continual instruction tuning on the same dataset.PromptBench experiments also show reduced response variation.
- COIN is evaluated on PromptBench, which varies instructions at character, word, sentence, and semantic levels.The method targets robustness when the same task intent is expressed through different language forms.
- The authors augment FLAN with contrastive instructions, releasing a dataset containing 52k entries and 104k instructions.The augmented collection is intended to facilitate contrastive instruction tuning and future work.
2 Related Work
Prior work shows that instruction-tuned LLMs remain vulnerable to variations in how instructions are expressed, while methods for improving this robustness remain underexplored. Related contrastive-learning approaches address other NLP goals, such as controllable attributes, rather than robustness to varied instructions of the same task.
- Instruction Tuning and Generalizability: Instruction-tuned LLMs can perform differently on unseen tasks when the same instruction is expressed through different wording, paraphrases, formats, or styles.This vulnerability motivates improving robustness to instruction variations.
- Robustness of Instruction-Tuned LLMs: Robustness studies report degradation under character-, word-, sentence-, paraphrase-, and format-level instruction perturbations.These findings establish varied instruction expression as a recurring robustness problem.
- Robustness of Instruction-Tuned LLMs: Enhancing robustness against varied instructions for the same task remains underexplored despite increased attention to evaluating instruction-tuned models.The paper positions its contribution as addressing this gap.
- Contrastive Learning: Contrastive learning trains models by contrasting positive and negative data pairs and has been adapted to several NLP tasks.Examples include sentence embedding, summarization, named entity recognition, and logical reasoning.
- Contrastive Learning: Within instruction tuning, prior contrastive learning with prefix-training steers outputs toward desired attributes rather than robustness to varied expressions of the same task.The targeted attribute is assumed known but can be difficult to specify across diverse tasks.
3 Contrastive Instruction Tuning
COIN improves robustness to instruction variations by contrasting hidden representations of semantically equivalent and distinct instruction-instance pairs. It combines paraphrased positives, hard negatives, contrastive learning, and generation loss to preserve task performance.
- 3.1 Overview: COIN defines contrastive instruction tuning as aligning hidden representations of semantically equivalent instruction-instance pairs while distinguishing semantically different pairs.The method targets robustness when equivalent instructions differ textually.
- 3.1 Overview: The method is motivated by inconsistent outputs from instruction-tuned LLMs when the same intent is expressed in different forms, limiting real-world reliability.COIN addresses this limitation by explicitly encouraging consistency across instruction expressions.
- 3.2 Contrastive Data Selection: COIN creates positive samples by perturbing instruction phrasing or template structure while keeping the original input and output unchanged.These positives teach the model to align instructions with the same underlying meaning.
- 3.2 Contrastive Data Selection: COIN uses hard negatives that retain the original instruction but pair it with different inputs and outputs, separating instances with different user intents.Negatives from different tasks already have low similarity and therefore provide weaker training signals.
- 3.3 Learning Objective: Each training batch matches an original sample with a perturbed-instruction positive and in-batch hard negatives sharing the instruction but using different instances.The batch construction operationalizes the positive and negative sampling strategy during optimization.
- 3.3 Learning Objective: COIN optimizes cosine-similarity contrastive loss to bring original and positive representations together while separating original and negative representations.The representations are taken from the decoder’s last-token hidden state, which performed better than max or average pooling in experiments.
- 3.3 Learning Objective: The objective includes standard cross-entropy loss for each instruction pair to preserve the language model’s generation ability.The generation term is computed over the desired output tokens.
- 3.3 Learning Objective: COIN balances contrastive and generation losses by scaling the contrastive term and imposing an upper bound on its weight.An excessively large contrastive weight adversely affects generation ability.
4 Experiment
Experiments evaluate COIN on PromptBench’s unseen instructions across 10 GLUE datasets and four perturbation levels. COIN consistently improves accuracy and reduces performance variation relative to the base model and continual instruction tuning.
- 4.1 Training Datasets: Training uses the FLAN Collection with paraphrased instructions, while evaluation samples unseen perturbed instructions and excludes training instructions to prevent contamination.The experiments use 25 deterministic-answer FLAN datasets for training and PromptBench for evaluation.
- 4.3 Evaluation Setting: PromptBench evaluates average accuracy and standard deviation over six instructions per GLUE dataset, using clean, character-, word-, sentence-, and semantic-level variants.The evaluation covers 10 GLUE datasets, and the instructions are unseen because their textual expressions do not occur in the instruction-tuning data.
- 4.4 Results: The base model reaches around 52% accuracy on clean instructions, with a 7.7% accuracy gap across perturbation groups and within-group variance from 16.9% to 19.0%.Its performance decreases further under character-, word-, and sentence-level perturbations.
- 4.4 Results: Continual instruction tuning raises accuracy but retains a performance gap of up to 6.1% between clean and word-level perturbation groups.The comparison uses the same data as COIN and cross-entropy loss only.
- 4.4 Results: COIN improves accuracy across all perturbation types, with gains of up to 4.4% for word-level perturbations over continual instruction tuning.The largest performance gap across groups falls to 3.6%.
- 4.4 Results: COIN reduces performance variance across clean, character-, word-, sentence-, and semantic-level instruction groups by 1.6%, 1.9%, 2.1%, 2.5%, and 1.2%, respectively.These reductions indicate more consistent performance across instructions within the evaluated groups.
5 Analyses
COIN analyses show that aligning representations of instruction variants corresponds with stronger robustness, with gains varying by task category and contrastive-loss weight.
- 5.1 Closer Representations of Instruction Variants: COIN brings character-, word-, and sentence-level instruction variants closer to clean representations, especially for word-level perturbations.The continually instruction-tuned model forms more distinct perturbation clusters, indicating greater sensitivity to formulation.
- 5.1 Closer Representations of Instruction Variants: Semantic perturbations remain relatively separated, but COIN still stabilizes representations of original and paraphrased instructions.Paraphrasing introduces new structure and wording, which can produce more varied hidden representations.
- 5.2 Impact on Different Tasks: Table 1 reports average accuracy and standard deviation by task category, with consistent improvements and larger gains on duplicate sentence detection and grammar correctness.The table summarizes performance across task categories rather than individual perturbation levels.
- 5.2 Impact on Different Tasks: +5.4% and +6.3% accuracy improvements appear on paraphrase identification and grammar correctness, respectively.Sentiment analysis and NLI show smaller gains of +1.4% and +1.7%.
- 5.3 Weighting of Contrastive Loss: COIN achieves its best average performance at λ = 1,000, whereas excessively large weights significantly reduce performance.The selected weight yields higher accuracy and smaller standard deviation.
6 Conclusion
The paper concludes that COIN aligns hidden representations of semantically equivalent instruction-instance pairs and improves robustness to instruction variations across four perturbation levels.
- 6 Conclusion: COIN aligns hidden representations of semantically equivalent instruction-instance pairs and improves robustness to character-, word-, sentence-, and semantic-level variations.The authors identify applying contrastive instruction tuning to other modalities and prompt components as future work.
Limitation
The paper identifies limitations in its positive instruction augmentation, experimental scale, and evaluation perspectives.
- COIN’s positive instruction augmentation currently considers only paraphrasing, leaving other semantic-invariant augmentation methods unexplored.
- The experiments could be expanded to more instruction-tuning datasets, instruction-tuned models, and downstream tasks for additional evidence of COIN’s effectiveness.
- Evaluating COIN from additional perspectives could improve understanding of contrastive instruction tuning beyond the current robustness evaluation.
A Datasets
The FLAN-based training collection covers diverse task categories, including language understanding, sentiment, paraphrase, reading comprehension, coreference, summarization, and question classification.
- The selected FLAN datasets span natural language inference, sentiment analysis, paraphrase detection, reading comprehension, coreference, summarization, and question classification.The collection includes 25 datasets with deterministic answers sampled according to the described selection rules.
- Natural language inference datasets assess relationships between sentence pairs, including ANLI, CB, MNLI, QNLI, and RTE.
- Sentiment-analysis datasets classify input text as positive or negative using IMDB, Sent140, SST2, and Yelp.
- Paraphrase-detection datasets assess semantic equivalence between sentences, while reading-comprehension datasets answer questions from passages containing the answers.
- Additional tasks include coreference resolution, summarization, and question classification across WSC273, AG News, TREC, CoLA, and WIC.
- Table 3 reports the number of entries filtered and selected from each dataset under the stated sampling rules.
B Detailed Experiment Results
The detailed experiment results section points to a comparison of COIN models trained with different contrastive-loss weightings.
- Table 4 reports average accuracy and standard deviation for COIN trained with different contrastive-loss weightings.
C GLUE Datasets Category
The GLUE datasets are categorized according to the task categories used by the PromptBench benchmark, with supporting tables documenting dataset sampling and contrastive-loss results.
- The GLUE datasets are split into four categories following the task categories defined in the PromptBench benchmark.
- Table 3 documents the number of entries sampled for each dataset from the FLAN collection.
- Table 4 reports average accuracy and standard deviation for COIN under different contrastive-loss weightings.
- Table 5 presents the GLUE task categories used in the PromptBench-aligned categorization.