PaperlayerView exact revision on arXiv

Source-linked AI summary

Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions

Taehyeon Kim, Joonkee Kim, Gihun Lee, Se-Young Yun

arXiv:2311.00233v2cs.CL

TL;DR

Instruction-tuned models can struggle with unfamiliar instructions, and task-specific prompt tuning is labor-intensive. Instructive Decoding contrasts next-token predictions from original and noisy instructions without parameter updates; across models and tasks, the opposite variant consistently shows notable performance gains.

  • Problem

    Instruction-tuned models often struggle with unfamiliar tasks, while prompt-engineering effectiveness can depend on task-specific prompt or initial-condition alignment.

    Source: §1 · p. 1

  • Method

    Instructive Decoding contrasts next-token logits from the original instruction with logits from a manipulated, potentially misleading but plausible noisy instruction during generation, without parameter updates.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    The opposite noisy-instruction variant consistently shows notable performance gains across various models and tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    ID-generated responses improve label adherence and coherence and help mitigate imbalances observed in standard decoding.

    Source: §1 · p. 3

    The risk of error propagation remains in ID’s two-step inference process: initial inaccuracies or biases may persist or be exacerbated in the subsequent inference.

    Source: §B.1 · p. 15

Abstract

from arXiv · show

While instruction-tuned language models have demonstrated impressive zero-shot generalization, these models often struggle to generate accurate responses when faced with instructions that fall outside their training set. This paper presents Instructive Decoding (ID), a simple yet effective approach that augments the efficacy of instruction-tuned models. Specifically, ID adjusts the logits for next-token prediction in a contrastive manner, utilizing predictions generated from a manipulated version of the original instruction, referred to as a noisy instruction. This noisy instruction aims to elicit responses that could diverge from the intended instruction yet remain plausible. We conduct experiments across a spectrum of such noisy instructions, ranging from those that insert semantic noise via random words to others like 'opposite' that elicit the deviated responses. Our approach achieves considerable performance gains across various instruction-tuned models and tasks without necessitating any additional parameter updates. Notably, utilizing 'opposite' as the noisy instruction in ID, which exhibits the maximum divergence from the original instruction, consistently produces the most significant performance gains across multiple models and tasks.

1 INTRODUCTION

Instruction-tuned models can struggle with unfamiliar tasks, while task-specific prompt tuning and dataset refinement demand substantial effort. The paper proposes Instructive Decoding, which contrasts predictions under original and noisy instructions; its reported gains are strongest with the opposite variant.

1 INTRODUCTION

  • Instruction-tuned models often struggle on unfamiliar tasks, while prompt tuning must be tailored to each task and dataset refinement requires substantial effort and computation.The authors also describe the model’s response to instructions as an area where many questions remain unanswered.Source: §1 · p. 1
  • Instructive Decoding contrasts next-token logits from the original instruction with logits from noisy variants during generation, without updating model parameters.The authors motivate this approach through the anchoring effect: different instructions can steer predictions in different directions.Source: §1 · p. 2
  • The opposite variant, designed to produce the greatest plausible deviation from the original instruction, consistently shows notable performance gains across models and tasks.ID-generated responses also improve label adherence and coherence and mitigate imbalances observed in standard decoding.Source: §1 · p. 3

2 INSTRUCTIVE DECODING

Instructive Decoding refines instruction-tuned models’ responses by contrasting predictions from the original instruction with predictions elicited by noisy variants. The section introduces the decoding setup, motivates this contrast, and describes automated and contrastive strategies for constructing noisy instructions.

2 INSTRUCTIVE DECODING

  • Instructive Decoding uses responses derived from noisy instructions in a contrastive technique intended to make generated responses more closely align with the provided instruction.The method is presented as response generation for instruction-tuned models.Source: §2 · p. 3

2.1 PRELIMINARY

  • In the autoregressive setup, the model predicts each next token from the instruction, context, and preceding output; token probabilities come from softmax over logits.The instruction serves as the guiding input for generalization to previously unobserved tasks.Source: §2.1 · p. 3

2.2 MOTIVATION AND OVERVIEW OF INSTRUCTIVE DECODING

  • ID contrasts logits from the original and noisy instructions, using their differences to refine outputs; this addresses the cost of dataset enrichment and the limits of two-model contrastive decoding.The authors hypothesize that differing instruction anchors can help refine responses; standard contrastive decoding compares a primary model with a less accurate amateur model.Source: §2.2 · pp. 3–4

2.3 A COLLECTION OF NOISY INSTRUCTIONS FOR INSTRUCTIVE DECODING

  • Noisy-instruction designs combine automated perturbations with prompts eliciting counter-intuitive but plausible responses, including truncation and shuffling, null instructions, random words, and opposite directives.Unless otherwise specified, experiments use one random word for Rand Words and a 0.6 truncation ratio for Trunc-Shuf.Source: §2.3 · p. 4

3 EXPERIMENTS

Experiments evaluate Instructive Decoding on unseen tasks across datasets, model families, and decoding conditions. ID generally improves performance over the baseline, with the opposite variant producing the largest gains, while results vary across task types and settings.

3.1 EXPERIMENTAL SETUP

  • SUPNATINST evaluation uses English data, 100 instances per task, and 119 tasks; UNNATINST evaluation uses 10,000 of its 64,000 samples.SUPNATINST includes 119 tasks, with 58 classification tasks used for LA and LC evaluation.Source: §3.1 · p. 5
  • Evaluations primarily use Tk-Instruct Large (770M), XL (3B), and XXL (11B), with T0, Alpaca, and OpenSNI also included.Unless specified otherwise, evaluations are zero-shot and use Rouge-L, with classification tasks also measured by EM, LA, and LC.Source: §3.1 · p. 5

3.2 PERFORMANCE ON UNSEEN TASK GENERALIZATION

  • ID consistently improves overall Rouge-L over the baseline across noisy-instruction variants, and opposite yields the largest gains across model sizes.Table 1 reports zero-shot results on held-out SUPNATINST tasks for Tk-Instruct models and OpenSNI-7B.Source: §3.2 · pp. 5–6, Table 1 · p. 6
  • Noisy instructions that cause greater Rouge-L degradation also tend to yield greater gains in ID; opposite causes the largest decline and boost.Figure 4(a) plots degradation against improvement and reports a strong positive correlation.Source: §3.2 · p. 6, Figure 4 · p. 6
  • Across 58 classification tasks, EM rises with model size, baseline LA and LC favor Tk-XL over Tk-XXL, and opposite notably improves XXL LC.Trunc-shuffle also significantly boosts LA as model size grows; Figure 5 shows performance changes across model sizes and noisy instructions.Source: §3.2 · p. 7, Figure 5 · p. 7

3.3 ABLATION STUDY

  • Cross-evaluation finds ID outperforming the baseline on UNNATINST with Tk-Large and on SUPNATINST with T0-3B and Alpaca-7B.The opposite variant is not the top performer in the Tk-Large UNNATINST evaluation, and Alpaca-7B shows a notable boost.Source: §3.3 · p. 7, Table 2 · p. 7
  • ID-null performance is stable for ϵ from 0.1 to 0.4, while negative values and values above 0.4 tend to reduce performance.The typical setting is ϵ = 0.3; Figure 6 shows Rouge-L across tested values from -0.5 to 0.5.Source: §3.3 · p. 7, Figure 6 · p. 7
  • Few-shot ID gains are more modest than zero-shot gains, and positive examples reduce the negative impact of noisy instructions.The few-shot evaluation uses two positive demonstration examples; Table 3 reports Rouge-L with ϵ set to 0.2.Source: Table 3 · p. 7, §3.3 · p. 8

4 DISCUSSION

The discussion connects ID’s opposite prompt to more varied label predictions and examines instruction embeddings as a possible source of differing anchoring effects. It also compares ID-amateur with standard contrastive decoding across model sizes.

Qualitative Analysis on ID with Opposite

  • Opposite diversifies predictions that baseline responses often concentrate on a single label, while combining opposite with the original instruction smooths responses without reversing predictions.The authors conjecture that the original instruction may alter response magnitudes but not directions when included in contrastive prediction.Source: §4 · p. 8

Visualization of Embeddings: Evidence of Anchoring Effect

  • Distinct instruction-embedding clusters accompany improvement in WA, whereas overlapping KT embeddings suggest weaker anchoring and lower Rouge-L.The visualization uses Tk-XXL encoder embeddings of concatenated instructions and inputs.Source: §4 · p. 8

On the Utility of ID over Contrastive Decoding

  • ID-amateur maintains performance across model sizes, while standard contrastive decoding weakens as the amateur model grows.With Tk-small as amateur and Tk-XL as expert, contrastive decoding modestly surpasses ID-amateur; the authors attribute this to limited understanding of noisy instructions in smaller models.Source: §4 · pp. 8–9, Figure 8 · p. 8

5 RELATED WORK

Related work examines instruction-tuned models’ generalization and sensitivity to instructions, alongside prior uses of contrast in text generation. This work applies contrast specifically to instructions, including noisy inputs that can degrade performance.

Instruction-tuned Language Models

Impact of Instructions on Generated Responses

Contrast in Text Generation

  • Prior contrastive text-generation methods include comparing less proficient and expert models, reweighting distant words, and addressing detoxification, knowledge conflicts, bias, or truthfulness (Ho & Salimans, 2022; Li et al., 2015; Shen et al., 2019; Yona et al., 2023; Liu et al., 2021; Li et al., 2022; Malkin et al., 2021; Shi et al., 2023).Contrastive Decoding contrasts a less proficient model with an expert-level model, while Coherence Boosting gives more weight to distant words.Source: §5 · p. 9
  • Unlike those approaches, this study focuses contrast on instructions and reports that even inputs causing severe performance degradation can improve unseen-task generalization when used for contrasting predictions.The passage describes this as extending prior contrastive work to the role of instructions in the input text.Source: §5 · p. 9

6 CONCLUSION

The paper concludes that Instructive Decoding improves instruction following by contrasting predictions from original and noisy instructions, with the opposite variant reported as especially effective. It also notes ethical concerns and points to materials describing the study’s reproducibility.

6 CONCLUSION

  • Instructive Decoding adjusts next-token predictions using a manipulated, plausible but deviating noisy instruction, and the opposite variant is reported as the most effective for improving performance.The method requires no additional parameter updates, and empirical results across multiple tasks confirm its efficacy.Source: §6 · p. 9
  • The work reports no direct ethical concerns but flags broader-impact implications related to systematic impact and possible misuse.Further discussion is located in Appendix A.Source: §6 · p. 10
  • The reproducibility statement directs readers to implementation details, experimental configurations, model origins, evaluation methods, and configurations for combining ID with other decoding techniques.It also says the appendix lists expected responses for instances using ID.Source: §6 · p. 10

Appendices

The broader-impact discussion presents ID as a decoding-level alternative to extensive retraining, with potential accessibility benefits and risks that warrant monitoring and safeguards.

A BROADER IMPACT

  • ID is presented as improving instruction adherence without extensive retraining, which the authors associate with reduced computational overhead and adaptability across tasks.The passage contrasts decoding-level optimization with traditional instruction fine-tuning that entails substantial resource commitments.Source: A BROADER IMPACT · p. 14
  • The authors state that ID can amplify instruction adherence for models up to 3B parameters without dependence on vast datasets, high-end compute, or specialized engineering teams.They frame this as making instruction-tuned models accessible to a wider range of users.Source: A BROADER IMPACT · p. 14
  • The authors say widespread adoption could help democratize instruction-following capabilities, while also requiring monitoring for unforeseen biases or unintended consequences.They describe potential streamlining and standardization of AI implementations across industries.Source: A BROADER IMPACT · p. 14
  • The authors warn that amplified instruction adherence could be exploited to disseminate misleading narratives or manipulate public discourse.They call for safeguards, ethical deployment standards, usage guidelines, and oversight.Source: A BROADER IMPACT · p. 14

B.1 LIMITATION

The limitations and future-work discussion identifies boundaries in generalization, robustness, and inference resources, and proposes broader evaluation and further analysis of ID.

B.1 LIMITATION

  • The method’s generality across instruction-following scenarios, languages, and cultures remains uncertain, and its effectiveness and potential biases in diverse contexts require validation.The authors describe current results as promising in specific tasks and datasets.Source: §B.1 · p. 14
  • The approach may be sensitive to slight changes in instructions or other input parameters, potentially yielding inconsistent outputs or varied performance.The authors call for testing across a range of inputs to assess stable operation.Source: §B.1 · pp. 14–15
  • Producing one output requires two separate inferences, increasing computational overhead and potentially affecting speed and storage requirements.The speed impact may be tangible in resource-constrained environments.Source: §B.1 · p. 15
  • The two-step process carries a risk that inaccuracies or biases from the initial inference persist or worsen in the subsequent inference.The authors say careful design and evaluation are needed to prevent initial errors from compromising final outputs.Source: §B.1 · p. 15

B.2 FUTURE WORK

  • Future work proposes reducing ID’s computational overhead and time complexity by optimizing perturbations, streamlining sampling, or using lightweight decoding heuristics.The stated aim is to improve resource efficiency.Source: §B.2 · p. 15
  • The authors propose evaluating ID across broader task types, including sequence-to-sequence and multimodal settings, as well as ambiguous tasks and unclear class boundaries.Their initial investigations are mainly focused on SUPNATINST.Source: §B.2 · p. 15
  • The authors identify applying ID to RLHF-enhanced models as a future direction, particularly for ambiguous or vague queries.They propose exploring a combination of RLHF’s human-feedback learning and ID’s instruction-anchoring capabilities.Source: §B.2 · p. 15
  • The authors call for theoretical analysis of how augmented instructions affect model predictions and the trade-offs among exact match, Rouge-L, and semantic coherence.They also propose analyzing the perturbation introduced into model decision-making.Source: §B.2 · pp. 15–16

C EXPERIMENTAL SETUP DETAILS

The setup uses several instruction-tuned models, with model-specific training and input-format choices. The paper evaluates open-ended instruction following with Rouge-L and classification behavior with label adherence and coherence measures.

C EXPERIMENTAL SETUP DETAILS

  • Tk-Instruct models use SUPNATINST training, while OpenSNI-7B uses the SUPNATINST template despite a different template in its training setup.Tk-Instruct training details include model variants, instance counts, and sequence-length limits; T0-3B and Alpaca-7B are also included in experiments.Source: C EXPERIMENTAL SETUP DETAILS · p. 16

D METRIC DETAILS

  • Rouge-L is the primary metric for open-ended zero-shot instruction following, which the authors consider better suited than label-likelihood ranking methods for this setting.The paper also notes that dialogue-focused evaluation frameworks are not ideally suited to zero-shot task generalization.Source: D METRIC DETAILS · p. 17
  • Across 58 classification tasks, label adherence checks whether outputs match the ground-truth label space, while label coherence uses an expanded space of semantically matching responses.The expanded label space is manually constructed and normalizes case and selected special characters.Source: D METRIC DETAILS · pp. 17–18, Table 5 · p. 17, Figure 9 · p. 18

E ADDITIONAL EXPERIMENTS

Additional experiments examine sampling, contrastive decoding, noisy-instruction variants, and an example of token-logit correction. Results indicate trade-offs in decoding and sensitivity to noise settings.

Decode by Sampling

  • Sampling-based decoding degrades performance across methods, with a particularly significant decline for ID, which the authors link to smoothing that makes sub-optimal tokens easier to sample.The comparison uses sampling-based and greedy decoding; the caption reports means and standard deviations over three seeds.Source: E ADDITIONAL EXPERIMENTS · p. 18, Table 6 · p. 18

CD ablations

  • ID-amateur reduces Contrastive Decoding’s sensitivity to expert–amateur model selection, with less degradation even when the expert is smaller than the amateur.The authors attribute this robustness to larger amateur models better understanding the opposite instruction and producing more significant noisy logits.Source: E ADDITIONAL EXPERIMENTS · pp. 18–19, Figure 10 · p. 19

Ablations on the Number of Random Words for Noisy Instruction

  • Increasing the number of random words in noisy instructions marginally lowers performance for Tk-Large and Tk-XL, suggesting a saturation point for added randomness.Table 7 presents performance across random-word counts for both models.Source: E ADDITIONAL EXPERIMENTS · p. 19, Table 7 · p. 19

Ablations on the Ration of Truc-Shuf

  • Neither extreme nor minimal truncation consistently maximizes performance, suggesting an intermediate Truc-Shuf truncation ratio may better balance task information and noise.The reported sensitivity varies between Tk-Large and Tk-XL.Source: E ADDITIONAL EXPERIMENTS · pp. 19–20, Table 8 · p. 19

Example on Logits Correction by ID

  • In the illustrated task, ID with the null instruction reinforces the correct “Correct” response by contrasting its predictions with the baseline’s ambiguous alternatives.The example shows baseline probability on “True” and “Correct” alongside some confusion with “Answer,” “Fal,” and “Yes.”Source: E ADDITIONAL EXPERIMENTS · p. 20, Figure 11 · p. 20

F QUANTITATIVE ANALYSIS ON TOKENS GENERATED BY ID

The token analysis finds that ID’s contrast varies by task and base-prediction confidence, while response-length gains are reported across the held-out set. Qualitative examples show instruction-focused errors, model-size effects, and limitations of fixed-reference evaluation.

F QUANTITATIVE ANALYSIS ON TOKENS GENERATED BY ID

  • In ID’s “opposite” decoding, classification tasks often show mismatched max-token indices between base- and noisy-instruction predictions, while generation has fewer differences, attributed to refined tokens influencing later noisy predictions.The comparison covers 58 classification and 61 generation tasks in the SUPERNATINST heldout dataset using ID with OpenSNI-7B; changed tokens are counted using ϵ = 0.3.Source: F QUANTITATIVE ANALYSIS ON TOKENS… · p. 20, Table 9 · p. 21
  • At epsilon = 0.3, confident base predictions tend to remain unchanged, whereas unconfident predictions may change under contrast; ID therefore does not always produce fully contrasting predictions.Figure 12 distinguishes altered from unchanged tokens by the maximum probability of the base prediction.Source: F QUANTITATIVE ANALYSIS ON TOKENS… · pp. 20–21, Figure 12 · p. 21
  • ID’s reported improvements are consistent across response lengths on the held-out SUPERNATINST set.Figure 13 compares response-length distributions for Tk-Instruct XL, XXL, and OpenSNI-7B.Source: F QUANTITATIVE ANALYSIS ON TOKENS… · p. 21, Figure 13 · p. 21

G.1 WRONG RESPONSE PATTERNS FROM ID

  • Wrong ID responses often reuse instruction words, while baseline errors more often reuse instance-sentence words; ID may therefore underperform when exact instance wording is needed.The authors identify Overlap Extraction and Coreference Resolution among categories with drops, alongside Keyword Tagging.Source: §G.1 · p. 22, Table 10 · p. 23, Table 13 · p. 25

G.2 HOW ID REFINES RESPONSE WHEN MODEL SIZE INCREASES

  • For Question Rewriting and Keyword Tagging, examples show progress with the larger Tk-XXL model, though ID does not improve the Tk-XL Question Rewriting score.In the cited rewriting example, Tk-XXL ID follows the instruction while maintaining the original intent; Tk-XL changes that intent incorrectly.Source: §G.2 · p. 22, Table 11 · p. 23, Table 12 · p. 24

G.3 LIMITATIONS OF USING FIXED REFERENCES IN ROUGE-L EVALUATION

  • Fixed-reference Rouge-L can mislead comparisons when ID produces contextually valid outputs that differ from the reference, as illustrated by Word Analogy examples.The examples include an emoji and “spatula” despite the reference “knife”; the authors note the difficulty of evaluating open-ended generation.Source: §G.3 · p. 22, Table 14 · p. 26

H EVALUATION ON MMLU BENCHMARK

The paper evaluates ID strategies in zero-shot MMLU and compares their exact-match scores with a baseline. It highlights the “Opposite” strategy’s consistent performance but supplies no score values in the provided text.

H EVALUATION ON MMLU BENCHMARK

  • MMLU is presented as a zero-shot multiple-choice evaluation spanning humanities, social sciences, and STEM questions.Each item has a query and four answer choices, with one correct choice.Source: H EVALUATION ON MMLU BENCHMARK · p. 28, Table 16 · p. 28
  • The comparison evaluates “Opposite,” “Opposite−,” “Opposite+,” and “Null−” variants on Tk-Large, Tk-XL, and OpenSNI.“Opposite” replaces the instruction, “Opposite−” combines it with answer options, “Opposite+” prefixes it to the given instruction, and “Null−” retains only answer options.Source: H EVALUATION ON MMLU BENCHMARK · p. 28
  • The “Opposite” strategy is singled out for consistent performance in the zero-shot MMLU comparison against a baseline on exact match.Table 16 covers 57 MMLU tasks across Tk-Large, Tk-XL, and OpenSNI with epsilon set to 0.3.Source: H EVALUATION ON MMLU BENCHMARK · p. 28, Table 16 · p. 28

Related papers

Loading 2311.00233v2…