PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions

Ruiyang Qin, Qingzhuo Wang, Tian Wang, Zhihua Wei, Wen Shen

arXiv:2608.18539v1cs.LGcs.AIcs.CL

TL;DR

Prompt sensitivity metrics based on final outputs reveal performance changes but not their internal causes. This paper uses interactions and the IPS metric to analyze those causes, finding that fine-tuning, larger scale, dense architectures, and few-shot learning reduce sensitivity mainly by stabilizing low-order interactions.

  • Problem

    Existing prompt-sensitivity evaluations compare final outputs, providing coarse evidence of changes but not explaining their underlying reasons.

  • Method

    The paper decomposes LLM output scores into interactions and quantifies prompt-induced interaction changes with the Interaction-based Prompt Sensitivity metric across 50 open-source LLMs.

  • Results

    Supervised fine-tuning, larger models, dense architectures, and few-shot learning reduce prompt sensitivity mainly by stabilizing low-order interactions, while high-order sensitivity remains relatively high.

  • Takeaways & Limitations

    Interaction analysis exposes prompt-related instability that can remain invisible when an LLM’s final output is unchanged.

  • Takeaways & Limitations

    The interaction analysis assumes input variables can be masked using a specific token or embedding.

Abstract

from arXiv · show

The remarkable capabilities of large language models (LLMs) are often undermined by their instability. Even subtle and semantically irrelevant changes in prompts can cause dramatic fluctuations in performance, a phenomenon known as prompt sensitivity. Previous studies typically evaluate prompt sensitivity by comparing the LLM's final outputs when prompts change. However, such coarse-grained metrics fail to explain the internal reasons for prompt sensitivity. In this paper, we introduce interactions as a fine-grained tool to analyze prompt sensitivity of LLMs. Specifically, we decompose the output score of the LLM into a set of interactions. Each interaction represents a nonlinear relationship involving a set of input variables. We discover that subtle changes to prompts can trigger severe instability in interactions, even when the outputs of the LLM remain the same. To this end, we propose an Interaction-based Prompt Sensitivity (IPS) metric by quantifying changes in interactions when we introduce subtle changes to prompts. We apply the IPS metric to 50 open-source LLMs and uncover four factors that reduce the prompt sensitivity of LLMs, including supervised fine-tuning, increased model scales, dense architectures, and few-shot learning. More crucially, we discover a common mechanism by which these four factors reduce prompt sensitivity: all four factors tend to reduce the prompt sensitivity of low-order interactions (i.e., interactions involving few input variables).

1. Introduction

The paper introduces interactions as a fine-grained framework for explaining prompt sensitivity beyond final-output comparisons. It proposes IPS, evaluates 50 open-source LLMs, identifies four sensitivity-reducing factors, and links them to reduced instability in low-order interactions.

  • Motivation: Semantically unimportant prompt changes can produce divergent LLM outputs, while output-based metrics do not explain the underlying causes.This motivates a fine-grained analysis of prompt sensitivity rather than relying only on final predictions.
  • Interaction framework: An interaction captures a nonlinear relationship associated with a specific combination of input variables and helps represent the DNN’s inference logic.For example, the joint presence of “green” and “hand” triggers an interaction corresponding to the idiom’s meaning.
  • Interaction framework: Interaction-based analysis distinguishes stable from unstable interactions, revealing how semantically equivalent prompt alterations can change the model’s behavior.The example changes the prediction from correct “B” to incorrect “A”; I{lay, eggs} shifts from 0.6242 to 0.6076 and is characterized as stable.
  • IPS metric: IPS quantifies changes in interaction patterns across prompts and is used to evaluate prompt sensitivity in 50 open-source LLMs.The paper notes that rankings alone make it challenging to determine which LLM families are more or less sensitive.
  • Empirical findings: Four factors reduce prompt sensitivity: supervised fine-tuning, larger parameter counts, dense architectures over MoE models, and few-shot learning over 0-shot learning.Instruct/chat models are less sensitive than base models, and few-shot learning considerably reduces sensitivity compared with 0-shot learning.
  • Mechanism: All four factors primarily reduce instability in low-order interactions, although high-order interactions tend to exhibit the highest sensitivity.Low-order interactions involve a small number of input variables, whereas high-order interactions involve many.

2. Related Work

Prior work shows that minor, semantically unimportant prompt changes can substantially alter LLM performance, threatening reliability. Existing explanation methods motivate a fine-grained interaction-based metric designed to reveal the mechanisms underlying prompt sensitivity.

  • Prompt sensitivity of LLMs: Minor or semantically unimportant prompt alterations can cause significant performance variation in LLMs, creating a reliability risk.
  • Prompt sensitivity of LLMs: Existing prompt-sensitivity metrics are complemented by a fine-grained interaction-based metric that evaluates sensitivity and uncovers underlying mechanisms.
  • Using game-theoretic interactions to explain DNNs: Traditional DNN explanation methods often lack mathematical guarantees of faithfulness, while game-theoretic interactions provide theoretical guarantees for explaining internal logic.

3. Interaction-Based Analysis of the Prompt Sensitivity of LLMs

The section defines interactions as nonlinear effects among input variables and establishes that they can faithfully represent LLM output scores. It then uses interaction instability to reveal prompt sensitivity that output-level comparisons miss.

  • 3.1. Preliminaries: Interactions: Interactions model nonlinear relationships among input variables, with each AND interaction contributing an effect when all variables in its set are present.The scalar output is defined from the ground-truth token probability, and masking any variable prevents the corresponding interaction from activating.
  • 3.1. Preliminaries: Interactions: The surrogate logical model can always match the LLM’s output score across all 2^n masked inputs, providing a faithfulness guarantee for interaction-based explanations.The model’s output is composed of the summed interaction effects.
  • 3.1. Preliminaries: Interactions: Only a small subset of the 2^n encoded interactions have salient effects, whereas most interactions are negligible noise patterns.This sparsity property supports interpreting salient interactions as the LLM’s meaningful inference patterns.
  • 3.2. Verifying the Faithfulness of Considering Interactions as Inference Patterns Used by LLMs: Experiments verify interaction sparsity and show that a logical model using the most salient interactions can match the LLM across masked inputs.Figure 2 reports negligible effects for most interactions and compares the resulting logical-model outputs with the LLM’s true outputs.
  • 3.3. Using Interactions as a Fine-Grained Tool to Analyze the Prompt Sensitivity of LLMs: Interaction-based analysis exposes latent unreliability that output-level analysis misses and quantifies stable and unstable interaction ratios for each sample and LLM.This provides a fine-grained tool for evaluating prompt sensitivity through changes in interaction patterns.
  • 3.3. Using Interactions as a Fine-Grained Tool to Analyze the Prompt Sensitivity of LLMs: When semantically equivalent prompt templates produce different outputs, nearly half of salient interactions reverse sign, 30%-40% change substantially, and only 10%-20% remain stable.Sign reversal is the most severe instability because it can reverse the LLM’s internal logic.
  • 3.3. Using Interactions as a Fine-Grained Tool to Analyze the Prompt Sensitivity of LLMs: Even when outputs remain identical, 60%-80% of salient interactions remain unstable under semantically equivalent prompt changes.This demonstrates that output stability can conceal substantial changes in internal interaction patterns.
  • 3.3. Using Interactions as a Fine-Grained Tool to Analyze the Prompt Sensitivity of LLMs: The analysis classifies interaction changes as opposite sign, same sign with different effect, or stable, enabling systematic measurement of instability.The first category reverses an interaction’s influence, while the second preserves its sign but changes its magnitude substantially.

4. Evaluating and Analyzing the Prompt Sensitivity of LLMs

The paper introduces IPS, a fine-grained interaction-based metric for measuring prompt sensitivity, and applies it to 50 open-source LLMs. Across four factors, reduced sensitivity is consistently associated with greater stability in low-order interactions, while high-order interactions remain difficult to stabilize.

  • 4.1. Evaluating Interaction-Based Prompt Sensitivity: IPS compares salient interactions extracted from the same input under semantically equivalent prompt templates and measures their symmetric mean absolute percentage error.The salient-interaction threshold is τ = 0.1 in all experiments.
  • 4.1. Evaluating Interaction-Based Prompt Sensitivity: The evaluation covers 50 open-source LLMs from six families on the ARC and MMLU multiple-choice benchmarks.The model set includes Llama, Mistral, Qwen, OLMo, and InternLM families.
  • 4.1. Evaluating Interaction-Based Prompt Sensitivity: IPS varies widely from 1.268 for Qwen2.5-72B-Instruct to 1.752 for Mistral-7B-v0.3, with no model family consistently dominating.The distribution indicates that multiple underlying factors influence prompt sensitivity.
  • 4.1. Evaluating Interaction-Based Prompt Sensitivity: IPS is negatively correlated with output consistency, aligning with the traditional metric while providing a more fine-grained view of prompt sensitivity.Output consistency is defined as the proportion of samples receiving identical predictions across prompt templates.
  • 4.2. Analyzing the Factors Impacting Prompt Sensitivity: Supervised fine-tuning, larger model scales, dense architectures, and few-shot learning each reduce prompt sensitivity in the evaluated comparisons.The factor analyses use ARC results, and few-shot learning reduces sensitivity across all tested LLMs.
  • 4.3. Explore the Underlying Mechanisms of Improved Stability for All Factors: Low-order interactions have the lowest prompt sensitivity, mid-order interactions are higher, and high-order interactions are the most sensitive to subtle prompt changes.The results characterize simple interaction patterns as comparatively robust.
  • 4.3. Explore the Underlying Mechanisms of Improved Stability for All Factors: All four factors significantly reduce low-order interaction sensitivity but fail to reduce high-order interaction sensitivity, revealing a common stabilization mechanism.The findings suggest that stable low-order interactions are easier for LLMs to learn than high-order interactions.
  • 4.3. Explore the Underlying Mechanisms of Improved Stability for All Factors: Experiments on Dolly-15k open-ended generation tasks consistently verify the main conclusions drawn from the multiple-choice experiments.The dataset includes open question answering, classification, and other non-MCQ tasks.

5. Conclusion · A. Masking Strategies of Input Variables · B. Proof of Theorem

The paper proposes interactions as a fine-grained metric for evaluating LLM prompt sensitivity and identifies model-training, scaling, architecture, and prompting factors that reduce it. It also adopts token replacement with a dedicated [MASK] token to mask input variables for interaction analysis.

  • 5. Conclusion: The paper proposes an interaction-based metric to evaluate prompt sensitivity in LLMs.The conclusion presents this metric as a contribution for analyzing prompt sensitivity.
  • 5. Conclusion: Four factors reduce LLM prompt sensitivity: supervised fine-tuning, increased model scale, dense architectures, and few-shot learning.The conclusion frames these findings as insights for improving LLM robustness through model design and prompting methods.
  • A. Masking Strategies of Input Variables: Attribution methods commonly mask input variables with tokens or embeddings and estimate their attributions from output changes.The passage notes that this practice appears in prior DNN attribution research.
  • A. Masking Strategies of Input Variables: Masking approaches are difficult to select because each method has weaknesses, including replacing inputs with a mean baseline value.The passage introduces masking-method selection as a complex design choice.
  • A. Masking Strategies of Input Variables: The paper adopts token replacement by substituting a target word with a dedicated [MASK] token at the embedding level.For example, “green” in “He is a green hand” becomes “[MASK]” in “He is a [MASK] hand.”
  • A. Masking Strategies of Input Variables: This masking approach nullifies the target word’s specific semantic contribution without introducing out-of-distribution artifacts.It therefore provides a clean and consistent baseline for interaction analysis.

B.1. Proof of Universal Matching Property … E.1. Computing Infrastructure

The paper proves that a complete AND-OR interaction surrogate can exactly match DNN outputs on every masked sample, while sparse salient interactions can approximate those outputs. It also formalizes OR interactions, extraction of sparse interactions, and the computational setup used for experiments.

  • B.1. Proof of Universal Matching Property: The surrogate logical model represents AND effects activated by jointly unmasked variables and OR effects activated when any variable in a set is unmasked.Each triggered interaction effect is added to the surrogate output; otherwise, it contributes nothing.
  • B.1. Proof of Universal Matching Property: The complete AND-OR interaction framework satisfies the Universal Matching Property, with the surrogate model exactly matching the DNN output for every masked sample.The framework generalizes the AND-only presentation and sums AND and OR interaction effects.
  • B.2. Proof of Sparsity Property: For well-trained DNNs under three stated conditions, a small set of salient AND and OR interactions can approximate outputs across all masked samples.The sparsity argument treats OR interactions as a special case of AND interactions.
  • C. OR Interactions Can Be Considered as Special AND Interactions: OR interactions can be regarded as special AND interactions by reversing which input state is treated as masked.The construction assumes vand(xS) = vor(xS) = 0.5v(xS) and uses the reversed masked-state definition.
  • D. Details of Extracting the Sparsest AND-OR Interactions: The extraction method decomposes each network score into AND and OR components using learnable γT parameters optimized with a LASSO-like loss to obtain sparse interactions.The components are vand(xT) = 0.5 · v(xT) + γT and vor(xT) = 0.5 · v(xT) − γT.
  • E.1. Computing Infrastructure: Experiments ran on four NVIDIA Tesla V100-DGXS GPUs with 32 GB of VRAM each, using NVIDIA Driver 570.133.07, CUDA 12.8, and torch.float16 inference.The reported software environment and datatype define the computing infrastructure for all evaluated LLMs.

E.2. Model Details … F. More Experimental Results

The experiments evaluate prompt sensitivity across 50 open-source LLMs using controlled model comparisons, deterministic generation, model-specific masking, and semantically irrelevant prompt variations. Few-shot prompts then test whether in-context examples mitigate the most severe 0-shot sensitivity.

  • E.2. Model Details: Experiments cover 50 open-source LLMs from 6 major model families, grouped into controlled subsets for alignment, architecture, and scale comparisons.The subsets compare instruct/chat with base models, dense with Mixture-of-Experts models, and same-family models with varying parameter counts.
  • E.2. Model Details: The study compares instruct/chat and base models, dense and MoE architectures, and parameter scales within matched model families.These designs aim to control other variables while analyzing alignment, architecture, and model-scale effects.
  • E.3. Generation Configuration of LLMs: Greedy decoding sets “do sample” to “False,” making each LLM select the highest-probability next token and produce identical outputs for repeated inputs.This removes randomness from sampling-based generation and supports reproducibility.
  • E.4. How to Mask Input Words For Different LLMs: Interactions mask words outside a selected variable set using each LLM’s <unk> token when available, otherwise falling back to <pad>.The exact token strings and IDs vary across model families.
  • E.5. Prompt Templates: Five prompt templates introduce semantically irrelevant changes in letter case, separators, and option-marker formats while preserving the underlying question and options.Examples include “Answers” versus “ANSWERS,” “:” versus “::,” and “A.” versus “A)”.
  • E.5. Prompt Templates: For every pair among five templates, prompt sensitivity is computed from changes in interactions among words in the question and options.This measures perturbation of the core input’s interaction patterns rather than only comparing final outputs.
  • E.6. Few-shot Learning Templates: Prompt Templates 1 and 4 were selected because they had the highest average 0-shot prompt sensitivity for testing few-shot mitigation.The follow-up uses 1-shot, 2-shot, and 3-shot prompts built from these two templates.
  • F. More Experimental Results: Few-shot examples were randomly drawn from datasets excluded from testing and formatted according to their corresponding prompt template.The procedure constructs prompts with one, two, or three in-context question-answer examples.

F.1. More Results on the Verification of the Sparsity of Interactions … H. Comparison between IPS and Other Metrics

The supplementary results validate that LLM outputs are governed by sparse salient interactions and that prompt sensitivity is concentrated in higher-order interactions. They also show that few-shot learning, threshold variation, dataset changes, perturbation type, and architecture affect sensitivity in systematic ways, while IPS provides a fine-grained alternative to coarse representation metrics.

  • F.1. More Results on the Verification of the Sparsity of Interactions: Only a small set of interactions has salient effects, while most interaction effects are negligible noise patterns.
  • F.2. More Results on the Verification of the Sparsity of Interactions: Using only the top 3% or top 5% of interactions yields minimal matching error, faithfully approximating the LLM output for all masked inputs.
  • F.3. Detailed Case Study: Semantically identical prompt templates can preserve the same input and output while substantially changing salient interaction patterns, revealing unstable interactions as an underlying cause of prompt sensitivity.
  • F.4. More Results on the Prompt Sensitivity of Different Orders; F.5. More Results on Relative Change in the Prompt Sensitivity of Low-, Mid-, and High-Order Interactions for Different Factors; F.6. More Results on the Prompt Sensitivity of Different Order Types across Different Model Scales: On ARC, low-order interactions are least prompt-sensitive, mid-order interactions are intermediate, and high-order interactions are most sensitive, indicating greater robustness for simple patterns.
  • F.7. More Results on the Prompt Sensitivity of Different Orders for Each Individual LLM when Applying Few-Shot Learning: Prompt sensitivity at all three order levels clearly drops with few-shot learning, with the supplementary comparisons including changes across model scales and learning settings.
  • F.8. Hyperparameter Experiments of the threshold τ; F.8.1. DETAILED MODEL RANKINGS UNDER VARYING THRESHOLDS: Across 16 thresholds, high correlation coefficients and low error rates show that IPS rankings and scores are robust to the choice of τ.
  • F.8.2. VERIFYING THE GENERALIZABILITY OF THE METHODS AND CONCLUSIONS ON DIFFERENT THRESHOLD τ; F.9. Results on MMLU Dataset: Across τ = 0.05 to τ = 0.20, absolute IPS scores fluctuate but relative model-stability rankings remain highly consistent, and the same conclusions hold on MMLU.

H.1. Empirical Inconsistency of Representation-based Metrics · H.2. Superiority of the Interaction-based Framework

Representation-based metrics inconsistently track prompt sensitivity and can contradict established robustness trends. IPS is superior because it explains unstable token combinations and faithfully reconstructs the LLM’s output score.

  • H.1. Empirical Inconsistency of Representation-based Metrics: Representation-based metrics show significantly less consistent trends than IPS when comparing Base with Instruct/Chat models.The comparison re-evaluates the Base-versus-Instruct/Chat factor using Cosine Similarity and L2 Distance.
  • H.1. Empirical Inconsistency of Representation-based Metrics: Cosine Similarity and L2 Distance fail to provide a reliable proxy for prompt sensitivity or a consistent Supervised Fine-Tuning effect.The metrics do not consistently reflect the impact of Supervised Fine-Tuning.
  • H.1. Empirical Inconsistency of Representation-based Metrics: For Mixtral-8x7B, Cosine Similarity favors the Base model’s stability, whereas L2 Distance favors the Instruct model.The two representation metrics therefore produce contradictory conclusions for the same model pair.
  • H.1. Empirical Inconsistency of Representation-based Metrics: For Llama-2-13B and Qwen3-8B, Base models show higher cosine similarity and lower L2 distance than Instruct counterparts despite IPS identifying Instruct/Chat models as more robust.This reverses the established robustness trend reported by the IPS analysis.
  • H.1. Empirical Inconsistency of Representation-based Metrics: For Llama-2-7B, Cosine Similarity makes the Chat model appear more stable, while L2 Distance makes it appear less stable.The results exhibit random fluctuations rather than a discernible pattern across model families.
  • H.1. Empirical Inconsistency of Representation-based Metrics: Global hidden-state change measures are too coarse to accurately indicate the model’s functional sensitivity.The contradictions across representation metrics motivate the theoretical advantages of IPS.
  • H.2. Superiority of the Interaction-based Framework: IPS explains why outputs fluctuate by identifying unstable combinations of input tokens, rather than merely measuring aggregate hidden-state changes.Its interaction decomposition exposes inference patterns that become unstable under prompt variation.
  • H.2. Superiority of the Interaction-based Framework: IPS faithfully evaluates decision-making because the Universal Matching Property guarantees that all interactions sum exactly to the LLM’s output score.Representation metrics lack this direct mathematical link to the final prediction.

I. Detailed Discussion on the Impact of Model Architecture

Controlled comparisons indicate that MoE architectures generally increase prompt sensitivity relative to dense architectures, beyond differences in activated parameters or training paradigms. The Mistral family is an exception, with dense models more sensitive than MoE models at low- and mid-order interaction levels.

  • Family-specific exception: In the Mistral family, dense models are more sensitive than MoE models at low-order and mid-order interaction levels.This reversal is attributed to the number of activated parameters, with Mixtral-8x7B activating more parameters than the compared dense models.
  • Controlled variable analysis: Controlled comparisons matched Qwen and OLMo models by activated parameters and training paradigm to isolate architecture’s effect on prompt sensitivity.The analysis paired base models with base models and instruct/chat models with instruct/chat models.
  • Controlled variable analysis: MoE models consistently exhibit higher prompt sensitivity than dense models, suggesting that MoE architecture inherently increases sensitivity beyond effects from fewer activated parameters.The result is based on the controlled analysis reported in Table 7.

J. Solutions for reducing the computational cost of the method … K.2. Selection of Input Variables

The interaction framework is computationally expensive because its cost grows exponentially with the number of input variables, especially for long texts. The paper reduces this burden through selective variable analysis, phrase aggregation, and approximation methods while preserving the analysis’s key conclusions and faithfulness.

  • J. Solutions for reducing the computational cost of the method: The method requires evaluating 2n masked inputs, causing exponential complexity that makes direct analysis of very long texts computationally costly.This is identified as the study’s main limitation.
  • J. Solutions for reducing the computational cost of the method: Phrase-level aggregation merges related words into phrasal units, reducing the number of variables while preserving key semantic meaning.The framework aggregates words expressing core concepts into single units.
  • J. Solutions for reducing the computational cost of the method: The applied selection strategies substantially reduce computational complexity without affecting the key conclusions in long-form open-ended experiments.The specific selection procedures are detailed in Appendices K and L.
  • J. Solutions for reducing the computational cost of the method: Approximation methods for efficiently computing sparse interactions can bypass exhaustive O(2n) evaluations and further reduce computational cost.The paper identifies these methods as a path toward extending interaction-based analysis to long-text NLP tasks.
  • K.1. Experimental Setup: Open-ended tasks use the Databricks Dolly-15k dataset, covering brainstorming, classification, closed QA, open QA, and summarization.The dataset provides diverse instruction-following records resembling real-world user scenarios.
  • K. Details of the Experiments on Open-Ended Generation Tasks: Because open-ended instructions are much longer than MCQs, the experiments apply selective input analysis and phrase-level aggregation to avoid exponential costs.These are the optimization strategies introduced to address long-text computation.
  • K.2. Selection of Input Variables: Selective input analysis keeps meaningful words or phrases as variables while treating stop words, punctuation, and generic instruction templates as fixed background context.Meaningful words are defined as terms that are neither NLTK stop words nor punctuation marks.
  • K.2. Selection of Input Variables: In the example prompt, “Walter Skinner” and “bald or balding” are aggregated as a key entity and critical condition.Functional words such as “from,” “the,” “who,” and “are” remain background context.

K.3. Experimental Results · L. Details of the Experiments Beyond Prompt Template Modifications · L.1. Experimental Setup

The experiments extend prompt-sensitivity analysis to realistic semantic paraphrasing and instruction reordering, while confirming the paper’s main conclusions across open-ended questions and optimized analyses. Results indicate that instruct/chat models, larger scales, dense architectures, and few-shot learning are associated with lower prompt sensitivity, particularly for low-order interactions.

  • K.3. Experimental Results: The Dolly experiment used input-variable selection strategies to reduce the computational cost of analyzing each long-text input.Methods 1 and 2 were applied to manage analytical complexity.
  • K.3. Experimental Results: Open-ended questions and the optimized setup produced results highly consistent with the main conclusions of the MCQ-based experiments.The experiment set τ to 0.1 and reported the same conclusions.
  • K.3. Experimental Results: Instruct/chat models are less prompt-sensitive than corresponding base models.This comparison was made within corresponding model pairs.
  • K.3. Experimental Results: MoE models generally exhibit greater prompt sensitivity than dense models in the same model family.The comparison concerns MoE and dense architectures.
  • K.3. Experimental Results: Prompt sensitivity systematically decreases as model scale increases within a model series.The paper also compares prompt sensitivity at the order level across model scales.
  • K.3. Experimental Results: Few-shot learning substantially reduces prompt sensitivity, with a substantial drop from 0-shot to 1-shot.The experiment compares 0-shot and few-shot learning conditions.
  • L. Details of the Experiments Beyond Prompt Template Modifications: The Dolly-15k experiment tested generalizability using semantic paraphrasing and instruction reordering as more realistic prompt perturbations.These perturbations were intended to evaluate robustness beyond superficial template modifications and authentic user interactions.
  • L.1. Experimental Setup: The longer open-ended instructions used Selective Input Variable Analysis and Phrase-level Aggregation to preserve core semantic elements while managing interaction-analysis complexity.This setup followed the input-variable selection strategy detailed in Appendix K.2.

L.3. Experimental Results

In a more challenging perturbation setup resembling authentic user interactions, the experiments replicate the paper’s main conclusions: supervised fine-tuning, larger scales, dense architectures, and few-shot learning reduce prompt sensitivity by stabilizing low-order interactions. These results extend beyond simple template variations and support the generalizability of the findings.

  • Main findings: The four factors—supervised fine-tuning, increased model scale, dense architectures, and few-shot learning—reduce prompt sensitivity primarily by stabilizing low-order interactions.The replicated experiment reports the same mechanism under more complex perturbations.
  • Main findings: Instruct/chat models are less prompt-sensitive than corresponding base models.This comparison supports the effect of supervised fine-tuning.
  • Main findings: MoE models generally show greater prompt sensitivity than dense models within the same model family.This comparison supports the advantage of dense architectures.
  • Main findings: Prompt sensitivity systematically decreases as model scale increases within a model series.The result is shown by comparing different scales across model series.
  • Main findings: Few-shot learning substantially reduces prompt sensitivity compared with 0-shot learning, with a marked drop from 0-shot to 1-shot.The experiment also reports the same overall conclusions with τ set to 0.1.
Loading 2608.18539v1…