Exploring Demonstration Ensembling for In-context Learning

1 INTRODUCTION

• Concatenation leaves each demonstration’s contribution to attention weights and can consume the context window, especially with many demonstrations or lengthy inputs.The introduction also notes that language models are sensitive to demonstration order, which long concatenated chains can exacerbate.Source: §1 · p. 1
• Demonstration ensembling partitions examples into non-empty buckets and combines predictions from the buckets, with three ensembling methods investigated.One method uses clustering to partition examples.Source: §1 · p. 1
• Experiments on 7 language tasks show that ensembling can outperform the standard concatenation approach.Source: §1 · p. 1

2 RELATED WORK

• Min et al. (2022a) applied demonstration ensembling to text classification in a limited setting, whereas this paper explores different bucket sizes and task types.Source: §2 · p. 2
• Wang et al. (2022) ensemble examples augmented with generated rationales; this paper instead ensembles the examples themselves.Source: §2 · p. 2
• Related approaches include mixtures of soft prompts for knowledge extraction and demonstration ensembling for reranking document paths in multi-hop question answering.The passage attributes these approaches to Qin & Eisner (2021) and Khalifa et al. (2022), respectively.Source: §2 · p. 2

3.1 BUCKET ALLOCATION

• DENSE allocates demonstrations to non-empty buckets, conditions the language model on each bucket with the test input, then combines the resulting probabilities into an output-label probability.The combining operator can also take the test example and may produce an unnormalized probability.Source: §3.1 · p. 2

3.2 ENSEMBLING METHOD

• The combination methods include product-of-experts, mixture-of-experts, and max ensembling, which uses the most confident bucket prediction.Bucket importance weights are normalized to sum to one.Source: §3.2 · pp. 2–3

3.3 BUCKET WEIGHTING

• Each bucket can be weighted by the average similarity of its demonstrations to the test input.The similarity uses cosine similarity between the test-input embedding and demonstration embeddings.Source: §3.3 · p. 3

3.4 CLUSTERING DEMONSTRATIONS

• Similar-together allocation uses k-means clusters of demonstration representations as buckets, while diverse allocation forms buckets from demos drawn across clusters.The authors state that clustering makes bucket assignment less sensitive to demonstration order; the diverse construction assumes n is divisible by b for simplicity.Source: §3.4 · p. 3

4.1 EXPERIMENTAL SETUP

• The experiments use GPT-j (6B), 12 datasets, 6 or 10 demonstrations, and bucket counts of 2, 3, or 6 for 6-shot and 2, 5, or 10 for 10-shot.Concat-sort orders demos by input similarity, with more similar demos later; concat uses an arbitrary order.Source: §4.1 · p. 4

4.2 COMPARING ENSEMBLING METHODS

• Unweighted ensembling methods outperform concat, but unweighted ensembling underperforms concat-sort; max gains 0.8 average points in 6-shot, while MoE gains 0.6 in 10-shot.The 0.8-point and 0.6-point comparisons are against concat.Source: §4.2 · p. 4
• In 6-shot, weighted max ensembling beats concat by 2.4 and concat-sort by 1.2; in 10-shot, it beats them by 1.9 and 0.75, respectively.The gains are reported for the corresponding setting and baseline pairs.Source: §4.2 · p. 4
• Similar-together clustering consistently boosts weighted methods in 6-shot, but no clustering strategy is best across tasks and demonstration counts.Diverse allocation always outperforms similar-together in 10-shot, unlike the 6-shot setting.Source: §4.2 · p. 4

4.3 BUCKETS COUNT

• In 6-shot, performance improves as bucket count increases across all ensembling methods, while a small bucket count seems to perform worse overall.The passage does not specify a corresponding 10-shot trend.Source: §4.3 · p. 5

• Across 12 language tasks, ensembling demonstrations in buckets and combining their predictions yields better performance than concatenation.Source: §5 · p. 5
• Max ensembling performs best compared with product-of-experts and mixture-of-experts.Source: §5 · p. 5
• The paper also analyzes how the number of buckets and bucket-construction strategies affect ensembling.Source: §5 · p. 5

• Table 1 lists the datasets used in the paper.Source: Table 1 · p. 8
• Table 1 identifies the task associated with each dataset.Source: Table 1 · p. 8
• Table 1 reports metrics and the number of evaluation examples for each dataset.Source: Table 1 · p. 8

• Few-shot inference uses fp16 half-precision.Source: B EXPERIMENTAL SETUP · p. 8
• Experiments run on a workstation with 4 Nvidia A100 GPUs and batch size 16.Source: B EXPERIMENTAL SETUP · p. 8
• The experiments use the GPT-j checkpoint from Huggingface; clustering uses sklearn K-means and a constrained K-means implementation.Source: B EXPERIMENTAL SETUP · p. 8

C DETAILED RESULTS

• Figure 4 reports relative improvement from weighted ensembling approaches over concatenation.The comparison is shown separately by task.Source: C DETAILED RESULTS · p. 8, Figure 4 · p. 8
• The figure presents the improvement per task.Results are aggregated across different example counts, bucket counts, and seeds.Source: Figure 4 · p. 8
• The aggregation includes experiments with 6 and 10 examples, different bucket counts, and different seeds.Source: Figure 4 · p. 8