Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Tasks

• NKI retrieval augmentation remains under-explored beyond language modeling, text generation, and machine translation, despite applications such as sentiment analysis and text classification.Source: §1 · p. 1
• NKI tasks require diverse relevance functions, such as sentence-level semantic similarity for classification and linguistic similarity for acceptability.Source: §1 · p. 2
• Jointly updating large external indexes can improve performance but is expensive, whereas static indexes reduce cost while limiting task-specific relevance.Source: §1 · p. 2
• PGRA first uses a task-agnostic retriever with a shared static index, then applies a prompt-guided PLM reranker using task-specific relevance before reader prediction.Source: §1 · p. 2, Figure 1 · p. 2
• The framework combines retrieval and reranking with textual prompts to maintain reasonably low training cost while addressing diverse relevance requirements across NKI tasks.Source: §1 · p. 2

2 Methods

• PGRA has three components: a shared static-index retriever, a prompt-guided PLM reranker, and a reader that uses top-d evidence as augmentation.Source: §2 · pp. 2–3

2.1 Task-Agnostic Retriever

• The task-agnostic retriever encodes queries and external texts, stores text representations as index keys, and shares the static index across tasks to reduce training cost.Source: §2.1 · p. 3
• Top-k candidates are selected from softmax retrieval scores and efficiently approximated with FAISS Maximum Inner Product Search before reranking.Source: §2.1 · p. 3

2.2 Prompt-Guided Reranker

• The prompt-guided reranker addresses task-specific relevance by using few-shot exemplars and manually written task descriptions with an autoregressive PLM.Source: §2.2 · p. 3, Table 1 · p. 3
• Prompt-guided PLM representations encode task-relevant features for queries and candidates, whose similarities determine reranking and selection of top-d evidence.Source: §2.2 · pp. 3–4

2.3 Reader

• The FiD reader independently encodes each query-evidence pair, applies decoder cross-attention, and predicts labels after reformulating NKI tasks as language modeling.Source: §2.3 · p. 4

3.1 Experimental Setups

• The evaluation covers sentiment analysis, linguistic acceptability, question classification, and subjectivity analysis, using accuracy except Matthew’s correlation for CoLA.Datasets include SST-2, SST-5, CR, MR, MPQA, CoLA, TREC, and Subj.Source: §3.1 · p. 4
• Baselines include in-context learning, supervised T5, k-nearest neighbour, LM-BFF, RAG, and FiD, with retrieval-augmented methods using the same T5 readers for fair comparison.The setup uses Wiki1M as the external resource, BERT-base-uncased as the shared retriever, OPT-13b as the reranker, and T5 readers.Source: §3.1 · p. 4

3.2 Results

• Across most tasks, PGRA outperforms supervised and retrieval-augmented baselines in both T5-base and T5-large setups.Table 2 compares the baselines and PGRA under separate T5-base and T5-large settings.Source: Table 2 · p. 5, §3.2 · p. 5
• PGRA significantly outperforms k-nearest neighbour and few-shot methods, including in-context learning with OPT-13b and LM-BFF.LM-BFF performs better than k-nearest neighbour and in-context learning but remains substantially below PGRA.Source: §3.2 · p. 5

4.1 Effects of Label Consistency

• More label-consistent evidence generally corresponds to higher accuracy, with PGRA retrieving more consistent evidence than FiD and approaching 100% accuracy above 13 consistent items.The small groups with 3–4 consistent items can also show high accuracy, but their results are statistically insignificant because few instances belong to those groups.Source: §4.1 · p. 6

4.2 Effects of k and d

• Larger k consistently improves average performance, whereas larger d produces a relatively stable trend because the top consistent evidence remains unchanged when candidates are fixed.The analysis therefore expects larger first-stage candidate pools to help PGRA when sufficient computing resources are available.Source: §4.2 · p. 6

4.3 Effects of OPT Model Sizes

• Across five downstream tasks, larger OPT models achieve better performance, and larger models with higher consistency scores result in better performance overall.The authors believe larger OPT models better apply task-specific features to encode representations and obtain more effective task-specific relevance scores for retrieving evidence.Source: §4.3 · pp. 6–7

4.4 Effects of Evidence Granularity

• Sentence-level external evidence significantly outperforms passage-level evidence across all evaluated NKI tasks.The authors argue that sentences more readily express task-specific signals such as sentiment orientation than paragraphs.Source: §4.4 · p. 7

4.5 Training Cost

• PGRA builds its 1M-item index once at about 0.5 hours, while refreshed-index models incur approximately c×n×h cost across n refreshes and h tasks.The comparison focuses on index-building and refreshing time because those operations account for a substantial portion of training cost.Source: §4.5 · p. 7

4.6 Generalization on Retrievers

• PGRA consistently outperforms FiD with BM25, BERT, or SimCSE first-stage retrievers, while BM25 performs worse than the BERT and SimCSE variants.This supports adaptation of the second-stage framework to different retriever types.Source: §4.6 · pp. 7–8

4.7 Case Study

• In an SST-2 case study, PGRA predicts correctly where FiD and RAG fail because its retrieved sentences align with the input’s negative sentiment orientation.FiD and RAG retrieve lexical or semantic overlaps that lack the sentiment alignment needed for the classification decision.Source: §4.7 · p. 8

Retrieval-Augmented Methods

• Retrieval-augmented systems commonly use BM25 or trained dual encoders followed by readers, especially for knowledge-intensive question answering requiring explicit knowledge [Izacard and Grave, 2021; Izacard et al., 2022].The section positions PGRA as targeting fine-grained textual information for non-knowledge-intensive tasks without training a separate retriever for each task.Source: §5 · p. 8
• Wang et al. (2022) keep a shared BM25 retriever across tasks while modifying the reader, whereas PGRA uses two retrieval stages to find task-specific information at low cost.PGRA’s distinction is task-specific reranking rather than only reader-side information filtering.Source: §5 · p. 8

Prompt-based Methods

• Prompt-based methods recast downstream tasks as language modeling, while other studies use prompts for instruction-guided retrieval or regex-pattern search [Liu et al., 2021; Gao et al., 2021a; Asai et al., 2022; van de Kar et al., 2022].PGRA differs by using pretrained models to retrieve according to task-specific relevance in an efficient retrieval-augmented framework.Source: §5 · pp. 8–9

6 Conclusion

• PGRA uses a task-agnostic retriever to build shared static indexes, then a prompt-guided reranker to select candidates using task-specific relevance for the reader.The two-stage design is intended to support wider NKI tasks while avoiding a separately trained retriever for each task.Source: §6 · p. 9
• PGRA overall outperforms previous state-of-the-art retrieval-augmented methods and shows generality across evidence granularities and retriever types.The analyses also examine how label consistency between inputs and retrieved evidence relates to performance.Source: §6 · p. 9
• The authors identify improving pseudo-label consistency as a future direction for enhancing model performance.This direction follows directly from the paper’s analysis of consistency between inputs and retrieved evidence.Source: §6 · p. 9

• PGRA has not been tested on sentence-pair tasks because Wikipedia contains few sentence pairs for retrieval.The authors suggest larger, more diverse corpora as a possible remedy.Source: Limitations · p. 9
• PGRA is restricted to classification tasks, while retrieving evidence for generation may require information related to both source and target.The authors identify machine translation as an example and leave this question for future work.Source: Limitations · p. 9
• PGRA has not been extensively tested on KI tasks beyond preliminary Appendix F analysis, limiting claims about generality.KI tasks may require passage-level knowledge matching and more specialized prompts.Source: Limitations · p. 9

• Retrieval augmentation can require substantial computation for indexing, retrieval, and index refreshing, limiting usability and potentially harming fairness.PGRA aims to balance retrieval performance and training cost without retraining a retriever or rebuilding an index for each new task.Source: Ethics Statement · p. 9
• Using Wikipedia and common tasks may reflect data-selection biases in the results, requiring additional attention to bias when scaling retrieved-text applications.Source: Ethics Statement · p. 9

A Multiple Runs of the Main Experiment

• PGRA's multiple-run results are reported in Table 8 for the main-experiment settings.Source: A Multiple Runs of the Main Experiment · p. 12

B Experiments with More Retrieved Evidence For FiD baselines

• PGRA still underperforms FiD despite FiD's higher average scores when more evidence is retrieved.The comparison is reported in Table 4.Source: B Experiments with More Retrieved… · p. 12, Table 4 · p. 12

C Impact of k, d and OPT Model Sizes

• Sections 4.2 and 4.3 analyze the effects of first-stage top-k, second-stage top-d, and OPT model size, keeping other hyperparameters identical to Table 2 settings.Tables 5, 6, and 9 report detailed performance for each task in these ablations.Source: C Impact of k, d and OPT Model Sizes · p. 12, Table 5 · p. 12

D Label Consistency

• Label-consistency scores for PGRA with different second-stage OPT models are reported for each task in Table 10.Source: D Label Consistency · p. 12, Table 6 · p. 12

E Generality on Retrievers

• The appendix evaluates FiD and PGRA across BM25, BERT, and SimCSE first-stage encoders.The results are shown in Table 11.Source: E Generality on Retrievers · p. 12

• 77.7% accuracy for PGRA versus 73.8% for FiD confirms a performance increase on FEVER with a sentence-level Wiki1m datastore.Both systems use T5-base and the same other hyperparameters as Table 2.Source: F Generalization on KI tasks · p. 12
• FiD with a traditional passage-level datastore can achieve better performance, highlighting a limitation of PGRA's sentence-level setup.Passage-level datastores require substantially different relevance metrics.Source: F Generalization on KI tasks · p. 12

G Datasets and Metrics

• Wiki1M from SimCSE is used as the external datastore, with task information reported in Table 12 and the same configuration and dataset splits as Gao et al. (2021a).Wiki1M is a subset of Wikipedia and was also used in Gao et al. (2021b).Source: G Datasets and Metrics · p. 13, Table 12 · p. 14

H Training Details

• Models are trained for 5000 steps after searching learning rates from 1e-5 to 1e-4 and batch sizes of 4 or 8.The best hyperparameters are reported in Table 7.Source: H Training Details · p. 13

I Prompts

• The appendix includes the prompts used across all 8 tasks.Prompt instances for in-context learning in the prompt-guided reranker are provided in Table 13.Source: I Prompts · p. 13, Table 13 · p. 15
• The impact of OPT model size is analyzed with k = 150 and d = 16.The detailed analysis appears in Table 9.Source: Table 9 · p. 14
• Pseudo-label consistency is analyzed across different OPT models, averaging consistency and performance over five tasks with k = 150 and d = 16.The analysis is summarized in Table 10.Source: Table 10 · p. 14
• Generalization performance is compared for FiD and PGRA using different first-stage encoders.The comparison is presented in Table 11.Source: Table 11 · p. 14