Memory-assisted prompt editing to improve GPT-3 after deployment

1 Introduction

• The motivating gap is that retraining is costly, while users otherwise lack a way to help a deployed model improve from repeated mistakes.Source: §1 · p. 1
• The approach injects knowledge through interactive end-user feedback and dynamically edits the prompt for each instance.The paper frames this as adding interactivity to prompt engineering.Source: §1 · p. 1
• Users can correct the model’s understanding of their intent without knowing the correct answer, so they need not be task experts.The example feedback clarifies that “similar to” means similar in meaning, rather than supplying the answer itself.Source: §1 · pp. 1–2
• MemPrompt stores feedback from prior misunderstandings and retrieves relevant corrections to adapt the prompt for each new query.The memory is a set of key-value pairs linking misunderstood questions to corrective feedback.Source: §1 · pp. 1–2
• Across four tasks, GPT-3’s accuracy significantly increased over time using corrective feedback, without retraining.The tasks included lexical relations, word scrambling, and two forms of ethical reasoning.Source: §1 · p. 2

2 Related work

• Unlike prompt methods that select or represent better prefixes, this work edits the input query by adding feedback from prior misunderstandings.Source: §2 · pp. 2–3
• Unlike retrieval from text corpora or prior question-answer pairs, MemPrompt retrieves from a dynamic memory of user feedback.Source: §2 · p. 3
• The work extends instruction learning with an adaptive component for cases where instructions are misinterpreted.Source: §2 · p. 3
• Compared with parameter-editing approaches, the goal is to generalize corrections to new problems without damaging the model’s basic problem-solving ability.The passage notes that prior model-editing approaches had been limited by uncontrollable out-of-scope changes.Source: §2 · p. 3
• MemPrompt frames dialogue-based debugging around the model’s understanding of the user’s intent.Source: §2 · p. 3

3.1 Memory enhanced GPT-3 architecture

• MemPrompt pairs GPT-3 with a growing query-feedback memory, retrieves feedback for new inputs, and assumes similar inputs incur similar errors and can share corrections.A combiner can ignore irrelevant retrieved feedback.Source: §3.1 · p. 3

3.2 Verbalizing Task Understanding

• By generating task understanding u alongside answer y, GPT-3 lets users critique intent without knowing y, though an incorrect u can mislead them.The paper reports such mismatches were rare on its tasks.Source: §3.2 · p. 4

3.3 Allowing GPT-3 to react to feedback

• Few-shot examples of (x, fb) → (u, y) teach GPT-3 to condition its generated understanding and answer on feedback.The model generates u and y as a continuation after the question and any retrieved feedback.Source: §3.1 · p. 3, Figure 4 · p. 4

3.4 Feedback on model’s understanding

• Feedback targets either task-instruction misunderstanding or nuanced input semantics; the experiments also explore settings with oracle feedback on labels.The primary goal is feedback on the model’s understanding of the task.Source: §3.4 · p. 5

3.5 Tasks; 3.5.1 Lexical Relations

• Tasks span lexical relations—synonym, antonym, homophone, definition, and sentence use—word scrambling, and ethics with categorical or natural-language feedback.The task collection includes two lexical tasks and two ethical-reasoning tasks.Source: §3.5 · p. 5, §3.5.1 · p. 5

3.5.2 Word Scrambling

• Word-scrambling queries cover reversal, cycling, random insertions, and two anagram variants; varied phrasings let failures accumulate as query-feedback pairs.The scrambling operations are drawn from the dataset by Brown et al. (2020).Source: §3.5.2 · p. 5

3.5.3 Ethical Reasoning (2 tasks)

• Ethical tasks elicit either a moral category or a natural-language rule of thumb alongside an ethical judgment, allowing users to critique the situation’s broader context.Categorical feedback uses one of ten categories, while natural-language feedback uses the dataset’s rule-of-thumb annotation.Source: §3.5.3 · pp. 5–6

3.6 MemPrompt Implementation

• Implementation uses similarity-thresholded retrieval and introduces GUD-IR to transform complex-task queries toward feedback-like representations for retrieval.Simple heuristics suit surface-similar lexical tasks; GUD-IR addresses lexically dissimilar situations that share an underlying principle.Source: §3.6 · pp. 6–7

4 Experiments

• The experiments compare MemPrompt with few-shot NO-MEM and, on lexical and word-scrambling tasks, GROW-PROMPT; they measure answer and instruction accuracy under varied clarification probabilities.GROW-PROMPT grows the prompt with recent memory that fits within 2,048 tokens; it is not used for ethical reasoning because the examples and initial prompt are too long.Source: §4 · p. 7

4.1.1 Ethical reasoning tasks

• MemPrompt achieves over 25% relative improvement on both ERT-NL and ERT-CAT, outperforming NO-MEM on categorical and more challenging natural-language ethical reasoning.The reported results use 1,000 DELPHI test points.Source: §4.1.1 · p. 7, Table 2 · p. 7
• ERT-CAT label accuracy rises over time for every clarification probability; NO-MEM saturates after 200 steps while MemPrompt continues improving.Higher clarification probabilities produce sharper early gains in instruction and label accuracy because feedback memory fills faster.Source: Figure 6 · p. 7, §4.1.1 · p. 8
• In ERT-NL error analysis, 30% of cases were correct lexical variants, 50% involved distracted understanding, and 18% were retrieval failures.Bad retrieval accounted for 30% of errors within the distracted-understanding category.Source: §4.1.1 · p. 8

4.1.2 Word Reasoning Tasks

• Within 300 examples, memory improves word-reasoning accuracy over no memory; GROW-PROMPT is intermediate, while MemPrompt leads lexical QA and both memory methods beat NO-MEM on word-scramble QA.GROW-PROMPT is about 3× more expensive and cannot scale beyond 2,048 tokens; retrieved feedback was effective 97% of the time.Source: §4.1.2 · pp. 8–9, Table 4 · p. 9, Table 5 · p. 9

4.2 Using dynamic prefix in prompts

• Combining KATE’s dynamic prompts with MemPrompt yields a consistent 10% improvement on ERT-CAT and ERT-NL, indicating complementary improvements.Source: §4.2 · p. 9

4.3 MemPrompt with label feedback

• On WEBQA factual question answering, MemPrompt is reported effective with label feedback, a setting where generating an understanding is difficult.Source: §4.3 · p. 9

4.4 Using MemPrompt for language and dialects based personalization

• In a simulated code-mixed Hindi-Punjabi use case, later queries using similar words succeed after users clarify language and dialect meanings.Source: §4.4 · p. 9

• MemPrompt has the model articulate its understanding of user intent, giving users an avenue to provide feedback.Source: §5 · p. 9
• The memory-enhanced GPT-3 system is presented as improving a deployed model through interaction, without retraining.Source: §5 · p. 9
• End-user interaction may improve system performance and broaden its utility.Source: §5 · p. 10

6 Limitations

• In practical deployments, memory may grow by orders of magnitude, creating scaling challenges; the authors envision it buffering retraining cycles of a week to several months.Between cycles, it could help avoid repeated mistakes and collect feedback for fine-tuning a later model version.Source: §6 · p. 10
• The current setup handles one user at a time, while multi-user deployments must manage feedback that may conflict across users and communities.Some feedback may apply across a community cluster, while interpretations and styles may differ.Source: §6 · p. 10
• For rich, noisy feedback collections, the authors suggest adapting diverse related past feedback into new feedback rather than retrieving only one item.They state that this could increase knowledge diversity and reduce the impact of erroneous feedback and noise.Source: §6 · p. 10
• The setting assumes users provide no adversarial feedback, but the authors warn that malicious feedback could control deployed systems and call robust mechanisms critical.They cite GUD-IR and memory adapters as examples of mechanisms relevant to deployment.Source: §6 · p. 10
• Recorded feedback may contain personal information, so deployment requires protecting memory from leaks and earning users’ trust.The paper gives a movie-preference query as an example of potentially personal feedback.Source: §6 · p. 10
• In multi-user deployment, feedback from communities with diverse beliefs, gender identities, and cultural backgrounds could conflict, requiring checks against harmful outputs.Source: §6 · p. 10

A Inside MemPrompt: Populating and using the memory

• MemPrompt adds a case to memory when a user corrects a misunderstood intent, then retrieves feedback from similar recorded cases for a new query.The memory stores misunderstood questions as keys and corrective feedback as values.Source: A Inside MemPrompt: Populating and… · p. 14

B.1 Introduction

• MemPrompt assumes similar inputs admit similar feedback, but ERT-NL challenges this when lexically dissimilar situations share feedback, risking missed or misleading retrieval.The paper’s example contrasts skating with poor balance and improving soccer skills; humans can see the shared improvement principle despite little lexical overlap.Source: §B.1 · p. 14

B.2 Initial approach: Learning a feedback similarity function

• The training objective aligns cosine similarity between input embeddings with cosine similarity between their corresponding feedback, bringing inputs with similar feedback closer in representation space.The resulting input representation is used as the key for retrieving feedback from memory.Source: §B.2 · p. 14
• Training the similarity function was unstable, which the authors attribute to highly dissimilar situations sharing identical feedback and limited training data.They therefore also experiment with a generative method.Source: §B.2 · p. 14
• The memory-population flow records user feedback after a question is misunderstood, while the retrieval flow adds feedback from a similar past case to a new question’s prompt.The retrieval figure describes GPT-3 using the question and feedback together to produce an understanding and answer.Source: Figure 9 · p. 15, Figure 10 · p. 15

B.3 Proposed approach: Training generative model for retrieving similar feedback

• The authors present GUD-IR as addressing sparsity in latent-space projection and making retrieval explainable and debuggable.They note that diverse situations can map to the same feedback, especially when feedback is specific and examples are limited.Source: §B.3 · pp. 15–16
• GUD-IR maps an input to a generated rough feedback, then searches the stored feedbacks for the closest match to use for that input.The approach replaces direct latent-space projection with a generated understanding and a two-step generate-then-retrieve process.Source: §B.3 · pp. 15–16, Figure 11 · p. 16
• For lexical reasoning and WEBQA, where similar keys indicate similar values, the system uses Sentence Transformers and cosine similarity with a 0.9 matching threshold.Source: §B.3 · p. 16
• Figure 12 and the accompanying text report that GUD-IR shifts the test-set distribution of true-versus-retrieved feedback similarity toward higher values than surface-level similarity.Source: §B.3 · p. 16, Figure 12 · p. 17

C Querying GPT-3-175B using OpenAI API

• For GPT-3 querying, the appendix describes appending the question and optional retrieved feedback to a prompt, then generating the understood intent and answer as a continuation.It reports temperature 0.0 for WEBQA and ERT experiments, and 0.7 for the other experiments; higher temperature caused ±10% accuracy variation across ERT-CAT and ERT-NL runs.Source: C Querying GPT-3-175B using OpenAI API · p. 17, D Prompt · p. 18

D Prompt

• The main prompt combines examples without feedback and examples with clarification, using formats that pair a question with its understood intent and answer.The prompt figure describes intent and answer together as a single sentence.Source: D Prompt · p. 18, Figure 13 · p. 19
• The appendix provides separate prompt figures for word scrambling, ERT-NL, and ERT-CAT; the ethical-task prompts ask for a morality judgment and a high-level task understanding.Source: Figure 14 · p. 20, Figure 15 · p. 21, Figure 16 · p. 22

E Datasets for lexical question-answering tasks

• The lexical datasets draw synonyms and antonyms from Nguyen et al. (2016), homophones from homz, definitions from the Online Plain Text English Dictionary, and sentence-usage examples from Commongen.For homophones, the authors use the closest homophone returned for each English-dictionary word.Source: E Datasets for lexical… · p. 23

E.1 Templates

• The authors manually created 15 task templates, with three question-phrasing variants for each task.Source: §E.1 · p. 23

E.2 Sample questions

• Sample x-y pairs illustrate linguistic variations in how a question can be phrased.Source: §E.2 · p. 23
• The listed examples also include Hindi queries.Source: §E.2 · p. 23
• Punjabi queries are included among the sample settings.Source: §E.2 · p. 23

F MemPrompt with label feedback

• Each WEBQA prompt combines k/2 fixed examples with k/2 retrieved question-answer pairs similar to the current query.The retrieved pairs come from questions the system previously answered incorrectly; experiments set k = 16 under budget constraints.Source: F MemPrompt with label feedback · p. 23
• An expert adds a query and its true answer to memory when GPT-3 answers incorrectly; questions in the 2,032-example WEBQA test set have multiple answers.The setup addresses factual QA where users may not be able to verbalize their understanding for feedback.Source: F MemPrompt with label feedback · p. 23
• Accuracy increases over time on more than 3,000 WEBQA test points as the memory grows with prior errors.The paper attributes this to retrieved examples helping avoid repeated mistakes.Source: F MemPrompt with label feedback · p. 23, Figure 17 · p. 24

F.1 Factual question answering Examples

• Examples retrieved for WEBQA domains such as alma mater, soccer, and language become more relevant as inference proceeds.For the alma-mater cluster, retrieved examples shift from initially irrelevant to mostly relevant near the end.Source: §F.1 · p. 24, F MemPrompt with label feedback · p. 24, Table 6 · p. 24

G Finding similar questions in low-resource settings

• English feedback clarifying Hindi and Punjabi queries yields significant gains in the low-resource setting.Similarity matching uses surface-form Levenshtein distance; FAISS or suffix trees are suggested for larger memories.Source: G Finding similar questions in… · p. 24

H Sample results

• The templates clarify that “similar to” requests a synonym and “unlike” requests an antonym.Source: H Sample results · p. 27
• Other templates pair definition and sentence-generation questions with clarifications naming the intended output.Source: H Sample results · p. 27
• English sample pairs show that the same task can be phrased in multiple ways, without the method assuming a particular question wording.Source: Table 10 · p. 28
• The section includes Hindi and Punjabi sample pairs, plus randomly sampled cases where GPT-3 without memory was incorrect.Source: Table 11 · p. 29, Table 12 · p. 29, Table 13 · p. 30