Training a Generally Curious Agent

Motivation

• The paper addresses interactive-agent challenges because naturally occurring data lacks interaction structure, while real-world collection can cause expensive and risky errors.These constraints motivate generating interaction data synthetically.Source: §1 · p. 1
• The framework trains agents to explore entirely unseen environments and gather information needed for the task, rather than relying on intrinsic-motivation curiosity.Its goal is amortized exploration: learning exploration strategies across many environments to make new-task exploration more efficient.Source: §1 · pp. 1–2

Method

• PAPRIKA creates diverse textual decision-making tasks, scores self-generated trajectories by task success, and applies sequential Direct Preference Optimization to favor successful trajectories.Its main computational bottleneck is sampling useful interaction data rather than performing model updates.Source: §1 · p. 2
• A curriculum-learning strategy improves sample efficiency by prioritizing trajectories from tasks with high learning potential.The framework is designed to teach a general problem-solving process that transfers across tasks.Source: §1 · pp. 1–2

Results

• PAPRIKA improves performance on unseen tasks by training on different subsets of synthetic interaction tasks, without task-specific fine-tuning.This supports learning general in-context reinforcement learning rather than memorizing particular problems.Source: §1 · pp. 1–2

• Each task τ is modeled as a partially observable Markov decision process, without relying on further POMDP formalism.Source: §2 · p. 2
• A task group G contains tasks that may share strategies without necessarily sharing the same optimal policy.Source: §2 · p. 2
• The agent interacts with each task as a black box that maps string actions and interaction history to string observations.Source: §2 · p. 2
• Episodes end upon task completion or the interaction limit, and each receives a single terminal score r(h) evaluating agent performance.Transition-level rewards are either zero or inferred from observations, while tasks may use different observation spaces and termination conditions.Source: §2 · p. 2
• Policy performance on a task group averages expected episode scores across its tasks, with training groups G_train distinct from unseen evaluation groups G_test.Source: §2 · p. 2

3. PAPRIKA

• PAPRIKA addresses limitations of bandit-focused fine-tuning by targeting complex strategic decision-making, unknown task algorithms, and data collection across many tasks.It uses diversity-encouraging sampling, preference-based fine-tuning, and curriculum learning, with behaviors intended to generalize to unseen task groups.Source: §3 · p. 3

3.1. Task Design

• The 10 task groups are text-based, multiturn, partially observable, and strategically diverse, requiring agents to balance information gathering with efficient exploitation.Environments provide intermediate observations and final rewards; LLMs or programs generate and verify trajectories, while chain-of-thought improves complex reasoning tasks.Source: §3.1 · p. 3

3.2. Dataset construction

• PAPRIKA generates high-temperature Min-p trajectories, then pairs each highest-scoring successful trajectory with a randomly selected lower-scoring trajectory to preserve dataset diversity.The preferred trajectory succeeds in the fewest turns, while the dispreferred trajectory fails or takes substantially longer to succeed.Source: §3.2 · p. 4

3.3. Optimization

• Optimization uses supervised fine-tuning on winning episodes, multiturn DPO on action tokens, and an RPO combination intended to mitigate DPO’s unintentional unalignment.DPO reduces compute requirements by decoupling data collection from policy improvement, while online RL is left as a potentially stronger alternative.Source: §3.3 · pp. 4–5

3.4. Scalable Online Curriculum Learning

• The curriculum prioritizes task groups whose trajectories offer high learning potential, because data generation—not model updates—is the primary training bottleneck.It assumes similar tasks provide similar learning signals, measures potential using reward-normalized trajectory variability, and selects groups sequentially with UCB.Source: §3.4 · pp. 5–6

4. Empirical Results

• 20 percentage points: Training only on other task groups raised Llama-3.1-8B-Instruct’s bandit success rate from 42.25% to 62.25%, without trajectories from the bandit group.This demonstrates transfer of generalizable decision-making strategies rather than reliance on task-specific optimal trajectories.Source: §4 · p. 7
• PAPRIKA’s leave-one-out models improved success on 9 of 10 task groups and sometimes matched or exceeded group-specific training, while full training outperformed single-group training on 7 of 10 groups.Transfer was uneven: mastermind showed minimal improvement, and wordle exhibited negative transfer, so generalization was not universal.Source: §4 · p. 7, Figure 3 · p. 7
• 1.4% and 3.3%: After three curriculum-training rounds, curriculum sampling exceeded uniform sampling in average and pass@4 accuracy, respectively.The curriculum grouped twenty-questions tasks by difficulty and sampled tasks using solution-turn statistics as a reward proxy.Source: §4 · p. 8, Figure 4 · p. 8

4.1. Analysis

• PAPRIKA reduced the average number of turns needed to solve tasks, indicating more optimal intermediate actions, and generally produced more sensible responses on twenty questions and wordle.These behavioral improvements were reported through quantitative and qualitative analyses.Source: §4.1 · p. 8
• PAPRIKA caused no noticeable degradation in the model’s regular capabilities on standard evaluations compared with Llama-3.1-8B-Instruct.This supports scaling the fine-tuning approach without sacrificing ordinary language-model performance.Source: §4.1 · p. 8

LLM alignment

• Standard LLM alignment commonly combines instruction tuning with reinforcement learning from human feedback using PPO or preference optimization using DPO, typically for single-turn responses.Paprika instead studies iterative interaction with an environment.Source: §5 · pp. 8–9
• Paprika focuses on multi-turn environment interaction for general sequential decision-making, rather than single-turn responses or task-specific human interactions.Prior work includes textual RL environments and scientific-task environments, but Paprika evaluates general exploration and exploitation across tasks.Source: §5 · p. 9

In-context reinforcement learning

• Existing in-context reinforcement learning studies typically use conventional RL environments and data from random policies or RL algorithms, whereas Paprika targets transfer to completely new environments.Paprika also differs from medical information-seeking work and LLM-assisted agents by applying general information seeking and using the LLM as the decision-making agent.Source: §5 · p. 9

Curriculum learning in RL

• Paprika’s curriculum uses task-group metadata and a constant number of rollouts per group, unlike related curriculum work that generates rollouts for every example to estimate learnability.Curriculum learning is motivated by ordering tasks from easier to harder so earlier skills can scaffold later learning.Source: §5 · p. 9

• Paprika improves LLMs’ multi-turn decision making and enables learned strategies to generalize zero-shot to unseen tasks.The method is presented as scalable fine-tuning for multi-turn decision making.Source: §6 · p. 9
• Performance depends on a good base model because rejection sampling requires self-generated behavior that is effective within a reasonable generation budget.Source: §6 · p. 9
• Offline preference tuning was used because of limited computational resources, while online RL on diverse tasks is proposed as a direction expected to improve in-context RL capabilities.Source: §6 · p. 9
• Future work includes scalable LLM-generated task creation and improving curriculum learning, whose performance depends heavily on imperfect task-group clustering.The environments required substantial human implementation effort, motivating automated task generation.Source: §6 · p. 9

Impact Statement

• Paprika could improve strategic exploration and decision-making in large language models, with potential real-world impact if agentic systems become widespread.Source: Impact Statement · p. 10
• Because experiments use relatively simple, controlled environments, the societal effects of genuinely agentic systems remain an open question.Source: Impact Statement · p. 10
• The paper aims to advance machine learning and does not identify any specific societal consequence as requiring emphasis.Source: Impact Statement · p. 10

A. Note on Curiosity

• Paprika defines curiosity as gathering only the information required to solve a task, rather than pursuing all knowable information through task-independent intrinsic motivation.Source: A. Note on Curiosity · p. 16
• This objective follows the exploration-exploitation trade-off by exploring enough to solve a problem without sacrificing efficiency through over-exploration.Source: A. Note on Curiosity · p. 16
• Unlike mostly tabula-rasa exploration methods, PAPRIKA learns exploration strategies from trajectories across environments, enabling more efficient exploration on new problems through amortized exploration.Source: A. Note on Curiosity · p. 16

B.1. Summary of Task Groups

• Paprika spans ten task groups with varied information-seeking demands, including questioning, diagnosis, deduction, exploration, logical reasoning, and sequential decision making.The task groups include Twenty Questions, Guess My City, Customer Service, Murder Mystery, Wordle, Cellular Automata, Mastermind, Battleship, Minesweeper, and Bandit Best Arm Selection.Source: §B.1 · pp. 16–17

B.4.1. TWENTY QUESTIONS

• Twenty Questions asks agents to identify a hidden topic using up to 20 adaptive yes-or-no questions, with 1,499 training topics and 367 test topics separated from one another and prior benchmark topics.GPT-4o-mini supplies constrained yes/no feedback and judges task success; the expanded topics cover people, events, concepts, locations, and other scenarios.Source: §B.1 · p. 16, §B.4.1 · p. 20

B.4.2. GUESS MY CITY

• Guess My City requires identifying a secret city within 20 questions through open-ended clues about geography, culture, landmarks, cuisine, and climate rather than directly requesting its name.The authors generated 500 training cities and 185 test cities, with a separate 91-city LMRL-Gym evaluation set; GPT-4o-mini simulates the environment and judges valid guesses.Source: §B.1 · p. 16, §B.4.2 · p. 22

B.4.3. CUSTOMER SERVICE

• Customer Service evaluates whether an agent can ask targeted, efficient questions to diagnose realistic technical problems and resolve them in as few interactions as possible.GPT-4o-mini simulates a customer with limited technical knowledge who provides relevant information without revealing the solution directly.Source: §B.1 · p. 16, §B.4.3 · p. 24

B.1.1. NOTE ON TASK PROMPTS

• Task instructions are supplied in the first user prompt, while every task group uses the same system prompt, “You are a helpful assistant.”Initial experiments found this placement more fruitful than putting the task instruction in the system prompt.Source: §B.1.1 · p. 18

B.2. Note on Text-based Games

• Paprika uses text-based games as simpler, reliably learnable information-seeking tasks while aiming to transfer the approach to more complicated domains.The paper implements its Murder Mystery environment separately because prior interactive-fiction environments depended on rigid action parsing; GPT-4o-mini simulation provides more dynamic responses.Source: §B.2 · p. 18

B.3. Comparison of action and observation spaces between the task groups

• The ten task groups differ in their initial states, action spaces, and observation spaces, as summarized in Table 3.The table is presented as a cross-task comparison of these interaction components.Source: Table 3 · p. 19, §B.3 · p. 19

B.4.3. CUSTOMER SERVICE

• Customer Service presents a refrigerator problem whose solution must be inferred by the agent, with an LLM judge filtering potentially hacked trajectories and validating successful suggestions.The agent is not told the solution directly; success requires proposing that the slightly ajar refrigerator door be properly closed.Source: §B.4.3 · pp. 24–25

B.4.4. MURDER MYSTERY

• Murder Mystery requires detectives to ask precise questions, inspect evidence, remain open to witnesses and background characters, and accuse the culprit only after gathering sufficient information.The environment reveals minimal, potentially misleading information while maintaining consistency, and GPT-4o-mini judges whether the final accusation identifies the true culprit.Source: §B.4.4 · pp. 26–28

B.4.5. WORDLE

• Wordle asks agents to guess a secret five-letter word within six attempts by strategically updating guesses from per-letter feedback about correctness and position.The environment is implemented with a hardcoded program that generates intermediate observations and eventual rewards, while prompts require concise chain-of-thought and answer formatting.Source: §B.4.5 · p. 29

B.4.6. CELLULAR AUTOMATA

• Cellular Automata requires inferring a 1D binary transition rule from input-output pairs and refining incorrect rules using the outputs they generate.There are 256 possible rules, edge neighborhoods wrap around, and success depends on matching all provided outputs rather than exactly recovering the hidden rule.Source: §B.4.6 · pp. 30–32

B.4.7. MASTERMIND

• Mastermind asks agents to recover a four-digit code, including repeated digits, using only aggregate feedback on exact-position and different-position matches.The agent must infer which digits and positions explain successive feedback because the environment does not identify the digits contributing to each match count.Source: §B.4.7 · p. 33
• Mastermind requires refining guesses of a hidden four-digit code using exact-match and partial-match feedback until all four positions are correct.The environment withholds which digits match, requiring deduction across successive guesses.Source: §B.4.7 · pp. 33–34

B.4.8. BATTLESHIP

• Battleship is a text-based single-player game on a randomly populated grid where agents must locate and sink three hidden ships within a turn limit.Feedback reports misses, hits, ship types, and sunk ships, requiring exploration followed by exploitation around discovered hits.Source: §B.4.8 · p. 35

B.4.9. MINESWEEPER

• Minesweeper places randomly hidden mines on a grid and requires revealing safe cells using adjacent-mine counts, recursive zero-cell expansion, and logical deduction.The first selected cell is guaranteed safe, while revealing a mine ends the game; numbered cells encode mine counts among up to eight neighbors.Source: §1 · p. 38, §2 · p. 38, §3 · p. 38

B.4.10. BANDIT BEST ARM SELECTION

• The bandit task samples one best Bernoulli arm and inferior arms, gives the agent 20 reward observations, and then asks it to identify the highest-mean arm.Training uses 81 predefined text scenarios, while evaluation uses one scenario with randomly sampled arm rewards.Source: §B.4.10 · p. 40
• The bandit evaluation averages performance over 100 trials, with four trajectories generated for each sampled arm-reward setting.This evaluation procedure is applied to the single evaluation task.Source: §B.4.10 · p. 40
• Unlike Nie et al. (2024), the bandit setting uses multi-turn interactions in which agents must interpret the full conversation history rather than make isolated single-turn decisions.Nie et al. (2024) supplies prior interaction history in each new prompt, whereas this setting maintains the interaction across turns.Source: §B.4.10 · p. 40

Trajectory generation

• Training data uses 20 trajectories per task, except 100 for mastermind, generated with temperature 1.5 and Min-p 0.3 to promote diverse, high-quality samples.The generation configuration was selected through an initial twenty-question ablation and then applied across task groups.Source: C. Details of Training Dataset… · p. 41

SFT dataset

• SFT retains successful trajectories with distinct turn counts while discarding examples exceeding 12000 tokens to avoid memory problems.The filtering addresses large-context memory demands despite using Flash-Attention.Source: C. Details of Training Dataset… · p. 41

DPO dataset

• DPO pairs each task’s fastest successful trajectory with a randomly selected lower-performing trajectory, using one pair per task and removing examples longer than 8192 tokens.Random dispreferred sampling improves dataset diversity and performance, while multiple pairs per task were avoided because they increased unintentional unalignment (Razin et al., 2024).Source: C. Details of Training Dataset… · p. 41

Filtering

• Trajectories that violate required answer formatting are terminated and filtered, while no other filtering is applied beyond the stated dataset rules.The authors leave additional filtering strategies, including those discussed by Razin et al. (2024), for future work.Source: C. Details of Training Dataset… · p. 41

Minesweeper limitation

• Minesweeper preference pairs may be invalid because the environment changes after the first action, although this had no significant observed performance effect; unpaired preference learning such as KTO may be more suitable.The authors did not control the first action during generation, so minesweeper pairs require filtering by first action before DPO.Source: C. Details of Training Dataset… · p. 41

D. Note about Task Environment Hacking

• Environment hacking can produce incorrect observations and success rewards in LLM-simulated tasks, especially open-ended ones where strict response matching is impossible.Strict string matching partially mitigates the issue for twenty questions, whose responses are limited to “yes,” “no,” and “Goal reached.”Source: D. Note about Task Environment Hacking · p. 42
• A separate GPT-4o-mini judge filters trajectories mistakenly marked successful by the simulated environment, reducing but not eliminating false successes.The judge operates in a separate conversation at every turn.Source: D. Note about Task Environment Hacking · p. 42
• 1.5% (3) of 200 manually sampled successful trajectories showed environment hacking, so environment errors should be considered when interpreting agent success rates.The authors argue that hacking cannot be fully mitigated.Source: D. Note about Task Environment Hacking · p. 42
• Using GPT-4o instead of GPT-4o-mini reduces observed hacking, but GPT-4o-mini was retained for environment simulation because of API-call budget constraints.GPT-4o-mini sometimes distinguishes Kiev from Kyiv incorrectly or reveals the secret answer, whereas GPT-4o was never observed making these mistakes.Source: D. Note about Task Environment Hacking · p. 42
• The paper identifies designing open-ended tasks less prone to environment hacking, potentially with smaller environment models, as future work.This direction is motivated by the residual errors observed with the current simulator and judge.Source: D. Note about Task Environment Hacking · p. 42
• Table 5 presents a guess-my-city trajectory where GPT-4o-mini reveals the answer and its judge fails to detect the resulting environment hacking.This occurs despite an explicit system instruction prohibiting the environment from revealing the city or country directly.Source: Table 5 · p. 42, D. Note about Task Environment Hacking · p. 42

E. More on LLM Inference Settings

• For four LLM-simulated task groups, environment and judge responses use temperature 0.0 and at most 1024 tokens per turn to support deterministic comparisons.The simulator and judge are GPT-4o-mini in these experiments.Source: E. More on LLM Inference Settings · p. 43
• Agents use Min-p 0.3, with maximum generation lengths of 128 tokens for twenty questions, 512 for guess my city, and 1024 for other task groups.These settings apply across the reported agent experiments.Source: E. More on LLM Inference Settings · p. 43

F. Additional Experimental Details

• Llama-3.1-8B-Instruct training used 8 NVIDIA L40S GPUs, Gemma-3-12B-IT training used 8 NVIDIA H100 GPUs, and inference/data generation used single NVIDIA A40 GPUs.The full project’s API cost was approximately 20,000 USD, while one complete experiment run was estimated at no more than 1000 USD.Source: F. Additional Experimental Details · p. 43

G. Public Release of Code, Model and Dataset

• The authors release code for reproducing the paper’s results.The codebase is publicly available through the project’s repository.Source: G. Public Release of Code, Model and… · p. 43
• They release both the supervised fine-tuning dataset and the dataset used for RPO fine-tuning.The passages identify separate public dataset locations for the two training stages.Source: G. Public Release of Code, Model and… · p. 43
• They additionally release a PAPRIKA-trained Llama-3.1-8B-Instruct model checkpoint to facilitate further research.The released checkpoint is explicitly identified as trained with PAPRIKA.Source: G. Public Release of Code, Model and… · p. 43

H. More Details on Curriculum Learning

• Curriculum difficulty levels for Twenty Questions are generated by asking gpt-4o-mini to classify topics as easy, medium, or hard based on guessability and estimated turns.The example classifies “Apple” as easy.Source: H. More Details on Curriculum Learning · p. 44
• Task difficulty is quantified from 20 trajectories using turns to success as a reward proxy, assigning failures the maximum of 20 turns and comparing tasks by coefficient of variation.Because the trajectory sample is small, the method uses an unbiased coefficient-of-variation estimator for normally distributed data.Source: H. More Details on Curriculum Learning · p. 44

I.1. Success Rate Comparison with More Baselines

• PAPRIKA improves Llama-3.1-8B-Instruct on both pass@4 and average success rate across the evaluated task groups.Figures 5 and 6 evaluate success across temperatures 0.3, 0.7, and 1.0; Figure 5 also compares against a WildChat-finetuned model that performs poorly on all tasks.Source: Figure 5 · p. 45, Figure 6 · p. 45, §I.1 · p. 45

I.2. Task Efficiency Comparison with More Baselines

• PAPRIKA generally lowers the average number of turns required to solve tasks, improving task efficiency and strategic exploration.The analysis averages four trajectories per task and all evaluation tasks within each group; bandit best-arm selection is excluded because its turn count is fixed.Source: §I.2 · p. 45

I.3. PAPRIKA Imporoves Task Success Rate on Gemma-3

• Figures 8 and 9 report Gemma-3-12B-IT’s pass@4 and average success rates after PAPRIKA fine-tuning.Source: §I.3 · p. 45

I.4. PAPRIKA Imporoves Task Efficiency on Gemma-3

• PAPRIKA’s reduced average turns quantify better strategic exploration abilities across task groups.The corresponding efficiency evaluation excludes bandit best-arm identification because its number of turns is fixed at 20.Source: Figure 7 · p. 46

I.3. PAPRIKA Imporoves Task Success Rate on Gemma-3

• PAPRIKA outperforms or matches GPT-4o-mini on 7 of 10 task groups in Gemma-3-12B-IT pass@4 success rate.It achieves improved or comparable success rates across all task groups, supporting applicability across models with different parameter counts and pre-training setups.Source: §I.3 · p. 46
• PAPRIKA produces improved or comparable success rates on all task groups for Gemma-3-12B-IT.The Gemma-3-12B-IT evaluation used the full pipeline with the same hyperparameters as the Llama-3.1-8B-Instruct experiments.Source: §I.3 · pp. 45–46

I.4. PAPRIKA Imporoves Task Efficiency on Gemma-3

• The Gemma-3-12B-IT results extend PAPRIKA’s success and efficiency improvements beyond the Llama-3.1-8B-Instruct setting.The Gemma model has 12B parameters and a different pre-training setup, while the efficiency trend mirrors the Llama experiments.Source: §I.3 · p. 46, §I.4 · p. 46
• PAPRIKA reduces Gemma-3-12B-IT’s average number of turns across all task groups.This reduction demonstrates improved information-seeking behavior learned through PAPRIKA fine-tuning.Source: §I.4 · p. 46

I.7. Evaluation on LMRL-Gym split

• On the disjoint LMRL-Gym split for guess my city and twenty questions, gains observed on PAPRIKA’s evaluation split translate to the external secret-topic sets.The paper’s training and evaluation sets have no overlap with LMRL-Gym, which is used only for fair comparison.Source: §I.7 · p. 47, Figure 14 · p. 49, Figure 15 · p. 50

I.8. Experiments on Modified Wordle to Further Test Generalization

• PAPRIKA outperforms Llama-3.1-8B-Instruct on modified Wordle tasks with 4-, 6-, 7-, 8-, 9-, or 10-letter words, despite never being trained on this task group.The agent must solve these tasks within 10 turns using Wordle-like environment feedback.Source: §I.8 · p. 47, Figure 16 · p. 50

I.5. More Performance Metrics

• PAPRIKA-finetuned Llama-3.1-8B-Instruct and Gemma-3-12B-IT models outperform their regular instruct counterparts for pass@k with k ∈{1, 2, 3, 4}.The additional metric complements previously reported Pass@4, average success rate, and average number of turns.Source: §I.5 · p. 46, Figure 11 · p. 48, Figure 12 · p. 49

I.9. Ablation Study over Different Finetuning Stages of PAPRIKA

• RPO using negative or dispreferred trajectories improves performance beyond SFT alone on all six evaluated task groups.The ablation compares an SFT checkpoint with the checkpoint obtained by further fine-tuning it with RPO.Source: §I.9 · p. 48, Figure 17 · p. 51

I.6. More Results on Generalization

• Leave-one-out experiments test PAPRIKA’s zero-shot performance on unseen task groups, while full multi-group training is compared with single-task-group training using pass@4 success rate.The comparison evaluates whether diverse trajectories from multiple task groups improve generalization over training on one task group.Source: §I.6 · p. 47, Figure 13 · p. 49

I.10. Finetuning on regular multiturn data does not help

• Fine-tuning Llama-3.1-8B-Instruct on 100,000 randomly sampled WildChat trajectories causes significant performance degradation across all task groups.The authors speculate that WildChat prioritizes coherence rather than information gathering, making strategic-exploration task data necessary for sequential decision-making gains.Source: §I.10 · pp. 49–50

I.11. Performance comparison between different starting models

• Across three representative task groups, average success ranks Llama-3.1-8B-Instruct above Qwen-2.5-7B-Instruct, which ranks above Mistral-7B-Instruct-v0.3.DeepSeek-R1 distilled Llama-8B and Qwen-7B produced no final answer even after 10,000 tokens, so their performance could not be evaluated; broader base-model fine-tuning is left for future work.Source: §I.11 · p. 50, Figure 18 · p. 51

I.12. Details on Standard Benchmarks

• Standard-capability evaluations cover MT-Bench, AlpacaEval, GPQA, Math, MMLU-Pro, and IFEval, using benchmark-specific scores such as length-controlled win rate, strict or exact-match accuracy, and instruction-level loose accuracy.MT-Bench uses its standard scores; AlpacaEval reports length-controlled win rate against GPT-4-turbo; Math uses the Math (Hard) subset.Source: §I.12 · pp. 51–52

Limitations

• PAPRIKA improves empirical regret over Llama-3.1-8B-Instruct for bandits with few arms, but the gap vanishes as the number of arms increases.This evaluation used the original bandit task proposed by Krishnamurthy et al. (2024).Source: Figure 19 · p. 52

Evaluation setup

• The evaluation reports empirical regret averaged across 100 trials using the reward difference between the optimal arm and the selected arm at each timestep.PAPRIKA-finetuned models were evaluated without explicit training on this bandit task group.Source: J. Limitations of PAPRIKA: Evaluation… · p. 52

• On the orca task, PAPRIKA asked more concise questions and reached the final topic quicker than Llama-3.1-8B-Instruct.Source: K. Example Trajectories · p. 53, Table 6 · p. 54
• PAPRIKA guessed the concept in 8 turns, while Llama-3.1-8B-Instruct failed within 20 turns in all 4 attempts.Source: K. Example Trajectories · p. 53, Table 7 · p. 55
• Overall, PAPRIKA improved quantitatively and qualitatively over Llama-3.1-8B-Instruct in this limited trajectory evaluation, while detailed behavior analysis remains future work.Source: K. Example Trajectories · p. 53
• On Wordle, PAPRIKA produced more concise, higher-quality guesses, whereas Llama-3.1-8B-Instruct made wrong guesses and reasoned incorrectly against the answer.Source: K. Example Trajectories · p. 53, Table 8 · p. 56