PaperlayerView exact revision on arXiv

Source-linked AI summary

From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions

Nathanaël Carraz Rakotonirina, Mohammed Hamdy, Jon Ander Campos, Lucas Weber, Alberto Testoni, Marzieh Fadaee, Sandro Pezzelle, Marco Del Tredici

arXiv:2502.13791v2cs.CL

TL;DR

LLMs are widely used in work, but effective collaboration over long interactions requires retaining and using relevant information across sessions. MEMORYCODE evaluates this with synthetic multi-session coding dialogues, and models that handle isolated instructions struggle with full histories, including GPT-4o. The results suggest that reasoning ability alone does not resolve the challenge.

  • Problem

    LLMs often solve individual tasks well, but their ability to retain relevant information across interactions for future tasks remains an important question for workplace collaboration.

    Source: §1 · p. 1

  • Method

    MEMORYCODE is a synthetic multi-session dialogue benchmark that tests whether models track and execute simple coding instructions amid irrelevant information, including instructions that are updated over time.

    Source: §1 · pp. 1–2

  • Results

    GPT-4o shows a 67% drop in accuracy with full dialogue history compared with instructions alone, while models generally handle isolated instructions well.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Even newer models with reasoning abilities struggle to reason over sequences of instructions, suggesting complementary or alternative capabilities are needed.

    Source: §5 · p. 6

    MEMORYCODE uses synthetic data, and the experiments do not establish a human performance upper bound.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used in working environments for a wide range of tasks, excelling at solving individual problems in isolation. However, are they also able to effectively collaborate over long-term interactions? To investigate this, we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information, simulating a realistic setting. While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions. Our analysis suggests this is due to their failure to retrieve and integrate information over long instruction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions.

1 Introduction

LLMs are increasingly used as workplace teammates, but collaboration requires retaining relevant information across interactions. MEMORYCODE tests whether models can retrieve and apply evolving coding instructions across sessions, where results show substantial difficulty when full dialogue histories are provided.

1 Introduction

  • MEMORYCODE is a synthetic multi-session benchmark testing whether models track simple coding instructions amid irrelevant dialogue and execute them in future coding tasks.Its mentor-mentee histories include unrelated content and instructions that can be updated multiple times.Source: §1 · pp. 1–2
  • The benchmark models workplace onboarding, where newcomers must follow coding conventions across sessions unless those rules are changed or deprecated.The motivating skill is retaining relevant information from interactions and leveraging it for future tasks.Source: §1 · pp. 1–2
  • Unlike cued retrieval of static information, MEMORYCODE requires spontaneous retrieval and integration of instructions from different parts of dialogue, using the latest update.Its rules are simple to execute, helping distinguish retrieval demands from complex coding skills.Source: §1 · p. 2
  • GPT-4o’s accuracy drops 67% with full dialogue history versus instructions alone, while models struggle to retrieve and incrementally update relevant information.Even small models can execute individual instructions without complex context, but performance deteriorates when instructions and irrelevant information are spread across sessions.Source: §1 · p. 2
  • The authors argue that solving MEMORYCODE requires dedicated capabilities, such as improved long-term memory retention, prospective memory, or additional reasoning mechanisms, rather than scaling alone.The dataset is released under the Apache 2.0 license.Source: §1 · p. 2

2 Related work

Prior work evaluates long-context retrieval, multi-turn dialogue, and synthetic conversations, but MEMORYCODE centers on applying current instructions to practical coding tasks in evolving business dialogues.

2.1 Long-Context Evaluation

  • Long-context benchmarks retrieve information from distractor texts, whereas MEMORYCODE retrieves the most up-to-date instructions dispersed across dialogue to accomplish a task.Prior frameworks include Needle-in-a-Haystack, LTM, RULER, and LOFT.Source: §2.1 · p. 2

2.2 Long-term Dialogue Evaluation

  • Existing dialogue benchmarks assess multi-turn conversations, tool use, or follow-up understanding; MEMORYCODE instead tests simple tasks amid constantly changing instructions in multi-session dialogues.MT-Bench, MT-Eval, MINT, and other frameworks cover dialogue evaluation strategies and capabilities.Source: §2.2 · p. 3

2.3 Synthetic Dialogue Generation

  • Unlike synthetic dialogue datasets focused on tracking facts or expanding conversational complexity, MEMORYCODE uses practical business coding contexts and evaluates following well-defined instructions.Its evaluation emphasizes instruction following rather than simply retrieving facts.Source: §2.3 · p. 3

3 Dataset

MEMORYCODE models onboarding as chronological mentor–mentee sessions where simple coding instructions appear amid irrelevant content and may change over time. Its histories are built from manually defined seeds, combined into templates, and rendered by an LLM.

3 Dataset

  • The dataset tests whether a new hire can apply relevant coding instructions across sessions despite irrelevant content and updates.Histories concatenate mentor–mentee sessions, and the instructions are evaluated on later coding tasks.Source: §3 · pp. 3–4

3.1 Seeds

  • Seeds include instructions, fillers, mentor and mentee personas, and names; authors manually defined their possible values to support quality.The dataset includes general-work and instruction-like fillers, which can be updated but are not evaluated; it uses six mentor and five mentee personas.Source: §3.1 · p. 4
  • Instructions are simple Python coding rules tied to specific objects; 16 of 51 manually crafted instructions can be updated up to eight times.Regex test functions check whether relevant rules were followed, not overall code quality.Source: §3.1 · p. 4

3.2 Dialogue Histories

  • They first create templates by sampling combinations of seeds and parameters, then generate histories from them with an LLM; for each session, a template-based prompt is used with Command R+.“Short” histories have fewer than 15 sessions (54%), while “long” histories have more than 15 (46%); the longest contains 63k tokens, and manual inspection of final generated dialogue histories confirmed overall quality and coherence.Source: §3.2 · pp. 4–5

4 Experiments

The experiments compare recent proprietary and open-weight LLMs across inputs ranging from a single instruction to a complete multi-session history. They score instruction adherence with regex tests and macro-averaged accuracy.

4 Experiments

  • INSTRUCTION supplies one coding rule, while SESSION supplies one whole session and requires adherence to all its instructions.These settings distinguish isolated coding performance from following multiple instructions in a session.Source: §4 · p. 5
  • The HISTORY setup supplies the concatenation of all sessions, testing recall of earlier information and its use with new information for a task.It is the most challenging setup and is intended to mimic long-term workplace interactions.Source: §4 · p. 5

4.1 Models

  • Models receive a score of 1 only when all relevant instances follow the instruction without syntax errors; overall performance is macro-averaged accuracy.The study includes Llama-3.1 at 8B, 70B, and 405B, Command R+, Command A, GPT-4o, DeepSeek-V3, and DeepSeek-R1.Source: §4 · p. 6, §4.1 · p. 6

5 Results

Models perform well on isolated instructions, but accuracy degrades when instructions must be retrieved from multi-session histories. Longer histories remain difficult even for strong and reasoning-capable models.

5 Results

  • All models achieve high or very high accuracy on INSTRUCTION, with large models approaching or exceeding 0.9 and DeepSeek-V3 achieving a perfect score.Even Llama-3.1-8B performs well on this relatively easy setup.Source: §5 · p. 6
  • Without the necessary instructions, models achieve average accuracy consistently below 0.01, showing that the instructions are crucial to correct task execution.This sanity check tests whether models succeed through their default behavior.Source: §5 · p. 6
  • In SESSION, larger Llama models, DeepSeek models, Command A, and GPT-4o perform similarly to INSTRUCTION, while Command R+ drops 0.22 (25%) and Llama-3.1-8B drops 0.34 (48%).The reported differences indicate that the two smaller or weaker-performing cases have more difficulty retrieving instructions across turns.Source: §5 · p. 6
  • Across models, HISTORY performance degrades even on short histories; Llama-3.1-8B drops 0.47 (67%) and Command R+ drops 0.48 (54%) from INSTRUCTION.The results indicate difficulty identifying and applying relevant instructions as session counts increase.Source: §5 · p. 6, Figure 3 · p. 7
  • One experiment and its associated analysis use only Command R+ because of budget limits; the authors expect the findings to represent other models.This limits the breadth of models directly tested in that experiment.Source: §5 · p. 6
  • GPT-4o reaches 0.30 accuracy on long histories, a 61% drop from short histories and a 67% drop from INSTRUCTION; DeepSeek-R1 drops 0.56 points from INSTRUCTION.The long-history setup includes 16 to 100 sessions.Source: §5 · p. 6, Figure 3 · p. 7

6 Analysis

Performance falls as dialogue histories lengthen, and even removing irrelevant dialogue leaves models struggling with instruction chains. The analysis also finds stable scores across update ranks but differences by instruction type.

6.1 Effect of Number of Sessions

  • As sessions approach 100, all models converge to around 0.1 accuracy, despite variation at lower session counts.Figure 4 presents score by number of sessions.Source: Figure 4 · p. 7, §6.1 · p. 7

6.2 Retrieval or Multi-Instruction-following Problem?

  • Retrieval-augmented generation does not improve performance over HISTORY.The authors tested retrieving relevant history portions instead of providing the whole history.Source: §6.2 · p. 8
  • INSTRUCTIONS-CHAIN remains difficult without irrelevant dialogue, with a trend similar to HISTORY, pointing mainly to difficulty reasoning compositionally over instruction sequences.The authors also report slightly higher performance in INSTRUCTIONS-CHAIN than HISTORY, indicating that retrieval plays a role too.Source: §6.2 · p. 7, Figure 5 · p. 8

6.3 Effect of Instruction Updates

  • GPT-4o’s score is stable across update ranks after controlling for session count, suggesting joint rule-following is harder than updating one instruction.The analysis computes scores for each update rank separately at each number of sessions.Source: Figure 6 · p. 8, §6.3 · p. 8

6.4 Instruction Difficulty

  • GPT-4o scores lower on less common instructions, such as digit-containing object names, than on familiar ones like docstrings or annotations.The authors suggest familiarity likely influences models’ ability to retrieve and update relevant instructions.Source: §6.4 · p. 8
  • Compositional interactions among mentor-provided instructions are not explicitly analyzed, though the authors argue their approach can be adapted to study them.Examples include combining parallel-computing comments with rules for merging comments on correlated code lines.Source: §6.4 · p. 8

7 Conclusions

The conclusions describe MEMORYCODE as a coding benchmark where simple instructions must be tracked across multi-session interactions. Models perform well on isolated tasks but struggle as instructions accumulate, which the authors connect to limits in long-term collaboration.

7 Conclusions

  • MEMORYCODE tasks are easily solved in isolation, but challenge models to keep track of and jointly apply simple instructions across multi-session interactions.The paper frames this as reasoning over a long chain of simple instructions.Source: MODEL OUTPUT · p. 9

MODEL OUTPUT

  • Even strong models such as GPT-4o show a dramatic performance drop as interactions grow to 100 sessions.The authors attribute the drop mainly to failure to reason over long chains of simple instructions.Source: MODEL OUTPUT · p. 9
  • The authors conclude that scaling model or dataset size, context windows, or test-time compute alone will not achieve effective long-term collaboration; they call for long-term memory mechanisms.MEMORYCODE is presented as a benchmark for developing and testing such methods.Source: MODEL OUTPUT · p. 9

Limitations

The authors identify synthetic data and a coding-only task domain as scope boundaries, and note that their experiments do not establish a human performance upper bound.

Limitations

  • MEMORYCODE uses synthetic data for cost and experimental-control reasons, so its interactions do not fully reflect unconstrained real-world exchanges.The authors suggest future work could relax the dataset’s constraints to explore more realistic interactions.Source: Limitations · p. 9
  • The experiments establish no human performance upper bound and cover coding tasks only, leaving human comparison and consistency across other domains untested.The authors identify both as directions for future work.Source: Limitations · p. 9

A Seeds

The appendix lists materials used to generate the dataset, including instructions, fillers, and names and personas for mentors and mentees.

A Seeds

  • Data generation used 51 instructions, each applied to a specific Python object.For pivots that can be updated, update values are shown in brackets.Source: A Seeds · p. 13
  • The dataset generation used 80 fillers, some of which can be updated over time.Source: A Seeds · p. 13
  • The appendix provides lists of mentor, mentee, and company names, plus mentor and mentee personas.The lists appear in Tables 7, 8, and 9.Source: A Seeds · p. 13

B Detailed scores

This section points to detailed scores for models evaluated on Instruction, Session, Short History, and Long History.

  • Table 4 contains detailed scores for the evaluated models across Instruction, Session, Short History, and Long History.Source: B Detailed scores · p. 13

C Hyperparameters

The dataset conversations were generated with Command R+ at temperature 0.9 and top-p 0.9; evaluation used temperature 0 for all models.

  • Conversation generation used Command R+ with temperature 0.9 and top-p 0.9, while all models were evaluated at temperature 0.Source: C Hyperparameters · p. 13

D RAG experiments

RAG offers only a marginal improvement over cumulative prompting on short conversations and converges with it on long ones. The appendices also report per-instruction and per-update averages, provide dialogue and prompt examples, and give computational costs.

D RAG experiments

  • RAG provides a marginal improvement over Cumulative for short conversations, then converges with Cumulative for long conversations.Retrieval uses previous sessions and dynamically sets top-k to the number of sessions with pivots.Source: D RAG experiments · p. 13

E Per-Instruction Scores

F Examples

  • The appendix gives two multi-session dialogue-history examples and prompts, plus prompts for the Instruction, History, and Instructions-Chains settings.The examples contain three pivots across three sessions and three pivots across four sessions.Source: F Examples · p. 13

G Computational budget

  • Dataset generation used approximately 0.9M prompt tokens and produced 4.6M dataset tokens; generation and single-model evaluation each cost approximately $50.The evaluation-cost estimate covers Instruction, Session, and Cumulative settings using online APIs.Source: G Computational budget · p. 13

H Ethics statement

This section describes MemoryCode’s synthetic, non-identifying dataset and absence of recruited human participants, and presents prompts and multi-session dialogue examples involving workplace and coding instructions. The examples show instructions being introduced or revised and prompts asking models to follow the latest guidelines in conversation histories.

H Ethics statement

  • The released dataset is synthetic, contains no personally identifiable information, and was manually validated without recruiting human participants.The authors say they manually validated dataset quality themselves.Source: H Ethics statement · p. 13

Prompt to generate dialogue history 70 SYSTEM PROMPT

  • The dialogue-generation prompt sets a business conversation between a mentor and intern at a software company, with Python as the main programming language.It specifies a formal, natural dialogue style and requires coding instructions to be mandatory and given without examples.Source: Prompt to generate dialogue history 70… · p. 24

SESSION 1

  • The first-session prompt has Jean-Aimé and Lucas discuss cultivating a feedback culture and performance improvement plans for continuous growth.It specifies a short conversation and that this is their first meeting.Source: SESSION 1 · p. 24

SESSION 2

  • The second-session prompt gives Lucas two coding instructions: add the @timer_class decorator from pedantic to all classes and start function names with x_.Lucas only acknowledges the instructions and asks no questions; the prompt forbids examples or other coding instructions.Source: SESSION 2 · p. 24

SESSION 3

  • The third-session prompts illustrate updates across sessions: Jean-Aimé changes Lucas’s function-name prefix to gn_; another prompt updates method names to end in _md and adds assert statements in functions.The second example also includes a non-coding update about using a whiteboard for brainstorming.Source: SESSION 3 · pp. 25, 27

Dialogue history 70 SESSION 1

  • The sample first-session dialogue introduces mentorship as reciprocal guidance and support, with Lucas expressing openness to constructive feedback for growth.Source: Dialogue history 70 SESSION 1 · p. 28
  • A comprehensive skills assessment establishes Lucas’s baseline and guides a performance improvement plan tailored to his needs and goals.The plan is collaborative and covers software engineers’ technical proficiency, problem-solving, and collaboration.Source: Dialogue history 70 SESSION 1 · p. 28

SESSION 2

  • The coding guidance first requires the @timer_class decorator on classes and x_ prefixes for function names.The decorator is from the pedantic module; the mentor later checks in regularly.Source: SESSION 2 · p. 29

SESSION 3

  • The mentor updates the function-naming instruction to require gn_ prefixes while framing feedback as a two-way process.Lucas agrees to use gn_ for function names.Source: SESSION 3 · p. 29

Dialogue history 108 SESSION 1

  • Pablo is initially told to avoid whiteboards for brainstorming and end method names with _o.The mentor advises diving straight into code and presents the naming convention as supporting consistency and readability.Source: Dialogue history 108 SESSION 1 · p. 31

SESSION 2

  • DEVS town halls are typically quarterly, open to all employees, and structured with submitted agenda items and a moderator.Sessions are recorded for people who cannot attend or want to revisit the discussion.Source: SESSION 2 · pp. 31–32

SESSION 3

  • The mentor revises Pablo’s guidance to use whiteboards, include assert statements, and end method names with _md.The mentor presents the changes as practices for robustness, consistency, and team efficiency.Source: SESSION 3 · p. 32

SESSION 4

  • Pablo suggests interactive town halls with Q&A or live polls, and the mentor asks him to organize the next meeting.The discussion also proposes agenda and time boundaries, collaboration tools, honest updates, and diverse presenters.Source: SESSION 4 · pp. 32–33

History Prompt Example SYSTEM PROMPT

Loading 2502.13791v2…