PaperlayerView exact revision on arXiv

Source-linked AI summary

RENDER: Controlling Reader-Facing Evidence in LLM Memory Evaluation

Yuan Si, Simeng Han, Daming Li, Jialu Zhang

arXiv:2608.23568v1cs.AI

TL;DR

Memory and RAG evaluations can conflate model ability with the artifact shown to the reader, even when the underlying history is the same. RENDER fixes the conversation and answer contract while varying reader-facing packets and deployed-style templates; matched-budget resolved packets then outperform recency-truncated raw dialogue by 42.4–72.6 points.

  • Problem

    Memory and RAG evaluations often compare systems whose readers receive different artifacts, so measured accuracy can conflate model ability with evidence rendering and related pipeline choices.

  • Method

    RENDER fixes the underlying conversation, question, and answer contract while varying the reader-facing artifact through a five-level packet ladder and deterministic memory-surface templates.

  • Results

    Under matched word budgets, streamlined resolved-P2 packets outperform recency-truncated raw dialogue by 42.4–72.6 points on every model.

  • Takeaways & Limitations

    Memory/RAG evaluations should report the reader-facing evidence artifact or include a fixed-artifact control rather than treating rendering as an implementation detail.

  • Takeaways & Limitations

    The deployed-template variants are controlled approximations, using gold-localized turns for ChatGPT- and MemGPT-style renderings and deterministic truncation for LangChain-style summaries.

Abstract

from arXiv · show

Memory and RAG evaluations often treat the answering model's input as an implementation detail, even though systems may render the same history as a memory entry, summary, typed record, or raw excerpt. We introduce RENDER, a benchmark control that fixes the conversation while varying the reader-facing artifact. RENDER combines a five-level packet ladder, localizing when answer-bearing content enters the input, with deterministic templates approximating ChatGPT-style entries, LangChain summaries, MemGPT-style typed records, and raw conversation. On 500 LongMemEval questions and nine models, matched-budget resolved packets beat recency-truncated raw dialogue by 42.4-72.6 points. In deployed-style templates, best-worst spread is 24.6-48.8 points per model; under the primary scorer, ChatGPT-style entries have higher point estimates than raw conversation on 7 of 9 models. Judge rescoring preserves the positive aggregate effect, but model-specific significance is mixed. Three models scoring 0 percent on formal ledger packets answer the same facts from natural-language entries at 45.4-53.4 percent. The effect persists under retrieval noise and transfers to HotpotQA, suggesting that memory/RAG evaluations should report or control the reader-facing artifact.

1 Introduction

RENDER treats the reader-facing evidence artifact as an evaluation variable: holding the conversation and answer contract fixed, it measures how rendering changes memory-QA accuracy. Across diagnostic and matched-budget controls, artifact choice produces large accuracy differences and supports reporting or controlling the artifact.

  • Motivation and contribution: Measured accuracy ranges from near zero to 82%, depending on which artifact reaches the answering model.The paper frames this range as evidence that surface form is not a neutral implementation detail.
  • Motivation and contribution: RENDER fixes the answer contract while varying only the evidence the answering model sees, distinguishing input-side rendering from output-format studies.The underlying conversation, question, and answer contract remain fixed while the reader-facing artifact changes.
  • Main findings: Answer-bearing content drives the main packet-ladder recovery: all nine models remain near 0% at P0/P1, while responsive models reach 15–25% at P2.Adding metadata above P2 changes accuracy by at most ±2 points.
  • Main findings: 42.4–72.6 points separate streamlined resolved-P2 packets from recency-truncated raw dialogue under matched word budgets on every model.Full raw conversation outperforms budgeted formal packets, but the matched-budget comparison reverses that conclusion.
  • Main findings: ChatGPT-style entries have higher primary-scorer point estimates than raw conversation on 7 of 9 models, while judge rescoring retains a positive aggregate effect with mixed model-specific significance.LangChain summaries score 4.4–10.6%, MemGPT-style typed records 14.8–33.0%, and the best–worst spread reaches 24.6–48.8 points per model.
  • Implications: Three models score 0% on formal ledger packets but answer the same facts from natural-language entries at 45.4–53.4%, consistent with artifact- or prompt-triggered abstention.The paper therefore recommends stating the reader-facing artifact or including a fixed-artifact control in memory/RAG evaluations.

2 Related Work

Prior work studies long-context robustness, memory organization, retrieval, RAG, and output-format sensitivity. RENDER differs by isolating the reader-facing evidence artifact while holding the answer contract fixed.

  • Long-context and memory evaluation: Long-context benchmarks primarily stress retrieval, context length, positional robustness, or long-horizon reasoning rather than rendering fixed evidence.Examples include RULER, InfiniteBench, L-Eval, ZeroSCROLLS, LongBench, Marathon, and BABILong.
  • Memory, retrieval, and RAG systems: Memory systems rewrite histories through hierarchies, graph indices, typed schemas, summaries, or agent-memory mechanisms before generation.The cited examples include MemGPT, HippoRAG, GraphRAG, ENGRAM, MemoryBank, Generative Agents, Reflexion, and LangChain utilities.
  • Memory, retrieval, and RAG systems: RAG research evaluates external stores, multi-hop retrieval, retrieval quality, faithfulness, and context use, generally as end-to-end systems.RENDER instead isolates the artifact presented to the reader.
  • Format sensitivity and judging: Output-side structured-generation studies change how models answer, whereas RENDER changes the evidence they read while keeping the answer contract fixed.This distinction situates RENDER among broader work on prompt-format sensitivity and judging.

3 System Design and Methodology

RENDER fixes the dialogue, question, and answer behavior, then varies the rendered evidence artifact shown to a tool-free reader. Its packet ladder progressively exposes answer-bearing information to diagnose content, resolution, and surface effects.

  • Design: RENDER asks how much the reader-facing artifact changes memory-QA accuracy when the conversation, question, and answer contract are fixed.The reader receives one rendered artifact and answers the same free-form question without external tools, search, provider memory, or hidden context.
  • Design: The system stores turn-addressed dialogue, identifies answer-bearing events, resolves conflicts, and renders alternative reader-facing artifacts.Prompt wrapping changes are labeled as part of the condition when applicable.
  • Packet ladder: P0/P1 expose witness addresses without answer text, P2 first writes the resolved current-state value, P3 adds blocker metadata, and P4 adds supporting entries.The ladder diagnoses whether failure reflects missing content, unresolved conflict, or the surrounding packet surface.
  • Query and conflict resolution: Questions use normalized slot keys, temporal and conversational scope, and precedence-ordered candidate events including updates, corrections, contradictions, and deletions.Scope resolution marks candidates as applicable, out_of_scope, or blocked.
  • Evaluation conditions: The structured evaluation crosses five packet levels, three reader prompt families, and two internal backends, yielding 5 × 3 × 2 = 30 conditions.Table 1 aggregates over the six reader/backend variants for each packet level.

4 Experimental Setup and Results

RENDER evaluates how reader-facing evidence artifacts affect memory-question answering across packet levels, budget controls, and deployed-style templates. On LongMemEval, resolved packets outperform recency-truncated raw dialogue under matched budgets, while surface choice produces substantial model-dependent score differences.

  • 4.1 Accuracy Rises When the Answer Enters the Packet: On LongMemEval, P0/P1 expose only witness addresses, while P2 first writes the resolved current-state value into the packet body; P3/P4 add metadata.The ladder separates missing answer content from effects of surrounding packet metadata.
  • 4.1 Accuracy Rises When the Answer Enters the Packet: 0.0–0.6% accuracy at P0/P1 rises to 15.0–25.3% for six responsive models at P2, while P3/P4 shift accuracy by at most ±2 points.Three models remain at 0.0% across all formal structured packets.
  • 4.2 Budget Control Reverses the Raw-vs-Structured Conclusion: Full raw conversation outperforms budgeted formal packets by 10.5–44.4 points, but this comparison mixes surface form with context length and recency cues.The unconstrained raw baseline preserves the entire dialogue, whereas structured packets are budgeted.
  • 4.2 Budget Control Reverses the Raw-vs-Structured Conclusion: 42.4–72.6 points: streamlined resolved-P2 packets outperform recency-truncated raw dialogue under matched word budgets on all nine models.Truncated raw scores 7.6–12.0%, whereas streamlined P2 scores 50.0–82.0%; all deltas are significant under paired bootstrap.
  • 4.3 Deployed Memory Templates Change Scores by up to 49 Points: ChatGPT-style entries have higher primary-scorer point estimates than raw conversation on 7 of 9 models, while judge rescoring shows mixed model-specific significance.MemGPT-style records score 14.8–33.0%, and LangChain summaries are weakest at 4.4–10.6%.
  • 4.3 Deployed Memory Templates Change Scores by up to 49 Points: 24.6–48.8 points: the best–worst surface spread varies across models, and three models answer the same facts from ChatGPT-style entries at 45.4–53.4% despite scoring 0.0% on formal packets.Under retrieval noise, raw accuracy drops 8.8 points from k=0 to k=9 distractors, while ChatGPT-style entries drop 1.5 points.

5 Analysis

Reader-facing artifacts affect both whether models answer and whether their answers are accurate. Formal packet prompts produce hard, cautious, or speculative abstention regimes across models.

  • 5 Analysis: 28.1–75.2 points: answer rates drop from raw conversation to full_render, separating models into three behavioral regimes.The regimes are hard abstention, cautious abstention, and speculative answering.
  • 5 Analysis: Three models have 0.0% answer rates on every formal packet but answer from ChatGPT-style entries at 45.4–53.4% and unresolved witness text at 18–19%.These models answer the same facts when evidence is presented in natural-language or unresolved witness form.
  • 5 Analysis: Answered-only accuracy can reward abstention: claude-sonnet-4 reaches 83.7% conditional accuracy at a 29.5% answer rate, while gemini-3-flash answers 43.9% of items at 15.2% overall.The study therefore counts abstentions as incorrect in accuracy over all questions.

6 Discussion

RENDER shows that artifact effects arise from both answer-bearing content and the surface used to present it. Its controls help distinguish missing evidence, unresolved conflict, and refusal-triggering structure, while results remain task- and model-dependent.

  • Content visibility is the largest driver: witness text alone lifts accuracy from near zero to 14.2–23.2%.For six responsive models, explicit conflict resolution adds only 0.8–6.1 points, significant for three.
  • A matched-content control separates answer-bearing text from upstream conflict resolution by withholding explicit slot resolution and active_value.
  • Hard-abstention models answer 18–19% of unresolved witness-text items but 0% of resolved formal packets.This pattern is consistent with artifact- or prompt-induced refusal rather than inability to reason from structured evidence.
  • Under tight budgets, resolved P2 packets preserve answer-bearing content that recency-truncated raw dialogue may remove.Unconstrained raw conversation retains recency ordering, update phrasing, and local coherence.
  • Compact natural-language entries combine discourse texture with shortness, helping retain relevant facts within constrained input budgets.
  • RENDER supports fixed-surface controls and failure localization, but its preferred representation is task- and model-dependent.NL entries lead deployed templates on LongMemEval, whereas some HotpotQA comparisons favor raw Wikipedia paragraphs or LangChain summaries.

7 Conclusion

Across models and tasks, the same questions receive different accuracies when the reader-facing evidence artifact changes. RENDER makes evidence rendering an explicit control for evaluation and deployment decisions.

  • 42.4–72.6 points: under matched word budgets, streamlined resolved packets beat recency-truncated raw dialogue.
  • ChatGPT-style memory entries have higher primary-scorer point estimates than raw conversation on 7 of 9 models.Judge rescoring preserves a positive aggregate effect but shows mixed model-specific significance.
  • 24.6–48.8 points: deployed-style artifact choices create large per-model accuracy spreads.
  • Three of nine models hard-abstain on formal packets while answering natural-language memory entries at 45.4–53.4%.
  • RENDER enables benchmark authors to add fixed evidence-surface conditions and system builders to distinguish missing content, unresolved conflict, and refusal-triggering surfaces.

8 Limitations

The evaluation isolates reader-facing rendering, but several design, measurement, scope, and statistical boundaries limit how its comparisons should be interpreted.

  • The token-matched baseline uses word-level truncation, so it does not isolate oracle evidence localization from surface form.An oracle raw-excerpt baseline would further separate localization from rendering.
  • RENDER controls the artifact after evidence selection, so it does not measure what a production system stores, updates, retrieves, or suppresses.
  • Deterministic deployed-style templates approximate common reader inputs rather than evaluating end-to-end product implementations.They omit real retrieval, summarization, and pipeline errors, and use gold-localized turns or fixed truncation.
  • The results are not evidence about output-side structured generation or constrained decoding because the answer contract remains fixed.
  • The study covers LongMemEval’s oracle tier, random retrieval distractors, conversational memory, and multi-hop Wikipedia QA rather than all RAG settings.It uses nine commercial APIs whose system prompts, safety policies, and calibration are not inspectable.
  • The study does not identify whether formal-packet refusal is driven by field names, abstention cues, ledger terminology, or the formal reader prompt.
  • The headline effects are robust across scoring variants, but finer metadata and formality claims are more sensitive to scoring choices and sampling noise.ChatGPT-style entries outperform raw conversation in point estimates on 7 of 9 models, while judge-based model-specific significance is mixed.
  • The gpt-5.2-pro comparison retains an endpoint confound because it uses v1/responses rather than the chatcompletions surface used for two other OpenAI models.

9 Ethical Considerations

The paper frames evidence rendering as an evaluation and safety concern rather than a deployment recommendation. It reports scorer validation and cautions against transferring benchmark refusal behavior directly to production.

  • Scope and release boundaries: RENDER studies evaluation artifacts, not deployed user-facing memory behavior, and releases audit materials rather than a deployment recipe.The released artifacts exclude private user data, credentials, provider logs with account identifiers, and provider-calling runner scripts.
  • Scope and release boundaries: Sensitive or stale personal information makes reporting and controlling evidence rendering preferable to exposing raw private histories.Production systems still require separate safety evaluation; the benchmark does not establish deployment safety claims.
  • Interpreting refusal behavior: Refusal on formal packets does not necessarily indicate deployment safety, while answering compact natural-language evidence does not necessarily indicate unsafety.The paper attributes refusal behavior to instructions, provider policy, and application context.
  • Experimental controls: The experiments disable provider-native tools, search, and memory, limiting direct transfer to systems with those capabilities enabled.Calls used temperature 0; output limits were 128 tokens except for gpt-5.2-pro, which used 1024 tokens.
  • Scoring and interpretation: Judge rescoring preserves positive aggregate packet effects, but model-specific ChatGPT-style versus raw differences are mixed.Under the judge, three models show significant positive differences, two significant negative differences, and four remain within sampling noise.

C.6 Late-abstention and substring-leakage cases in the primary substring scorer

The appendix examines leakage from late abstentions and substring matching, showing that stricter abstention handling changes aggregate scores little while exposing concrete false positives.

  • Leakage cases: The primary scorer can count a response as correct when it mentions the gold string before or alongside a late ABSTAIN.Its prefix-only check misses refusals appended after answer-related text.
  • Scorer sensitivity: A stricter abstention-anywhere check reclassifies 1 to 30 items per condition and shifts aggregate accuracies by less than 1 point.The stricter check addresses late-abstention cases but does not remove all substring false positives without an explicit abstention token.
  • Headline robustness: Despite these scorer defects, the headline token-matched gap remains +61.1pp and the real-system ChatGPT-minus-raw aggregate remains essentially unchanged.No qualitative headline conclusion changes under the strict scorer.
  • Retrieval-noise check: Under retrieval noise, natural-language entries are more stable than raw inputs on 7 of 9 models at k=9.The significant NL-over-raw gaps range from +11.5 to +22.5pp for four models, with heterogeneous nonsignificant results elsewhere.

E HotpotQA Evidence-Surface Transfer: Per-Model Detail

HotpotQA tests whether evidence-surface effects transfer beyond conversational memory. Natural-language renderings generally lead, but model-specific preferences and the lower-order ranking remain heterogeneous.

  • Per-model comparisons: NL beats typed on all 9 models, with significant advantages for 6 models ranging from +6.5 to +10.5pp.Three remaining comparisons are within sampling noise.
  • Per-model comparisons: NL beats summary on 8 of 9 models, with gpt-5.4-mini showing a nonsignificant −2.0pp difference.The result indicates a broad but not universal preference for NL over summaries.
  • Per-model comparisons: NL beats raw on 6 of 9 models, including significant gains for gpt-5.4-mini (+10.5pp), gemini-2.5-flash-lite (+12.5pp), and gemini-3.1-flash-lite (+5.5pp).Claude-opus-4.1 and claude-sonnet-4 significantly prefer raw Wikipedia paragraphs, while gpt-5.2-pro narrowly prefers raw nonsignificantly.
  • Rendering setup: The summary and typed renderings are generated once by claude-opus-4.6 and fixed across evaluator models, while NL uses rule-based supporting-fact extraction.Using Opus-4.6 as both generator and evaluator creates a possible generator/reader bias.
  • Scope and limitations: The retrieval-noise experiment uses random memory-store distractors, so dense retrieval at fixed recall budgets remains a needed realism check.The paper identifies query-correlated retrieval errors as a boundary of the current experiment.
  • Aggregate transfer: Aggregate HotpotQA accuracy follows NL > raw > summary > typed, while the summary-versus-typed ordering differs from LongMemEval.The top-level NL preference transfers across tasks, but lower-order surface rankings do not fully transfer.
Loading 2608.23568v1…