PaperlayerView exact revision on arXiv

Source-linked AI summary

Ask Before You Answer: A Question-Driven Evidence Reasoning Architecture (QERA) for In-Context Learning

Jinhu Qi, Minda Hu, Wentao Zhang, Zixing Song, Irwin King

arXiv:2610.35083v1cs.CL

TL;DR

Real-life context tasks often require information the final request leaves implicit, even though it is present in the conversation. QERA asks structured questions before drafting and reuses them to check evidence. Across CL-bench Life, it achieved the highest solving rate on each of four frontier LLMs, averaging 21.0% against 18.2% for the strongest baseline.

  • Problem

    On CL-bench Life, over 80% of rubrics share less than 60% of their content words with the final request, while direct-answering LLMs pass 67% of implicit rubrics versus 81% of explicit ones.

    Source: §3.1 · p. 3

  • Method

    QERA creates a code-filtered requirement checklist and typed context-information needs, drafts from the whole conversation, then checks the same questions with audited quotes and guarded repair.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    QERA had the highest solving rate on all four LLMs, averaging 21.0% versus 18.2% for the strongest baseline, PSCI, and 14.6% for direct answering.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For assistants reasoning over users’ web traces, inference is best spent recovering what a request leaves unsaid before generating.

    Source: §5 · p. 8

    Because the same underlying LLM generates questions, answers, and checks, it can still misread messy context or hallucinate, and those errors can carry into the answer.

    Source: §5 · p. 8

Abstract

from arXiv · show

AI assistants are increasingly asked to reason over a user's own web traces: group chats, community threads, activity logs and revision histories. On CL-bench Life, a benchmark of such real-life context learning, frontier LLMs solve fewer than one task in five. We find that the request is not the specification: over 80% of the benchmark's rubrics share less than 60% of their content words with the final request, as for a rule set two turns earlier or every record that meets the user's criterion, and LLMs answering directly pass only 67% of these implicit rubrics against 81% of the others. We propose QERA, a Question-driven Evidence Reasoning Architecture that asks before it answers. QERA first turns the request into a requirement checklist and typed information needs, filtered by code for faithfulness to the user's words; it then answers from the whole conversation, asks the same questions again with quotes verified by code, and applies a guarded local repair. It needs no training or rubrics and makes about four LLM calls per task. On all 405 tasks with four frontier LLMs, QERA attains the highest solving rate on every LLM (21.0% on average, against 18.2% for the strongest of three inference-time baselines and 14.6% for direct answering) and leads every baseline significantly when pooled across LLMs, while using fewer tokens than PSCI and 4-5% of the tokens of Ctx2Skill. Ablations on GPT-6-Luna show that the gain is settled before the answer, by a fixed answer protocol (+3.8 rubric points) and the request-specific questions (+1.6), and that it is concentrated on implicit rubrics, with checking afterwards refining mainly numeric requirements.

1 Introduction

On CL-bench Life, correct replies often require recovering specifications left implicit in earlier conversation turns. QERA asks structured questions about those requirements before answering, then uses them to check the reply against evidence.

1 Introduction

  • Over 80% of CL-bench Life rubrics share less than 60% of their content words with the final request; direct answering passes 67% of these implicit rubrics versus 81% of explicit ones.Implicit requirements can include earlier rules, later corrections, or every record matching a user’s criterion.Source: §1 · p. 1
  • QERA turns the request into a requirement checklist and typed information needs, filters them for faithfulness, answers from the whole conversation, then checks quoted evidence and guards local repair.It requires no training or rubrics and makes about four LLM calls per task.Source: §1 · p. 2
  • 21.0% of tasks were solved by QERA on average, versus 18.2% for the strongest baseline and 14.6% for direct answering, across four frontier LLMs.QERA ranked first on all four LLMs, led every baseline significantly when pooled, and used fewer tokens than PSCI and 4–5% of Ctx2Skill’s tokens.Source: §1 · p. 2
  • Ablations on GPT-6-Luna attribute most of the gain to steps before the answer: the fixed answer protocol adds 3.8 rubric points and request-specific questions add 1.6.The two-call draft already solves more tasks than every baseline on every LLM; checking mainly refines numeric requirements.Source: §1 · p. 2

2 Related Work

Related work covers context learning, inference-time methods for organizing context, and approaches that decompose or verify reasoning. QERA differs by asking typed questions of the context and reusing them to check evidence.

Context learning

  • CL-bench Life extends context learning to real-life chats, personal archives, and behavioral traces [6], while conversational-memory work studies assistants that store or retrieve user history [18] [21] [24] [27] [35].QERA neither retrieves nor stores: the whole conversation remains in each call.Source: §2 · p. 3

Inference-time methods for context learning

  • CCA compiles context into typed rules [26], Ctx2Skill discovers context-specific skills through self-play [29], and PSCI induces a specification contract before checking and repairing an answer [40].QERA’s questions are anchored to the user’s words, code-filtered, and include typed information needs whose findings are verified against the conversation.Source: §2 · p. 3

Asking, decomposing and verifying

  • Unlike user-directed clarification [1], query expansion [9] [33], or post-answer verification [5] [12] [20] [28], QERA asks typed questions of the context both before and after drafting.Its enumerations and computations are auditable; the paper notes that most work is done once the right questions have been asked before answering.Source: §2 · p. 3

3 QERA: Ask Before You Answer

QERA represents a conversation task with a request and context, then recovers what the reply must deliver and find through questions. It drafts from the full conversation, checks those questions against quoted evidence, and repairs only under safeguards.

3.1 Problem Formulation

  • A task is a conversation ending in a user request; its context includes other material the reply draws on, and benchmark rubrics judge whether the reply satisfies specified criteria.CL-bench Life tasks average 13.2 rubrics, and 59.8% are multi-turn.Source: §3.1 · p. 3

3.2 Ask: What Must the Reply Deliver, and What Must Be Found?

  • QERA asks what the reply must deliver in checklist O and what must be found in context in typed needs N, then code filters questions to keep them faithful to user wording.Needs can look up, enumerate, count, compute, compare, or judge; code adds an enumerate need when a list, count, timeline, summary, or classification requirement lacks one.Source: §3.1 · p. 3, §3.2 · p. 4

3.3 Answer: Generate from the Whole Conversation

  • The draft is generated in one call from the whole conversation, guided by the questions and a fixed answer protocol that checks coverage, specificity, qualifying items, verdicts, calculations, and format.Nothing is retrieved or extracted before generation, so evidence is not lost before drafting.Source: §3.3 · p. 4

3.4 Check: Ask the Same Questions Again, with Evidence

  • The check tests each requirement against the draft and searches the whole conversation for quoted findings; code audits quotes and recomputes itemized findings.It flags latent gaps and limits count errors to cases where the check names a missed, double-counted, or wrongly included item.Source: §3.4 · p. 4

3.5 Guarded Repair

  • A local repair uses checked facts and quotes, while a deterministic guard keeps it only if it preserves required specifics and passes structural and user-constraint checks.QERA makes three LLM calls when no repair is needed and four otherwise, averaging 3.8–4.0 calls including retries, without training, demonstrations, or rubrics.Source: §3.5 · p. 4

4 Experiments

Across 405 CL-bench Life tasks, QERA leads every tested LLM and every pooled baseline comparison. Ablations locate most of its rubric gain before the answer, while its strongest advantage is on implicit requirements.

4.1 Experimental Setup

  • The evaluation covers all 405 CL-bench Life tasks, 5,348 rubrics, and four frontier LLMs; contexts average 19.4K tokens.Tasks span 404 distinct real-life contexts, and inference-time methods receive no rubric access.Source: §4.1 · pp. 5–6

4.2 RQ1: Effectiveness and Cost

  • 21.0% of tasks are solved by QERA on average, versus 18.2% for PSCI, the strongest baseline; QERA leads every LLM and every pooled baseline comparison.Pooled comparisons are significant for both solving rate and rubric pass rate against every baseline.Source: Table 2 · p. 6, §4.2 · p. 6
  • QERA uses 10–26% fewer tokens than PSCI and 4–5% of Ctx2Skill’s tokens, while solving more tasks than Ctx2Skill on every LLM.QERA also improves on PSCI in solving rate while using fewer tokens on every LLM.Source: §4.2 · p. 6

4.3 RQ2: Asking Before vs. Checking After

  • On GPT-6-Luna, the answer protocol (+3.8 rubric points) and request-specific questions (+1.6) account for 89% of QERA’s rubric gain before answering.The draft alone solves more tasks than every baseline; checking after the answer refines numeric requirements.Source: §4.3 · p. 7

4.4 RQ3: Where Do the Gains Come From?

  • On implicit rubrics, QERA passes 74.0%, 2.5 points above CCA; on explicit rubrics, it ties the best baseline.Implicit rubrics make up 82% of classified rubrics, and QERA’s lead holds in every lexical-overlap band below 60%.Source: §4.4 · pp. 7–8
  • QERA ranks first in every context-length quartile and context category, but passes 71.4% of number-or-count rubrics versus 79.2% of other rubrics.Number and count rubrics comprise 71% of rubrics in Behavioral Records & Activity Trails, where QERA’s lead over PSCI is smallest.Source: §4.4 · p. 8

5 Conclusion and Limitations

QERA recovers requirements implicit in conversation before generating, and performs best among the compared inference-time methods on CL-bench Life. Its remaining constraints include LLM-dependent errors, added token cost, and evaluation limited to text traces.

Conclusion

  • On all 405 CL-bench Life tasks, QERA solves the most tasks among the compared inference-time methods, with gains concentrated on implicit requirements across context lengths and types.The authors conclude that the gain is settled before the answer, through a fixed answer protocol and request-specific questions, rather than the subsequent check.Source: §5 · p. 8
  • For assistants handling personal web traces, the authors conclude that inference is best spent recovering what a request leaves unsaid before generating.QERA processes conversations at inference time and stores nothing, while deployment data-retention terms remain with the LLM provider unless models are hosted locally.Source: §5 · p. 8

Limitations

  • QERA’s LLM-written questions and self-checks can still misread messy context, and errors may carry into answers; asking and checking use about five times the tokens of direct answering.The study covers text traces only; extending QERA to photos, screenshots, voice messages, and videos is left for future work.Source: §5 · p. 8
  • Each stage sends the conversation to the LLM provider, so deployment inherits that provider’s data-retention terms; locally hosted LLMs avoid this exposure.Consent and data governance remain the responsibility of whoever deploys the assistant.Source: §5 · p. 8

A Prompts

The appendix records QERA’s prompts for turning a user request into requirements and information needs, answering from the conversation, checking the draft, and making guarded repairs. Its prompt rules emphasize complete, specific replies while preserving the user’s requested format and exact wording.

Ask

  • The Ask prompt splits the last user request into requirements, retaining its exact triggering words, scope, kind, and answer-format constraints.User-defined rules are copied word for word, and earlier-turn requirements are retained when the last message relies on them.Source: A Prompts · pp. 9–10
  • The information-needs prompt lists facts to find or derive across the conversation, with each need tied to the requirements it serves and where to look.List or count requests first require an enumerate need covering every item that meets the user’s criterion.Source: A Prompts · p. 10

Answer

  • The answer prompt directs the model to read all relevant records and later corrections, then deliver each requirement with the materials’ specific names, identifiers, values, and qualifiers.For list, count, and compute requests it calls for item-by-item review and explicit arithmetic.Source: A Prompts · p. 10
  • The answer rules require every qualifying item and its requested fields, preserve attribution and quotation wording, and follow the user’s requested format and structure.They also call for definite counts, rankings, and totals rather than approximate answers.Source: A Prompts · pp. 10–11

Check

  • The Check prompt tests requirements and information needs against the draft and conversation, identifies latent gaps and unsupported claims, and proposes concrete repairs while preserving correct parts.Enumerate and count needs are checked against every qualifying item, not just items already present in the draft.Source: A Prompts · p. 11

Repair

  • The Repair prompt applies actionable fixes using checked facts and quotes, while skipping changes contradicted by the user’s words or materials.Patch rules preserve the draft where no problem was found and keep definite answers definite; the appendix also gives deterministic rules for filters, audit, and the repair guard.Source: A Prompts · pp. 9, 11

B Filters, Audit and Guard

Deterministic filters remove checklist and information-need content that is not faithful to the user’s words. The audit and guard then constrain which findings trigger repair and when a proposed repair is rejected.

Filters

  • Checklist filters remove exclusions, caution clauses, unsupported brevity cues, narrowed date windows, absent required strings, and unrequested output formats when they lack grounding in the user’s words.The rules also govern when earlier-reply templates and time-window constraints are retained.Source: B Filters, Audit and Guard · p. 11

Filters and audit

  • Information-need filters remove unsupported narrowing terms and date windows, and add an enumerate, count, or compute need when a list-like requirement lacks one.The rules also specify which findings reach repair and when quoted findings or unsupported claims are dropped.Source: B Filters, Audit and Guard · p. 11

Guard

  • The guard rejects repairs that are empty, overly short, add out-of-range years, drop required strings, leak labels, or violate constraints the draft already met.Other rejection checks cover word limits, sentence or bullet counts, and excessive loss of numbers and times.Source: B Filters, Audit and Guard · p. 11

C Baseline Implementation

The baseline implementation uses shared evaluation settings and describes implementation-specific adaptations for CCA, PSCI, and Ctx2Skill. The appendix’s result-table captions identify the cost and draft-versus-final measures reported, but provide no cell values.

Shared protocol

  • All methods receive the original conversation, use a frozen LLM with high reasoning effort, and are graded by the same GPT-5.1 judge and request builder.The shared protocol specifies retry handling and reports four of 8,100 runs ending without an answer.Source: C Baseline Implementation · p. 11

Baseline implementation

  • The CCA pipeline runs with unchanged prompts and correction logic, while PSCI uses its five prompt templates and specified demonstration, retry, and repair settings.For Ctx2Skill, the released code runs five self-play iterations with five probing tasks per context; on three LLMs it uses 2.4M–3.1M tokens per task.Source: C Baseline Implementation · p. 11

Reported comparisons

  • Table 6 reports API tokens and LLM calls per task, while Table 7 compares QERA’s draft with its final reply using paired intervals and repair counts.The captions define the reported measures but do not supply numerical table values.Source: Table 6 · p. 12, Table 7 · p. 12

D Cost Breakdown

QERA uses four sequential stages, with each LLM call rereading the conversation; input tokens dominate its per-task token use.

D Cost Breakdown

  • 65–79% of QERA’s tokens are input tokens because every call rereads 22k–25k input tokens.Tokens split across Ask (21–23%), Answer (22–27%), Check (29–35%), and Repair (17–23%).Source: D Cost Breakdown · p. 12
  • QERA’s sequential calls give it the latency of three or four calls of the same LLM, while code steps take milliseconds.The authors do not report wall-clock times; their runs were dominated by provider rate limits under high concurrency.Source: D Cost Breakdown · p. 12
  • Repair runs on 76–95% of tasks, while JSON repairs occur only on GLM-5, at 0.08 calls per task.Tokenizers differ, so token comparisons are meaningful only within an LLM.Source: D Cost Breakdown · p. 12

E Draft vs. Final

QERA’s draft alone solves more tasks than every baseline on every LLM, while checking and repair chiefly improve rubric pass rates for numeric requirements. The check and repair also have documented failure cases, including unsupported substitutions and garbled text.

E Draft vs. Final

  • The draft alone solves more tasks than every baseline on every LLM, with best-baseline solving rates of 22.5%, 25.9%, 14.1%, and 14.1%.It uses fewer tokens than CCA, PSCI, and Ctx2Skill counting its self-play; its rubric pass rate trails CCA’s only on Grok 4.7.Source: E Draft vs. Final · p. 12
  • Check and repair raise rubric pass rates for numbers by 0.9 points pooled across 2,513 rubrics, while other rubrics are unchanged.The reported intervals are +0.1 to +1.8 for numeric rubrics and −0.9 to +0.3 for other rubrics.Source: E Draft vs. Final · p. 12
  • The check can replace correct book titles with placeholders, the guard can miss garbled text, and a per-item length limit can be misread as a whole-reply limit.The reported examples include a Grok 4.7 task where correct titles were replaced and two Kimi K3 repairs that reduced passing rubrics to none.Source: E Draft vs. Final · p. 12

F Full Results

The paper defines explicit rubrics by how much of their content appears in the final request, and reports that QERA leads on most subcategories. Its explicit–implicit rubric split is based on a 60% content-word threshold.

F Full Results

  • A rubric is explicit when at least 60% of its content words occur in the final request; 981 rubrics are explicit and 4,352 are implicit.Fifteen rubrics with fewer than two content words are excluded.Source: F Full Results · p. 12
  • QERA has the highest solving rate in seven of nine subcategories, with two ties, and the highest rubric pass rate in seven.Table 5 gives the per-LLM tests of §4.2.Source: F Full Results · p. 12

Related papers

Loading 2610.35083v1…