Lost in the Request: How Communication Variation Disrupts Retrieval and Action in Email Agents

1 Introduction

• Most benchmarks use one canonical request per task, leaving whether wording changes affect completion of interactive work an open question.A fluent response can still omit a required action, retrieve incorrect evidence, or take an unsupported action.Source: §1 · p. 1
• A single end-to-end success score cannot distinguish failures such as missing a required tool call, ignoring tool results, or producing unsupported output.The paper therefore examines what the system stopped doing correctly, not only whether its performance changed.Source: §1 · pp. 1–2
• The paper evaluates email assistants in both retrieval-augmented answering and tool-using settings while preserving each request’s underlying task and expected outcome.It validates communication-style and English-variety variants to ensure they preserve the original intent.Source: §1 · p. 2
• The paper contributes a validated variant-generation framework, an evaluation across RAG and agentic settings, and an analysis of distinct assistant failure types.The analysis separates retrieval and answer errors from omitted required actions and unsupported actions.Source: §1 · p. 2

2 Evaluation Framework

• The framework tests whether RAG-based and agentic assistants fulfill the same task when only communication style or English variety changes.It examines both returned information and performed actions.Source: §2 · p. 2

2.1.1 Communication-Style Axes

• Five communication-style axes pair reference rewrites with contrastive poles, except formality, whose formal and casual poles are both contrastive.The axes cover directness, formality, verbosity, emotion, and grammaticality.Source: §2.1.1 · p. 2

2.1.2 Dialect Conditions

• Four dialect conditions apply documented features of Indian, Singapore, Black South African, and Nigerian English to synthetic variants.These variants are stress tests, not samples of how individual speakers write.Source: §2.1.2 · p. 2

2.2 Variant Construction

• Style variants use an LLM rewriter, while dialect variants use a rule-based transducer; candidates are admitted only after clearing applicable gates.The style process samples candidate framings for contrastive poles and selects among candidates that clear the gates; dialect transformations preserve masked operative literals.Source: §2.2 · pp. 3–4

2.3 Variant Validation

• Across each base request and its variants, the information need or action, evidence, named entities, numbers, dates, protected literals, and expected outcome remain fixed.Only wording may differ, and every candidate must clear all gates applicable to its pole.Source: §2.3 · p. 4
• Validation gates check factual preservation, intent, target style or dialect features, and human-likeness where applicable.Two authors agreed completely on intent (κ=1.00), with 82.5% agreement on style and 80.0% on human-likeness.Source: §2.3 · p. 4

3.1 Evaluation Tasks

• The study uses EnronQA for RAG, ToolTalk for multi-turn tool use, and WorkBench for sandboxed workplace tasks.EnronQA uses 400 questions in four isolated complexity bins; ToolTalk uses 50 hard-split conversations, and WorkBench uses 90 email-domain tasks.Source: §3.1 · p. 4

3.2 Models and Retrieval Systems

• Headline results use gpt-5.6-terra, and EnronQA retrieval compares BM25, SPLADE, and BGE-M3.SPLADE is the headline retriever because it provides the strongest baseline retrieval among the tested systems; the cross-model comparison excludes WorkBench because the full grid exceeded available compute.Source: §3.2 · pp. 4–5

3.3 Evaluation Metrics

• Each benchmark has its own success metric, including judged answer accuracy for EnronQA, conversation success for ToolTalk, and task accuracy for WorkBench.The metrics measure different quantities, so percentages and magnitudes should not be compared across benchmarks.Source: §3.3 · p. 5

3.4 Statistical Analysis

• EnronQA uses exact McNemar tests on paired binary outcomes; the agent benchmarks use paired percentile bootstraps over item-level differences.For the agent benchmarks, metrics are averaged over three runs per condition before testing, with 10,000 bootstrap resamples and 95% intervals.Source: §3.4 · p. 5

4.1 Effects of Style and Dialect Variation

• Indirect phrasing costs −3.2 points of EnronQA answer accuracy, −22.4 points of ToolTalk conversation success, and −10.7 points of WorkBench task accuracy.It is the only condition that degrades all three benchmarks.Source: §4.1 · p. 5, Table 2 · p. 6
• Formal requests significantly reduce ToolTalk and WorkBench performance, while all four dialect varieties harm EnronQA and dialect effects vary by variety.Formality is null on EnronQA; Singapore is significant on all three benchmarks, while Indian is significant on two.Source: §4.1 · p. 5
• Verbosity and emotional, non-grammatical, and reference-pole effects are selective rather than broadly consistent across benchmarks.Verbosity is significant only on EnronQA, emotionally charged phrasing only on WorkBench, and non-grammatical phrasing only on ToolTalk; only one of 12 reference cells is significant.Source: §4.1 · pp. 5–6

4.2.2 Agentic Setting

• On ToolTalk, required-call recall falls −11.28 points under indirect and −10.32 under formal, while incorrect-action rates do not significantly rise; WorkBench shows a similar omission pattern.For formal requests on WorkBench, tasks with no tool calls rise from 0.4% to 12.2%, while unwanted side effects fall from 8.9% to 4.8%.Source: Figure 3 · p. 7, §4.2.1 · p. 7, §4.2.2 · p. 7

4.2.1 RAG Setting

• Under BM25, verbose requests cost −26.0 points of accuracy and −28.0 Recall@5; under SPLADE, verbose recall changes −0.2 (p=.89) and accuracy falls −3.0.With BM25, Recall@5 falls from 95.7% in the least-padded quartile to 44.6% in the most-padded quartile.Source: §4.2.1 · p. 7
• Under SPLADE, indirect phrasing costs −3.2 points of accuracy against −1.8 recall, and pooled dialect costs −3.0 accuracy against −0.9 recall.Both variations remain harmful when retrieval changes little, indicating that the loss is downstream of retrieval.Source: §4.2.1 · p. 7
• Among 118 retrieved-email cases with wrong answers, unsupported content was the largest error category, and rewrite-induced errors more often involved misreading than refusal.Contrastive rewrites broke 4.26% of previously correct 3-hop slots and 4.08% of multi-question slots, versus 1.62% for simple controls.Source: §4.2.1 · p. 7

4.3 Generality Across Models and Systems

• Across three models and two systems, indirect phrasing is negative in all six model-by-system cells and significant in five; formal and non-grammatical phrasing are each significant in two cells.Both formal and non-grammatical effects are significant in two cells on the agentic benchmark.Source: §4.3 · p. 8, Table 3 · p. 8

5.1 Request Explicitness Affects Performance

• The indirect pole retains the request’s head verb as its main predicate least often on all three benchmarks, from 12.5% of slots on EnronQA to 4.7% on WorkBench.The formal pole is also among the bottom three poles on all three benchmarks, often placing the retained head verb in a subordinate clause.Source: §5.1 · p. 8

5.2 Incomplete Fulfillment as an Agentic Failure Mode

• In both agentic settings, indirect and formal requests mainly reduce completion of required work without a corresponding increase in unsupported actions.The discussion distinguishes omission from taking an unwanted action as separate failures within end-to-end success.Source: §5.2 · p. 8
• In WorkBench, formal requests raise tasks with no tool call from 0.4% to 12.2%, while indirect requests reduce call F1 by 8.42 points.The formal change is close to the −12.2-point accuracy loss; under indirect wording, the agent acts but its calls land in the wrong state.Source: §5.2 · p. 8

5.3 Dialect Robustness and Fair Access

• Dialect variation reduces performance on all three benchmarks, but the dialect conditions use feature density 1.0, a deliberate worst case.The transformations apply grammatical features documented for each variety; the authors suggest that variation beyond isolated grammatical slips carries part of the effect.Source: §5.3 · p. 9

5.4 Implications for Evaluation and System Design

• The discussion argues that evaluations should use more than one canonical wording because request variation also changes whether agents carry out the requested work.This extends the evaluation concern from retrieval and RAG to the agentic setting.Source: §5.4 · p. 9
• Stage-aware evaluation should precede mitigation: replacing BM25 with SPLADE removes most verbosity penalty, while indirect and dialect penalties persist with gold evidence retrieved.The model with the highest baseline on both systems where all three models were run also changes significantly in the most conditions.Source: §5.4 · p. 9
• Completeness and unsupported action should be reported separately; trajectory measures can expose incomplete work without a gold final answer, but do not replace final grading.The cited measures include required-call recall, failure to act, and read-without-write behavior.Source: §5.4 · p. 9

6 Conclusion

• Indirect requests most consistently reduce performance; verbosity mainly disrupts lexical retrieval, dialect effects persist after gold-email retrieval, and indirect or formal requests mainly reduce required-action completion.The conclusion recommends varying request formulation, inspecting intermediate retrieval and execution signals, and reporting required-action completion separately from unsupported action.Source: §6 · p. 9

Limitations

• Dialect variants use feature density 1.0 as a deliberate worst case, not an estimate of typical use.Other limitations include lower coverage for WorkBench indirect requests, an imperfect ToolTalk reference-pole control, qualitative cross-benchmark comparisons, incomplete cross-model coverage, and no tested mitigation.Source: Limitations · p. 9

A Communication-Style Axes in Detail

• Directness contrasts explicit questions or commands with requests conveyed through a need, hint, observation, or uncertainty.The contrast follows the on-record/off-record distinction.Source: A Communication-Style Axes in Detail · p. 12
• Formality contrasts a polished professional register with a casual, chat-like one.The paper treats neither register as neutral in its reference poles.Source: A Communication-Style Axes in Detail · p. 12
• Verbosity contrasts a concise request with the same request embedded in generic chatter, hedging, or digression that adds no task facts.The added material introduces no new evidence or expected-answer information.Source: A Communication-Style Axes in Detail · p. 12
• Emotion contrasts neutral wording with wording that expresses frustration, stress, anxiety, or urgency while holding task, evidence, and expected outcome fixed.The axis tests whether emotional wording changes model behavior on the otherwise fixed task.Source: A Communication-Style Axes in Detail · p. 12
• Grammaticality contrasts standard written English with understandable English carrying systematic grammatical deviations, including restructuring and slips in articles, number, tense, or agreement.The pole is defined by text properties, not writer attributes.Source: A Communication-Style Axes in Detail · p. 12

B EnronQA Sampling Design

• EnronQA questions were classified by reasoning depth, multiplicity, context dependency, and reasoning type, then binned so each complex bin varied only one hard property.The simple control was at most two hops, single-question, and self-contained; each other bin raised only depth, multiplicity, or context dependency.Source: B EnronQA Sampling Design · p. 12
• The sample contains 400 questions from 400 emails and 99 users, with 100 questions per bin, selected from eligible EnronQA training data using stratification and screening.Selection restricted users to those with at least 25 emails, deduplicated questions, limited each email to one question, stratified by reasoning type, and applied near-duplicate and refusal filters.Source: B EnronQA Sampling Design · pp. 12–13

C Verbalized Sampling

• To increase phrasing diversity, verbalized sampling generated six framed rewrites per call, then selected a valid candidate with the least-used opening words and seeded tie-breaking.Framing rotation was deterministic by item identifier, making runs reproducible and order-independent; failed candidate sets triggered retries.Source: C Verbalized Sampling · p. 13

D.2 Indirectness Mechanisms

• Indirect prompts preserved the exact information need while implying the ask through varied mechanisms, including uncertainty, unmet preconditions, motivation, and understatement.The prompt required a specific, answerable request without new facts; the menu also included preparatory knowledge, associative hints, presupposition, and other indirect framings.Source: §D.1 · p. 13, §D.2 · p. 14

D.3 System Prompt: formal

• Formal prompts changed only register, requiring natural, courteous professional language, complete sentences, and retention of every specific detail.The framing menu varied the professional approach, from direct courteous requests to clarification and verification requests.Source: §D.3 · p. 14, §D.4 · p. 14

D.5 System Prompt: casual

• Casual prompts required lowercase, chat-like rewrites with informal wording while preserving the ask and real identifying details.The framings ranged from quick Slack-style pings to rushed or friendly one-liners, sometimes dropping nonessential email-attribution wording.Source: §D.5 · pp. 14–15, §D.6 · p. 15

D.7 System Prompt: verbose

• Verbose prompts lengthened and rambled around the request, burying the unchanged ask in the middle without adding facts about the email.Framings used unrelated asides, tangents, backtracking, or narration of searching for the detail.Source: §D.7 · p. 15, §D.8 · p. 15

D.9 System Prompt: charged

• Charged prompts infused the request with a stressed persona, while non-grammatical prompts required visibly non-native sentence structure and systematic learner errors.The charged prompt treated stress as the writer’s state rather than a new email fact; the learner prompt prioritized structural changes over surface-level mistakes.Source: §D.9 · p. 16, §D.11 · p. 16

D.11 System Prompt: non-grammatical; D.12 Non-Grammatical Structural Profiles

• The non-grammatical prompts require readable plain English, preserve the exact ask and specific details, and prohibit cultural markers or appended awkward sentences.The structural profiles cover six transformations: topic-fronting, question-inversion, word-order and tense shifts, clause reordering, fragments or run-ons, and spelling combined with sentence rebuilding.Source: §D.11 · pp. 16–17, §D.12 · p. 17

D.13 Reference-Pole Rewrite System Prompt; D.14 Reference-Pole Rewrite Instructions

• Reference-pole rewrites must preserve intent and details without adding requests or facts; the four poles use a single plain rewrite, unlike the contrastive poles’ pole-specific prompts.Instructions specify explicit, terse, calm, and fluent native-English rewrites.Source: §D.13 · p. 17, §D.14 · p. 17

D.15 Gate A: Intent Preservation

• Gate A judges whether a variant requests exactly the same information as the original, ignoring differences in tone, politeness, directness, length, and wording.It returns a one-line yes-or-no verdict on whether the two questions request the same information.Source: §D.15 · p. 18

D.16 Gate B: Style Realization

• Gate B checks only whether a rewritten question exhibits its specified communication style, not whether its meaning or content is correct.It returns a one-line yes-or-no verdict on style realization.Source: §D.16 · p. 18

D.17 Fidelity Criteria for All Ten Poles

• The fidelity criteria operationalize the ten poles with definitions distinguishing direct from indirect requests, formal from casual register, and fluent native from non-native English.The criteria also define poles for brevity versus buried, lengthy asks and calm versus emotionally charged tone.Source: §D.17 · p. 18

D.18 Gate C: Human-Likeness

• Gate C assesses whether a question reads as genuinely human rather than artificial, without treating length, detail, formality, or learner-like errors alone as evidence of artificiality.It focuses on authenticity and naturalness, not fluency, correctness, or politeness.Source: §D.18 · pp. 18–19

D.19 Verbalized-Sampling User Template

• The verbalized-sampling template requests exactly k distinct rewrites that retain the same single answer and every specific detail, returned only as a JSON array.It asks for varied openings and sentence shapes and forbids prefixes such as style names or numbers.Source: §D.19 · p. 19

E Validation Gates

• Deterministic checks reject empty or unchanged variants and those missing an original numeric token; missing capitalized tokens are recorded but do not reject.Names may legitimately be rephrased, so missing capitalized tokens are only a soft signal.Source: E Validation Gates · p. 19
• Rubric Gates A, B, and C check intent preservation, target-style realization, and human-likeness, respectively.Gate C is skipped for verbose variants and recorded without filtering for non-grammatical variants, avoiding selection against the properties under test.Source: E Validation Gates · p. 19
• WorkBench indirect admits 64 of 90 variants, with losses concentrated at Gate A; yield exceeds 80% in 29 of 30 dataset-pole cells.WorkBench instructions identify a specific record to modify, and indirect rewrites may omit that identifier along with the operation.Source: E Validation Gates · p. 19
• Table 5 reports per-pole gate-passing rates by dataset, with A for intent, B for style, and C for human-likeness.It reports slots per pole and distinguishes reference from contrastive poles; Gate C is not applied to reference poles or verbose variants.Source: Table 5 · p. 20

F EnronQA Error Labeling

• The analysis identifies 118 generation-error slots where the gold email was retrieved but the answer was wrong, and 237 retrieval failures where it never reached the reader.The analysis covers the headline condition: gpt-5.6-terra with SPLADE, judged by gpt-5.4.Source: F EnronQA Error Labeling · p. 20
• Generation errors receive one precedence-assigned code, including wrong value, wrong role or event, wrong semantic type, incompleteness, and false abstention; retrieval failures use three separate codes.E6 covers defective gold answers or judge strictness; E3 covers an answer drawn from another retrieved email.Source: F EnronQA Error Labeling · p. 20
• Claude-opus-5 assigned the labels, then one author reviewed all 118 and recoded 23 whose rationale did not support the assigned code.Source: F EnronQA Error Labeling · p. 20
• Unsupported content is the largest generation-error category in both the style-induced slice and the pooled persistent-error and baseline slice, at 43.2% and 47.3%.Style-induced errors are items correct under the unperturbed request but broken by the rewrite; persistent errors were already wrong at baseline.Source: F EnronQA Error Labeling · p. 20
• Among style-induced errors, the reader reports the fact absent in 55 of 237 retrieval failures (23.2%), hedges in 6 (2.5%), and answers confidently anyway in 176 (74.3%).This behavior is the same whether the item was never retrievable at baseline or the rewrite pushed the gold email out of the top five.Source: F EnronQA Error Labeling · p. 20

G Estimators

• The paired difference is the variant metric minus the base metric for each item present in both conditions.Source: G Estimators · p. 21
• For replicated agent benchmarks, the paired bootstrap resamples item indices with replacement to test the mean paired difference.It uses 10,000 resamples with a fixed seed; the 95% interval uses the 2.5th and 97.5th percentiles, and the two-sided p-value is based on resampled means on either side of zero.Source: G Estimators · p. 21
• For EnronQA, the exact McNemar test uses discordant pairs: b items correct at baseline but wrong under the variant, and c with the reverse outcome.Each item has one binary outcome from a single run, so paired differences are −1, 0, or +1.Source: G Estimators · p. 21
• McNemar conditions on b + c and is exact at any sample size; across the EnronQA grid, b + c has a median of 18 and falls below 10 in 20 of 114 cells.The paper notes that a bootstrap over all items can be anti-conservative when only a handful of same-signed events occur.Source: G Estimators · p. 21

H Compute Resources

• Evaluations use inference-time API calls, with retrieval experiments run locally on a 64 GB RAM machine and no model training or finetuning.Source: H Compute Resources · p. 21
• All 14 conditions are measured in every reported benchmark–model cell, but WorkBench is reported only for the headline model.The cross-model comparison therefore covers the two benchmarks where all three models were measured.Source: Table 6 · p. 21
• Table 7 pairs each headline-model result with item counts, matched baselines, condition values, intervals, and raw p-values; Tables 8–9 organize error and fragility profiles.Source: Table 7 · p. 22, Table 8 · p. 23, Table 9 · p. 23
• Tables 10–15 describe accuracy, retrieval, dialect, verb-retention, verbosity, and intermediate-signal analyses across their stated conditions.Source: Table 10 · p. 24, Table 11 · p. 25, Table 12 · p. 25

NeurIPS Paper Checklist

• The checklist states that the abstract and introduction reflect the paper’s claims and identifies limitations including incomplete benchmark coverage and absent mitigation experiments.Source: NeurIPS Paper Checklist · p. 28

• The limitations section identifies single-density dialect generation, incomplete benchmark coverage, imperfect reference controls, compute constraints, and no mitigation experiments.Source: §2 · p. 29
• Dialect variants use a single feature density of 1.0, leaving other feature densities outside the evaluated setting.Source: §2 · p. 29
• The submission combines public benchmarks with proprietary model APIs and includes no anonymized code or artifacts.Source: §2 · p. 30
• The checklist says the paper specifies benchmark, model, retrieval, prompting, validation, metric, and statistical-testing details, with implementation details in the experiment section and appendix.Source: §2 · p. 31