PaperlayerView exact revision on arXiv

Source-linked AI summary

WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation

Amey Varhade, Ananya Sutradhar, Ravishankar Krishnaswamy, Navin Goyal

arXiv:2609.12171v1cs.AI

TL;DR

Enterprise QA often requires reconstructing evolving state from evidence scattered across artifacts, while existing benchmarks may not capture the complexity of workplace settings. WINSYN generates grounded synthetic enterprise email datasets from seed scenarios and evaluates agentic systems on them. The evaluated systems continue to struggle, especially with synthesis across long-running projects.

  • Problem

    Enterprise QA requires reconstructing evolving organizational state from partial, distributed, and sometimes conflicting artifacts, while existing benchmarks often have limited real-world complexity.

    Source: §1 · pp. 1–2

  • Method

    WINSYN expands compact seed scenarios into a structured scaffold and generates grounded communication artifacts and question–answer pairs for simulated enterprise projects.

    Source: §1 · p. 2

  • Results

    Current agentic systems struggle particularly with questions requiring evidence synthesis across long-running projects rather than retrieval of isolated facts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Temporally structured synthetic enterprise datasets can serve as stress tests for enterprise search and question-answering systems.

    Source: §1 · p. 2

    The dataset scope is limited to email, and its statistical similarity to actual enterprise data has not been measured.

    Source: §5 · p. 9

Abstract

from arXiv · show

Enterprise settings provide a challenging environment for question-answering agents, which often rely on Retrieval-Augmented Generation, Deep Research (DR), and related techniques. Much of this challenge comes from the complexity of enterprise data: information is often spread across evolving and potentially conflict- ing emails, chat messages, documents, and other artifacts. Existing benchmarks typically have limited real-world complexity, short-form responses, and unnatural queries, so they often fail to capture the challenges of enterprise settings. In this work, we introduce an automated pipeline for generating synthetic datasets of emails reflecting realistic workplace scenarios, along with long- and short-form questions and gold answers grounded in the data. Our method simulates long-running enterprise projects spanning several months and involving up to 25 interacting employees across multiple roles. The data emphasizes ambiguity, distributed information, and naturally occurring queries. To validate the pipeline, we evaluate few standard agentic baselines on our datasets using the latest frontier models. We find that aggregate scores averaged over all queries remain below 80% for each dataset, indicating significant room for improvement. These findings suggest that more work remains to be done for enterprise deployment and underscore the importance of realistic, high-complexity evaluation data for developing stronger real-world enterprise DR systems.

1 Introduction

Enterprise QA requires reconstructing evolving organizational state from evidence distributed across artifacts, while realistic synthetic enterprise data must preserve temporal, causal, and organizational structure. WINSYN generates grounded enterprise QA datasets from seed scenarios and evaluates agentic systems, which continue to struggle with cross-project synthesis.

1 Introduction

  • Enterprise answers may depend on decisions introduced, revised, justified, and reflected across different artifacts, requiring reconstruction of evolving organizational state.Evidence can be partial, distributed, and sometimes conflicting, so retrieving one relevant passage is insufficient.Source: §1 · p. 1
  • Realistic synthetic enterprise data must preserve temporal consistency, causal dependencies, role-specific communication, implicit context, and auditable grounding.Useful synthetic data is challenging to generate, and existing benchmarks trade off realism, scalability, domain flexibility, and answer verifiability.Source: §1 · p. 2
  • WINSYN expands compact seed scenarios into a hidden structured scaffold, then generates communication artifacts and grounded question–answer pairs from it.The scaffold is separated from the evidence evaluated systems see, testing whether they can recover relevant structure from fragmented workplace artifacts.Source: §1 · p. 2
  • The four email datasets cover multi-month projects with more than twenty employees and include grounded questions, reference answers, and citation sets.The pipeline is designed for extension to chats, meetings, documents, tickets, and code repositories, though the paper instantiates it with email corpora.Source: §1 · p. 2
  • Current systems struggle particularly when questions require synthesizing evidence across long-running projects rather than retrieving isolated facts.The evaluation covers representative agentic RAG and Deep Research baselines using retrieval and answer-quality metrics.Source: §1 · p. 2

2 WINSYN: Automated Pipeline for Enterprise Data Generation

Enterprise data is a distributed, evolving discourse in which answering questions can require connecting evidence across people, time, and artifacts. WINSYN builds such environments through staged expansion from a seed, using structured project dependencies and grounded intermediate records to generate emails and QA.

2.1 Enterprise data.

  • Answering questions such as why a decision was made or who knew what and when requires linking evolving, sometimes conflicting evidence across artifacts.The section frames enterprise data as a distributed, multi-agent, temporally extended discourse rather than a collection of documents.Source: §2.1 · p. 3
  • Enterprise data combines causal event chains with implicit relationships, shared context, communication norms, and artifacts that do not all survive equally.Its realism also spans word and response-time distributions, pragmatic meaning, and traces left by ephemeral communication.Source: §2.1 · p. 3

2.2 Our pipeline

  • WINSYN addresses the challenge of automatically constructing richly structured enterprise datasets, avoiding unconstrained employee-agent simulation because coherence and answer grounding are difficult to ensure.The pipeline is designed to handle realism, consistency, context-size limits, instruction-following errors, and hallucinations.Source: §2.2 · p. 4
  • The paper focuses on email and emphasizes causal structure and consistency, without explicitly accounting for statistical or pragmatic realism.The approach may extend to other modalities, but adding them requires more pipeline stages.Source: §2.2 · p. 4
  • The pipeline progressively expands a seed narrative through company and employee information, dependent Epic and Task DAGs, and lower-level artifacts.Epics and Tasks encode dependencies and handoffs; the hierarchical structure lets generation use pertinent context rather than the whole prior dataset.Source: §2.2 · p. 4, Figure 1 · p. 5
  • Daily diaries record task-level activity, progress, blockers, resolutions, and collaboration, serving as the concise source of truth for generated artifacts and answers.The diaries are expanded from task descriptions across each task’s timeline.Source: §2.2 · p. 5
  • Questions and answers are generated with controlled context, while emails add lower-level detail without changing the answers and leave evaluated agents without the pipeline’s underlying structure.Email attribution, distractors, validation checks, and reflection–refinement loops are also part of the pipeline.Source: §2.2 · p. 5

3 Related Work

Related work spans synthetic enterprise benchmarks and AI-generated story frameworks. WINSYN’s design is closer to planned, ground-truth-oriented story construction, while HERB and DRBench offer related but differently constrained enterprise settings.

Story writing

  • WINSYN’s planned approach resembles “plotters,” prioritizing coherence and firm ground truth; a hybrid with agent simulation may improve realism, but is left for future work.The paper says fiction-generation frameworks are not directly adaptable to synthetic enterprise data.Source: §3 · p. 6

Existing benchmarks

  • Related benchmarks span information retrieval, RAG, and deep research, including CRMArena-Pro and DataMorgana.The paper also names HotpotQA, OfficeQA Pro, DRACO, APEX-Agents, PRBench, and TheAgentCompany.Source: §3 · p. 6

Synthetic enterprise benchmarks

  • HERB [6] covers 39,190 synthetic artifacts across 30 software products, with traceable, multi-hop queries, but its software-lifecycle templates require expert redesign for other domains.Source: §3 · p. 6
  • DRBench [1] evaluates open-ended enterprise research across private files and the public web, but its enterprise files are static and non-conflicting, and the benchmark is not designed for new contexts.Its core challenge is multi-source internal-data reasoning rather than web retrieval.Source: §3 · p. 6

4 Experiments

Experiments evaluate two agentic baselines across four workplace datasets using judge-based and retrieval metrics. ReAct has higher final scores across datasets, while performance and metric trade-offs vary by query form.

4.1 Datasets

  • The four datasets simulate distinct teams and projects, using synthetic newsletters for two domains and public Stripe and VS Code technical documents for two others.Source: §4.1 · p. 6

4.2 Evaluations

  • The evaluation uses a hybrid BM25+dense ReAct baseline and a sandboxed Onyx Deep Research agent restricted to each question’s visible email corpus.Onyx plans sub-questions, then runs up to ten act–observe–reflect cycles with local tools; gpt-5.4 is the underlying model and judge.Source: §4.2 · p. 7, §4.3 · p. 7

4.3 Evaluation Metrics

  • Six judge metrics score answer completeness, soundness, relevance, faithfulness, readability, and retrieval accuracy; the weighted final score emphasizes completeness and soundness.The composite assigns 0.30 to completeness and soundness each, 0.15 to relevance and faithfulness each, and 0.10 to readability; separate retrieval metrics include MRR, Hit Rate@k, Recall@k, and Citation Coverage Recall.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 Results

  • Final scores are within 0.11 across both systems on all four datasets, with ReAct higher on each; Onyx often scores higher in relevance and soundness.MRR remains modest overall, though Onyx has stronger MRR on some datasets.Source: Table 3 · p. 8, §4.4 · p. 8
  • Long-form queries score below 2.8 holistically for both systems; ReAct’s final score is 0.717 vs. 0.679, with stronger completeness and retrieval accuracy.These queries require synthesizing information across many documents into a coherent narrative.Source: §4.4 · p. 9, Table 4 · p. 9
  • Short-form final scores converge at 0.765 vs. 0.766, while ReAct has higher completeness (0.665 vs. 0.601) and Onyx higher relevance (0.680 vs. 0.571).Short-form queries target specific facts or events.Source: §4.4 · p. 9, Table 4 · p. 9

5 Conclusion and limitations

WINSYN builds synthetic workplace corpora through staged, validated generation and finds that strong agentic baselines still struggle with retrieval, reasoning, and grounding. Its current scope is email, with realism measurement and larger, longer simulations left open.

5 Conclusion and limitations

  • Staged generation, reflection–refinement loops, and validators produce coherent, temporally consistent corpora, while agentic baselines still struggle with retrieval, reasoning, and grounding.Source: §5 · p. 9
  • The benchmark covers emails, lacks measured statistical realism and external interactions, and leaves larger organizations and longer time spans to future work.Source: §5 · pp. 9–10

A Pipeline details

  • A hierarchical pipeline expands a seed into workplace structures, diaries, communications, and QA, using reflection, validation, and refinement across its stages.Source: A Pipeline details · p. 13

A.1 Overview and Notation

  • Generation proceeds through company context, employee hierarchy, project and task graphs, diaries, QA, emails, and sentence-level attribution; gold answers are frozen before emails.Source: §A.1 · p. 13

A.2.1 Company, Employees and Projects

  • The company stage creates a roster of 20–25 employees with roles, responsibilities, skills, and a reporting hierarchy that guides later assignments.Source: §A.2.1 · pp. 13–14

A.2.2 Hierarchical Epic Generation

  • Epics form a precedence DAG generated in opening, intermediate execution, and closing stages, with the final stage conditioned on preceding epics.Source: §A.2.2 · p. 14

A.2.3 Task Decomposition

The pipeline decomposes projects into dependent tasks and daily records, then creates diary-grounded questions before generating communications constrained by their answers. Attribution, distractors, and layered validators support traceability and consistency.

A.2.3 Task Decomposition

  • Each epic becomes a task DAG with scoped descriptions, employee assignments, timelines, dependencies, expected outputs, and downstream unlocks.Source: §A.2.3 · p. 14

A.2.4 Daily Diary Synthesis

  • Daily diaries record work, progress, blockers, resolutions, and collaboration, with hand-off context supplied to keep downstream artifacts consistent with prior days.Source: §A.2.4 · pp. 14–15

A.2.5 Question and Answer Generation

  • QA is created from diary evidence before emails; queries include briefings, project reports, and cross-task synthesis, with candidate answers checked for entailment and grounding.Source: §A.2.5 · pp. 15–16

A.2.6 Communication Synthesis with Locked QA

  • Emails are generated last as chronological task threads, conditioned on locked QA so they support prior answers without leaking future information.Source: §A.2.6 · p. 17

A.2.7 Sentence Level Attribution

  • Attribution maps answer sentences to substantive diary evidence, then diary sentences to supporting communications, preserving provenance without direct corpus-wide matching.Source: §A.2.7 · p. 17

A.2.8 Distractor Generation

  • Post-QA distractors add irrelevant noise, corrected misstatements, or tangential-project messages; validation checks that they do not change gold answers.Source: §A.2.8 · pp. 17–18

A.3 Reflection-Refinement Loops

  • Each artifact stage uses reflection, rubric-based critique, and deterministic validation, with separate critic and generator prompts intended to reduce self-reinforcing errors.Source: §A.3 · p. 18

A.3.1 Validators and Rubrics

  • Rubrics check seed-span coverage, graph acyclicity and timelines, task hand-offs, employee assignments, and project closure across epic and task levels.Source: §A.3.1 · p. 18

A.4 Validation and Quality Assurance Summary

The validation framework combines programmatic checks with iterative LLM reflection to keep generated artifacts complete, consistent, and grounded. It applies across generation stages and provides traceability to source material.

A.4 Validation and Quality Assurance Summary

  • Validation spans seed-document coverage, graph refinement, answer generation, and attribution, supporting syntactic and semantic consistency.Source: §A.4 · p. 19
  • Coverage checks and grounding mechanisms, including claim verification and sentence-level attribution, support complete artifacts and traceability to source documents.Source: §A.4 · p. 19
  • Table 6 presents validation mechanisms as layered across the pipeline; the framework combines programmatic checks with iterative LLM-based reflection.Source: Table 6 · p. 19, §A.4 · p. 19

B Additional Experiments and Results

Additional experiments examine query-type performance, retrieval configurations, model-family variation, and generation cost. The supplied passages describe evaluation scope and setup, but do not provide the tables’ cell values.

B Additional Experiments and Results

  • ReAct retrieval combines BM25 and dense vectors, while Onyx mirrors its Vespa hybrid profile, using different chunk sizes and top-k settings.ReAct retrieves up to 60 chunks of up to 256 tokens; Onyx uses 1,600-character email chunks and top-k 50.Source: B Additional Experiments and Results · p. 19
  • Table 7 breaks down answer-quality, retrieval, and ranking metrics by query type, grouping long-form synthesis tasks separately from short-form factual retrieval.The breakdown includes MRR, Recall, and HR@25; the supplied passages do not state individual table values.Source: B Additional Experiments and Results · p. 19, Table 7 · p. 20
  • Experiments using Grok 4.1 Fast as agent and Kimi K2.6 as judge show similar or greater answering difficulty across distractor and non-distractor variants.The additional experiments use different model families for the agent and evaluator.Source: B Additional Experiments and Results · p. 20
  • Generation cost grows approximately quadratically with the number of employees, epics, and tasks, making larger datasets prohibitively expensive.Cost scales linearly with the project timeline.Source: B Additional Experiments and Results · p. 20

C Dataset Qualitative Analysis

The example shows why a workstream status answer must reconstruct a protocol across artifacts and distinguish its locked plan from what has been implemented and what remains open. The gold answer combines those layers while preserving the protocol’s safety constraints.

C Dataset Qualitative Analysis

Q3 Multi Domain Catch Up; 2025-07-28

  • The workstream remains in orchestration: the planned sequence is locked, while the implemented coordinator has only a first-pass lifecycle model.The plan includes recomputation, old-definition live processing, buffering, patching, replay, cutover, and post-cutover cleanup; the lifecycle distinguishes recomputation as submitted versus running.Source: Q3 Multi Domain Catch Up; 2025-07-28 · p. 21

Citations: Orchestration Update; 2025-07-25)

D Dataset Examples

This appendix presents prompts used in the generation pipeline and evaluation prompts that map specific inputs to structured scores or retrieval metrics. The examples document the prompt and output formats for several evaluation dimensions.

D.1 Prompts

  • The appendix introduces the actual prompts used at various pipeline stages.The supplied passage does not describe their contents.Source: §D.1 · p. 23

D.1.1 Data Generation Prompts

  • A figure presents a prompt used to generate Epics.The supplied caption does not provide the prompt text.Source: Figure 6 · p. 24

D.2 Evaluation Prompts

  • Evaluation prompts enforce explicit reasoning steps and a JSON output schema for consistency and interpretability.They evaluate system outputs across multiple dimensions.Source: §D.2 · p. 28

D.2.1 Completeness / D.2.2 Soundness

  • Completeness and soundness prompts take a question, reference answer, and candidate answer, and return a score in [0, 1].Source: §D.2.1 · p. 28, §D.2.2 · p. 29

D.2.4 Readability

  • The readability prompt takes a question and candidate answer and returns a readability score on a 0–5 scale.Source: §D.2.4 · p. 30

D.2.5 Retrieval Accuracy

  • The retrieval-accuracy prompt takes a question, retrieved chunks, and gold attribution documents, and returns chunk precision and document recall.Source: §D.2.5 · p. 31

D.2.6 Faithfulness

  • The faithfulness prompt scores a candidate answer against the question and retrieved chunks on a [0, 1] scale.Source: §D.2.6 · p. 32

D.2.7 Overall Score

  • The overall-score prompt uses the question, reference and candidate answers, retrieved chunks, gold documents, and sub-scores to return an integer score on a 0–5 scale.Source: §D.2.7 · p. 33

D.3 Synthetic Data Output

The appendix introduces generated synthetic data and shows an example Epic titled “Agent Sessions, Session Search, and Plan-First Workflow Implementation.”

D.3 Synthetic Data Output

  • The appendix section contains actual data generated by the pipeline.Source: §D.3 · p. 34

Epic

Task

The task specifies a centralized Agent Sessions view and its default layout and placement behavior. The diary describes the initial implementation and a placement ambiguity encountered while wiring its setting.

Task

  • The centralized Agent Sessions view is the primary management surface for active sessions, with separate local and background sections and configurable placement beside Chat.The view is enabled by default; the optional single-view location is in the Secondary Side Bar.Source: Task · p. 35
  • The task assigns Elena Morozova and Maya Chen to work scheduled for August 22–25, 2025.Source: Task · p. 35

Daily Diary

  • Elena Morozova and Maya Chen began implementation from locked T1 decisions, establishing the baseline view structure, default behavior, and placement-setting model.Source: Daily Diary · p. 36
  • While wiring the placement setting, Elena found that the decision log defined the optional consolidated layout as adjacent to Chat in the Secondary Side Bar.Source: Daily Diary · p. 36

D.3.2 Final Dataset

The examples show how workplace artifacts capture implementation decisions, daily priorities, project status, and cross-person ownership. They pair realistic questions with answers grounded in distributed project details.

Email Thread

  • The email thread establishes the sectioned local-and-background layout as the default, while tying optional adjacency to Chat to a setting and tracking consolidated-layout differences.Source: Email Thread · p. 38

Example of Daily Briefing Question and Answer

Example of Project Status Question and Answer

Example of Multi Hop People Question and Answer

Related papers

Loading 2609.12171v1…