MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records

1 Introduction

• Earlier records remain unchanged as information is shared and corrected, creating cross-source disagreements and queries that require recovering a specific historical state.Reliable answers must use both the correct information across modalities and its appropriate temporal state.Source: §1 · p. 1
• MMEnterprise-Bench introduces 1,613 QA pairs over evolving multimodal workplace records, including questions that explicitly target revision histories.It evaluates Evidence Reasoning, Conflict Resolution, and Unanswerability across eight subtasks.Source: §1 · pp. 2–3, Table 1 · p. 2
• The benchmark simulates 30 collaboration projects and retains superseded records, preserving inconsistent evidence across sources and multiple temporal states.Its retrieval corpus contains 7,140 textual and visual records.Source: §1 · p. 2
• 52.6% accuracy with 10 retrieved records and 68.0% with 50; GPT-5.6 Sol also reaches only 27.5% on multiple-image questions versus 83.3% on single-text questions.Conflict-resolution accuracy ranges from 11.8% to 36.3% on cross-source conflicts and from 13.2% to 33.5% on temporal conflicts.Source: §1 · p. 2

Multimodal models for document understanding

• Document-understanding benchmarks extend from visual QA to multi-page and long-document evidence integration, with multimodal LLMs and agentic pipelines addressing text-rich images and crossmodal aggregation.The cited examples include scanned documents, infographics, charts, slides, tables, and figures.Source: §2 · p. 3

Enterprise question answering benchmarks

• Enterprise QA benchmarks simulate workplace corpora, but are largely text-only and rarely treat earlier states of revised records as valid answers.EnterpriseRAG-Bench accepts only up-to-date values, while WinSyn uses correction-chain distractors without changing gold answers.Source: §2 · p. 3
• Temporal knowledge benchmarks primarily target public knowledge, personal conversations, or formally versioned documents and are limited to text, unlike MMEnterprise-Bench’s multimodal workplace revisions.MMEnterprise-Bench evaluates cross-source and temporal conflicts while preserving earlier states that may be valid answers.Source: §2 · p. 3

3.1 Enterprise Workspace Simulation

• The workspace aggregates records from projects grounded in seed documents, with collaborative events planned as actions and realized as records.Projects define the simulated collaboration; the shared workspace is the retrieval corpus across projects.Source: §3.1 · pp. 3–4
• Earlier records and document versions are retained rather than overwritten, preserving historical states and the progression of changes over time.Teams, roles, channels, access relationships, and deadlines constrain who performs events, when, and where records appear.Source: §3.1 · p. 4

3.2 QA Annotation

• Questions cover Evidence Reasoning, Conflict Resolution, and Unanswerability, with tasks spanning extraction, attribution, computation, comparison, interpretation, cross-source resolution, temporal recovery, and insufficient evidence.Rule-based and LLM-based generation are used; unanswerability examples are created by perturbing answerable questions and retaining those unanswerable from the complete project records.Source: §3.2 · pp. 4–5
• Answerable questions have reference answers and gold evidence records, annotated for evidence count, source types, and modality composition.The evaluation retrieves individual records from the shared corpus and provides them as context for answering.Source: §3.2 · p. 5

3.3 Quality Control and Human Validation

• None of 50 LLM-accepted annotations was rejected by human reviewers, and all 29 human-rejected annotations were also rejected by the LLM validator.Three annotators reviewed each of 100 items without access to the LLM decision.Source: §3.3 · p. 5, Table 4 · p. 5

3.4 Dataset Statistics

• The benchmark contains 1,613 questions across 30 projects and 7,140 retrieval records, including 5,725 text and 1,415 visual records.There are 1,526 answerable and 87 unanswerable questions; 461 answerable questions (30.2%) span multiple source types.Source: Table 3 · p. 5, §3.4 · p. 6

4.1 Experimental Setup

• For each question, Qwen3-VL-Embedding-2B retrieves top-k records, which are given to the model in timestamp and ID order; GPT-5.6 Luna judges responses.Overall and task-level accuracy are reported across all 1,613 questions.Source: §4.1 · p. 6

4.2 Main Results

• 68.0% overall accuracy at k = 50 is GPT-5.6 Sol’s highest result; Gemini 3.8 Flash reaches 63.9%, Gemini 3.1 Pro 62.5%, and Qwen3.5-27B 50.3%.GPT-5.6 Sol also improves from 52.6% at k = 10 to 68.0% at k = 50; Qwen3.5-27B rises from 38.6% to 50.3%.Source: §4.2 · p. 6
• GPT-5.6 Sol reaches 65.5% accuracy on unanswerable questions, while its accuracy is 36.3% on cross-source conflicts and 32.5% on temporal conflicts at k = 10.The results identify conflict resolution and recognizing insufficient evidence as distinct challenges despite strong overall performance.Source: §4.2 · p. 7

4.3 Analysis of Gold Evidence Records

• For GPT-5.6 Sol, accuracy falls from 83.3% to 51.5% on text-only questions and from 52.9% to 27.5% on image-only questions when multiple evidence records are required.Accuracy is also 25.6% for multi-record questions requiring mixed text and image evidence; visual or mixed evidence remains challenging across models.Source: §4.3 · p. 7, Table 6 · p. 7

4.4 Analysis on Record Retrieval

• More retrieval does not guarantee higher QA accuracy: with the 2B retriever, recall rises from 57.0% to 64.8% as k increases from 50 to 100, while Qwen3.5-2B accuracy falls from 25.0% to 22.2%.The 8B retriever likewise raises recall from 59.6% to 67.3%, while InternVL3.5-8B accuracy falls from 38.0% to 37.6%; larger-retriever recall gains also do not consistently improve QA accuracy.Source: Table 7 · p. 7, §4.4 · p. 8

4.5 Qualitative Analysis

• In the cross-source case, GPT-5.6 Sol answers 34 instead of the newly added row’s 23; in the temporal case, it gives the revised deadline when asked for the preceding one.The examples show errors in selecting the target value among revisions and recovering the historical state requested.Source: §4.5 · p. 8, Figure 3 · p. 8

• MMEnterprise-Bench contains 1,613 questions across 30 simulated collaboration projects grounded in public papers and reports, spanning Evidence Reasoning, Conflict Resolution, and Unanswerability.The benchmark targets question answering over evolving multimodal workplace records.Source: §5 · p. 8
• Evaluations of 10 proprietary and open-source agents reveal persistent challenges integrating evidence across records and modalities, resolving conflicts, and recognizing insufficient information.Source: §5 · p. 8
• Improved retrieval does not consistently translate into better QA performance, and MMEnterprise-Bench provides a testbed for developing reliable enterprise assistants as workplace information evolves.Source: §5 · p. 9

• Simulated workplace records may not capture every aspect of real enterprise environments.Source: Limitations · p. 10
• Human validation showed high agreement for LLM-accepted annotations, but LLM-based generation and validation may still leave residual errors.Source: Limitations · p. 10

Ethical Considerations

• Benchmark performance does not guarantee reliability in real deployments, where privacy, access controls, and human oversight remain important.Source: Ethical Considerations · p. 10
• Workplace communications are simulated, and the authors performed human validation without recruiting external annotators.Source: Ethical Considerations · p. 10
• LLMs assisted with data generation, validation, answer evaluation, manuscript drafting and revision, and code development; the authors retain responsibility for the work.Source: Ethical Considerations · p. 10

A.1 Workspace Size

• Each of 30 projects contains 63.2k–123.8k tokens, averaging 88.8k; the workspace totals approximately 2.66 million tokens across 7,140 retrieval records.Token counts use the Qwen3.5-2B tokenizer and image processor, and evaluation retrieval draws from the entire workspace.Source: §A.1 · p. 13

A.2 Domain Coverage

• The projects draw on 12 research papers and 18 government publications across artificial intelligence, healthcare, economics, cybersecurity, transportation, and environmental policy.Seed documents provide textual and visual information, including numerical results, tables, and figures.Source: §A.2 · p. 13

A.3 Question and Evidence Characteristics

• Tab. 2 reports question and reference-answer lengths by task type: questions contain an average of 24.9 words, while reference answers contain an average of 18.0 words, respectively.Source: §A.3 · p. 13
• Gold evidence averages 1.94 records, with multiple records in 50.9% of sets; cross-source and temporal conflict questions average 2.23 and 2.93 records, respectively.Multiple records occur in 71.7% of cross-source conflict questions and 75.9% of temporal conflict questions.Source: §A.3 · pp. 13, 15
• Gold evidence spans multiple source types for 461 questions (30.2% of answerable questions), with 47 distinct combinations; frequent pairings include Notion with PDFs, Slack, or standalone images.The most frequent combinations are reported among the 461 questions with multi-source evidence.Source: §A.3 · p. 13, Table 3 · p. 15
• Visual records appear in gold evidence for 54.6% of answerable questions; text-only evidence dominates Attribution (97.5%), while visual-only evidence is most prevalent in Comparison (56.1%).Cross-source and temporal conflict questions have the highest mixed text-and-visual evidence proportions, at 32.5% andSource: §A.3 · p. 13, Table 4 · p. 15
• For 777 answerable questions with at least two records, gold evidence has a median temporal span of 24.0 hours; 50.7% span at least 24 hours and 35.8% at least 48 hours.Span is measured as the difference between the latest and earliest record timestamps.Source: §A.3 · p. 15, Table 5 · p. 16

B Benchmark Construction Prompts

• The benchmark construction prompts cover enterprise workspace simulation and QA annotation; each project’s pipeline proceeds from source extraction through workspace construction, event generation, and multimodal record generation.Source: B Benchmark Construction Prompts · p. 15, §B.1 · p. 15

B.1.1 Source Information Extraction

• Source extraction captures textual and visual facts, including image-only facts, labels, and key numbers, to support workplace-record construction and questions requiring visual evidence.Textual extraction also gathers system, configuration, claims, and method steps from seed documents.Source: §B.1.1 · p. 16

B.1.2 Workspace Construction

• Workspace construction defines each project’s organization, participants, channels, access relationships, and milestones, selecting an organizational archetype based on the seed document.The prompts distinguish research labs from analysis firms and tailor terminology to the selected archetype.Source: §B.1.2 · pp. 16–17

B.1.3 Collaborative Event Generation within a Project

• Collaborative events encode facts that drift over time, including ownership, deadlines, choices, sizes, budgets, and names, with each version ordered chronologically.Changes can reflect either an honest slip or a changed situation.Source: §B.1.3 · p. 18

B.1.4 Multimodal Record Generation

• Reports use information available at their writing time, and Slack messages are generated afterward from planned collaborative events, including reports, corrections, and decisions.Source: §B.1.4 · p. 18

B.2 QA Annotation

• Questions target plausible workplace information needs and receive a reference answer, reproducible solution, and representative sufficient evidence set.Annotation guidelines require checking ambiguity and alternative interpretations, using visual records for visual claims, and avoiding unnecessary evidence.Source: §B.2 · pp. 19–20

C.1 Workspace Validation

• Workspace validation checks generated formats, values, entities, event order, documents, and Slack messages; low naturalness scores trigger a feedback-based rewrite.Channels and days with at least six planned events receive a naturalness review; scores of 2 or lower on a five-point scale trigger one additional rewrite.Source: §C.1 · p. 21

C.2.1 Evidence-grounded Answering

• An independent solver answers using only the cited records and indicates whether those records settle the question.The solver must rely on records rather than general knowledge and should not present an unsettled fact as the answer.Source: §C.2.1 · p. 21

C.2.2 Candidate–Gold Comparison

• The candidate–gold comparison grades only the requested object, allowing equivalent wording and appropriate rounding while flagging substantive disagreements or missing qualifiers.It treats abstention as a mismatch when the gold answer is established, and commitment to a value as a mismatch when the gold is unanswerable.Source: §C.2.2 · pp. 21–22

C.2.3 Full-project Validation

• The full-project audit checks for competing interpretations, unsupported gold values, answerability errors, and insufficient cited evidence before passing, fixing, or dropping an item.The independent solver’s response informs the audit but does not determine it.Source: §C.2.3 · p. 22
• Full-project validation logs each question, gold answer and status, solution-stage JSON, deduplicated cited records, and the Stage-1 answer’s knowability, grade, cause, and reason.The record template also includes all project records.Source: §C.2.3 · p. 22

C.3 Human Validation Interface

• The human interface presents one item at a time, with expandable cited-evidence cards, source metadata, and a searchable browser that highlights cited records.Metadata includes application, filename, page, message author, and timestamp; the browser provides additional context.Source: §C.3 · pp. 22–23, Figure 2 · p. 23
• Annotators assess whether the answer follows from highlighted evidence, answers the question correctly, and is absent from the question text; specified premise, challenged, and run-ID values are not leaks.C1 and C2 allow Yes, No, or Cannot judge; C3 records whether the answer is present in the question.Source: §C.3 · p. 23

D.1 Record Retrieval

• Qwen3-VL-Embedding-2B retrieves the top-k workspace records for each question, with k ∈{10, 50} in the main experiments and k = 100 plus the 8B retriever variant in retrieval analysis.The query and document instructions frame retrieval as finding records or document pages that help answer an organizational-record question and as representing the user's input.Source: §D.1 · pp. 23–24

D.2 Answer Generation

• Retrieved records are ordered by timestamp, with stored order and unit ID breaking ties, and visual records are supplied as stored text followed by their images.Records retain stored text, source headers, and available metadata; the question follows the records.Source: §D.2 · p. 24
• Generation uses temperature 0, with maximum output lengths of 8,192 tokens for Qwen3.5 and proprietary API models and 2,048 tokens for InternVL3.5-8B and Kimi-VL-16B-A3B-Instruct.Local inference uses vLLM 0.24.0 with bfloat16 weights.Source: §D.2 · p. 24
• The answering instruction requires concise plain-text answers based only on supplied workplace records and images, with abstention when information is not established and brief explanations of contradictions.Records are treated as evidence rather than instructions.Source: §D.2 · p. 24

D.3 Answer Evaluation

• GPT-5.6 Luna judges correctness by comparing the question, reference answer, and model response, and the resulting decisions determine overall and task-level accuracy.Semantically equivalent answers are accepted when they include all requested components; equivalent abstentions count as correct for unanswerable questions.Source: §D.3 · p. 24
• The judge evaluates the final answer, accepts semantic equivalents, requires requested components, and treats refusal as incorrect when the reference provides a concrete answer.When the reference says information is unavailable, an equivalent abstention is correct; extra explanation found only in the reference is not required.Source: §D.3 · p. 24
• The judge returns one JSON object with exactly the keys correct and reason, while treating user fields as untrusted data rather than instructions.The JSON input includes the question, reference answer, and candidate answer.Source: §D.3 · p. 24

E Qualitative Analysis

• Figures 3–10 illustrate eight task types across Evidence Reasoning, Conflict Resolution, and Unanswerability, each with a question, reference answer, and relevant workplace records.The examples cover extraction, attribution, computation, comparison, interpretation, cross-source conflict, temporal conflict, and unanswerability.Source: E Qualitative Analysis · p. 24
• Figures 11–13 show simulated Slack, email, Notion, and attached-file records, with collaborator exchanges reflecting specified roles and communication styles.The figures include Slack interactions, an email with an attached file, and Notion entries.Source: E Qualitative Analysis · p. 24, Figure 11 · p. 33, Figure 12 · p. 34

Decision Log

• A logged decision says not to use one headline number, instead presenting a per-dataset table and the screened-study and interview counts in a sentence.The example specifies 193 screened primary studies narrowed to 38 final empirical cases, alongside 69 stakeholder interviews supporting 11 implementation concepts.Source: Decision Log · p. 35, N Tandem Grove · p. 35