xDailyBench: Benchmarking LLMs on Professional Consultation for Real-Life Problems

• Existing benchmarks largely emphasize autonomous professional workflows, leaving professional assistance for underspecified problems in work, learning, and everyday life underexplored.Such assistance requires models to infer implicit and personalized constraints, integrate domain knowledge with user context, and provide actionable recommendations.Source: §1 · pp. 1–2
• xDailyBench contains 248 authentic, open-ended tasks across 51 scenarios and four domains, preserving contextual, constraint-rich, and personalized user requests.Tasks came from more than 1,000 participants who submitted tasks they had completed or genuinely intended to accomplish with AI, along with original instructions and relevant workspace files.Source: §1 · pp. 2–3
• 75.6% average score was achieved by the best-performing model, and strong conventional leaderboard performance did not necessarily translate into reliable everyday task completion.The benchmark also diagnoses failures by evaluation criterion and capability dimension, offering actionable guidance for model development.Source: §1 · p. 2
• A rubric-based framework scores open-ended tasks against task-specific criteria mapped to an interpretable capability taxonomy, without requiring a unique reference answer.It measures end-to-end task success and fine-grained model capabilities.Source: §1 · pp. 2–3
• The benchmark’s contributions include evaluating representative frontier LLMs to reveal practical limitations and identify capability bottlenecks behind failures.Its criteria support fine-grained diagnosis of model weaknesses.Source: §1 · pp. 2–3

2.1 Evolution of LLM Benchmarks

• Earlier benchmarks tested verifiable expertise, while newer end-to-end benchmarks improved realism but emphasized professional workflows; xDailyBench evaluates life-centered problems requiring specialized knowledge.Representative question benchmarks cover law [9] [15], medicine [13] [38], and advanced mathematics [10]; newer work includes office tasks [23] [33], expert-curated tasks [35], data-rich professional environments [32], and long-horizon industry workflows [27].Source: §2.1 · p. 3

2.2 Interactive Agent Benchmarks

• Interactive-agent benchmarks assess tool use and sequential decisions, but xDailyBench adds genuine user requests, fine-grained scenario coverage, and binary criteria mapped to capabilities.Existing evaluations include general assistance [19], web interaction [6] [37], planning [34], and application ecosystems [31], but often use constructed workflows or simulated environments and emphasize terminal states or task success.Source: §2.2 · p. 4

2.3 Evaluation of Open-Ended Outputs

• LLM-as-a-Judge supports open-ended evaluation [18] [36], but bias and robustness concerns [28] [30] motivate structured criteria validated against human judgments [16].xDailyBench pairs independently judged binary criteria with a shared five-dimensional capability taxonomy, supporting task scoring, cross-scenario comparison, and capability-level failure analysis.Source: §2.3 · p. 4

3.1 Benchmark Construction

• The construction pipeline is designed to preserve the ambiguity, personalization, and contextual richness of real-world requests rather than reduce them to narrowly specified tasks.It organizes the collection, annotation, and validation of authentic open-ended tasks across diverse domains.Source: §3.1 · p. 4

3.1.1 Task Sourcing

• A survey of over 1,000 participants collects tasks they used or genuinely intended to use AI for, retaining more than 500 authentic tasks across four broad domains after initial filtering.Requests may distribute requirements across explicit instructions, workspace materials, and user context; context helps test inference of unstated needs and preferences.Source: §3.1.1 · pp. 4–5

3.1.2 Rubric Construction

• Task-specific rubrics define success conditions rather than a single reference output, balancing alignment, completeness, and openness to multiple valid solutions.Weighted rubric criteria determine task-level scores, and each criterion is associated with a capability dimension.Source: §3.1.2 · p. 5

3.1.3 Quality Assurance

• Quality assurance combines model-assisted screening for authenticity, completeness, clarity, and difficulty with contributor cross-review for rubric consistency and verifiability.Cross-review requires atomic, clear criteria that support deterministic binary judgments, with iterative revision until requirements are met.Source: §3.1.3 · p. 6

3.1.4 Pre-Validation of Automatic Evaluation

• Each rubric criterion must achieve at least 90% agreement between automatic and human judgments across trial outputs, or the rubric is manually reviewed and revised.The pilot evaluates end-to-end outputs from diverse model–harness combinations using the same rubric protocol for automatic and contributor assessments.Source: §3.1.4 · p. 6

3.2 Benchmark Statistics

• Rubrics average 13.3 binary items across five capability dimensions, with 70.5% explicit, 18.5% general implicit, and 11.0% user-specific implicit requirements.User-specific implicit requirements must be inferred from a user’s role, background, or context.Source: §3.2 · p. 6
• 248 real-world tasks span four C1 domains and 51 fine-grained C2 scenarios, with no scenario containing more than 16 tasks.Tasks are also classified as easy, medium, or hard based on contributors’ professional judgment.Source: §3.2 · p. 6, Figure 3 · p. 7, Table 2 · p. 7

4.1 Experimental Setup

• 11 models were evaluated with standardized task instructions and budgets (up to 120 steps), using GLM-5.1 to judge text and file-based outputs.Nanobot was the default harness; Kimi-K3, Kimi-K2.6, and Gemini-3.1-Pro used OpenCode because of compatibility issues.Source: §4.1 · p. 8, §4.2 · p. 8

4.2 Main Results

• 75.6%: Seed-2.1-Evolving and Kimi-K3 tie for the highest task-level score, ahead of GPT-5.5 (72.6%) and GPT-5.6-sol (72.4%).Claude-Opus-4.7 scores 70.8%, while Gemini-3.1-Pro is lowest at 51.3%.Source: Figure 4 · p. 8, §4.2 · p. 8, Table 3 · p. 9
• Every model scores at least 9 points lower on implicit than explicit requirements; DeepSeek-V4-Pro has the largest gap, at 19.0 points.Both general and user-specific implicit requirements lag explicit ones, making contextual inference about unstated needs a persistent challenge.Source: §4.2 · pp. 8–9
• No model leads all four domains: Seed-2.1-Evolving leads White-collar Work (78.8), Kimi-K3 leads Study & Research (80.5), and GPT-5.6-sol leads Others (82.4).Average domain scores are relatively close, but GPT-5.6-sol ranges from 82.4 on Others to 66.5 on White-collar Work.Source: Table 3 · p. 9, §4.2 · p. 9

5.1 Capability Dimension Analysis

• 65.8 for GPT-5.5 and 62.6 for GPT-5.6-sol on Quantitative Reasoning contrast with their 76.7 and 76.6 on Instruction Following, respectively.Quantitative Reasoning is the lowest-scoring dimension for nearly all models, indicating a persistent weakness despite stronger request-following performance.Source: §5.1 · pp. 9–10
• 70.7 for Instruction Following and 70.2 for Artifact Quality lead the average capability scores, followed by 68.2 for Information Understanding, 67.6 for Analytical Reasoning, and 60.1 for Quantitative Reasoning.Scores aggregate rubric results sharing each capability tag; the table reports averages across all evaluated models.Source: §5.1 · p. 9, Table 4 · p. 10
• Kimi-K3 leads Information Understanding at 77.1, GPT-5.5 leads Analytical Reasoning at 75.3, and Seed-2.1-Evolving leads three dimensions; Seed-2.1-Evolving and Kimi-K3 both score 75.6 overall.Seed-2.1-Evolving’s leading scores are 78.4 in Instruction Following, 74.7 in Quantitative Reasoning, and 81.5 in Artifact Quality.Source: §5.1 · p. 10
• GPT-5.5 scores 74.1 versus GPT-5.6-sol’s 67.1 on Artifact Quality, while Gemini-3.1-Pro scores 45.8 on Analytical Reasoning and 41.5 on Quantitative Reasoning.Similar overall scores can mask different strengths, and lower-performing models generally show broader degradation across dimensions.Source: §5.1 · p. 10

5.2 Failure Mode Analysis

• A renovation plan schedules 14 weeks despite a three-month move-in deadline and leaves only RMB 10,000 for contingencies, yet presents the project as feasible.This failure mode arises when a model produces detailed plans without checking whether scattered space, budget, schedule, construction, and risk constraints can hold simultaneously.Source: §5.2 · p. 10, Figure 5 · p. 11
• In open-ended information tasks, agents may keep searching or revising intermediate materials instead of judging whether evidence is sufficient, resolving conflicts, and producing a final artifact.Repeated preparation can replace synthesis and exhaust the interaction budget because relevant search results do not establish that key sources are covered or evidence is sufficient.Source: §5.2 · pp. 10–11

6 Conclusion

• xDailyBench is a rubric-grounded benchmark for evaluating whether language-model agents satisfy open-ended requirements in everyday user scenarios.It is proposed as a practical testbed for future agent evaluation.Source: §6 · p. 11
• Current agents can use tools and produce complete artifacts, yet failures reveal weaknesses in grounding, reasoning accuracy, feasibility checking, and knowing when to stop exploring and synthesize.In one trajectory, a model failed to open an official statute page returned by its first search, made 94 increasingly redundant queries, and exhausted its 120-step budget without a deliverable, scoring 0.Source: §6 · p. 11, Figure 6 · p. 12
• xDailyBench shifts evaluation from surface task completion toward reliable, verifiable, and useful outcomes under real-user constraints.The authors present it as a practical testbed for future agent evaluation.Source: §6 · p. 11

A Typical Cases of xDailyBench from Different Domains

• The appendix selects representative tasks from four primary domains to illustrate failure modes including neglected requirements, unmet constraints, superficial reasoning, factual fabrication, and non-compliant risk warnings.All examples are sanitized and do not reflect real user information.Source: A Typical Cases of xDailyBench from… · p. 17
• Medical pickup planning requires filtering designated pharmacies, checking travel distances and dispensing windows, and proposing primary and alternative options aligned with the user’s available time.Seed-2.1-Evolving identifies the earliest eligible pickup time but selects incorrect pharmacies and details; the baseline invents pharmacies and falsely claims noon pickup availability.Source: A Typical Cases of xDailyBench from… · p. 17, Figure 7 · p. 17
• The education-and-housing task requires comparing two school pathways under strict budget, residency, commute, occupancy, enrollment, and exit-strategy constraints.The evaluation prioritizes verifiable data, full constraint alignment, and risk control rather than a comprehensive report.Source: A Typical Cases of xDailyBench from… · p. 18, Figure 8 · p. 18
• Seed-2.1-Evolving answers all 15 questions but misstates tuition, proposes prohibited mortgage borrowing, and undercounts expenses for two children.It also gives actionable instructions for evading residency audits and claims Monte Carlo analysis without disclosing its distributions, sampling logic, or intermediate results.Source: A Typical Cases of xDailyBench from… · p. 18
• The Sanxingdui guide task requires using only supplied materials, distinguishing established archaeological conclusions from speculation, and adapting explanations for junior-high students and elderly visitors.Required outputs include concept flashcards, travel narration, on-site reminders, and family group-chat announcements.Source: A Typical Cases of xDailyBench from… · pp. 18–19, Figure 9 · p. 19
• Models inconsistently obey the source-only rule, overlook detailed formatting requirements, and produce generic rather than scenario-customized guide materials.The case highlights weak evidentiary grounding and superficial scenario adaptation despite basic structured-generation ability.Source: A Typical Cases of xDailyBench from… · p. 19
• The admissions-rubric audit requires line-by-line review of Rules R01–R06, actionable detection parameters, culturally aware revisions, and distinct standards for institutional tiers.Seed follows the output format but changes key trigger words and misses a concern in R04; the baseline preserves triggers better but remains abstract and fails to differentiate R06 standards.Source: A Typical Cases of xDailyBench from… · pp. 19–20

B.1 Annotation Workflow of xDailyBench

• Tasks come from AI requests contributors actually used or genuinely intended, preserving natural underspecification and relevant user background, materials, and constraints.Requirements may remain distributed across the query, workspace materials, and user background when that reflects the original interaction.Source: §B.1 · p. 21
• Task-specific rubrics cover explicit, generally expected, and context-inferred requirements using weighted, independently judgeable binary criteria without prescribing one solution.Contributors are instructed to ensure coverage of essential requirements and uphold alignment, completeness, and openness.Source: §B.1 · p. 21
• Cross-reviewers check task sufficiency and rubric clarity, atomicity, evaluability, coverage, and evidential support, returning flawed annotations for revision until quality requirements are met.The process is intended to reduce reliance on one contributor’s interpretation and support consistent independent evaluation.Source: §B.1 · p. 21
• At least 90% criterion-level agreement between human and automatic judgments is required; failing criteria are inspected, revised when needed, and re-evaluated.Human judgments are compared with automatic judgments on sampled trial outputs; the goal is rubric consistency, not calibration of humans to the automatic judge.Source: §B.1 · p. 21
• Contributors provide authentic tasks, construct rubrics, cross-review others’ annotations, and validate rubric reliability for automatic evaluation.The workflow aims to preserve naturally occurring user requests while maintaining consistency for benchmark-scale assessment.Source: §B.1 · pp. 20–21

B.2 Rubric Design Principles

• Each rubric criterion records its description, scoring type, importance level, and capability dimension, turning ambiguous judgments into reusable, quantifiable scoring units.Source: §B.2 · p. 22

C Rubric Dimensions of xDailyBench

• The five dimensions are Information Understanding, Instruction Following, Analytical Reasoning, Quantitative Reasoning, and Artifact Quality.They cover interpreting task inputs and context; following requirements and constraints; integrating information and reasoning under constraints; numerical reasoning; and the completeness, correctness, usability, and form of the deliverable.Source: C Rubric Dimensions of xDailyBench · p. 22

D Model Performance on General and User-specific Implicit Requirements on xDailyBench

• Table 5 reports detailed model performance for different types of implicit requirements.Source: D Model Performance on General and… · p. 22

E Comparison between GPT-5.5 and GPT-5.6-sol

• 72.6 vs. 72.4 overall task-level scores for GPT-5.5 and GPT-5.6-sol are nearly identical and do not indicate that either model is uniformly stronger.Their differences instead reflect a redistribution of strengths across requirement types and task domains.Source: E Comparison between GPT-5.5 and… · pp. 22–23
• 77.2 vs. 75.0 on explicit requirements favors GPT-5.5, while 66.0 vs. 62.3 on implicit requirements favors GPT-5.6-sol.The explicit–implicit gap is 14.9 points for GPT-5.5 and 9.0 points for GPT-5.6-sol; both score lower on implicit requirements.Source: E Comparison between GPT-5.5 and… · pp. 22–23
• 74.5 vs. 69.7 favors GPT-5.6-sol on Personal Life, while 72.5 vs. 66.5 favors GPT-5.5 on White-collar Work.The models score similarly on Study & Research (72.8 vs. 72.3) and Others (81.8 vs. 82.4); their differences are concentrated in particular task domains.Source: E Comparison between GPT-5.5 and… · p. 23
• Decomposing performance by requirement explicitness and task domain reveals behavioral differences that a single task-level score would obscure.Which model is preferable may depend on whether a user needs strict execution of explicit instructions or inference of underspecified, personalized requirements.Source: E Comparison between GPT-5.5 and… · p. 23