Source-linked AI summary
AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios
Kaiyuan Chen, Qimin Wu, Taiyu Hou, Tianhao Tang, Xueyu Hu, Yuchen Hou, Bikun Li, Chengming Qian, Guoyin Wang, Haolin Chen, Haotong Tian, Haoye Zhang, Haoyu Bian, Hongbing Pan, Hongkang Zhang, Hongyi Zhou, Jiaqi Cai, Jiewu Rao, Jiyuan Ren, Keduan Huang, Lucia Zhu Huang, Mingyu Yuan, Naixu Guo, Qicheng Tang, Qinyan Zhang, Shuai Chen, Siheng Chen, Ting Ting Li, Xiaoxing Guo, Yaocheng Zuo, Yaoqi Guo, Yinan Wang, Yinzhou Yu, Yize Wang, Yuan Jiang, Yuan Tian, Yuanshuo Zhang, Yuxuan Liu, Yvette Yan Zeng, Zenyu Shan, Zihan Yin, Xiaobo Hu, Yang Liu, Yixin Ren, Yuan Gong
TL;DR
Current evaluations provide limited evidence about agents handling the diverse, file-centered tasks of everyday work, life, and learning. AgentIF-OneDay introduces a 104-task benchmark with three user-centric task types and instance-level judging, finding high overall performance among leading agents while exposing annotation and verification challenges.
Problem
Current evaluations insufficiently cover the diversity of agentic tasks needed for daily work, life, and learning.
Method
AgentIF-OneDay evaluates multimodal, attachment-based daily tasks through Open Workflow Execution, Latent Instruction Inference, Iterative Refinement, and instance-level rubrics.
Results
Manus achieves the highest overall success rate at 0.645, followed by Genspark at 0.635, ChatGPT-Agent at 0.626, and Minimax-Agent at 0.562.
Takeaways & Limitations
The benchmark supports analysis of general-agent strengths and bottlenecks across sustained instruction adherence, file understanding, and stateful collaboration.
Takeaways & Limitations
Verification is difficult because daily-life scenarios span highly diverse contexts, making it challenging to recruit experts who can assess all tasks.
Abstract
from arXiv · showhide
The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow, demonstrating exceptional performance in coding, deep research, and complex problem-solving evaluations. However, in daily scenarios, the perception of these advanced AI capabilities among general users remains limited. We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks necessary to cover the daily work, life, and learning activities of a broad demographic. To address this, we propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks. These tasks require not only solving problems through dialogue but also understanding various attachment types and delivering tangible file-based results. The benchmark is structured around three user-centric categories: Open Workflow Execution, which assesses adherence to explicit and complex workflows; Latent Instruction, which requires agents to infer implicit instructions from attachments; and Iterative Refinement, which involves modifying or expanding upon ongoing work. We employ instance-level rubrics and a refined evaluation pipeline that aligns LLM-based verification with human judgment, achieving an 80.1% agreement rate using Gemini-3-Pro. AgentIF-OneDay comprises 104 tasks covering 767 scoring points. We benchmarked four leading general AI agents and found that agent products built based on APIs and ChatGPT agents based on agent RL remain in the first tier simultaneously. Leading LLM APIs and open-source models have internalized agentic capabilities, enabling AI application teams to develop cutting-edge Agent products.
1 Introduction
AgentIF-OneDay addresses the limited diversity of current evaluations by testing whether general users can use agents for varied daily work, life, and learning tasks. It combines three interaction types with file-centered outputs and instance-level evaluation.
- Current evaluations do not adequately reflect the diversity of real-world tasks across work, life, and learning.
- AgentIF-OneDay evaluates general agents assisting users with daily tasks through natural-language instructions and tangible work outputs.
- Open Workflow Execution tests precise adherence to explicit, detailed procedures while resisting instruction forgetting and hallucinations.
- Latent Instruction Inference requires agents to infer unstated rules or constraints from attachments and apply them to new tasks.
- Iterative Refinement measures precise updates to existing outputs after supplementary or corrective instructions, emphasizing state maintenance and collaboration.
- 104 tasks cover 767 scoring points, supporting analysis of agent strengths and bottlenecks and potentially providing data for reinforcement learning.
2 Related Work
Related benchmarks evaluate instruction following, agent capabilities, and economic automation, but they differ in scope and emphasis. Existing work spans verifiable instructions, domain-specific agent tasks, and end-to-end labor automation.
- Instruction-following benchmarks test precise execution through verifiable instructions, domain classifications, and format requirements.
- Agent benchmarks assess combinations of reasoning and tool use in real-world tasks, web environments, and dedicated tool-use settings.
- User-conversation analyses report that 57% of AI interactions focus on augmenting human capabilities rather than full automation.
- Frontier agents achieve only a 2.5% automation rate on complex, end-to-end workflows.
- Instruction-following reinforcement-learning methods use verifiable signals, self-supervision, and decomposition of complex instructions.
3 AgentIF-OneDay
AgentIF-OneDay is a domain-agnostic, end-to-end benchmark for general agents that combines multimodal attachments, long-tail daily tasks, three interaction categories, and workflow-based judging. Its rubric system evaluates final outputs across content, form, and execution.
- Current instruction-following evaluations target individual models or vertical-domain systems, leaving complete general-agent systems underassessed.
- AgentIF-OneDay assesses end-to-end outcomes from multimodal inputs and attachments without prescribing the evaluated system’s architecture or solution method.
- The benchmark covers authentic needs across office routines, hobbies, and specialized professional tasks, including long-tail deliverables such as image editing and research.
- Tasks are categorized as Open Workflow Execution, Latent Instruction Inference, and Iterative Refinement according to user interaction patterns.
- LLM judges evaluate rubric satisfaction, aggregate bonus and penalty points, normalize scores, and use web search or rendered HTML when factual or visual verification is needed.
- 53.8% of tasks are Open Workflow Execution, while Work accounts for 59.6% of domains and tasks may include up to 10 files.
- Content receives 385 scoring points, followed by Form with 244 and Execution with 138, covering output quality and procedural accuracy.
4 Building AgentIF-OneDay
AgentIF-OneDay combines expert-authored tasks with an automated, file-centered synthesis pipeline. The process extracts workflows, gathers attachments, generates queries and rubrics, then filters and manually refines the resulting instances.
- Human annotators were instructed to submit difficult, objective, and search-resistant questions within verified areas of expertise.
- Submitted questions pass editorial screening, expert validation, revision, and final review before inclusion.
- The synthetic-data pipeline expands human-authored seed tasks while preserving their workflows across new scenarios, attachments, and evaluation rubrics.
- Workflow extraction identifies steps, inputs, outputs, and dependencies that structure subsequent task generation.
- Attachment searching retrieves and analyzes domain-specific documents and images to ground synthetic tasks in realistic contexts.
- Query generation preserves the seed workflow while varying content, domains, and contexts across the benchmark’s three task categories.
- Rubrics separate independently verifiable Bonus Criteria and Penalty Criteria, while filtering requires complex, measurable, valid, and ethically acceptable tasks.
5 Experiments
The evaluation benchmarks four leading agents across overall performance, capability dimensions, quality rubrics, efficiency, and attachment-heavy tasks. Results show close overall scores but distinct strengths, persistent difficulty with implicit instructions, and imperfect multimodal execution.
- 0.645 was Manus’s overall success rate, followed by Genspark at 0.635, ChatGPT-Agent at 0.626, and Minimax-Agent at 0.562.
- Genspark led implicit instruction inference, Manus led open workflow execution, and Minimax-Agent led iterative editing capabilities.
- Implicit condition inference was the weakest capability overall, with leading systems often trading off format accuracy against content coverage.
- Genspark scored highest for Instruction Following at 0.766, Negative Constraints at 0.824, and attachment handling at 0.691, while Manus led Factuality at 0.731.
- 1416.2 s was Minimax-Agent’s average latency, compared with 484.1 s for Genspark and 500.0 s for Manus.
- 80.1% was Gemini-3-Pro-preview’s agreement with human judges, versus 63.8% for GPT-5.1.
- In latent-instruction tasks, ChatGPT followed the page format but listed too few articles, while Genspark missed a citation marker and added irrelevant information.
- Genspark accurately inferred the golf-course background and met the price constraint, whereas ChatGPT ignored visual context and missed shaft-stiffness specifications.
6 Discussion and Limitation
AgentIF-OneDay faces substantial challenges in scaling task creation and verification because daily-life scenarios are complex, diverse, and costly to annotate. The paper partially addresses these bottlenecks through synthetic generation and identifies longer-horizon evaluation as a future direction.
- Three hours per annotator is the average cost of designing a single valid task in the pilot study.
- Task formulation depends heavily on personal life scenarios and imagination, limiting how much any individual annotator can generate.
- Daily-life contexts are highly diverse, making it difficult to recruit experts who can verify tasks across the full topic spectrum.
- A synthetic generation pipeline uses models to brainstorm diverse topics for human refinement, reducing dependence on individual creativity and specialized background knowledge.
- The paper presents transfer from daily to weekly scenarios as a future direction for extending agents to longer time horizons.
7 Conclusion
AgentIF-OneDay evaluates general AI agents in realistic daily-use conditions through diverse task types and highlights sustained instruction adherence, file-centered understanding, and stateful collaboration. Results indicate convergence in baseline agentic competence while implicit constraint inference and long-horizon consistency remain challenging.
- AgentIF-OneDay is a task-level instruction-following benchmark for general AI agents under realistic daily-use conditions.
- Its tasks span open workflow execution, latent instruction inference, and iterative refinement.
- Effective agent behavior depends on sustained instruction adherence, file-centered understanding, and stateful collaboration across time.
- Leading API-driven and RL-based agents show notable convergence in baseline agentic competence.
- Performance gaps across task categories and rubric dimensions persist in implicit constraint inference and long-horizon consistency.
- The benchmark provides a diagnostic tool and a data source for future agent training.
8 Contributors
The contributor information lists the paper’s authors and specifies authorship conventions. Contributors are listed alphabetically by first name, and some authors are marked as contributing equally.
- The listed contributors include Kaiyuan Chen, Qimin Wu, Taiyu Hou, Tianhao Tang, Xueyu Hu, and Yuchen Hou.
- Some authors are identified as having contributed equally to the work.
- Xiaobo Hu, Yang Liu, Yixin Ren, and Yuan Gong are also listed as contributors.
- Contributors are listed alphabetically by their first name.
A Prompts
The appendix contains prompt tables for synthetic data generation and LLM-based judging. These tables cover workflow extraction, attachment searching, query generation, and rubric generation.
- A.1 Synthetic Data Generation: Table 4 contains the prompt for workflow extraction.
- A.1 Synthetic Data Generation: The appendix labels Table 4 under synthetic data generation.
- A.2 LLM as Judge: Table 5 contains the prompt for attachment searching.
- A.2 LLM as Judge: Table 6 contains the prompt for query generation.
- A.2 LLM as Judge: Table 7 contains the prompt for rubrics generation.