Source-linked AI summary
The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios
Daocheng Fu, Jianbiao Mei, Rong Wu, Xuemeng Yang, Jia Xu, Ding Wang, Pinlong Cai, Yong Liu, Licheng Wen, Botian Shi
TL;DR
Existing research largely evaluates workflow agents in static, controlled settings, leaving robustness in stochastic deployment insufficiently assessed. The paper introduces Trainee-Bench, a dynamic benchmark for scheduling, exploration, and continual learning, and finds that current agents struggle especially with uncertainty and continuous learning. It argues for evaluation and development focused on robust exploration and experience internalization.
Problem
Existing research predominantly targets performance upper bounds in controlled environments, while realistic stochastic deployment requires assessing robustness across dynamic tasks, uncertainty, and prior experience.
Method
Trainee-Bench simulates a trainee agent in a novel workplace setting using streaming tasks, concealed clues, and rule-based dynamically generated scenarios to assess three core competencies.
Results
State-of-the-art agents struggle in dynamic environments, particularly with active exploration and continual learning, with a significant performance gap compared to human-guided execution.
Takeaways & Limitations
The findings support shifting agent evaluation from static laboratory tests toward stochastic, production-oriented assessment of robust exploration and experience internalization.
Takeaways & Limitations
Task diversity is constrained, meta-task rules are manually crafted, and evaluation covers selected agent frameworks within one workplace simulation context.
Abstract
from arXiv · showhide
The rapid evolution of Multi-modal Large Language Models (MLLMs) has advanced workflow automation; however, existing research mainly targets performance upper bounds in static environments, overlooking robustness for stochastic real-world deployment. We identify three key challenges: dynamic task scheduling, active exploration under uncertainty, and continuous learning from experience. To bridge this gap, we introduce \method{}, a dynamic evaluation environment that simulates a "trainee" agent continuously exploring a novel setting. Unlike traditional benchmarks, \method{} evaluates agents along three dimensions: (1) context-aware scheduling for streaming tasks with varying priorities; (2) prudent information acquisition to reduce hallucination via active exploration; and (3) continuous evolution by distilling generalized strategies from rule-based, dynamically generated tasks. Experiments show that cutting-edge agents have significant deficiencies in dynamic environments, especially in active exploration and continual learning. Our work establishes a framework for assessing agent reliability, shifting evaluation from static tests to realistic, production-oriented scenarios. Our codes are available at https://github.com/KnowledgeXLab/EvoEnv
1 Introduction
Existing agent research often evaluates upper-bound performance in controlled settings, leaving robustness in stochastic, realistic workflows underexamined. Trainee-Bench addresses this gap by evaluating scheduling, active exploration, and continual learning, and finds notable deficiencies in current agents.
- Existing research emphasizes performance upper bounds in controlled environments rather than robustness in stochastic real-world deployment.
- Realistic deployment requires dynamic scheduling, active exploration of hidden information, and learning from prior tasks to avoid repeated errors.
- Trainee-Bench simulates a trainee agent exploring a novel setting through streaming tasks, concealed clues, and dynamically generated rule-based scenarios.
- Experiments show state-of-the-art agents have substantial deficiencies in active exploration and continual learning within dynamic environments.
- The benchmark assesses temporal scheduling, prudent decision-making under uncertainty, and long-term strategic evolution of MLLM agents.
2 Trainee-Bench
Trainee-Bench is a dynamic workplace simulation that evaluates agents on multi-task scheduling, active exploration under partial observability, and generalization across stochastic task instances. It composes rule-based meta-tasks into temporally structured scenarios and verifies progress through checkpoints.
- Benchmark Overview: Trainee-Bench simulates corporate interns performing realistic workplace routines while evaluating dynamic multi-tasking, proactive exploration, and continuous self-evolution.Its bottom-up design spans atomic skills through holistic workflows using human-crafted meta-tasks.
- Environment Formalization: The benchmark formalizes workplace interaction as a state-transition system in which tool actions transform environment states containing agents, NPCs, files, and databases.The transition function supports procedural task generation and concurrent multi-threaded workflow orchestration.
- Dynamic Task Instantiation: Randomized NPC profiles and environmental data instantiate abstract meta-tasks into diverse task instances with objectives, verifiable checkpoints, and latent clues.Changing the random seed creates multiple instances from one rule, requiring generalized reasoning and exploration rather than memorization.
- Task Diversity and Coverage: Its 181 meta-tasks cover information analysis, scheduling, proactive inquiry, and strategic optimization, while stochastic variables prevent fixed layouts or hard-coded solutions.The separation of logical rules from randomized data expands task coverage and operational diversity.
- Partial Observability: Partial observability withholds essential clues from the initial prompt, requiring agents to explore files or conduct multi-turn interactions to complete tasks.This design measures information acquisition under uncertainty instead of passive execution.
- Dynamic Composite Scenarios: Composite scenarios combine tasks along timelines with conflicts, preemptive deadlines, and inter-task dependencies that require context switching and adaptive replanning.The benchmark uses these constraints to test priority scheduling, memory management, and updates based on real-time observations.
3 Experiments
The experiments evaluate LLM agents on dynamic workplace scenarios spanning scheduling, uncertainty, task difficulty, capability asymmetry, continual learning, and human-guided exploration. Results show substantial weaknesses in complex and stochastic settings, especially autonomous exploration and experience use.
- The benchmark evaluates whether current LLMs handle dynamic workplace environments, learn consistently from experience, and close the gap between proactive exploration and human guidance.
- Overall Performance: 35% Success Rate is achieved by the best-performing model, Gemini-3-Flash, across scenarios, while it records 90 average steps and 232 average tool calls.
- Impact of Task Workload: 50% to 36% is Gemini-3-Flash’s Success Rate decline when scenario workload increases from 2 to 6 tasks.GPT-4o and Grok-4 show similar downward trends, whereas Claude-4-Sonnet and GPT-5.1 show no strong correlation between performance and task count.
- Impact of Task Difficulty: 34% to 7% is Grok-4’s Success Rate drop from Easy to Hard tasks, compared with 32% Success Rate for Gemini-3-Flash on Hard tasks.Claude-4-Sonnet declines from 37% to 8%, while GPT-4o declines from 26% to 3%.
- Impact of Task Types: GPT-4o and Qwen3-VL show polarized capability profiles, performing better in Time Management than in Active Exploration and Task Modeling.The breakdown identifies asymmetry under complex, coherent task inputs and highlights autonomous decision-making and logical reconstruction as weak areas.
- Continual Learning: Accumulated experience degrades easy-task performance but improves hard-task performance marginally, because Day 1 failures do not reliably match Day 2 failures in stochastic environments.The MUSE agent’s experience is extracted from unreached checkpoints, which may differ across days.
- Benefits of Human Guidance: 0.24 to 0.83 is the average score increase for GPT-4o on hard tasks with human-provided tiered hints, while self-evolution improves performance by only +0.04.The comparison indicates shortcomings in autonomous exploration and in experience summarization and utilization.
4 Conclusion
Trainee-Bench bridges static evaluation and dynamic, uncertain workplace scenarios by assessing multi-tasking, exploration, and continuous self-evolution. Experiments show SOTA agents struggle particularly with uncertainty and continual learning, with a significant gap from human-guided execution.
- Trainee-Bench evaluates dynamic multi-tasking, proactive exploration under uncertainty, and continuous self-evolution in workplace scenarios.It uses rule-based meta-task templates, composite time-constrained scenarios, and automated verification.
- The benchmark is designed to bridge static setups and dynamic, uncertain workplace scenarios.
- SOTA agents struggle with uncertainty and continuous learning, showing a significant performance gap compared to human-guided execution.The findings motivate shifting from isolated-skill optimization toward robust exploration and experience internalization.
5 Limitations
The benchmark is limited by constrained task diversity, manually crafted meta-task rules, restricted framework coverage, and a single workplace simulation context.
- Task composition lacks complex causal inter-dependencies, limiting the benchmark’s simulation of dynamic realities.Future iterations will incorporate rigid causal chains.
- Manually crafted meta-task rules limit benchmark scalability, motivating automated rule-generation methods.
- Resource and time constraints restricted evaluation to selected agent frameworks within one workplace simulation context.Future work will broaden framework coverage and extend simulations to production and industrial domains.
A Related Works
The related-work review situates Trainee-Bench within language-agent research and examines existing benchmark paradigms to identify evaluation gaps.
- The review covers the evolution of agent systems and their core capabilities.
- It analyzes existing agent-benchmark paradigms to identify gaps that Trainee-Bench is designed to address.
A.1 Evolution of Agent Systems
Agent research progressed from reasoning and tool-use foundations toward memory, self-reflection, and open-ended exploration. ReAct established interleaved reasoning and environmental action as a core modern-agent paradigm.
- Chain-of-Thought prompting enhanced reasoning, while ReAct established interleaved Thought and Action in environmental interaction.
- Subsequent work added long-term memory systems and self-reflection mechanisms for complex simulations and iterative plan refinement.
- Open-ended exploration in complex environments became another frontier of agent development.
A.2 Paradigms in Agent Evaluation
Existing agent benchmarks largely use static, information-complete, one-shot tasks, leaving dynamic scheduling, active exploration, and continual learning insufficiently evaluated. Trainee-Bench addresses these gaps with streaming tasks, hidden information, and repeated similar tasks across several days.
- Existing benchmarks use fixed task sets and cannot assess managing a continuous stream of tasks.
- Information-complete settings provide no mechanism to measure active exploration under uncertainty.
- One-shot task designs cannot evaluate whether agents apply lessons from prior tasks to improve future performance.
- Trainee-Bench introduces a dynamic task stream, hidden information, and similar tasks across several days to evaluate scheduling, exploration, and continual learning.
- Its scenarios combine rule-based task templates into complex workflows, while the surrounding framework tracks goals, findings, persistent state changes, files, code, and communications.
B.3 Failure Case
Llama-4-maverick repeatedly failed to follow tool schemas and generated invalid parameters during tool invocation. A representative call passed an unsupported command argument, producing an execution error.
- Llama-4-maverick exhibits recurring tool-invocation difficulties involving schema violations and invalid parameters.
- An OpenFolderInCloudDisk call supplied a command argument that the tool did not accept.
- The invalid argument caused OpenFolderInCloudDisk to return an unexpected-keyword error.
B.4 Case of Human Guidance
Human guidance experiments isolate whether failures arise from missing data, strategy, or precise execution. Correct data and algorithmic guidance improved advertising performance, while event planning remained limited by calculation accuracy after strategic guidance.
- Case 1: Advertising Campaign Planning: In Advertising Campaign Planning, the unaided agent achieved a Checkpoint Score of 0.31 after failing to calculate cost and exposure metrics correctly.
- Case 1: Advertising Campaign Planning: Providing the correct population density matrix raised the Checkpoint Score to 0.62, but channel selection remained non-optimal.
- Case 1: Advertising Campaign Planning: Adding the correct algorithm suggestion produced a Checkpoint Score of 0.85, although the agent still used incorrectly derived population-density data.
- Case 1: Advertising Campaign Planning: Combined guidance on the correct data source and algorithm achieved a perfect Checkpoint Score of 1.00.
- Case 2: Event Planning: In Event Planning, the unaided agent scored 0.17 because it failed to select a valid date from the common availability period.
- Case 2: Event Planning: Providing valid dates and necessary files raised the score to 0.72, but further data-use and full-strategy hints did not improve it because precise calculations remained inaccurate.
B.5 Case of Daily Feedback
Daily feedback identifies incomplete checkpoints and gives targeted instructions for contacting responsible colleagues, explaining problems, and obtaining authorization. The example focuses on resolving a website-maintenance issue through the appropriate internal roles.
- The incomplete checkpoints involved failing to ask HR for the website-maintenance contact, discuss the problem, or request an authorization code.
- Feedback directs the agent to ask HR for the person responsible for maintaining the company website.
- The agent should clearly report the discovered problem, such as an almost-full website database, to that responsible person.
- The agent should seek authorization from the Engineering Managers.