PaperlayerView exact revision on arXiv

Source-linked AI summary

Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

Yijun Pan, Yukun Lian, Kunyu Shi, Junbo Li, Hongwei Xue, Sicong Xie, Guannan Zhang, Xiaoying Xing

arXiv:2608.08621v1cs.AI

TL;DR

Business-related capabilities remain underrepresented in agent benchmarks, despite the risks and complexity of end-to-end business operation. Business Arena evaluates autonomous cross-border shops using realistic commercial data, diagnostic metrics, attribution, and ablations, finding substantial differences across 15 frontier models and a 9.0 times gap in mean final net worth, with the best model behind expert-designed strategies.

  • Problem

    Existing agent benchmarks provide limited coverage of realistic, end-to-end business operation, motivating controlled evaluation of agents as economic operators.

  • Method

    Business Arena evaluates autonomous cross-border B2B shops over long horizons using real commercial data, skill metrics, action-level attribution, and mechanism ablations.

  • Results

    9.0 times gap: mean final net worth ranges from $20,856 to $188,488 across 15 frontier models, while the strongest expert-designed strategy earns more than twice the best model mean.

  • Takeaways & Limitations

    Business Arena shows that end-to-end business operation remains challenging for LLM agents and that profitability alone does not capture their differing operating skills and decisions.

  • Takeaways & Limitations

    The benchmark abstracts external operating systems and currently covers only cross-border B2B commerce rather than broader business settings.

Abstract

from arXiv · show

Running a business is a challenging form of intelligent work. Operators must infer opportunities from partial signals, commit capital under uncertainty, adapt to delayed outcomes in a changing market, and satisfy regulatory obligations before trading legally. Frontier LLM agents can increasingly complete complex workflows, yet business-related capabilities are rarely evaluated in existing agent benchmarks. We introduce \textbf{Business Arena}, a controlled environment where an AI agent runs a cross-border shop, buying from suppliers and selling to buyers over a long horizon. We ground the arena in real Alibaba.com sourcing data and market conditions calibrated from authoritative sources. Delayed and coupled consequences make individual business decisions difficult to judge, but their combined outcome is measurable through profit. Because profit alone cannot explain why an agent succeeds or fails, we compare agents with human-designed strategies to estimate available opportunity, use skill-level metrics to reveal underlying strengths and weaknesses, and trace realized gains and losses to the actions that produced them. We use mechanism ablations to establish that strong results reflect genuine business intelligence rather than neglect or simulator-specific shortcuts. We evaluate 15 frontier models and find a ninefold difference in mean final net worth. Even the best model falls behind human-designed strategies, indicating that business operation remains challenging for LLM agents. Skill-level analysis reveals operating styles, from margin-focused premium sellers to high-turnover wholesalers and customer-service specialists, while action-level attribution identifies the sourcing, pricing, and recovery decisions that create or destroy value. Together, Business Arena takes a first step toward a realistic and trustworthy testbed for evaluating end-to-end business agents.

1 Introduction

Business Arena is a controlled, long-horizon testbed for evaluating AI agents that operate cross-border businesses under realistic uncertainty, delayed feedback, changing conditions, and regulatory obligations. It combines measurable financial outcomes with skill-level analysis, action attribution, and mechanism ablations, revealing substantial performance gaps and headroom relative to expert-designed strategies.

  • Testbed: The arena preserves business difficulty by requiring decisions from partial evidence, capital commitments before outcomes, adaptation to evolving markets, and compliance with unavoidable obligations.Agents must also satisfy approval requirements, provide timely and accurate buyer service, and absorb operating costs regardless of sales outcomes.
  • Testbed: Business Arena evaluates agents running a cross-border business-to-business shop through an end-to-end loop covering research, sourcing, inventory, pricing, sales, service, compliance, and finance.Products, supplier offers, prices, minimum order quantities, and lead times come from real Alibaba.com listings, while demand cycles are calibrated from authoritative market conditions.
  • Evaluation: Mechanism ablations, skill-level metrics, and action attribution are used to distinguish genuine business competence from simulator-specific shortcuts and explain realized gains and losses.The attribution toolkit traces outcomes back to the actions that produced them, supporting fine-grained model analysis and future training-data construction.
  • Results: 15 frontier models produce mean final net worth from $20,856 to $188,488, a 9.0 times gap.Over the same long horizon, 51% of all runs lose money, and only four models preserve their starting capital in every trial.
  • Results: The strongest expert-designed strategy earns more than twice the best model mean, while successful models combine disciplined capital deployment, sell-through, margin-preserving pricing, and continued market learning.These results indicate substantial headroom for improving business operation by LLM agents.

2 Related Work

Prior agent benchmarks have advanced toward long-horizon execution, but typically retain fixed environments and verifiable outcomes. Existing business benchmarks cover narrower workflows, whereas Business Arena evaluates the interacting physical-commerce cycle in a data-grounded marketplace with diagnostic analysis.

  • Agent evaluation: Agent evaluation has progressed from short-horizon, verifiable tasks to long-horizon benchmarks requiring sustained execution across repositories and failing test suites.These longer benchmarks test coherent execution over extended horizons, but their environments remain fundamentally fixed.
  • Business simulations: Business simulations expose a realism–evaluability tension because real markets involve hidden state, changing conditions, coupled decisions, and no single ground-truth trajectory.Simplified environments are easier to evaluate but remove much of the difficulty of business decision-making.
  • Commercial benchmarks: Recent benchmarks study only partial commercial settings, including narrower workflows in VendingBench, ShopBench, and YC-Bench and simulated SaaS management in CEO-Bench.These benchmarks capture parts of business operation rather than the full physical-commerce cycle.
  • Business Arena: Business Arena addresses this gap by jointly evaluating sourcing, inventory, landed costs, pricing, demand, customer service, logistics, and compliance in a data-grounded, autonomously operated marketplace.Its diagnostic evaluation examines both the opportunities available and the decisions that capture them.

3 Designing a Marketplace for Business Intelligence

Business Arena models an agent-operated shop that must make end-to-end decisions under uncertainty, delayed consequences, market change, and persistent compliance obligations. Its design spans the business cycle and evaluates four capabilities: uncertain decision-making, constrained planning, insight-to-action alignment, and cooperation and competition.

  • Episode structure: The episode starts with shop-focus selection, opportunity inspection, initial inventory or capital retention, market-entry requirements, and listing configuration.Inventory acquired during setup is available on day 0, while market entry may wait for required approvals.
  • Episode structure: Each simulated day combines market inspection, offer adjustment, sourcing, advertising, liquidity management, and stakeholder responses before autonomous market progression.Buyers purchase, competitors update operations, shipments progress, costs are deducted, and new events or signals may emerge after the agent advances the day.
  • End-to-end coverage: Business Arena covers market research, sourcing, logistics, compliance, pricing, advertising, sales, stakeholder interactions, financial management, and recovery in one marketplace.The integrated design lets decisions in one business stage affect outcomes elsewhere across an end-to-end cycle.
  • Arena mechanisms: The arena creates difficulty through incomplete evidence, delayed economic consequences, changing markets, and persistent operational obligations.These mechanisms include noisy signals, overhead and inventory costs, shifting demand and trade conditions, competitor adaptation, disruptions, and compliance requirements backed by fines.
  • Business intelligence capabilities: The paper organizes business intelligence into four capabilities: Decision-Making Under Uncertainty, Strategic Planning Under Constraints, Insight-to-Action Alignment, and Cooperation & Competition.Strong strategies gather and update evidence, coordinate resource and risk decisions, execute economically sound actions, and manage buyers, suppliers, and competitors.

4 Diagnosing Model Performance

Business Arena diagnoses model performance beyond final profit by comparing agents with coherent deterministic strategies and analyzing both capabilities and the actions that produce gains or losses. Stateful evaluation restores the same business context so alternative decisions can be compared fairly.

  • Motivation: Final scores are difficult to interpret because similar outcomes can arise from different operating choices whose value depends on available market opportunities.The evaluation is therefore bounded around what opportunity was available and how it was pursued.
  • Reference strategies: Deterministic strategy libraries provide credible references for what agents could achieve under identical market conditions.The strategies use only information available to evaluated agents and encode coherent operating choices rather than isolated heuristics.
  • Stateful evaluation: Stateful evaluation compares alternative continuations from the same business state to isolate the effects of replacing decisions.The save–fork–load pipeline restores model-visible context, the OS-level workspace, and exact marketplace state.
  • Skill-level diagnosis: Skill-level metrics reveal strengths and weaknesses across business capabilities and operational subtasks.Metrics are economically grounded; advertising, for example, is assessed through full-funnel ROAS and ROI.
  • Action-level attribution: Action-level attribution traces every realized gain and loss through economic transitions to the sourcing, pricing, service, financing, or recovery action that produced it.This links capability-level diagnosis to concrete decisions that can be reinforced or corrected.

5 Experiments

The experiments evaluate frontier proprietary and open-weight models in isolated, source-inaccessible sandboxes, using layered documentation and maximum thinking effort. Results are reported as mean final net worth across 10 runs.

  • Testing Environment: Each evaluation uses a fresh isolated sandbox with a preinstalled OpenClaw runtime and layered agent-facing documentation.Filesystem permissions block access to arena source code, the database, and hidden simulator state.
  • Models Evaluated: 15 frontier models are evaluated, spanning proprietary and open-weight systems.The proprietary cohort includes GPT 5.6 Sol, GPT-5.5, Claude Fable 5, Opus 4.6 and 4.8, Gemini 3.1 Pro and 3.5 Flash, and Qwen 3.7 Max; the open-weight cohort includes GLM 5.2, Kimi K2.6 and K3, DeepSeek V4 Pro, MiniMax M2.5 and M3, and Qwen-3.8-Max-Preview.
  • Models Evaluated: 10 runs determine leaderboard results using mean final net worth, with all models receiving maximum thinking effort.GPT 5.6 Sol is specifically configured with pro effort.

6 Results

Business Arena reveals large performance gaps among frontier models, substantial headroom relative to expert strategies, and distinct operating styles across capital deployment, selling, service, and compliance. Skill- and action-level analyses, together with mechanism ablations, explain why agents succeed or fail and support interpreting scores as business intelligence.

  • Overall performance: Mean final net worth spans $188,488 for Gemini 3.1 Pro to $20,856 for MiniMax M2.5, a 9.0× difference; 51% of runs lose money.Only four models preserve their $80,000 starting capital in every trial.
  • Overall performance: The strongest expert-designed strategy reaches $436,195, more than twice the best model mean, by coordinating evidence, portfolio selection, sourcing, pricing, compliance, service, and capital allocation.Other real-world seller doctrines also surpass most models.
  • Skill-level profiles: Models exhibit distinct operating styles: Gemini 3.1 Pro reaches 199% cumulative capital utilization and protects a 52.0% average order margin, while volume wholesalers exceed 93% sell-through.GPT-5.6 Sol and Fable 5 reach 167% and 150% cumulative capital utilization, respectively, while maintaining inventory turnover near 1.0.
  • Skill-level profiles: Customer-service rankings differ from final net worth: Opus 4.8 converts 84% of inquiries, Fable 5 converts 81%, and Qwen 3.8 converts 74%.Opus 4.8 behaves more like a customer-service specialist than a full-cycle operator.
  • Skill-level profiles: Compliance failures carry substantial fines: Fable 5 and GPT-5.5 incur no penalized violations, whereas DeepSeek V4 Pro averages 17.4 violations and $39,650.The results show profitability alone is insufficient for deployment without reliable compliance and safety behavior.
  • Mechanism ablations: Across all five mechanisms, intended policies outperform neglect and misuse, supporting higher Business Arena scores as evidence of business intelligence rather than simulator-specific shortcuts.Demand inference recovers hidden opportunity rankings with a pooled correlation of 0.972 and identifies the strongest country in 93.3% of cases.

7 Limitations

Business Arena evaluates end-to-end business decision-making through structured tools rather than production operating systems. Future work can connect tasks to production-like systems and extend the decision framework to other forms of business.

  • Production-system coverage: The benchmark abstracts external operating systems behind structured tools and evaluates an agent’s choices rather than live execution.It does not test reliably updating a live storefront, submitting third-party offers, or managing customer conversations through a real inbox and GUI.
  • Production-system coverage: Production-system evaluation would test the complementary capability of executing business decisions through real systems.This includes storefront updates, third-party offer submission, and customer conversations through an inbox and GUI.
  • Future extensions: Future work can connect benchmark tasks to production-like systems and adapt its decision framework to other forms of business.These extensions would broaden the benchmark along both dimensions.

8 Conclusion … F Financial System and Capital Tools

Business Arena is a realistic, open-ended benchmark for autonomous cross-border business operation, combining grounded market conditions, broad business tools, and diagnostic evaluation. Its mechanisms test sourcing, demand adaptation, competition, logistics, compliance, and financially realized performance over extended episodes.

  • 8 Conclusion: Business Arena benchmarks 15 frontier models on profitability and capital, pricing, and service strategies in a realistic autonomous marketplace.The arena grounds an evolving marketplace in real commercial data and permits open-ended strategies with minimal supervision.
  • A Agent Runtime and Interface: Episodes run for 30 simulated operating days without evaluator-defined daily turns, allowing agents to interleave observations, tools, and business actions.Agents can inspect the market, manage files, execute programs, and advance each day through end_round.
  • A Agent Runtime and Interface: The isolated runtime enforces an information boundary: agents modify their workspace freely but access market information only through the public arena interface.The arena service, source code, database, and hidden state are protected under a separate system identity.
  • A Agent Runtime and Interface: The business interface covers observation, sourcing, selling, customer and supplier interaction, marketing, compliance, finance, and inventory recovery through typed MCP calls.These tools complement general workspace capabilities for reading, writing, editing, execution, and process management.
  • B Sourcing: Sourcing spans 965 supplier offers from 831 masked suppliers across 135 SKUs, with due diligence, hidden reliability risks, and changing wholesale prices requiring continued reassessment.Offers retain country, unit cost, MOQ, stock, lead time, and advertised quality, while discrepancies in quality or delivery may emerge only after ordering.
  • C Realistically-grounded Demand: Demand realism combines tariff shocks, seasonal retail series, Google Trends, and a 30-day calendar of major commercial festivals across represented markets.These mechanisms require agents to estimate demand trends while responding to recurring and event-driven market variation.
  • D Autonomous NPC Sellers: NPC sellers create heterogeneous competition through 60 scripted sellers, ten archetypes, scale tiers, repricing, restocking, advertising, festival responses, and inactivity decay.The population includes 10 baselines and 50 population NPCs, with capital ranging from $5k micro sellers to $1M enterprise sellers.
  • E Logistics, Tariffs, and Compliance: Cross-border operation evaluates landed-cost reasoning and compliance planning, while the financial system scores salvaged net worth and makes borrowing, factoring, and liquidation costly recovery choices.Landed cost includes supplier, freight, tariff, compliance, and selling costs; repeated compliance violations can reach $20,000 across ten floor-level violations.

G Mechanism Ablations · H Reliability Under Business Variance · H.1 Ranking Stability Under Repeated Sampling

Business Arena’s mechanism ablations show that evidence-guided business behavior improves outcomes while blind, misused, or shortcut-based strategies are penalized. Repeated-run analysis further shows that model rankings remain stable despite long-horizon business variance.

  • G Mechanism Ablations: Mechanism ablations construct competent and systematically incorrect policies using only agent-visible information, then compare them on matched market seeds.The comparisons form behavioral ladders showing which strategies are rewarded or punished.
  • G Mechanism Ablations: Evidence-guided sourcing reaches $144,069 in mean final net worth, compared with $80,433 for cheap-product concentration.Competent sourcing combines demand, selling cost, quality, lead time, and observed sales, then adjusts purchases using sell-through.
  • G Mechanism Ablations: Disciplined pricing reaches $144,069, versus $93,806 for static high-markup pricing and $35,697 for near-cost pricing.The mechanism tests whether agents preserve margin after full costs while remaining competitive enough to sell.
  • G Mechanism Ablations: Across mechanisms, competent use of public evidence improves intended economic or operational outcomes, while blind behavior, misuse, and benchmark-hacking strategies are punished.The results support strong scores reflecting correct mechanism use rather than static shortcuts.
  • H Reliability Under Business Variance: Long-horizon outcomes vary because early decisions alter later capital, inventory, and opportunities, allowing small trajectory differences to compound before final scoring.Business Arena therefore evaluates whether conclusions persist across repeated runs.
  • H.1 Ranking Stability Under Repeated Sampling: Each of the 15 models runs ten separately launched episodes under the same world seed and configuration, with leaderboard scores based on mean final net worth.Split-half resampling independently partitions each model’s ten runs into two disjoint five-run sets.
  • H.1 Ranking Stability Under Repeated Sampling: 0.898 mean Spearman correlation and 98.7% leading-three agreement across 50,000 partitions show rankings are not driven by exceptional trajectories.The two five-run rankings are computed independently, with no trajectory appearing in both halves.

H.2 Between- and Within-Model Variation · I Seller-Buyer Interface

Model identity explains most performance variation, while repeated-run noise remains substantial but can be reduced through averaging. The marketplace interface supports seller sourcing and listing alongside buyer discovery and purchasing, without evaluating GUI-agent use.

  • H.2 Between- and Within-Model Variation: 64.8% of observed score variation is explained by model identity, while 35.2% remains among repeated runs of the same model.The fixed-effects decomposition shows that economically different outcomes can arise across repeated trajectories of one model.
  • H.2 Between- and Within-Model Variation: A single episode contains substantial trajectory-level variation, so repeated evaluation is needed to estimate model performance reliably.The analysis models persistent model-level deviation separately from within-model trajectory variation.
  • H.2 Between- and Within-Model Variation: 5-run means reduce the within-model contribution according to the reported averaging relationship, and 10-run means reduce it further.For a mean of k runs, the within-model contribution decreases by a factor of k.
  • H.2 Between- and Within-Model Variation: ICC(1, 5) = 0.893 and ICC(1, 10) = 0.944, supporting highly reliable performance estimates from repeated evaluation.The agreement with empirical split-half analysis further supports the stability of the main performance ranking.
  • H.2 Between- and Within-Model Variation: Gemini 3.1 Pro achieves the highest mean but falls below starting capital in one run, whereas four named models preserve initial capital in every trial.The four models are GPT-5.6 Sol, Fable 5, Gemini 3.5 Flash, and GPT-5.5.
  • H.2 Between- and Within-Model Variation: The reliability results apply only to the shared leaderboard world, while robustness across market realizations requires evaluating models across multiple world seeds.Under the present setting, reported performance bands primarily reflect persistent model differences rather than isolated successful trajectories.
  • I Seller-Buyer Interface: The seller-buyer interface illustrates a marketplace in which an agent sources and lists a product while buyers discover and purchase the resulting offer.The illustration is dedicated to the marketplace and does not evaluate GUI agent use.

J Expert-designed Strategies · K Stateful Evaluation and Controlled Continuations

Expert-designed strategies estimate competent operation from public evidence while coupling beliefs, capital allocation, sourcing, compliance, and execution through feedback. Their strongest variants recycle cash, preserve margins, and diversify across products more effectively than leading model families.

  • J Expert-designed Strategies: The strategy reserve uses only the public interface and available evidence, excluding hidden demand parameters, future events, simulator seeds, and database internals.It is organized as a hierarchy of coupled decisions rather than independently selecting the best rule for each arena feature.
  • J Expert-designed Strategies: Each cycle combines five public information groups with bounded memory of market, sales, inventory, advertising, customer, and portfolio history.The snapshot covers the external market, available supply, route economics, current firm state, and realized operating history.
  • J Expert-designed Strategies: The controller converts evidence and memory into beliefs about demand, routes, inventory, and customer value while recording uncertainty.These beliefs combine public signals with realized velocity, sell-through, visible costs, stock quality, and service outcomes.
  • J Expert-designed Strategies: The coordinating plan adapts operating phase, reserves, budget, portfolio breadth, allocation, and risk exposure as feedback reveals winners and weaknesses.Early cycles fund diversified probes, winners become eligible for scaling, weak portfolios trigger repair, and late cycles prioritize cash recovery.
  • J Expert-designed Strategies: The operating tree links portfolio constraints to sourcing, route economics, compliance approval, inventory, listings, advertising, and promotion.Compliance gates affected routes, while advertising and promotion activate only after inventory and offer economics are viable.
  • J Expert-designed Strategies: The reserve includes distinct operating styles, including Bayesian export compounding, relationship-led wholesaling, velocity market-making, event trading, brokering, and premium account selling.These strategies respectively emphasize uncertain route updates, relationships, turnover, time-sensitive opportunities, low ownership risk, or quality and account value.
  • J Expert-designed Strategies: 2.06–3.10 times was the strongest strategies’ capital recycling range, compared with 1.67–1.99 times for the two leading model families.The strongest strategies also purchase across 26–52 SKUs, with compounding described as recycling returned cash into later inventory.

K.1 Implementation … L.5 Compliance Reliability

Business Arena uses exact state restoration to support long-horizon evaluation and trace search, while diagnostic profiles decompose business performance into operational, financial, selling, customer-interaction, and compliance measures.

  • K.1 Implementation: Business Arena restores the marketplace, workspace, model-visible context, and tool boundary through a white-box checkpoint harness that verifies digests and reconstructs continuation state.The harness checkpoints post-compaction messages, tool-operation boundaries, provenance, and a content-addressed workspace archive, while hiding checkpoint internals from the model.
  • K.2 Trace Search Case Study: Trace search forks retained traces into continuations and keeps the global top K states, reaching $108,878 with five-day forking versus $93,111 with daily selection.The comparison uses Qwen 3.8 Max Preview with K = 2 and M = 3; branching from retained states outperforms independent episodes under the same rollout budget.
  • L Model Diagnostic Profiles: Figure 4’s diagnostic profiles report repeated-run means and cohort-relative standing to explain where models succeed or fail beyond final net worth.The profiles summarize operational strengths and weaknesses across the business cycle.
  • L.1 Operating Fluency: Operating fluency combines arena-call volume, world-check frequency, and call-failure rate to distinguish active monitoring and reliable interface use from stale assumptions and execution mistakes.Arena calls include operations issued through model-authored scripts.
  • L.2 Capital Deployment: Capital deployment is assessed through peak capital utilization and holding cost, distinguishing active capital use and disciplined turnover from idle capital or stranded, slow-moving inventory.Capital utilization may exceed 100% when capital is recycled.
  • L.3 Selling Performance: Selling performance covers sell-through, order margin, route cost, advertising return, and market share, linking demand capture with full-cost pricing and efficient cross-border operations.These metrics identify stranded inventory, omitted costs, costly routes, unprofitable advertising, and weak competition against NPC sellers.
  • L.4 Customer Interaction: Customer interaction measures buyer conversion, factual-reply quality, and RFQ success, capturing preference inference, evidence-grounded communication, and feasible bulk-offer negotiation.Low scores indicate replies that rarely close sales, unsupported claims, or missed, rejected, and infeasible opportunities.
  • L.5 Compliance Reliability: Compliance reliability is reflected in fines: low or zero penalties indicate reliable operating gates, whereas high fines can overwhelm otherwise plausible commercial performance.Fines sum monetary penalties caused by compliance violations.

M Model Behavior Comparisons

Model comparisons show that business intelligence depends on converting evidence into sustained, action-linked decisions across market shocks, compliance, customer service, and advertising. The strongest behaviors close these loops, while weaker behaviors recognize relevant signals without consistently acting on them.

  • Market shocks: Under the same market shock, Gemini 3.5 Flash updates route-specific pricing and market access, whereas MiniMax M2.5 reopens the route without refreshing cost assumptions.Each decision is linked to its realized order outcome.
  • Compliance: GPT-5.6 Sol checks certification, obtains and verifies the required credential, then reopens US listings with zero compliance violations and no fines.DeepSeek V4 Pro recognizes the missing certification but does not complete corrective action, resulting in substantial fines.
  • Customer service: GLM-5.2 answers buyer questions with supported commercial information, replies to all 31 inquiries, and achieves an 80.65% conversion rate.Gemini 3.1 Pro instead uses largely templated responses that omit central decision criteria and loses the inquiry.
Loading 2608.08621v1…