PaperlayerView exact revision on arXiv

Source-linked AI summary

KiranaBench: Can a Language Model Run a Neighbourhood Shop?

Bhuvan Arora, Ankita Karmakar, Chaitanya Jain, Dilesh Churiwal, Harshit Jain, Shivansh Manish Dubey, Manish Prasad, Raja Piyush, Kshitij Tayal, Dhruv Kumar, Jagat Sesh Challa, Raunak Manekar, Yash Sinha

arXiv:2610.22980v1cs.CLcs.MA

TL;DR

Existing business-operation benchmarks are largely English-language, Western, and single-agent, leaving a gap for long-horizon, multilingual Indian retail settings. KiranaBench simulates a multi-role Indian general store with survey-calibrated demand and English, Hindi, or Hinglish interactions. The evaluated models fall far short of the scripted shopkeeper, and supplier communication performs worse in Hindi and Hinglish than English.

  • Problem

    Existing business-operation benchmarks are largely English-language, Western, single-agent, and provide direct access to business state, unlike the multi-person, multilingual context of Indian kirana shops.

    Source: §1 · p. 1

  • Method

    KiranaBench simulates an Indian general store run by four LLM roles, with English, Hindi, or Hinglish interactions and demand calibrated to India’s Household Consumption Expenditure Survey.

    Source: §1 · p. 1

  • Results

    The best model earns only one-eighth of the fixed-rule shopkeeper’s profit, two models finish below an unmanaged shop, and supplier emails are lost to language about three times as often.

    Source: §1 · p. 1

  • Takeaways & Limitations

    KiranaBench provides evidence that current LLM agents are not yet reliable for autonomously operating small businesses in India.

    Source: §6 · p. 4

    The scripted shopkeeper has direct stock access and writes to suppliers in their own languages, so its gap from the models is not a head-to-head comparison.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly evaluated on long-horizon tasks, including autonomously running a business. Existing benchmarks for this setting are almost exclusively English-language, Western in retail context, and single-agent, with the agent observing the business directly. We study how these results transfer to a multi-agent Indic setting. We introduce KiranaBench, a simulated Indian general store in which four roles (manager, cashier, helper, and bookkeeper) are each played by a separate LLM agent. Interactions with customers and suppliers occur in English, Hindi, or Hinglish, and product demand is calibrated to India's national Household Consumption Expenditure Survey. Evaluating four open-weight LLMs, we find that all fall far short of a fixed-rule shopkeeper baseline: the best model earns only one-eighth of the baseline's profit, and two models finish below an unmanaged shop. Performance is also consistently lower for Hindi and Hinglish than for English supplier communication. KiranaBench offers evidence that current LLM agents are not yet reliable for autonomously operating small businesses in India, and provides a testbed for developing agents suited to settings that are long-horizon, collaborative, and multilingual.

1 Introduction

KiranaBench tests whether LLM agents can run a small Indian shop in a setting that combines long-horizon business decisions with staff-mediated, multilingual interactions. Across four models, performance trails the scripted shopkeeper, and supplier communication is worse in Hindi and Hinglish than English.

1 Introduction

  • Earlier business-operation benchmarks are English-language, Western, single-agent, and give agents direct access to business state, unlike the multi-person, multilingual reality of Indian kirana shops.Table 1 highlights staff-mediated access and report scoring as benchmark dimensions.Source: §1 · p. 1, Table 1 · p. 2
  • KiranaBench simulates a kirana store with four LLM roles—manager, cashier, helper, and bookkeeper—and demand calibrated to India’s Household Consumption Expenditure Survey.Code settles prices, credit, deliveries, spoilage, and the cash book.Source: §1 · p. 1
  • The best model earns only one-eighth of the fixed-rule shopkeeper’s profit, two models finish below an unmanaged shop, and supplier emails are lost to language about three times as often.The benchmark compares four open-weight models with scripted shopkeepers.Source: §1 · p. 1
  • Staff report only to the manager, who cannot observe the shop directly; code computes outcomes and provides ground truth for checking reports.Customers and suppliers communicate in English, Hindi, or Hinglish through a separate fixed model.Source: §1 · p. 1, Figure 1 · p. 2

2 Related Work

Prior work evaluates business-operation agents, delegation and report fidelity, and agentic language abilities, but the cited language-focused evaluations do not run a business over a long horizon.

Agents that run a business

  • Business-operation benchmarks span vending machines, storefronts, supermarkets, enterprise planning, and live commerce, but agents read business state directly and nearly all operate in English.Vending-Bench models have forgotten orders and misread delivery schedules (Backlund and Petersson, 2025).Source: §2 · p. 2

Delegation and the fidelity of reports

Agents in Indian languages

  • Indic-language work covers knowledge, generation, multilingual function calling, translated agent benchmarks, Hindi tool-calling, and short-round bargaining, but none runs a long-horizon business.The cited studies include MILU, IndicGen-Bench, and BhashaBench (Verma et al., 2024; Singh et al., 2024; Devane et al., 2025).Source: §2 · p. 2

3 The Shop

The shop is a four-role operation in which the manager relies on staff reports, while customer demand, supplier interactions, and operating costs are simulated.

Who does what

  • One model fills the manager, cashier, helper, and bookkeeper roles, each with separate tools, a clock, and about 30,000 tokens; the manager learns the day by asking staff.Actions consume working minutes, so counting the storeroom prevents the helper from restocking shelves during that hour.Source: §3 · pp. 2–3

Customers and suppliers

  • About 289 routine customers daily draw baskets from the national consumption survey through a choice model with a no-purchase option, while about 65 live customers interact at the counter.Live customers are voiced in English, Hindi, or Hinglish and remember how they were treated.Source: §3 · p. 3
  • Suppliers communicate by email, may misread orders, send short, or refuse items, and fixed daily rent, electricity, and wages total Rs 4,157.Program code fills factual reply details; only bargaining sentences are generated.Source: §3 · p. 3

4 Setup and Measures

The setup defines profit from coded net worth and evaluates four open-weight models over matched three-seed, 30-day simulations.

Measure

  • Profit is day-30 net worth minus day-1 net worth, with net worth including cash, stock, goods paid for but undelivered, and aged-discounted khata credit, less liabilities and GST owed.The code computes net worth, so models cannot report this measure incorrectly.Source: §4 · p. 3

Models and runs

  • Across three 30-day seeds, four open-weight models faced matched conditions; auditors mechanically checked the cash book, supplier emails, and staff reports daily.Each seed fixes opening stock and cash, customers, and suppliers.Source: §4 · p. 3

5 Results

Models trail scripted management, can lose more than an unmanaged shop, struggle with non-English supplier emails, and vary in the accuracy of staff reports.

5.1 RQ1: against a scripted shopkeeper

  • Qwen3.8’s best average is Rs 13,683, about one-eighth of the scripted shopkeeper’s Rs 1,03,434; gpt-oss and LFM2.5 average below the unmanaged shop.gpt-oss averages −Rs 1,93,411 and LFM2.5 −Rs 1,26,554, versus −Rs 1,14,995 for the unmanaged shop.Source: §5.1 · p. 3, Figure 2 · p. 4

5.2 RQ2: what the wrong language costs

  • Language refusals cost Qwen3.8 5%, Gemma 4 7%, and gpt-oss 46% of all supplier e-mails; Qwen3.8, Gemma 4, and LFM2.5 never fail with English suppliers.Qwen3.8 loses 19% of its Hinglish e-mails, while Gemma 4 loses 42% of its Hindi e-mails.Source: §5.2 · pp. 3–4
  • LFM2.5 is excluded from the language ranking because none of its developers’ ten listed supported languages is Indic.Its Hindi and Hinglish results are reported as floor cases, while its English result is a control.Source: §5.2 · p. 4

5.3 RQ3: are the staff’s reports true?

  • The manager sees the day through staff reports, and sales-report accuracy is scored against the register total within a one-rupee tolerance.A report is counted exact when it agrees with the true register total within one rupee.Source: §5.3 · p. 4
  • Qwen3.8 and Gemma 4 report sales exactly every day (100% each), while gpt-oss is wrong on 6% of days and LFM2.5 on 27%.The worst single-day misstatements are Rs 16,854 for gpt-oss and Rs 15,612 for LFM2.5.Source: §5.3 · p. 4
  • A gpt-oss helper reported a Rs 10,000 payment absent from the cash book, despite having no role tool to make it.Its own munshi contradicted the report that night.Source: §5.3 · p. 4
  • Named failures include writing to nonexistent suppliers, duplicate purchases from two suppliers, and losing the morning dairy van while claiming to take it.The paper traces each loss to a named action and counts each kind per day.Source: §5.3 · p. 4

6 Conclusion

KiranaBench tests whether agents can carry a long business task through staff and unfamiliar languages. The best model averages Rs 13,683 against a scripted shopkeeper’s Rs 1,03,434, and two models finish below an unmanaged shop.

6 Conclusion

  • The best model averages Rs 13,683 against a scripted shopkeeper’s Rs 1,03,434, and two of four finish below doing nothing.The conclusion notes that every identified failure has a name, while a benchmark scoring one session can miss them.Source: §6 · p. 4

Availability

The release includes the environment, scripted shopkeepers, scoring code, 360 day traces, and scripts for rebuilding the paper’s tables and figures.

Availability

  • The release provides the environment, scripted shopkeepers, scoring code, all 360 day traces, and scripts that rebuild every table and figure.The artifact is anonymised for review.Source: Availability · p. 4

Limitations

The reported results are bounded by three-seed comparisons, an advantaged scripted-shopkeeper yardstick, and a limited set of models and one simulated shop. Report fidelity measures whether reports are false, not their financial impact or whether managers check them.

Limitations

  • Every gpt-oss run ranks below every LFM2.5 run, and every LFM2.5 run below every Qwen and Gemma run; Qwen and Gemma cannot be separated confidently.Their spread across seeds is wider than the gap between their means.Source: Limitations · p. 5
  • The scripted shopkeeper has direct stock access and supplier-language matching, so its profit gap is not a head-to-head comparison with the models.A scripted manager using the staff channel is identified as the needed comparison.Source: Limitations · p. 5
  • Report fidelity scores how often reports are false, but does not measure the rupee cost of a false report or whether a manager checks it against the books.Qwen3.8 and Gemma 4 report sales exactly every day, so the channel is truthful in practice for those two models that keep the shop running.Source: Limitations · p. 5
  • The profit ordering covers four open-weight models from four developers, while the language ordering covers three because LFM2.5 is reported but not ranked.The authors say widening the model set is needed to settle whether the orderings agree more broadly.Source: Limitations · p. 5
  • The results concern one Delhi kirana, with one catalogue, supplier set, and festival calendar; customer and supplier wording also varies between runs.The environment can be repointed by changing its demand tables, catalogue, and language mix.Source: Limitations · p. 5

AI Use Statement

The paper reports its AI-assisted writing tools were limited to brainstorming, outlining, and grammar refinement. Its scripted-shop results establish ordered performance anchors, while individual model outcomes vary across seeds.

AI Use Statement

  • AI-assisted writing tools were used only for brainstorming, outlining, and grammar refinement; the authors state that scientific content, analysis, and conclusions are their own.The authors take responsibility for all content, facts, and claims.Source: AI Use Statement · p. 5

A The Score Scale

  • Rs 1,03,434 over 30 days is the sensible shopkeeper’s profit, while an unmanaged shop ends at −Rs 1,14,995.The intermediate scripted policies fall in the expected order, supporting the claim that the environment rewards management rather than a fixed policy.Source: A The Score Scale · p. 6

B Every Run, Seed by Seed

  • Gemma 4’s seed profits range from Rs 37,226 to −Rs 40,762, and every gpt-oss run ranks below LFM2.5, which ranks below Qwen and Gemma.Figure 3 displays all twelve model runs and both scripted shops across the three seeds.Source: B Every Run, Seed by Seed · p. 6

C Measures Scored Apart from Profit

Behavioral measures distinguish models that keep the shop running from those that do not, but revenue-derived measures partly overlap with profit. The measures do not reproduce the profit ordering.

C Measures Scored Apart from Profit

D A Tool-Use Score and the Month

The section tests whether reported tool-use competence transfers to month-long shop operation, then details the role-specific tools, environment model, and evaluation setup. LFM2.5-8B finished below the unmanaged shop on every seed despite its developers’ reported τ 2-bench telecom score.

D A Tool-Use Score and the Month

  • LFM2.5-8B finished below the unmanaged shop on every seed at −Rs 1,26,554, despite its developers’ highest reported τ 2-bench telecom score among these models.The authors limit this claim to transfer between settings, not a ranking, because they did not rerun τ 2-bench and independent reports differ.Source: D A Tool-Use Score and the Month · p. 7
  • gpt-oss repeatedly used nonexistent supplier names and reported an unrecorded Rs 10,000 payment; helpers also mishandled deliveries, and Gemma misread a quantity tool eight times.A duplicated cash-on-delivery order cost Rs 1,60,657 against a till of Rs 75,080.Source: D A Tool-Use Score and the Month · p. 7
  • The behavioral measures separate the two models that keep the shop running from the two that do not, but do not reproduce the profit ranking.LFM2.5 scores worse on most behavioral measures, while gpt-oss loses more money.Source: Table 4 · p. 8

E Tools and Time

  • Role-specific tools consume simulated minutes, while the manager can inspect truthful shop books but must learn staff actions and reports from the other roles.Checking reports against the books also costs time in an already short day.Source: E Tools and Time · pp. 8–9, Table 6 · p. 9

F The Environment Model

  • The environment model was selected through e-mail and customer-language tests; it parses supplier messages into structured intents that the deterministic kernel acts on.The selected Mistral model shared no model family with the systems under test.Source: F The Environment Model · p. 9

G How the Models Were Run

  • The four open-weight entrants were chosen from reported tool-use scores and different model families; LFM2.5 was kept out of the ranking because its listed languages omit Indic languages.The runs used developer-recommended thinking settings, large context and reply limits, and counted tool-call errors as simulated time.Source: G How the Models Were Run · pp. 9–10
  • Runs used multiple machines, but the environment was deterministic for a given seed, and resumed days were checked for byte-identical results.Source: G How the Models Were Run · p. 10

H The Scripted Shops

The section defines fixed scripted shops as anchors for judging management, then describes the demand, financial rules, and measures used to score the benchmark. The shop policies and household demand are specified separately from model behavior.

H The Scripted Shops

  • The reference shop uses fixed stocking, purchasing, cash-buffer, festival, and customer-request rules, providing a policy-based comparison on the same environment and seeds as the models.It earns money under sensible management, while a strict printed-price cashier earns less and pricing shelves at MRP loses money.Source: H The Scripted Shops · p. 10
  • The scripted shop is a yardstick for shop operation, not customer conversation: it uses fixed bargaining and supplier-language templates and avoids language-related e-mail refusals.The reference cashier never goes below cost and grants credit under a fixed visit-and-balance rule.Source: H The Scripted Shops · pp. 10–11

I Demand and Customers

  • Routine and live-customer demand draws on household baskets calibrated to India’s HCES urban Delhi data, with a no-purchase option and a pre-run check within 10% of survey quantities.Routine customers buy only available goods at acceptable prices; live customers bargain, request credit, complain, and ask for new stock.Source: I Demand and Customers · p. 11
  • The 30-day calendar covers Navratri and Dussehra, with demand shifts that require stocking ahead of festival dates.Households also keep depleting pantries, and some live customers request products the shop does not yet stock.Source: I Demand and Customers · pp. 11–12
  • Festival demand changes by season, while loyalty reflects treatment and affects later visit frequency and basket size; the festival multipliers and loyalty magnitudes are stated assumptions.The authors report no research-grade series for Indian festival retail demand, and describe loyalty magnitudes as estimates rather than measurements.Source: I Demand and Customers · p. 12

J How Money Works in the Shop

  • Credit repayment is fixed by household records, and some reliable borrowing late in the 30-day run is not repaid before it ends.Net worth discounts khata balances by age rather than treating every uncollected balance as bad credit.Source: J How Money Works in the Shop · p. 12
  • Daily running costs are Rs 4,157 regardless of trade, while spoilage, GST liabilities, payables, stock, and age-valued credit enter the net-worth calculation.Profit is day-30 net worth minus day-1 net worth.Source: J How Money Works in the Shop · p. 12

K Measures

  • Profit is the headline measure; separate measures track reporting accuracy, credit decisions, fair service, captured shelf sales, and customers lost.The audit also records spoilage, transport, overdue bills, wait times, and whether customer requests were relayed and acted on.Source: K Measures · p. 13

L Language in Detail

The section specifies how supplier e-mails are classified by language and refused when they do not match a supplier’s language. Transcript examples and a day-level timeline illustrate operational errors and the roles’ time costs.

L Language in Detail

  • E-mails are classified as Hindi using Devanagari counts; otherwise, after removing order forms and identifiers, romanised Hindi words distinguish English from Hinglish.Suppliers refuse messages classified in a language they cannot read.Source: L Language in Detail · p. 13
  • The three ranked models had more supplier e-mails refused in Hindi and Hinglish than English, and refusal rates followed the same model order as profit.Three models never failed in English, while gpt-oss failed on 23% of its English e-mails.Source: Figure 4 · p. 13

M Transcripts

N One Simulated Day

  • Figure 14 presents one simulated day as four role lanes, where bar width represents minutes spent and gaps in the manager’s lane represent time it chose to wait.Customer purchases and supplier e-mails are marked around the lanes; an early missed delivery or unattended counter cannot be recovered later that day.Source: N One Simulated Day · p. 15

O How Often Each Failure Occurs

Failure patterns differ by model, while cash and stock trajectories show how poor operating decisions can leave shops unable to pay bills. The shared seeds and suppliers make reporting fidelity a model-specific result.

O How Often Each Failure Occurs

  • The passage attributes sales-report fidelity to the model rather than the testbed: all four comparison columns were produced with the same shop, seeds, and suppliers.Source: O How Often Each Failure Occurs · p. 16

P Cash and Stock Over the Month

O How Often Each Failure Occurs

  • Invalid actions and below-cost sales dominate gpt-oss, while wrong sales reports and empty-shelf losses dominate LFM2.5.Figure 15 counts failures per simulated day on a logarithmic scale; Qwen3.8 and Gemma 4 fail rarely enough that their bars sit near the floor.Source: O How Often Each Failure Occurs · p. 15, Figure 15 · p. 17

Q Audit Method

This section describes the audit setup for the multi-agent shop, including role-specific prompts and tools, environment responsibilities, and rules for parsing supplier emails. It also reports the audit’s separation of model mistakes from environment faults and its finding of no environment money-rule errors in the analyzed runs.

Q Audit Method

  • The audit followed days in order, checked cash, supplier messages, staff reports, and environment rules, and recorded faults or model mistakes with timestamps and trace quotations.Source: Q Audit Method · pp. 17–18
  • No environment money calculation or rule errors were found, and the cash book balanced on every analyzed day.Source: Q Audit Method · p. 18

R Prompts

  • Prompts were rendered from frozen run code; models received Devanagari in its original script, although it is elided in the displayed prompts.Appendix M shows the prompts as the models saw them.Source: R Prompts · p. 18

How a call is made

  • Each model call combines a role prompt, relevant prior turns, the latest event, and native tool definitions; tool results return to the model until it ends its turn.Prompts have no worked examples; a failed call costs one minute, and three consecutive errors end the turn.Source: How a call is made · p. 18

Manager (the model under test)

  • The manager prompt makes the agent responsible for running the Delhi shop through tools, directing staff, managing prices and suppliers, and pursuing net worth despite limited persistent memory.Old exchanges are dropped when memory fills; scratchpad notes persist.Source: Manager (the model under test) · pp. 18–19

Cashier (the model under test)

  • The cashier serves customers one at a time, may bargain only down from the bill, must wait for a khata request before deciding credit, and reports daily sales.The cashier is instructed to speak briefly in each customer's language.Source: Cashier (the model under test) · p. 20
  • The cashier can check khata histories, negotiate bills only downward, offer substitute packs, and approve khata only for the whole agreed bill.Manager instructions can be saved as lasting rules; urgent messages reach and wake the manager.Source: Cashier (the model under test) · p. 21
  • The cashier can review sales and cash-book entries, report daily cash-and-khata sales, and must answer the manager using only verified facts.The cash book records rupee inflows and outflows in time order, including sales, supplier payments, rent, wages, and GST.Source: Cashier (the model under test) · p. 21

Helper (the model under test)

  • The helper follows the manager’s shelf plan, tops up shelves from the godown, and handles deliveries under rules for ordered goods, capacity, and payment.Milk stays in the fridge, while the helper may choose how much of the daily dairy delivery to take; supplier deliveries generally must be accepted up to what arrived.Source: Helper (the model under test) · p. 22
  • The helper receives the manager’s shelf plan and is instructed to carry it out with tools, using set_plan to record the requested quantities.The example request is to set doodh_500ml to 30, bread to 10, and atta_5kg to 4.Source: Helper (the model under test) · p. 24

Munshi (the model under test)

  • The munshi keeps the books, reviews records after closing, and reports sales, customer khata owed, and GST owed to the manager.The cash book records money movements, including sales at both counters, supplier payments and transport, overheads, and GST; many payments occur automatically from the till.Source: Munshi (the model under test) · p. 24

The environment model (customers, suppliers, parser)

  • The environment model writes words only: the simulator supplies numbers and decisions, and checks each generated line with one retry before using a template.Its prompts specify customer speech and parse supplier emails in English, Hindi, or Hinglish into structured order intents and item lines.Source: The environment model (customers,… · pp. 25–26
  • Supplier emails are classified by whether they place a new order, follow up on an existing order, or ask for information, and only requests for the named supplier count.A request for goods not already on order is an order even if phrased as a confirmation; prior unread or refused emails do not establish an order.Source: The environment model (customers,… · p. 26
  • The parser reports pack quantities and unit prices only when written in the new email, using null for unstated values and unknown for ambiguous or unlisted goods.It does not estimate or convert values, and reports only what the new email asks rather than earlier context.Source: The environment model (customers,… · p. 26
  • Customer prompts vary among English, Hindi, and Hinglish and specify persona and register, while supplier prompts specify tone and language.Hindi is prompted in Devanagari and Hinglish in Latin script.Source: The environment model (customers,… · pp. 26–27
Loading 2610.22980v1…