PaperlayerView exact revision on arXiv

Source-linked AI summary

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

Remco Hendriks

arXiv:2609.10016v1cs.LGcs.AIcs.CL

TL;DR

Transit kiosks require policy logic that is correct, renderable, adaptable, and auditable under changing rules and adversarial passenger scenarios. MetroLLM-Bench evaluates language models in this setting with structured tools, terminal-state outputs, deterministic and semantic scoring, and held-out testing. On the bounded task, a 2.6 GB 4B Qwen 3.5 PEFT student matches or exceeds the cited frontier baselines on held-out Tier 1, while the authors delimit that conclusion to the tested task and training recipe.

  • Problem

    Transit-kiosk policy changes traditionally require code deployment, while existing transit and general agent benchmarks provide limited evidence about operational decisions under disruption, multi-turn context, and adversarial input.

  • Method

    MetroLLM-Bench evaluates models on 955 cases across six metro systems using structured tool calls, renderable terminal states, fourteen deterministic Tier 1 components, eight semantic Tier 2 components, and a stratified held-out split.

  • Results

    91.32 Tier 1 points from the 4B Qwen 3.5 PEFT student exceed GPT-5.6 luna at 90.63 and sol at 90.00, while matching GPT-5.4 full at 91.37.

  • Takeaways & Limitations

    On this bounded kiosk task, an open-weight model adapted to the setting can match a frontier API on core executable requirements without sacrificing deterministic performance.

  • Takeaways & Limitations

    The capacity-ceiling result is specific to the tested scoring contract and recipe—QLoRA rank 16, 600 traces, and three epochs—and need not hold for other tasks, datasets, or adapter settings.

Abstract

from arXiv · show

We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk. It covers six real metro systems, ranging from 37 to 414 stations, and eleven categories that include routing, fare calculation, disruptions, accessibility, and adversarial input. In each case, the model must call structured tools and submit a machine-renderable terminal state containing an outcome, a per-ticket fare quote when applicable, and a kiosk action. Fourteen deterministic scoring components form Tier 1; eight semantic-quality components form Tier 2, six of which use a language-model judge. We report Tier 1 and the combined score of both tiers. A stratified 75/25 split reserves 717 cases for training-data generation and 238 for held-out evaluation. We evaluate twenty-six models from six vendors, of which twenty-three are ranked. On the held-out partition, a 4B Qwen 3.5 student trained through parameter-efficient fine-tuning (PEFT) exceeds both GPT-5.6 tiers on Tier 1 (91.3 against 90.6 and 90.0) and matches GPT-5.4 full at maximum reasoning effort (91.4), with a 2.6 GB Q4_K_M footprint. Larger 9B and 27B students provide no further Tier 1 improvement over the 4B student at this training scale. Across the four Qwen sizes, the PEFT gain over the corresponding base model decreases from +7.03 points at 2B (three training seeds) to -0.91 at 27B; every seed shows the same direction at every size. A deterministic rule-based baseline reaches 84.6 on Tier 1, with the remaining language-model advantage concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning. Muse Glimmer 30B leads the composite ranking, and serving configuration alone moves the Qwen 3.5-to-3.8 comparison by 2.7 Tier 1 points. The benchmark, harness, reproduction guide, and fine-tuned students are released at https://github.com/continker/metrollm-bench.

1 Introduction

MetroLLM-Bench examines whether language models can replace coded transit-kiosk policy logic while remaining correct, renderable, adaptable, and auditable under routine and adversarial conditions. It introduces a benchmark and reports strong held-out performance from a compact fine-tuned student.

  • Motivation: MetroLLM-Bench tests language-model policy runtimes that read natural-language system descriptions, call structured tools, and emit kiosk-renderable terminal states.The alternative replaces policy logic that would otherwise require code changes and deployment cycles.
  • Motivation: Correctness, fixed display constraints, changing operational rules, and auditability make transit kiosks a focused testbed for this evaluation.The interaction is short and goal-directed, while operational rules change often enough for code-defined logic costs to matter.
  • Research gap: Disruption, multi-turn context, and adversarial input remain open evaluation questions because passenger-facing kiosks routinely encounter all three.Prior transit benchmarks address planning, travel choice, or data semantics but not the full operational-decision setting.
  • Contributions: 955 cases across six metro systems separate deterministic scoring from semantic quality and use a fixed system-stratified held-out split.The benchmark also calibrates its deployed scoring stack against two independent human annotators.
  • Results: A 4B PEFT student fits in 2.6 GB, exceeds both GPT-5.6 tiers on held-out Tier 1, and matches GPT-5.4 full at maximum reasoning effort.Nothing trained above 4B measurably improves on that result, while the language-model advantage is concentrated in decision-heavy categories such as policy adaptation, accessibility, temporal reasoning, and compound scenarios.

2 Benchmark Design

The benchmark constructs controlled kiosk interactions across diverse metro systems, runs models through a bounded tool-calling loop, and scores both deterministic execution and semantic quality. Its held-out design and calibration expose important limits around template overlap, judge disagreement, and interpretation of safety.

  • Runtime: Each case assembles a framebook and scenario events, permits up to twenty tool rounds, and ends with a structured terminal state containing an outcome, kiosk action, and conditional route and fare fields.The mock server validates submissions and returns field-level errors that the model may correct within the remaining budget.
  • Benchmark coverage: Six metro systems span three fare models, four currencies, and networks ranging from 37 to 414 stations.The systems include MARTA, Doha Metro, BART, Taipei MRT, CTA, and Beijing Subway.
  • Benchmark coverage: The benchmark combines routine requests with system-specific disruptions, policies, terminology, languages, and operating patterns across eleven categories.Framebooks provide operational rules at runtime so the same model can operate under six rule sets without code changes.
  • Scoring: Fourteen deterministic Tier 1 components measure executable correctness, while eight Tier 2 components measure semantic quality and include model-judged criteria.Composite scores are unweighted means across the six per-system means, whereas category and bootstrap summaries use different aggregation units.
  • Evaluation design: 40 percent of 149 held-out cases with defined station pairs have no training-set neighbour sharing that pair, but structural template overlap remains possible.The held-out matrix is the primary generalisation evaluation; the full matrix includes cases used for training-data generation and is not independent evidence.
  • Scoring calibration: κw = 0.53 for judge–author agreement exceeds κw = 0.25 between the two human raters, but safety-rubric disagreements reveal a substantive interpretation split.The clearest disagreement concerns strict constraint adherence versus outcome utility in adversarial scenic-route cases.
  • Scoring calibration: Class imbalance reduces κ on some rubrics, and cross-judge calibration with a non-Anthropic model remains future work.The central deployment comparison therefore relies on deterministic Tier 1 rather than Tier 2 alone.

3 Evaluation

The evaluation compares twenty-six models across held-out leaderboard, category, baseline, and serving-configuration results. A compact 4B PEFT student is highly competitive, while category strengths, benchmark limitations, and serving choices materially affect interpretation.

  • Leaderboard: Muse Glimmer 30B leads the composite ranking, while Qwen3.6-27B leads Tier 1 at 93.63 and the top eleven rows span only 3.18 composite points.No adjacent gap among the top eleven exceeds 0.68 points, and only two of the six highest-ranked rows are proprietary OpenAI models.
  • Leaderboard: 91.32 Tier 1 points from the 2.6 GB 4B PEFT student exceed both GPT-5.6 rows and nearly match GPT-5.4 full at xhigh effort.The student scores 90.63 against GPT-5.6 luna, 90.00 against GPT-5.6 sol, and 91.37 for GPT-5.4 full at xhigh effort.
  • Results by category: Category leadership is split: Qwen 27B base leads Fare, Disruption, Accessibility, Cultural, and Policy, whereas GPT-5.4 xhigh leads Routing, Adversarial, Temporal, and Compound Stress.The Multi-turn and Tool-Hallucination scores differ by less than one point, and the largest reported gaps favor Accessibility for Qwen and Temporal for GPT-5.4.
  • Results by category: The 4B PEFT student has the best Fare score at 97.7, shares the best Policy score at 96.9, and trails only sol on Routing.Its main weaknesses are Temporal and Tool-Hallucination; the 27B PEFT student shows no category improvement over its base beyond the single-run interval.
  • Rule-based baseline: 84.6 Tier 1 points from the scripted baseline show strong routine routing and fare performance, but language-model gains concentrate on decision-heavy categories.The baseline scores 93.5 on Routing and 91.7 on Fare, versus 52.1 on Temporal, 68.9 on Compound Stress, 69.7 on Accessibility, and 73.3 on Policy.
  • Serving-configuration sensitivity: 2.7 of the apparent 3.6-point Qwen3.5-to-Qwen3.8 regression is attributable to serving configuration rather than capability.Longer output budgets prevent truncation, while recommended sampling mitigates repetition loops; the configuration optimum differs across generations.

4 PEFT and the Capacity-Ceiling Curve

The PEFT sweep shows a capacity ceiling: gains decline monotonically with model size, while the 4B student reaches the measured Tier 1 plateau. This pattern is consistent across seeds and evaluation partitions but remains specific to the tested recipe.

  • PEFT gain by size: +6.03, +1.72, +1.09, and −1.07 are the full-matrix Tier 1 PEFT deltas for 2B, 4B, 9B, and 27B, respectively.Every seed agrees on the direction at every size: improvement through 9B and regression at 27B.
  • Seed sensitivity: Full-matrix seed spread falls from 3.96 points at 2B to 0.33 at 4B, 0.17 at 9B, and 0.09 at 27B.The 2B result depends materially on seed, whereas the two 27B runs are nearly identical.
  • Scope: The capacity-ceiling conclusion is limited to QLoRA rank 16, three epochs, and 600 teacher traces from the training partition.The 27B student also uses a 2048-token maximum sequence length because of the hardware constraint.

5 Related Work

MetroLLM-Bench occupies a middle ground between broad agent evaluation and short-prompt structured-output benchmarks. It narrows the operational domain while requiring tool-grounded decisions and a consistent terminal state.

  • Positioning: MetroLLM-Bench narrows the action space to six tools in one operational domain, enabling diagnostic categories and reuse of deterministic components during fine-tuning.This contrasts with broader agent benchmarks that test multi-step tool use across many domains.
  • Structured evaluation: Unlike schema-validity benchmarks, it requires the route, fare, outcome, and purchase decision to agree with preceding tool calls.The terminal contract is evaluated together with the decisions that produced it.
  • PEFT comparison: Prior PEFT studies report gains from distilled trajectories, but this work contributes a four-size sweep showing that adapter-training benefits can change sign.The sweep uses two to three seeds per size.
  • Transit-specific gap: Earlier transit studies use language models for trip planning, travel-choice prediction, or GTFS understanding rather than as a kiosk policy layer.MetroLLM-Bench evaluates tool calling followed by commitment to a machine-renderable operational state.

6 Discussion and Limitations

The discussion places the results within a deliberately bounded kiosk task: 4B reaches the measured plateau, but aggregate parity should not be generalized beyond this scoring contract, recipe, or task scope.

  • Task boundary: The task uses six tools, typically three to seven calls within twenty rounds, and a constrained terminal state, with deterministic ground truth for routing and fare arithmetic.The setting does not strongly reward long-horizon reasoning.
  • Category variation: GPT-5.4 full at xhigh reaches 87.2 composite in Temporal reasoning against 73.5 for Qwen 27B base and 68.6 for GPT-5.6 sol.The widest category spread favors one frontier configuration rather than reasoning effort generally.
  • Measured plateau: The 4B, 9B, and 27B students score 91.32, 91.03, and 91.41 on held-out Tier 1, while the 4B model fits in 2.6 GB.The benchmark does not establish an end-to-end kiosk latency requirement.
  • Statistical limits: Most non-PEFT models have one run, and bootstrap half-widths are about one Tier 1 point on the full matrix and two on the held-out partition.Differences below one point should not be treated as meaningful.
  • Benchmark resolution: The leading eleven models span 3.18 composite points, while Accessibility and Temporal still separate leaders by 13.5 and 18.6 points.Extending the measured range upward would require harder compound and temporal cases.
  • Generalization boundary: The capacity-ceiling result is specific to this scoring contract and recipe, including QLoRA rank 16, 600 traces, and three epochs.It need not hold for other tasks, datasets, or adapter settings.
  • Scaling interpretation: The 2B-to-4B jump is reproducible within Qwen and points in the same direction for Gemma, but it is not a general law because architecture and training data remain confounders.Qwen 35B-A3B ranks seventh overall despite having only 3B active parameters.

7 Conclusion

On this bounded kiosk task, a 2.6 GB open-weight 4B student reaches frontier-level deterministic performance without proprietary infrastructure. The result supports sufficiency at 4B, not general replacement of frontier models.

  • Deployment result: The 2.6 GB Qwen 3.5 4B PEFT student scores 91.32 on Tier 1, above GPT-5.6 luna at 90.63 and sol at 90.00.It matches GPT-5.4 full at xhigh effort at 91.37.
  • Sufficiency: Neither 9B nor 27B PEFT improves on the 4B Tier 1 plateau, while 27B adaptation significantly reduces Tier 1 relative to its base on the full matrix.The nine runs took 9.4 GPU-hours on one high-end consumer GPU.
  • Boundary: The 4B student trails GPT-5.6 luna, GPT-5.6 sol, and GPT-5.4 xhigh on composite score, with frontier advantages on Adversarial and Temporal cases.The 27B regression is specific to the tested recipe, including its shorter training sequence length.
  • Reproducibility: The benchmark, harness, reproduction guide, and fine-tuned students are released, while local inference and PEFT training use a single RTX 5090.The repository maintains the full command sequence from partition construction through evaluation and statistics.

B.4 Model configurations

Evaluation-time configurations vary by model family, endpoint, sampling, reasoning effort, and serving build, so these settings materially qualify cross-model comparisons.

  • Model and endpoint settings: GPT-5 rows use temperature 1.0 because the author’s Azure subscription tier accepts no other value for those deployments.GPT-5.4 full was evaluated at three reasoning-effort levels, while nano and mini used medium effort.
  • Model and endpoint settings: Local and Mistral evaluations use temperature 0.0, while Qwen3.8-27B is the only local row served with sampling.The evaluation does not separately measure run-to-run variance; local server spread is measured at 0.23 Tier 1 points per system.
  • Model and endpoint settings: GPT-5.6 uses Azure OpenAI’s Responses API because the chat-completions endpoint rejects function tools combined with reasoning effort.The harness converts the Responses payload to the chat-completions shape before recording, leaving scoring unchanged.
  • Serving configuration: Rows added after v1 use llama.cpp b10398, whereas v1 rows use b8642, making the serving build part of the reported configuration.Re-running Qwen3.5-27B on b10398 produced 93.08 held-out Tier 1 versus 92.32 on b8642 under otherwise unchanged settings.
  • Harness and case generation: The evaluation harness uses per-model serving configurations with twenty tool rounds, while cases are generated from system metadata, disruption templates, route graphs, and fare calculators.Generation-time checks cover required fields, identifiers, station references, and graph-valid routes.
  • Hardware measurements: Exploratory Apple Silicon throughput measurements are separate from the RTX 5090 setup and do not constitute an end-to-end kiosk latency study.The fanless M2 Air sustains 39 tokens per second for 2B Q4_K_M decoding after thermal stabilisation, while the M2 Max reaches 108.

C.1 Training-set composition

Training data is built from high-scoring teacher traces on the training partition, with category coverage determined by teacher success rather than balanced sampling.

  • Trace selection: 600 deduplicated teacher traces are retained after requiring at least 90 percent Tier 1 performance on the 717-case training partition.The final set contains 540 traces from the dense 27B teacher and 60 from the 35B-A3B teacher, with mean Tier 1 of 99.0 percent.
  • Coverage: The Tier 1 threshold is a capability filter, not a category filter, so training coverage reflects where the teachers succeed.Temporal reasoning is undersampled because the base teachers score about 83 percent there, while Compound Stress has high proportional coverage but only nineteen training cases.
  • Training recipe: The maximum sequence length is reduced from 4096 to 2048 for the 27B model to fit within 32 GB of VRAM.
  • Training recipe: All four students use QLoRA with 4-bit NF4 base quantisation, LoRA rank 16, three epochs, and effective batch size 8.Training uses seeds 42 and 43 at every size, with seed 44 added at 2B.
  • Training variability: Re-running the 2B artefact shows about one held-out point of run-to-run variation, several times the 0.23-point per-system spread measured for 27B models.The reported 2B reruns move held-out scores by 2.1, 0.3, and 1.1 points across the tested runs.
  • Artefacts: Training time and final artefact sizes are reported per seed and student size, with merging, GGUF conversion, and Q4_K_M quantisation following scripted repository procedures.
  • Partitioning: A system-stratified seed-42 split assigns 717 cases to training-data generation and reserves 238 cases for held-out reporting.Fifteen gap-audit cases added after the first PEFT cycle are pinned to the held-out partition.

D Statistical methodology

The analysis combines conservative case-level uncertainty estimates with paired bootstrap intervals, while calibrating semantic judges against independent human ratings and documenting remaining limitations.

  • Uncertainty estimates: At 90 percent accuracy, the Bernoulli standard-error bound is approximately 0.97 points for 955 cases and 1.9 points for 238 held-out cases.The corresponding 95 percent half-widths are about 1.9 and 3.8 points, and differences below one point remain within measurement noise on either partition.
  • Bootstrap procedure: PEFT rows use the per-case mean over two seeds, and bootstrap intervals are reproduced with 5,000 resamples for the full and held-out partitions.
  • Bootstrap intervals: Full-matrix Tier 1 intervals have half-widths of about 1.0 point, while held-out intervals are roughly twice as wide because only 238 cases are available.The wider held-out intervals explain why individual held-out differences do not reach significance.
  • Paired comparisons: The paired bootstrap preserves model pairing, and an interval excluding zero is treated as significant at α = 0.05.Table 10 reports four principal comparisons on both full and held-out partitions.
  • Paired comparisons: The full-matrix 4B PEFT gain and 27B regression are significant on both Tier 1 and composite score, while every held-out interval includes zero.The 4B student ties GPT-5.4 full xhigh on Tier 1 at +0.01 [−0.93, +0.94].
  • Scope of inference: Bootstrap uncertainty captures case sampling only; training-seed variation is analysed separately, and per-category differences are not paired-tested.Held-out category counts range from 11 to 32, so some apparent category differences may be sampling noise.
  • Judge calibration: Six Tier 2 components use Claude Haiku 4.5 language-model judgments, with all judgments cached during evaluation.No Anthropic model is evaluated as an agent, avoiding an Anthropic agent–judge comparison.
  • Judge calibration: Judge calibration used two independent annotators rating the same 100 case-rubric pairs sampled across six systems and ten categories.The second annotator saw no judge output during rating.

F Limitations and threats to validity

The benchmark’s claims are bounded by its authored scenarios, English-only interaction, selected model matrix, answer-key defects, and uncertainty in judges, contamination, and generalisation.

  • Measurement scope: The benchmark tests structured transit-tool use and terminal-state synthesis, with Tier 2 adding semantic-quality measures through six judged and two programmatic components.
  • Measurement scope: It does not measure end-user satisfaction, payment or ADA-device integration, multi-session continuity, PII handling, strict-latency behaviour, or general adversarial robustness.All responses are in English, including cases for non-English systems.
  • Claim scope: The results do not establish PEFT as universally better than other adaptation methods or show that very small models replace large ones.Performance falls sharply below the 4B Qwen base.
  • Answer-key validity: Four answer-key issues were retained, including two non-adjacent disruption closures that leave routes unchanged and could affect the expected outcome by at most 0.3 composite points.
  • Model inclusion: Gemma 4 E2B and E4B are excluded because 28 to 33 percent of cases exhaust the twenty-round budget without a valid terminal submission.Their resulting floor-effect zeros are not comparable with normally terminating models.
  • Model inclusion: The model matrix was assembled from systems available locally or through APIs accessible to the author, with Claude agents omitted partly to avoid same-family agent–judge comparison.
  • Generalisation across systems: Held-out results do not show a clear US-versus-non-US pretraining familiarity advantage, but per-system samples contain only about forty cases and remain noisy.Four models score more than a point higher on non-US systems, while the 4B and 27B students are level within 0.05 points.
  • Training dependence: The PEFT students are same-family distillations from Qwen teachers, and no student exceeds either teacher on held-out cases.The 4B student scores 91.3 Tier 1 versus 92.3 for the 27B teacher, indicating compression to teacher-class quality rather than teacher surpassing.

G Data and model artefacts

MetroLLM-Bench releases the benchmark, evaluation infrastructure, training pipeline, and model artefacts needed to reproduce its experiments end to end.

  • Benchmark and evaluation artefacts: The repository includes six system datasets and framebooks, 955 committed cases with train/held-out variants, the partition specification, and the complete evaluation harness.The harness includes the mock server, runner, scorer, judge, and rule-based agent.
  • Training and model artefacts: The release provides PEFT scripts, a step-by-step reproduction guide, and four Qwen3.5 students spanning 2B, 4B, 9B, and 27B.Each student includes a LoRA adapter, Q4_K_M GGUF, and model card; an Apple Silicon measurement package is also released.
  • Reproducibility: Per-case scores, judge caches, and the 600-trace training set are archived or regenerable, making every reported table and figure traceable to underlying evaluation records.The paper states that the scored result files and judge caches underlying every number are archived by the author, while the training set is regenerable from the repository.
Loading 2609.10016v1…