CostNav: A Navigation Benchmark for Real-World Economic-Cost Evaluation of Physical AI Agents

• Existing benchmarks emphasize success rate, collision rate, path length, and navigation time while leaving the cost-revenue question of profitable deployment unanswered.Source: §1 · p. 2
• CostNav introduces an Economic Navigation Benchmark that evaluates embodied physical AI agents through economic viability rather than task metrics alone.The benchmark frames navigation around cost-revenue outcomes for autonomous food delivery.Source: §1 · p. 2
• Profit per run combines delivery revenue with operating cost, while the benchmark models CAPEX, OPEX, revenue, and break-even time using public industry references.Inputs include SEC filings, AIS injury reports, and commercial delivery pricing rather than simulator-defined values.Source: §1 · pp. 2–3
• CostNav grounds economic metrics in Isaac Sim collision and cargo dynamics, linking collision impulse and Δv to injury data while modeling food spoilage and pedestrian liability.A sim-trained CANVAS policy also shows SLA compliance close to simulation on a physical Segway E1 route.Source: §1 · p. 2
• The benchmark is fully open-sourced with scenarios, cost models, baselines, training resources, datasets, and evaluation code.Source: §1 · p. 3

Navigation Benchmarks and Systems

• Existing embodied-navigation benchmarks mostly score success rate, SPL, and collision-related measures, while DeliveryBench uses simulator-defined costs without autonomous-robot CAPEX/OPEX references.CostNav combines collision and cargo dynamics with industry-grounded cost references.Source: §2 · p. 3

Cost-Aware Robotics

• Cost-aware robotics studies optimize isolated components such as energy, while industry filings and deployment reviews do not provide a unified simulation-driven economic model.Source: §2 · p. 3

Economic Evaluation of AI

• Economic evaluations of AI attach monetary or labor value mainly to digital tasks, whereas CostNav applies value-oriented evaluation to embodied navigation.Source: §2 · p. 3

Simulation Platforms for Robotics

• Robotics simulators span game engines, physics platforms, and hybrid systems, while CostNav uses Isaac Sim to unite high-fidelity physics and rendering for cost estimation.Source: §2 · p. 3

3 CostNav: Cost-Aware Navigation Benchmark for Embodied Agents

• CostNav translates simulation logs into profit curves and break-even analysis, complementing conventional task-oriented navigation metrics with financial performance.The benchmark connects simulator behavior to real-world economic models for direct comparison of navigation methods.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Economic Model

• The economic model spans capital expenditure, operational expenditure, and delivery revenue to compare navigation methods by economic viability.Capital costs occur before deployment, while operational costs and revenue are evaluated per delivery.Source: §3.1 · p. 4, §3.1.1 · p. 4

3.1.1 Cost Framework Overview

• Negative contribution margin makes break-even undefined because each delivery increases cumulative losses; lower positive break-even values indicate faster recovery of upfront investment.Contribution margin is revenue minus operating expenditure, and break-even measures deliveries required to recover capital expenditure.Source: §3.1.1 · p. 4

3.1.2 Capital Expenditure (CAPEX)

• CAPEX includes upfront hardware investment and learning-based data-collection costs, while occupancy-map and environment-survey costs for rule-based methods remain future extensions.Hardware includes the robot, sensors, compute, chassis, battery, and supporting electronics.Source: §3.1.2 · pp. 4–5

3.1.3 Operational Expenditure (OPEX)

• OPEX links navigation behavior to electricity, repairs, customer compensation, pedestrian liability, and property damage incurred during delivery runs.Service compensation accounts for spoiled cargo, timeouts, and physical-assistance failures; pedestrian injury costs use AIS severity probabilities derived from impact-speed changes.Source: §3.1.3 · p. 5

3.2 Simulation Environment

• CostNav simulates a commercially deployed Segway E1 in a 200 m × 200 m urban sidewalk network with pedestrians and priced infrastructure obstacles.The platform’s publicly documented hardware and optional LiDAR and GPS prices ground configuration-specific capital expenditure in real market values.Source: §3.2 · p. 6

4.2 Simulation Results

• All seven methods yield negative contribution margins, so none reach break-even in the urban sidewalk evaluation.The economic results are derived from 100 episodes and convert physical measurements into CAPEX, OPEX, revenue, and contribution margin.Source: §4.2 · p. 6, Table 3 · p. 7
• CANVAS attains 70% SLA compliance and 0% timeout, whereas Nav2 reaches 43% with AMCL and 46% using GPS-only localization.CANVAS learns direct velocity commands from RGB observations and a sketch goal, while Nav2 uses classical planning and LiDAR-based costmaps.Source: §4.2 · p. 6
• Nav2’s forward-speed tuning makes pedestrian encounters trigger lateral evasive maneuvers, raising food spoilage to 29% in the GPS-only variant.Source: §4.2 · p. 7
• GNM, NoMaD, and NavDP achieve 0%, 1%, and 0% SLA compliance with timeout rates of 84%, 78%, and 91%, while ViNT reaches 10% SLA compliance but needs physical assistance in 36% of runs.Image-based policies lose localization when deviations move observations out of distribution; NavDP lacks intermediate guidance for long-horizon routes.Source: §4.2 · p. 7

4.3 Sim-to-Real Performance Experiment

• CANVAS achieves 63% SLA compliance across eight real delivery scenarios, close to its 70% simulation rate.Failures involved vegetation or fence contact, and one vegetation contact spoiled cargo.Source: §4.3 · p. 8, Table 4 · p. 8

• Across seven rule-based and learning-based baselines, every evaluated method yields a negative contribution margin, even the method with the highest task success.Source: §5 · p. 8
• CostNav connects simulator-measured behavior to real-world referenced costs and revenue, enabling contribution-margin and break-even analysis for autonomous delivery.Source: §5 · p. 8

6 Limitations

• Simulation event counts are only partially validated against deployment, and pedestrian injury costs use a mass-rescaled proxy, so close rankings should not be over-interpreted.The viability conclusion remains negative for every method even when pedestrian safety cost is removed.Source: §6 · p. 8
• The economic model omits training compute, rule-based mapping costs, demand, dynamic pricing, and failures such as battery depletion.Source: §6 · pp. 8–9
• Evaluation covers only the Segway E1, popcorn cargo, an urban sidewalk map, and daytime conditions, so results are not each method’s absolute ceiling.Further method-specific optimization may improve performance.Source: §6 · p. 9

A Cost Parameter Details

• Electro-mechanical efficiency is derived as 0.73 from battery, motor, inverter, and charge efficiencies.Source: A Cost Parameter Details · p. 16
• The benchmark uses public cost parameters for pedestrian injury, property damage, robot operation, delivery timing, distance, and energy efficiency.Examples include AIS injury costs, municipal property-repair costs, manufacturer robot specifications, and SEC-derived delivery distance.Source: A Cost Parameter Details · pp. 15–16

A.1 Source Data Figures

• The appendix provides source figures for robot and sensor costs, operational rates, delivery performance, damage costs, robot dynamics, and energy-conversion parameters.Source: Figure 3 · p. 17, Figure 4 · p. 17, Figure 5 · p. 17

B Baseline Details

• Table 6 summarizes sensor inputs, pretraining data, CostNav finetuning, and goal conditioning for seven evaluated baselines.Source: Table 6 · p. 31, B Baseline Details · p. 31

B.1 Rule-Based Navigation Baseline

• Nav2 is evaluated with AMCL and GPS-only localization, using classical map-based planning and LiDAR costmaps for obstacle avoidance.Source: §B.1 · p. 31

B.2 Learning-Based Baselines: Pretraining Source Datasets

• GNM, ViNT, and NoMaD share many real-world multi-robot navigation datasets, whereas NavDP and CANVAS each use one self-collected simulation dataset.Source: §B.2 · p. 31, Table 7 · p. 31

B.3 Finetuning Procedure

• All five learning baselines are finetuned from official pretrained checkpoints on 40.71 hours of CostNav teleoperation data.Source: §B.3 · p. 31

B.4 Goal Conditioning Specifications

• GNM, ViNT, and NoMaD use image goals with intermediate sub-goal images from a topological map, while NavDP combines an image goal with a local-frame point goal.Source: §B.4 · p. 32
• CANVAS is conditioned on a top-down sketch tracing the simulation shortest path to the destination.Source: §B.4 · p. 32

C Simulation Details

• Simulation metrics include contact events, timeout duration, robot speed, momentum, delivery distance, and data-collection quantities.Source: C Simulation Details · pp. 34–35
• The evaluation uses 65.63 hours of working time and 40.71 hours of collected data.Source: C Simulation Details · p. 35

D Aggregated Cost Formula

• The seven reported contribution margins are negative: −44.07, −37.34, −24.04, −50.52, −32.08, −15.75, and −28.40 $/run.Source: D Aggregated Cost Formula · p. 37
• No method reaches break-even, so break-even runs and days are undefined across every evaluated configuration.Source: D Aggregated Cost Formula · p. 37

E Per-Component Break-Even Thresholds

• AIS 3 contacts above 0.12% exhaust the 3.49$/run revenue, compared with thresholds of 0.52% for AIS 2 and 3.7% for AIS 1.Each 1% of contacts adds 28.79$/run at AIS 3, 6.67$/run at AIS 2, and 0.94$/run at AIS 1.Source: E Per-Component Break-Even Thresholds · p. 38
• Spoilage above 9.8% exhausts the budget because each 1% costs 0.35$/run in refunds and lost delivery-fee revenue.The threshold is calculated as 3.49/35.42 = 9.8%.Source: E Per-Component Break-Even Thresholds · p. 38
• Combined timeout and physical-assistance rates above 50% exhaust the budget at a cost of 0.0698$/run per percentage point.The same 50% threshold applies to the stated timeout-and-assistance cost calculation.Source: E Per-Component Break-Even Thresholds · p. 38
• A property contact rate above approximately 3.0% exhausts the budget based on mean damage of 116.25$/contact.The property types include mailbox, trash bin, bollard, and building glass.Source: E Per-Component Break-Even Thresholds · p. 38
• Physical assistance above 10.6% exhausts the budget through repair alone, using an estimated repair cost of 32.88$/run per unit assistance rate.The threshold is 3.49/32.88 ≈ 10.6%.Source: E Per-Component Break-Even Thresholds · p. 38

F Sim-to-Real Experiment: Per-Scenario Details

• Eight real-world CANVAS scenarios were attempted on a Segway E1, with per-scenario outcomes and physical measurements summarized in the experiment results.The platform used a Livox MID-360 LiDAR and an off-board laptop running the sim-trained policy.Source: F Sim-to-Real Experiment: Per-Scenario… · p. 38, Table 10 · p. 38, Figure 43 · p. 39
• Contacts occurred only in S3, S5, and S7, while S5 and S7 were stopped by a safety operator before full collision development.The raw velocity-drop measurements therefore underestimate the unprevented impact severity for those scenarios.Source: F Sim-to-Real Experiment: Per-Scenario… · p. 38, Figure 44 · p. 40
• The analysis assigns 50 N·s to S5 and 100 N·s to S7 as physics-based would-be impulses, while S3 uses the measured velocity drop and 50 kg robot mass.The S7 value corresponds to an upper-bound near-elastic collision at approximately 1 m/s.Source: F Sim-to-Real Experiment: Per-Scenario… · p. 38
• Distances are computed from the LiDAR-localized pose stream, and aggregated real-world variables are means over the eight scenarios for comparison with simulation.The real-world aggregate uses an R* prefix parallel to the simulation S* schema.Source: F Sim-to-Real Experiment: Per-Scenario… · p. 38, Table 11 · p. 41