PaperlayerView exact revision on arXiv

Source-linked AI summary

TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving

Victoria Smirnova, Viktoriia Zinkovich, Gregorii Bukhtuev, Artem Belyaev, Andrey Kuznetsov, Denis Shepelev, Vlad Shakhuro

arXiv:2609.38463v1cs.ROcs.CV

TL;DR

Aggregate planner metrics do not explicitly measure traffic-rule compliance, and traffic-rule supervision is limited for rule-critical cases. TrafficSignBench combines sign-focused rules, real-map scenes, and targeted scenarios, while expert-supervised fine-tuning raises PlanT-2’s reported compliance outcomes. The benchmark currently evaluates sign-associated rules at the planning level, not the complete perception-to-control pipeline.

  • Problem

    Traffic-rule compliance remains underexplored in planner evaluation, and expert demonstrations underrepresent rule-specific corner cases.

    Source: §I · pp. 1–2

  • Method

    TrafficSignBench combines real-map scenarios, automatic checkers for sign rules, rule-targeted generation, and oracle-selected rule-compliant expert trajectories.

    Source: §I · p. 2, B. Baselines · p. 4

  • Results

    Standard planners achieve only 2.9–9.0% overall SCD; expert-supervised fine-tuning raises PlanT-2 SCD from 5.9% to 72.3% and yields 96.6% sign compliance.

    Source: §VI · p. 7

  • Takeaways & Limitations

    PlanT-2-FT episodes more often fail by obeying the sign but missing the destination than by violating a sign, making interactive right-of-way and longer-horizon routing the next challenge.

    Source: §VI · p. 7

    The benchmark focuses on sign-associated rules, assumes structured sign information, and does not systematically test multiple simultaneously active rules.

    Source: §VII · p. 7

Abstract

from arXiv · show

Autonomous driving planners are typically evaluated using aggregate metrics such as driving score, destination rate, and collision rate, which do not explicitly measure compliance with traffic rules. As a result, planners can achieve high benchmark scores while still exhibiting unsafe or illegal behaviors, limiting their applicability to real-world deployment. To address this gap, we introduce TrafficSignBench, a large-scale, traffic sign-centric benchmark for systematic and interpretable evaluation of traffic-rule compliance in autonomous driving. Our framework combines real-map-based simulation for realistic road layouts with rule-targeted procedural scenario generation for scalable and balanced coverage of underrepresented rules. We implement traffic rules corresponding to 34 traffic signs, each equipped with an automatic rule checker for detecting violations during closed-loop execution. This design yields 29,000 diverse road scenes and 29 distinct testing scenario types, enabling controlled evaluation of rule-specific planner behavior. We construct 5,800 testing scenes and demonstrate that current autonomous driving planners can exhibit poor traffic-rule compliance despite strong performance on standard evaluation metrics. To address this limitation, we transform existing planners into rule-compliant trajectory experts via explicit traffic-sign constraints, enabling scalable generation of high-quality oracle trajectories for fine-tuning.

I. INTRODUCTION

TrafficSignBench targets the gap between aggregate planner evaluation and explicit traffic-rule compliance assessment. Its rule checkers and targeted scenarios support systematic testing, while results show that standard planners can remain poorly compliant and expert-supervised fine-tuning can improve performance.

I. INTRODUCTION

  • Open-loop benchmarks cannot capture violations by design, while closed-loop benchmarks typically offer partial rule coverage and rely on aggregate metrics.Planners can therefore score strongly while still violating fundamental traffic regulations.Source: §I · p. 1
  • TrafficSignBench extends MetaDrive with executable rules for 34 signs, automatic online checkers, and rule-targeted interactions requiring explicit planning decisions.The benchmark is presented as an open-source framework for systematic, interpretable evaluation.Source: §I · p. 2
  • The benchmark combines 29,000 scenarios across 29 functional types with parameters calibrated to real-world driving data and controlled rule-critical interactions.Its scenes are constructed from a pool of 26,020 real-map crops.Source: §I · p. 2
  • Standard planners attain only 2.9–9.0% under the joint compliance-and-destination criterion; oracle-selected expert fine-tuning raises PlanT-2 from 5.9% to 72.3%.The benchmark evaluates 17 planners and reports that conventional metrics obscure systematic rule failures.Source: §I · p. 2

II. RELATED WORK

Existing benchmarks and simulators provide only partial support for systematic, sign-centric rule evaluation. The related work also contrasts interpretable but less scalable rule-based planning with flexible learning-based methods that lack explicit rule adherence.

A. Planning Benchmarks

  • TrafficSignBench combines explicit sign and road-marking semantics, deterministic per-rule checkers, and targeted scenarios where prior benchmarks lacked sign-centric per-rule evaluation.Open-loop evaluation cannot observe violations, and several closed-loop benchmarks focus on broader driving abilities or route performance.Source: A. Planning Benchmarks · p. 2

B. Simulators for Autonomous Driving

  • The work extends MetaDrive with traffic signs from real-world databases and map data, plus automatic checkers for detecting violations.The related-work discussion notes that many simulators lack explicit traffic-rule representations and rely on manually designed scenarios.Source: B. Simulators for Autonomous Driving · p. 2

C. Rule-Based and Learning-Based Planning

  • Rule-based planners are interpretable but less scalable, while learning-based planners are flexible but do not explicitly encode traffic regulations.The paper identifies these limitations as motivation for systematic compliance evaluation.Source: C. Rule-Based and Learning-Based… · p. 2

III. TRAFFICSIGNBENCH

TrafficSignBench organizes sign-imposed rules into a closed-loop benchmark built from real-world map crops and rule-targeted scenarios. Its 34 implemented signs have substantial semantic alignment with other jurisdictions, supporting adaptation through modular changes.

III. TRAFFICSIGNBENCH

  • The benchmark proceeds from rule formalization to a functional testing taxonomy, real-map sign placement, and calibrated rule-targeted closed-loop scenario generation.These stages are described as the structure of TrafficSignBench.Source: §III · pp. 2–3

A. Rule Formalization and Generalizability

  • The benchmark formalizes 34 signs that impose explicit mandatory rules, excluding warning or informational signs and signs not observed in practice.Each implemented sign has explicitly defined and verifiable rule logic.Source: A. Rule Formalization and… · p. 3
  • 92% average semantic overlap across Vienna Convention signatories indicates that adapting the benchmark to other jurisdictions requires small modifications.Table I reports semantic equivalence percentages for the implemented signs against Russian traffic signs.Source: A. Rule Formalization and… · p. 3, Table I · p. 3

B. Testing Taxonomy

TrafficSignBench consolidates the signs into functional closed-loop tests, places them on real-map crops, and generates rule-targeted scenarios with varied traffic conditions. The resulting benchmark contains 29,000 scenarios across 29 testing types.

B. Testing Taxonomy

  • The 34 signs are consolidated into 29 scenario types, grouped by priority, speed, obstacles, and routing capabilities.The groups contain 6, 4, 8, and 11 scenarios, respectively.Source: B. Testing Taxonomy · p. 3

C. Real-Map Scene Construction

  • The benchmark uses 26,020 real-world map crops, placing signs according to semantic requirements across junction, dual-path, and corridor layouts.Dual-path maps pair a shorter prohibited route with a longer compliant route to the same destination.Source: C. Real-Map Scene Construction · p. 3

D. Rule-Targeted Scenario Generation

  • Rule-targeted interactions make compliance decisions non-trivial, including releasing a convoy as the ego approaches a conflict and assigning a destination that tempts use of a prohibited branch.Crosswalk pedestrians use controlled presets, including late arrivals and chained groups.Source: D. Rule-Targeted Scenario Generation · pp. 3–4
  • Each of 2,900 maps expands into 10 variants across five dimensions, producing 29,000 scenarios: 23,200 train and 5,800 test.Traffic density, initial speed, and background-agent parameters are sampled from nuPlan-estimated distributions.Source: D. Rule-Targeted Scenario Generation · p. 4

IV. EVALUATION PROTOCOL

Standard driving metrics do not directly measure traffic-rule compliance, so the protocol adds explicit rule-adherence measurements. Its primary metric combines compliance with destination completion and is reported at three aggregation levels.

A. Rule-Aware Evaluation Metrics

  • Rule-adherence measurements from automatic verifiers complement Driving Score, Destination Rate, Efficiency, and Collision Rate, which do not explicitly measure regulatory compliance.A planner can score near-perfectly on conventional benchmarks while systematically violating traffic rules.Source: A. Rule-Aware Evaluation Metrics · p. 4
  • Sign Compliance × Destination (SCD) requires both target-sign compliance and destination completion, avoiding ambiguity when a maneuver is unfinished.A crash or becoming stuck before the maneuver ends can make isolated sign compliance impossible to determine.Source: A. Rule-Aware Evaluation Metrics · p. 4
  • SCD is reported per scenario type, within four semantic groups, and as an overall macro-average across all 29 scenario types.The groups are Priority, Speed, Obstacles, and Routing.Source: A. Rule-Aware Evaluation Metrics · p. 4

B. Baselines

The evaluation groups planners by access to explicit traffic rules, from base planners to rule-compliant experts, an oracle, and a fine-tuned baseline. Background traffic calibration and expert contributions are also characterized.

B. Baselines

  • Empirical CDFs compare nuPlan [11] traffic data with TrafficSignBench recordings, while dashed guides mark three controlled density and speed probes and D denotes the two-sample KS statistic.The figure characterizes background-traffic calibration.Source: Figure 2 · p. 4
  • Base planners operate without explicit traffic-rule access and include IDM variants, PPO, CaRL, and PlanT-2.IDM is evaluated in default and four stochastic variants; PPO uses Proximal Policy Optimization, CaRL uses sparse route-completion rewards, and PlanT-2 is a transformer-based imitation-learning planner.Source: B. Baselines · p. 4
  • Rule-compliant experts add explicit action constraints in sign-governed regions without changing the underlying base-planner architecture.Examples include hard braking before a stop line.Source: B. Baselines · p. 4
  • The Oracle Expert aggregates rule-compliant experts and selects the best-performing one for each scene, serving as an upper-bound oracle and a source of ground-truth trajectories.Because selection is post-hoc across multiple experts, it is not a direct evaluation baseline.Source: B. Baselines · p. 4
  • PlanT-2-FT is a supervised baseline fine-tuned exclusively on trajectories generated by the Oracle Expert.Its architecture and training details are provided in Section IV-D.Source: B. Baselines · p. 5
  • The Oracle Expert dataset’s high-quality trajectories come from eight rule-compliant experts across four semantic sign groups, with no single planner dominating.The caption describes the experts’ strengths as complementary.Source: Figure 3 · p. 5

C. Oracle Trajectory Collection

Oracle trajectories are collected by comparing rule-compliant expert rollouts under matched closed-loop conditions, then filtering and ranking successful demonstrations. The resulting collection contains 36,828 high-quality trajectories from experts with complementary strengths.

C. Oracle Trajectory Collection

  • All eight experts run from the same initialization, and a trajectory succeeds only if the ego completes the route without violating the target rule (SCD = 100%).When an expert fails, that expert’s scenario configuration is iteratively resampled until it produces a successful trajectory.Source: C. Oracle Trajectory Collection · p. 5
  • The ranking keeps the fastest successful IDM-based rollout, then compares it with successful non-IDM experts using a weighted F-score of normalized speed and comfort.IDM-based controllers are treated as consistently smooth and as differing primarily in efficiency.Source: C. Oracle Trajectory Collection · p. 5
  • 36,828 high-quality training trajectories are retained, averaging 1.6 successful trajectories per scenario, with different experts contributing across sign categories.The top two trajectories per training scenario are selected using Q(τ), which prioritizes efficiency over comfort when β < 1.Source: C. Oracle Trajectory Collection · p. 5

D. Rule-Supervised Fine-Tuning

PlanT-2-FT fine-tunes a PlanT-style planner on oracle trajectories, adding sign-conditioned inputs and a speed head while leaving the backbone unchanged. Training uses sign-specific targets and balanced, perturbed examples across all scenario types.

D. Rule-Supervised Fine-Tuning

  • PlanT-2-FT adds sign object tokens, a persistent sign-state token, and a learnable speed token with a discrete speed head, while retaining the unchanged backbone.The extensions add 147,976 parameters, or 0.4%, to the 37.2 M-parameter planner.Source: D. Rule-Supervised Fine-Tuning · p. 5
  • Training jointly optimizes path and waypoint L1 losses with cross-entropy over soft two-hot speed targets, using 29,462 trajectories for training and 7,366 for validation.The split spans all 29 scenario types; fine-tuning lasts 26 epochs, and checkpoints are selected by closed-loop validation performance.Source: D. Rule-Supervised Fine-Tuning · p. 5
  • Targets are tailored to sign rules, sign-relevant decision windows are up-weighted, sign families are batch-balanced, and perturbed ego poses retain the original trajectory targets.Speed-regime targets use minimum expert speed over the next 4 seconds; maneuver signs use the expert 4-second trajectory, and right-of-way targets are frame-local.Source: D. Rule-Supervised Fine-Tuning · p. 5

V. EXPERIMENTAL RESULTS

Planner comparison shows that standard planners score poorly on rule-conditioned completion, while expert-supervised fine-tuning substantially improves PlanT-2. The failure analysis separates sign compliance from destination completion, with remaining errors concentrated in route execution and interactive rules.

A. Planner Comparison

  • 72.3% overall SCD for PlanT-2-FT is 66.4 points above PlanT-2 and 63.3 points above the strongest standard baseline.Table II reports closed-loop evaluation across all 29 scenario types.Source: A. Planner Comparison · p. 5, Table II · p. 6
  • Obstacle SCD rises from 3.7% to 90.8%, while Routing SCD rises from at most 1.1% to 58.4% under PlanT-2-FT.Source: A. Planner Comparison · p. 5
  • PlanT-2-FT comes within 7.7 points of the strongest privileged expert at 80.0%, but Priority and Routing remain its localized gaps.It surpasses every expert on Obstacle scenarios; the remaining behaviors are interactive right-of-way and sign-conditioned route selection.Source: A. Planner Comparison · p. 5

B. Failure Analysis

  • PlanT-2-FT sign compliance is 96.6%, but joint SCD is 71.9%; 24.7% of episodes obey the sign but miss the destination, versus 3.4% that violate a sign.Sign 4.1.4 has 100% compliance but 36.0% SCD on difficult dual-path maps; crosswalk and roundabout compliance is approximately 81%, and these classes account for 36.4% of violation events.Source: B. Failure Analysis · pp. 5–6

C. Correlation with Conventional Metrics

Conventional metrics do not provide a category-independent proxy for rule-conditioned completion. Removing sign identity from PlanT-2-FT sharply reduces performance on paired scenes, supporting the role of explicit sign information in its learned behavior.

C. Correlation with Conventional Metrics

  • No conventional metric proxies SCD across categories: Collision Rate correlates −0.93 overall but +0.07 with Routing, while Driving Score and Destination Rate correlate +0.81 with Routing.Efficiency correlates −0.75 with overall SCD but +0.49 with Routing; six of nine planners have zero Routing SCD.Source: C. Correlation with Conventional Metrics · pp. 6–7

D. Sign-Channel Ablation

  • Across 580 paired episodes, removing sign identity cuts SCD from 72.8% to 15.4% and compliance from 90.3% to 28.9%.Every functional group is affected; crosswalk sign 5.19 is the sole compliance exception, remaining at 100% because road markings and visible pedestrians provide direct cues.Source: D. Sign-Channel Ablation · p. 7

VI. CONCLUSION

TrafficSignBench evaluates sign-rule compliance in closed loop and finds that expert-supervised fine-tuning substantially improves PlanT-2’s measured compliance. Its current scope excludes some regulations, perception errors, and systematic testing of simultaneous rules.

VI. CONCLUSION

  • TrafficSignBench provides automatic checkers for 34 traffic signs and 29 diverse testing scenarios, with 29,000 scenarios derived from real-map geometry and rule-targeted generation.Source: §VI · p. 7
  • Standard planners achieve only 2.9–9.0% overall SCD, while expert-supervised fine-tuning raises PlanT-2 from 5.9% to 72.3% and yields 96.6% sign compliance.The strongest privileged expert reaches 80.0%.Source: §VI · p. 7
  • 24.7% of PlanT-2-FT episodes obey the sign but miss the destination, versus 3.4% that violate a sign.The stated challenge is preserving measured compliance while completing interactive right-of-way and longer-horizon routing maneuvers.Source: §VI · p. 7

VII. LIMITATIONS

  • The benchmark excludes regulations not tied to explicit signs, assumes structured sign information, and does not systematically test combinations of simultaneously active rules.It therefore evaluates planning-level rule compliance rather than the complete perception-to-control pipeline.Source: §VII · p. 7
  • TrafficSignBench’s modular design permits future extensions for additional regulations or multi-rule interactions without changing the evaluation protocol.Source: §VII · p. 7
Loading 2609.38463v1…