Source-linked AI summary
A Closed-Loop Evaluation of Capability Loss and Recovery in Compressed Driving Policies
Ahmad Alfan Alfian Irfan, Nur Ahmad Khatim, Mansur Arief
TL;DR
Compressed driving policies must fit embedded resource limits, but aggregate fidelity scores may miss failures in closed-loop behavior. The paper evaluates a belief-state driving actor stage by stage across five curricula and finds that pruning first destroys capability, rehearsal-dependent distillation can recover it, and later integer quantization can remove difficult behaviors again.
Problem
Aggregate numerical scores may not reveal whether a compressed driving policy retains safe closed-loop behaviors when interacting with other road users.
Method
The paper formulates driving as a POMDP, trains a belief-state PPO policy in Gym-Duckietown, and evaluates its actor through sequential compression stages on fixed curricula.
Results
Structured pruning first loses capability across all five curricula; distillation recovers behavior according to rehearsal coverage, while integer quantization of the improved actor loses the two hardest curricula again.
Takeaways & Limitations
Closed-loop task acceptance should be measured directly because action-level fidelity cannot guarantee preserved driving capability.
Takeaways & Limitations
The study uses one policy, simulator, and compression toolchain, and its operational acceptance checks are ad hoc rather than standard.
Abstract
from arXiv · showhide
Many automobile and mobility companies deploy learned driving policies on embedded computers with limited memory and power. Pruning, knowledge distillation, and quantization are the standard methods to reduce the size and the inference cost of these policies. However, these methods are commonly assessed by aggregate numerical scores, and such scores may not reflect the ability of the policy to drive safely when interacting with other road users. In this study, we propose a stage-wise closed-loop evaluation approach to follow a driving policy through a compression pipeline. We formulate the driving task as a partially observable Markov decision process (POMDP) and train a belief-state policy with proximal policy optimization (PPO) in Gym-Duckietown. We then extract the actor, compress it one stage at a time, and evaluate it on five driving curricula. We show that structured pruning is the stage at which the driving capability is first lost. Meanwhile, distillation improves the pruned actor, but the improvement is limited by its rehearsal data. Integer quantization of the improved actor loses some of the curricula that require the vehicle to stop and then resume. Interestingly, the same procedure on the unpruned actor preserves all five curricula. Our study thus provides an empirical analysis aiming to answer the currently active discussions on how to accept a compressed driving policy, so as to achieve a safe and statistically reliable deployment of automated driving functions.
I. INTRODUCTION
The paper argues that aggregate action or accuracy scores can miss driving-capability losses in closed-loop interaction. It therefore follows a learned driving actor through sequential compression stages and finds capability loss at pruning, partial recovery through rehearsal-dependent distillation, and renewed failures after later quantization.
- Embedded deployment motivates compressing learned driving policies under memory, power, and thermal limits.
- Aggregate scores may conceal losses in behaviors such as lane following, stopping, resuming, or yielding to pedestrians.
- The study asks when capability first breaks, whether pruning damage is recoverable, how rehearsal data shapes recovery, and whether recovery survives precision reduction.
- The evaluation formulates driving as a POMDP, trains a belief-state PPO policy in Gym-Duckietown, and evaluates the isolated actor at each compression stage.
- Structured pruning reduces the actor below 10% of its original size, and 64-unit hidden layers lose driving capability on all five curricula.
- Distillation recovers behavior according to rehearsal coverage, while later quantization can remove recovered curricula; therefore acceptance must assess driving rather than action reproduction.
II. RELATED WORK
The related work positions structured pruning and knowledge distillation as practical compression tools, while motivating empirical localization of capability loss in driving policies.
- Structured pruning removes whole filters or units, preserving dense networks that can run faster on ordinary hardware.
- The study aims to pinpoint capability loss empirically because pruning benchmarks and practices have been criticized.
- Knowledge distillation trains a compact student to reproduce a larger teacher’s outputs, using teacher actions rather than class scores for control policies.
- Prior policy-distillation results indicate that student capability depends on which experts and states are rehearsed.
C. Quantization and Reduced Precision
Quantization changes numerical precision through several routes, and prior work shows compression-stage ordering matters. For driving policies, final-model evaluation cannot identify which stage removed behavior because closed-loop actions affect future inputs.
- Quantization lowers the precision of weights and activations, with integer schemes enabling inference using integer arithmetic alone.
- Post-training quantization calibrates an already trained network, whereas quantization-aware training simulates reduced precision during optimization.
- Reduced floating-point formats provide a separate precision-reduction route from integer quantization.
- Compression studies disagree on ordering details but support placing quantization late, with one line of work specifically favoring pruning before quantization.
- The value of distillation in the studied position depends on the data it rehearses on.
- Because driving policies close the loop, evaluating only the final compressed model cannot reveal which stage removed a behavior.
III. PROBLEM FORMULATION
The task is a visuomotor driving problem in Gym-Duckietown, modeled as a POMDP with camera observations, simulator dynamics, and curriculum-specific rewards. Five curricula serve as both training progression and fixed evaluation scenarios.
- A. Driving Task and Evaluation Curricula: The section defines the task as visuomotor lane following with pedestrian avoidance and stop-line compliance in Gym-Duckietown.
- A. Driving Task and Evaluation Curricula: Five driving curricula increase in difficulty and serve jointly as training curricula and fixed evaluation scenarios.
- B. POMDP Formulation: Each curriculum is modeled as a POMDP with simulator state, actions, transitions, rewards, observations, and discounting.
- B. POMDP Formulation: The simulator state includes vehicle pose and velocity, lane geometry, and curriculum-specific pedestrian or stop-sign configurations.
- B. POMDP Formulation: The policy receives monocular camera frames and must infer state through belief because pedestrians and stop signs can become intermittently observable.
- B. POMDP Formulation: Actions map to linear velocity capped at 0.4 m/s and yaw-rate commands spanning 8 rad/s.
- B. POMDP Formulation: The reward combines progress, lane keeping, pedestrian proximity, stop compliance, smoothness, and terminal outcomes, with weights set per curriculum.
- B. POMDP Formulation: Rewards train the original policy, whereas acceptance is determined directly from task outcomes.
C. Perception and EKF-Based Belief
The policy converts camera observations into metric measurements and fuses them into a fixed-dimensional belief representation for control.
- Belief construction: The system fuses measurements over time with EKFs and assembles their posteriors into a fixed 29-dimensional policy input.This factorized belief representation replaces conditioning the policy directly on raw frames.
- Measurement extraction: YOLO11n detects pedestrians and stop signs, while MobileNetV3-small estimates lane offset, heading error, and curvature.The detector supplies object measurements; the regressor outputs d, φ, and κ in normalized units.
- Measurement extraction: Detected object locations are projected onto the ground plane to obtain range measurements in the robot frame.A calibrated camera model performs the ray-ground intersection before range correction and noise modeling.
- Evaluation inputs: The five driving curricula provide evaluation rollouts with onboard camera frames for assessing the resulting policy behavior.The curricula are rendered through camera frames collected during evaluation rollouts in Gym-Duckietown.
3) Belief initialization, prediction, and correction:
The belief state is initialized conservatively, updated with lane and object filters, and augmented with stop-obligation information before controlling the vehicle.
- Initialization: At reset, lane validity is zero, object existence probabilities use their priors, and the stop mode is set to none.The lane filter initializes on the first lane measurement with preset covariance values.
- Prediction and correction: Separate EKFs track lane state and object position and velocity while ego-motion re-expresses predictions in the current robot frame.Object tracks initialize with zero velocity and broad velocity uncertainty, then receive corrections in polar coordinates.
- Stop reasoning: A stop-obligation state machine combines stop-sign belief and route prior to track whether stopping is not required, required, or satisfied.It also supplies the distance to the stop line for the policy input.
- Policy input: The policy reads the filter posteriors as sufficient statistics in a fixed 29-dimensional factored belief vector.The belief-state formulation supports model-free training with PPO under the POMDP representation.
- Policy input: The extracted actor maps the 29-dimensional belief state through two 256-unit hidden layers to a two-dimensional deterministic mean action.The actor contains 73,986 parameters, while perception and belief processing remain unchanged during compression.
A. Pipeline Configurations
The study evaluates structured pruning, rehearsal-data variants for distillation, and reduced-precision routes across a controlled set of pipeline candidates.
- Candidate configurations: The experiment evaluates ten candidates spanning meaningful placements of pruning, distillation, and quantization.A0 is the original reference actor; A4 and A5 serve as quantization and pruning-plus-quantization controls, respectively.
- Pruning: Structured pruning removes whole hidden units and reduces the width-64 actor to 6,210 of 73,986 parameters, a 91.6% reduction.Units are ranked using the sum of incoming and outgoing L2 connectivity plus absolute bias.
- Pipeline structure: The study overview separates policy construction from actor compression, showing two distillation branches and three precision routes from a shared checkpoint.The upstream policy fuses YOLO and MobileNetV3-small measurements into an EKF-based belief state before Belief-PPO training.
- Distillation: The historical rehearsal set samples C4 development states, whereas the balanced set contains 62,176 public states distributed across all five curricula.Teacher, loss, optimizer, batch size, learning rate, and epoch budget are held constant between the two sets.
- Quantization: Post-training quantization uses eager static INT8 with per-channel symmetric weights and per-tensor affine activations calibrated on development states.The FP16 control casts the A3 weights to 16-bit floats without other changes and requires native half-precision validation.
C. Closed-Loop Acceptance Criterion
Closed-loop acceptance requires every predefined behavioral check to pass on each curriculum, with comparisons against the original actor on matched seeds.
- Evaluation design: Each candidate drives every curriculum on the same eight seeds, producing 40 episodes per candidate and 400 episodes overall.The fixed seeds support paired comparison across the candidate set.
- Acceptance rule: A candidate passes a curriculum only when all predefined acceptance checks hold simultaneously.The criterion is defined over a set K of checks and is applied per candidate and curriculum.
- Acceptance checks: The checks cover completion, progress, collisions, unsafe proximity, stop behavior, lane failures, invalid poses, and pedestrian clearance.Stop behavior includes both stop violations and stop completion with restart.
- Acceptance rule: Paired comparison with the original actor prevents pre-existing weaknesses from being charged to compression unless they worsen beyond the preset margin.A high completion count alone cannot establish acceptance because every check must pass.
A. Capability Loss under Pruning
Structured pruning is the first compression stage to destroy task-level driving capability, while later recovery and quantization outcomes depend on rehearsal coverage and the compression pathway.
- Structured pruning to 64-unit hidden layers is the first stage at which driving capability is lost across all five curricula.The pruned actor fails every curriculum, with lane, pose, and stop violations across different tasks.
- Balanced rehearsal restores all five curricula, whereas C4-focused rehearsal recovers only C3 and C4.The two distilled actors share the same source, teacher, loss, and training budget; their rehearsal coverage differs.
- Direct quantization of the pruned actor fails all five curricula, while balanced distillation before quantization preserves C0 through C2.This comparison uses fixed tested pathways rather than a factorial proof of operation ordering.
- Post-training quantization of the recovered actor fails C3 and C4 despite the FP32 source passing all five curricula.A6 completes three of eight C3 episodes, compared with eight of eight for A3, and stops without issuing a later driving command.
- The two quantized branches fail stop-and-resume curricula from opposite sides: A6 remains stopped, whereas A8 crosses the line.A6 emits vcmd = 0.000 while parked before the line; A8 emits vcmd = 0.077 while crossing it and violates stops.
D. Task Decision, Action Fidelity, and Cost
The two precision routes diverge in closed-loop behavior: FP16 preserves all five curricula, whereas INT8 loses the two stop-and-resume curricula despite acceptable action fidelity on some tasks. Cost measurements show that FP16 trades latency for memory savings, while INT8 is fastest but has a larger serialized file.
- Task decisions: FP16 preserves all five curricula, while INT8 fails C3 and C4, the curricula requiring stopping and resuming.The two quantized branches fail the stop curricula from opposite sides, while lane keeping survives continuous small errors.
- Cost: FP16 halves parameter memory but adds 26 % median actor latency over FP32, whereas INT8 runs at 12.9 µs versus 19.4 µs for FP32.The FP16 latency penalty is consistent with the absence of a native half-precision compute path.
- Task decisions: The stop curricula expose a sharp driving-to-holding-to-driving decision boundary that small quantization perturbations can disrupt.The data locate this pattern but do not establish the internal mechanism in the compressed networks.
- Action fidelity: Fidelity-only acceptance produces both false negatives and false positives: A8 would be accepted over A6, while A4 would be rejected despite driving every curriculum.Closed-loop task acceptance therefore cannot be inferred from action-fidelity metrics alone for an acting policy.
A. Implications for Deployment Practice
Deployment should assess compressed driving policies at every intermediate stage, with rehearsal coverage and stop-resume scenarios treated as safety-relevant evidence. The study also cautions that its findings are bounded by a small, single-policy, single-simulator evaluation.
- Deployment practice: Release-gate evidence should be collected at every intermediate compression stage, not only for the final configuration.Stage-wise reporting distinguishes where behavior was removed from where recovery failed.
- Deployment practice: Rehearsal coverage is a safety-relevant configuration because it determined which driving behaviors returned after distillation.The study contrasts limited-task rehearsal with balanced coverage across curricula.
- Deployment practice: Stop-and-resume scenarios should be first-class acceptance cases because both integer routes failed there while continuous lane keeping survived.The unpruned actor preserved all curricula, whereas integer quantization of the improved actor lost the two hardest curricula.
- Deployment practice: Table IV records the state and action at each candidate's first failure, with stop violations highlighted and a dash indicating no stop line was involved.This format supports diagnosing whether failures involve stopping, lane behavior, or other first-failure conditions.
- Limitations: The conclusions are limited to one small policy, one simulator, and one fixed compression toolchain, so transfer to larger policies or full-scale driving stacks remains untested.The authors specifically identify the 6,210-parameter perceptron, Gym-Duckietown, and the fixed integer-quantization procedure as scope boundaries.