SteerQuant: Steering Quantization Error with Action-Guided Scaling in World-Action Models

1 INTRODUCTION

• Comparable quantization errors across video, proprioceptive, and action streams can propagate differently and produce distinct final-action deviations, so numerical accuracy alone does not determine control fidelity.Shared weights mean one weight-quantization choice affects multiple streams; independent stream scales generally require different quantized weight matrices.Source: §1 · pp. 1–2
• Stream tolerance varies across layers and denoising steps, motivating calibration guided by downstream action impact.The paper identifies this variation as a reason to adapt error trade-offs while retaining shared weights and fixed bit-widths.Source: §1 · p. 2
• SteerQuant uses an Action-Impact Stream Map to guide shared channel scaling and stream-specific activation calibration across denoising steps, retaining one static quantized weight matrix per layer and fixed bit-widths.Its calibration prioritizes computations with greater action impact.Source: §1 · p. 2
• Rudder fuses calibrated scaling and output compensation into 4-bit kernels to reduce extra kernel launches and intermediate memory traffic.This is designed to preserve quantization's computational and memory benefits.Source: §1 · p. 2
• Across three WAMs, W4A8 and W4A4 preserve near-BF16 LIBERO success, outperform evaluated same-precision RoboLab baselines, and reach up to 2.23× denoising speedup over BF16.On a real dual-arm robot, W4A8 achieves 1.35× end-to-end inference speedup while maintaining average task success relative to BF16.Source: §1 · p. 2

2 RELATED WORK

• Existing PTQ approaches either reduce numerical error through transformations or incorporate action-related criteria into quantization.Examples include SmoothQuant's equivalent channel scaling [1], ActQuant's action-guided bit allocation and weight-scale optimization [Akbari et al., 2026], and QuantWAMs' weight-precision allocation using joint video–action loss sensitivity [Zhou et al., 2026].Source: §2 · p. 2
• Prior criteria do not directly characterize how coupled WAM-stream weight and activation errors propagate to final actions across denoising steps.The paper uses this gap to motivate calibration at fixed 4-bit.Source: §2 · p. 2
• Inference accelerators such as Fast-WAM and DreamZero reduce computation by omitting future-video generation or reusing intermediate results, whereas this method lowers precision in retained operations.The paper describes quantization as complementary to computation reduction and caching while preserving joint world–action denoising.Source: §2 · p. 2

3.1 WAM QUANTIZATION

• Symmetric k-bit quantization of activations and weights jointly determines how much a layer's output deviates from full precision.The defined normalized local reconstruction error measures relative output-error magnitude, but similar local errors can yield different final-action deviations.Source: §3.1 · p. 3
• Similar local reconstruction errors can lead to different final-action deviations as errors propagate through coupled computation.This is why local error magnitude alone is insufficient to characterize quantization's action effects.Source: §3.1 · p. 3

3.2 OBSERVATIONS

• The observation study injects rescaled W4A8 quantization residuals into selected full-precision Cosmos-Policy layer outputs, matching normalized local error while keeping other computation full precision.It measures standardized action RMSE and closed-loop task success on three LIBERO tasks, with three initial states and nine paired rollouts per condition.Source: §3.2 · p. 3

A. Shared-Weight Error Routing

• Weights are shared across timesteps.The passage provides no further description of the section's error-routing mechanism.Source: A. Shared-Weight Error Routing · p. 4

B. Stream-Specific Activation Modulation 𝛾𝐴

• Figure 2 depicts Γℓ,τ as token-wise stream gains and distinguishes offline weight preparation from fused inference kernels.Source: Figure 2 · p. 4
• At Block 14, matched video errors yield 2.9× the action RMSE of action errors; at Block 24, action errors yield 7.2× video errors, with success 4/9 versus 9/9.The success comparison is for video-stream errors versus action-stream errors at Block 24.Source: B. Stream-Specific Activation… · p. 4
• At Block 24, matched action-stream errors at Step 4 yield 109.6× Step 0 action RMSE, with success 4/9 versus 7/9 or 8/9 at earlier steps.The errors are matched across Steps 0–4.Source: B. Stream-Specific Activation… · p. 4
• Each layer shares weights across streams and steps, so targeting one stream at one step can affect others under fixed 4-bit precision.Source: B. Stream-Specific Activation… · p. 4

4 METHODOLOGY

• SteerQuant estimates local error impact with an Action-Impact Stream Map, calibrates shared then stream- and step-specific scales, and fuses execution in Rudder.Source: §4 · p. 4

4.1 ACTION-IMPACT STREAM MAP

• The map scores each layer–step–stream region by RMS first-order action displacement from local quantization error, using Rademacher projections rather than dense Jacobians.Calibration inputs come from a full-precision forward pass to exclude upstream quantization errors.Source: §4.1 · p. 5

4.2 SHARED-WEIGHT ERROR ROUTING

• A shared positive diagonal Dℓ rescales weights and inversely scales activations; map-weighted reconstruction loss trades off regions while preserving one quantized weight matrix across streams and steps.The map penalizes errors more strongly in regions with greater action impact.Source: §4.2 · p. 5

4.3 STREAM-SPECIFIC ACTIVATION MODULATION

• At each layer and step, a shared clipping threshold sets a reference range, while bounded stream gains adjust effective activation ranges to accommodate changing priorities.The calibrated gains and thresholds are stored in a layer–step lookup table, without online optimization.Source: §4.3 · pp. 5–6

4.4 RUDDER

• Rudder’s fused kernels apply scaling and gains during activation quantization, then perform low-bit GEMM with dequantization, inverse-gain compensation, and bias; only integer activations and row scales pass between kernels.Table 1 reports closed-loop task success on LIBERO and RoboLab; LIBERO Avg. covers the four suites.Source: §4.4 · p. 6, Figure 3 · p. 6, Table 1 · p. 7

5.1 SETUP

• The evaluation covers three WAMs on LIBERO and RoboLab, comparing W4A8 and W4A4 against precision-matched quantization baselines and FP references.Control is assessed by closed-loop success rate; efficiency measures include denoising latency, peak GPU memory, and packed weight storage.Source: §5.1 · p. 7

5.2 TASK SUCCESS

• 0.8 pp is the maximum average SR decrease versus FP across LIBERO models and precision settings; on RoboLab, SteerQuant beats all evaluated same-precision baselines.LIBERO W4A8 averages reach 98.6% on Cosmos-Policy and 98.1% on FastWAMJoint; RoboLab W4A8 reaches 14.67% and W4A4 reaches 17.33%.Source: §5.2 · p. 7

5.3 INFERENCE EFFICIENCY AND MEMORY FOOTPRINT

• Under W4A4, SteerQuant achieves 2.229×, 1.859×, and 1.690× speedups over FP on Cosmos-Policy, FastWAMJoint, and Cosmos-Edge, respectively.It has the lowest evaluated denoising latency at both precisions on all three models; Table 2 reports latency and speedup alongside memory and storage.Source: §5.3 · p. 8, Table 2 · p. 8
• Across models and precision settings, SteerQuant reduces peak GPU memory by 24.7–72.2% and packed weight storage by 31.3–73.4% relative to FP.Peak memory includes resident weights, activations, and temporary buffers; packed storage covers stored model weights only.Source: §5.3 · p. 8, Table 2 · p. 8

5.4 ABLATION STUDIES

• The full method reduces standardized action RMSE by 24.79% versus the base quantizer, and uniform weighting increases RMSE relative to map-derived weighting.Shared-Weight Error Routing alone beats Stream-Specific Activation Modulation alone, while combining both further improves action fidelity.Source: §5.4 · p. 8, Table 3 · p. 8
• In 935 of 1,120 layer–step units (83.48%), map-derived weighting lowers high-impact-group error while increasing low-impact-group error.The groups are the top and bottom three streams ranked by reference action-impact scores estimated on the calibration set; Figure 4 plots changes relative to uniform weighting.Source: §5.4 · pp. 8–9, Figure 4 · p. 9

5.5 PERFORMANCE ON REAL-WORLD TASKS

• On real dual-arm tasks, W4A8 reaches 82.41% average task score versus 75.93% for BF16, with a 1.35× end-to-end inference speedup.The median latency measurement excludes robot motion execution time.Source: §5.5 · p. 9, Table 4 · p. 9

• Efficiency evaluation is limited to an NVIDIA RTX 4090; extending Rudder to other GPU architectures and edge devices would broaden deployment opportunities.Source: §6 · p. 9
• Evaluation covers three WAMs; additional architectures and scales remain future directions for assessing applicability.Source: §6 · p. 9
• Real-world experiments cover three manipulation tasks on a dual-arm robot; more tasks and robot platforms would broaden evidence of practical benefits.Source: §6 · p. 9

7 CONCLUSION

• SteerQuant guides error allocation by downstream action impact, coordinating shared weights with stream-specific activation ranges across denoising steps while retaining one static quantized weight matrix per layer at fixed bit-widths.Rudder integrates the transformations into fused low-bit kernels.Source: §7 · pp. 9–10
• Across three WAMs, SteerQuant preserves near-BF16 LIBERO success, outperforms evaluated same-precision baselines on RoboLab, and reaches up to 2.23× denoising speedup with reduced GPU memory usage.Real-world W4A8 deployment delivers a 1.35× end-to-end inference speedup while maintaining average task success.Source: §7 · p. 10
• The reported findings support downstream action impact as a useful guide for efficient quantization of coupled world–action computation.Source: §7 · p. 10

A EXPERIMENTAL DETAILS FOR PRELIMINARY ANALYSIS

• The controlled intervention evaluation uses matched predictions and sampler seeds, with only intervened actions executed; action RMSE is averaged over queries and then summarized across nine episodes.The setup targets one Linear output in the 28-block Transformer across specified tasks, initial states, and denoising conditions.Source: A EXPERIMENTAL DETAILS FOR PRELIMINARY… · p. 12, Table 5 · p. 13

B.1 CALIBRATION SETUP

• Calibration uses 320 LIBERO observations with fixed conditioning and diffusion draws; its 28 blocks, five steps, and ten Linear families define 10,360 stream-aligned regions.The map analysis uses Cosmos-Policy W4A8 and 16 Rademacher projections.Source: §B.1 · pp. 12–13

B.2 EFFICIENT MAP ESTIMATION

• Projected reverse passes estimate regional action-impact scores without dense Jacobians, using O(NcalP) reverse passes rather than O(NcalR) downstream replays for R regions.The estimator targets a first-order response under a Lipschitz condition on the action mapping along the perturbation segment.Source: §B.2 · p. 13

B.3 MAP STRUCTURE

• Action-stream impact varies with location: it dominates from Block 17 onward, while the highest-scoring stream shifts from current proprioception at Steps 0–2 to future wrist image at Step 3 and action at Step 4.Figure 5 normalizes each column by its maximum; colors compare streams within a column, and its marker indicates Block 17.Source: §B.3 · p. 13, Figure 5 · p. 14

B.4 MAP STABILITY

• Across 10,360 regions, 200 disjoint 160/160 splits yield median Spearman correlation 0.967 and top-10% overlap 0.979; Table 6 reports convergence to the 320-observation map.The reported percentile intervals are 0.953–0.974 for correlation and 0.974–0.984 for overlap.Source: §B.4 · p. 14, Table 6 · p. 14

C CALIBRATION WEIGHTS AND ACTIVATION CONSTRAINTS

• Calibration converts squared map scores into positive weights, with η ensuring positivity and ρ < 1 tempering large scores; shared-weight and activation calibration normalize across different groups.Source: C CALIBRATION WEIGHTS AND ACTIVATION… · p. 14

C.1 EFFECTIVE ACTIVATION RANGES

• Increasing a stream gain narrows its effective activation clipping range and rounding intervals, while shared-gain bounds and token-weighted normalization constrain the available range choices.The map weights guide trade-offs when streams prefer incompatible settings; the activation-error target uses unquantized activations and fixed quantized weights.Source: §C.1 · pp. 14–15

Quantization

• Weights use 4 bits and activations use 8 or 4 bits in W4A8 or W4A4, respectively, while normalization, RoPE, Softmax, GELU, the KV cache, and residual additions remain in BF16.Quantization covers attention query, key, value, and output projections, plus FFN up- and down-projections.Source: D ADDITIONAL EXPERIMENTAL DETAILS · p. 15

Calibration and evaluation

• Calibration uses 320 LIBERO observations for Cosmos-Policy, 100 for FastWAMJoint, and 512 DROID observations for Cosmos-Edge, with evaluation sets kept separate and matched initial states and seeds across methods.Evaluations use 3,000, 2,000, and 1,200 rollouts for Cosmos-Policy, FastWAMJoint, and Cosmos-Edge, respectively.Source: D ADDITIONAL EXPERIMENTAL DETAILS · p. 15

Efficiency

• Efficiency benchmarks run on an NVIDIA RTX 4090, with Cosmos-Policy latency measured over 20 timed sequences after five warm-ups and speedup reported relative to each model’s BF16 reference.Source: D ADDITIONAL EXPERIMENTAL DETAILS · p. 15

Real-world evaluation

• Real-world evaluation compares BF16 and W4A8 SteerQuant using the same fine-tuned FastWAMJoint checkpoint and 12 trials per task under matched initial conditions.Pack Objects uses a score of k/3 for k successfully handled objects; other tasks use binary success.Source: D ADDITIONAL EXPERIMENTAL DETAILS · p. 15