RawVLA: Embodied Neural Image Signal Processor For Robotic Manipulation

1 INTRODUCTION

• Existing VLA systems typically use camera-rendered RGB and treat the upstream ISP as fixed, task-neutral preprocessing.The camera transforms RAW sensor measurements into RGB before model inference.Source: §1 · p. 2
• ISP choices can change VLA behavior for an unchanged scene, especially under challenging illumination where processing may obscure task-relevant details.The cited conditions include low light, high dynamic range, backlighting, and unusual illumination.Source: §1 · p. 2
• RAW measurements retain sensor information that rendering may weaken or lose and allow RAW-to-RGB processing to be adapted to the scene, task, and downstream model.The paper identifies how strongly ISP choices affect VLA behavior and whether adaptive processing can improve manipulation as open questions.Source: §1 · p. 2
• ISP dimensions affect VLA behavior to substantially different degrees, motivating RAW-domain evaluation and adaptive ISP design.The study varies five dimensions while holding the scene, task, and robot state fixed.Source: §1 · p. 2
• RawVLA adaptively renders RAW inputs for downstream VLA models, while RawVLA-Bench makes image processing an explicit variable across clean and challenging acquisition conditions.RawVLA adapts behavior-relevant imaging factors without modifying the VLA backbone.Source: §1 · p. 2

2 ANALYZING VLA SENSITIVITY TO ISP

• The study synthesizes pseudo-RAW inputs and isolates five ISP factors through a factorized RAW-to-RGB mapping while keeping the observation fixed.The experiments are conducted on LIBERO and RoboTwin 2.0.Source: §2 · p. 3

Exposure

• Exposure is the most influential axis: performance stays high only near default settings and collapses under severe underexposure or overexposure.WAM approaches have the narrowest stable range, and fine-tuned models on RoboTwin 2.0 remain vulnerable at extreme exposure settings.Source: §2 · p. 3

Sensor Noise

• VLA models tolerate mild sensor noise after brightness restoration but degrade rapidly from moderate noise onward; WAMs degrade earlier and more sharply.On RoboTwin 2.0, Qwen3-OFT retains partial success under high noise whereas FastWAM approaches complete failure.Source: §2 · p. 3

Chromatic Response

• Chromatic response reveals policy-dependent color preferences: WAM agents prefer warmer LIBERO renderings, and RoboTwin policies retain white-point-dependent bias after visual randomization.VLA PI-family models are comparatively robust to chromatic changes; the RoboTwin policies remain stable across relative-color transformations.Source: §2 · p. 3

Tonal Response

• VLA policies retain high success near the default tonal response but degrade abruptly at very flat or highly contrasted settings; WAMs are more sensitive across the range.Both model families on RoboTwin 2.0 lose substantial performance at tonal extremes.Source: §2 · p. 4

Bit Depth

• Bit depth affects both model families less than other ISP dimensions: VLA models remain stable to three bits and decline at two bits, while WAMs weaken earlier.Performance is largely stable above two bits and drops only under the most aggressive quantization.Source: §2 · p. 4

3 RAWVLA

• RawVLA selectively adapts behavior-relevant photometric controls without changing the downstream VLA policy.The method is organized around streaming formulation, recurrent luminance–chroma conditioning, structured decoding, and action-driven optimization.Source: §3 · p. 4

3.1 CAUSAL STREAMING FORMULATION

• RawVLA denoises consecutive RAW frames and recurrently updates ISP parameters to produce policy-facing observations while retaining temporal context.The recurrent state supports illumination and color adaptation, and the explicit parameter set controls the current RAW-to-RGB processing.Source: §3.1 · p. 4

3.2 RECURRENT PHOTOMETRIC CONDITIONING

• Decoupled luminance and chroma states address ambiguity between brightness and color controls; luminance features inform exposure and tone, while scale-invariant chroma statistics limit brightness leakage into color estimation.The luminance pathway combines spatial structure and intensity-distribution descriptors; the chroma pathway uses RAW-channel chromaticities and log-chrominance ratios.Source: §3.2 · pp. 4–5

3.3 STRUCTURED NEURAL ISP DECODING

• The decoder predicts exposure, white balance, color correction, and a shared monotonic tone curve from the recurrent state, following fixed six-frame burst denoising.The tone curve is achromatic, endpoint-preserving, and identity-mapped when its logits are zero.Source: §3.3 · pp. 5–6

3.4 ACTION-DRIVEN OPTIMIZATION

• RawVLA is trained through the downstream policy’s native action objective with the VLA parameters fixed, so pixel-wise supervision is unnecessary.Pathway-specific brightness and chroma regularizers address learning difficulties under extreme illumination.Source: §3.4 · p. 6

4 RAWVLA-BENCH

• The benchmark varies simulator illumination before RAW formation across five regimes: ExtremeLow, Low, Normal, Over, and ExtremeOver.Environment-specific ranges are calibrated for comparable visual difficulty.Source: §4 · pp. 6–7
• LIBERO success rates are reported for Spatial, Object, Goal, Long (LIBERO-10), and Avg, which averages all four suites.SOG averages Spatial, Object, and Goal.Source: Table 1 · p. 7
• LIBERO pairs normal- and target-light observations at identical states and actions, retaining 1,771 successful trajectories from 2,000 replayed demonstrations.MuJoCo RGB8 is unprocessed into three-channel pseudo-RAW10 for 256 × 256 agent and wrist views.Source: §4 · p. 7
• RoboTwin 2.0 provides 632 successful trajectory pairs from 650 demonstrations, with synchronized target-light frames formed from pre-tonemap HDR buffers and heteroscedastic sensor noise.The benchmark uses head and dualwrist views at 240×320.Source: §4 · p. 7

Experiments

• RawVLA trains per frozen policy for 2,000 updates, while single-image ISP baselines are jointly optimized over 8-frame trajectory windows to address temporal inconsistency.RawVLA instead unrolls recurrent endpoints and applies full BPTT.Source: §5 · p. 7
• The comparison uses a fixed Default ISP and four learning-based neural ISPs, evaluating task success with 50 rollouts per task using distinct random seeds and initial positions.The evaluated backbones include PI0, PI-0.5, and Qwen3-OFT on LIBERO, and PI-0 and PI-0.5 on RoboTwin 2.0.Source: §5 · p. 7

Evaluation on LIBERO

• RawVLA raises LIBERO cross-backbone average success from 43.01% for the strongest baseline to 68.82%, with gains across task types and horizons and largely preserved normal-light performance.Gains are driven primarily by adverse illumination, especially Qwen3-OFT under overexposure.Source: §5 · pp. 7–8

Evaluation on Real-World Dual-Arm

• Under real-world normal illumination, RawVLA achieves 75.00% average success versus 76.50% for Default; under low light, it reaches 66.50% versus 0.00% for Default and 11.00% for DarkISP.The evaluated ISP modules run faster than the cameras’ native frame rate, with RawVLA reaching 168 FPS.Source: Table 3 · p. 8, §5 · p. 8, Table 4 · p. 8

Evaluation on RoboTwin 2.0

• On RoboTwin 2.0, RawVLA reaches 54.98% cross-backbone average success, an 85.81% relative improvement over the strongest neural ISP baseline.It ranks first in every adverse illumination regime for both policies, with the largest gains under low light and improvements under overexposure.Source: Table 2 · p. 8, §5 · p. 8

Photometric Descriptors

• The complete photometric descriptor set averages 62.24% success on LIBERO and 57.12% on RoboTwin 2.0, versus 36.35% and 41.93% after removing spatial feature gt.The spatial feature captures luminance structure.Source: §6 · p. 8
• Removing chroma causes the largest descriptor-related drop on LIBERO, while global luminance is especially important on RoboTwin 2.0.Together, the findings support separate descriptors for intensity distribution, spatial structure, and color.Source: §6 · p. 8

Recurrent Estimator

• Removing either hidden state or previous-parameter conditioning sharply degrades LIBERO performance and consistently lowers RoboTwin 2.0 success.A shared GRU underperforms the decoupled design, particularly on RoboTwin 2.0; luminance-only and chroma-only states are each inferior to their combination.Source: §6 · pp. 8–9
• The recurrent ablations indicate that temporal context, parameter feedback, and factorized luminance–chroma estimation each contribute to robust adaptation.Additional ablations cover training losses, individual ISP operators, and burst-denoising settings in Appendix G.Source: §6 · p. 9

VLA & WAM Visual Robustness

• Visual robustness work evaluates changes in appearance, lighting, viewpoint, scene composition, and physical conditions, and methods edit regions, enforce action consistency, restore observations, or learn invariance.Additional sensing modalities can improve robustness but require extra sensors and cross-modal data; this work instead uses the existing camera signal and studies RAW and ISP processing.Source: §7 · p. 9

Task-Oriented and Neural ISP

• Task-oriented and neural ISP research has co-optimized RAW processing with recognition, developed compact or content-adaptive pipelines, and addressed adverse-condition RAW perception.This work extends that direction to sequential VLA behavior and manipulation success.Source: §7 · p. 9

8 CONCLUSION

• RawVLA is a lightweight neural ISP for frozen VLA policies.The paper also introduces RawVLA-Bench, a RAW-domain manipulation benchmark.Source: §8 · p. 9
• RawVLA maintains performance under nominal conditions across simulation and real-world tasks.Source: §8 · p. 9
• RawVLA substantially improves robustness to adverse illumination, demonstrating adaptive camera processing as support for robust robot control.Source: §8 · p. 9

RAW Input Representation

• RawVLA takes linear RAW: a black-level-corrected, white-level-normalized, demosaiced three-channel signal.Simulated observations use three-channel pseudo-RAW in the same representation, without a Bayer mosaic.Source: A RGB UNPROCESSING AND DEFAULT ISP · p. 10

RGB Unprocessing

• The simulated pseudo-RAW pipeline inverts the default tone curve and sRGB transfer function, removes fixed RGB gains, and clips the result.The resulting RAW retains the corresponding renderer’s available precision.Source: A RGB UNPROCESSING AND DEFAULT ISP · p. 10

Default ISP

• The paired default ISP reverses the fixed photometric transformations using its default parameter setting.Source: A RGB UNPROCESSING AND DEFAULT ISP · p. 10

Scope of the representation

• The baseline pseudo-RAW is not a camera-specific Bayer measurement and does not simulate mosaicing, sensor noise, or a camera-specific color-correction matrix.The sensor model is introduced separately for the noise axis.Source: A RGB UNPROCESSING AND DEFAULT ISP · p. 10

B RAWVLA-BENCH CONSTRUCTION

• The benchmark pairs target-light RAW sequences with normal-light RGB references under the same actions and physical states, with 2,403 training trajectories and 13,250 evaluation rollouts.Each training trajectory is assigned one illumination regime; evaluation expands each task initialization across all five regimes.Source: B RAWVLA-BENCH CONSTRUCTION · p. 10

B.1 LIBERO CONSTRUCTION

• LIBERO construction retains 1,771 successful trajectories, then forms 10-bit pseudo-RAW with synthetic heteroscedastic noise across five illumination regimes.Its pseudo-RAW is not Bayer data and contains no additional renderer highlight information beyond the source RGB8.Source: §B.1 · pp. 11–12

B.2 ROBOTWIN 2.0 CONSTRUCTION

• RoboTwin construction yields 632 complete paired trajectories; its three-channel pseudo-RAW is stored as float32 without 10-bit quantization.Evaluation spans 13 tasks, 50 seeds per task, and five illumination regimes for 3,250 rollouts.Source: §B.2 · pp. 12–13

C.1 IMPLEMENTATION DETAILS

• RawVLA predicts explicit ISP controls and analytically renders RGB after burst fusion, exposure, white balance, color correction, and a shared monotonic tone curve.The implementation has 886,966 trainable parameters and omits a spatial image decoder or residual branch.Source: §C.1 · p. 14

C.2 BURST DENOISING

• Six-frame causal burst denoising uses frequency-wise reliability to down-weight motion and misalignment, retaining sharper dynamic content than direct averaging.The figure compares both approaches on LIBERO and RoboTwin 2.0 using K = 6 frames.Source: §C.2 · p. 14, Figure 8 · p. 15

C.3 LUMINANCE DESCRIPTOR; C.4 CHROMA DESCRIPTOR

• The luminance descriptor summarizes intensity histograms, quantiles, and boundary ratios, while the chroma descriptor summarizes distributions of chromaticity and log-chrominance maps.Source: §C.3 · p. 15, §C.4 · p. 15

C.5 STRUCTURED ISP PARAMETERIZATION

• White balance uses two degrees of freedom for relative channel gains, separated from global exposure.Source: §C.5 · p. 15

D.1 FROZEN-POLICY TRAINING PROTOCOL; D.2 BASELINE NEURAL ISP CONFIGURATIONS

• Training freezes each VLA backbone and action head, gives all ISP methods the same pseudo-RAW input, and uses a shared 2,000-step optimization budget.Baseline training samples trajectory-local windows and averages action losses across observations without recurrence or explicit temporal-consistency loss.Source: §D.1 · p. 16, §D.2 · p. 16

D.3 RAWVLA TRAINING AND TEMPORAL SAMPLING

• RawVLA trains on six recurrent endpoints, each with a six-frame causal burst, and backpropagates through the full six-update sequence without detaching state or prior ISP parameters.The final policy image comes from the terminal burst; earlier calls contribute through recurrent state.Source: §D.3 · p. 16
• Policy-native action losses are scaled against fixed brightness and chroma regularizers, with paired default-lighting RGB used only for checkpoint selection.Paired RGB does not contribute to the training loss.Source: §D.3 · p. 17

E RAWVLA-BENCH RENDERING VISUALIZATIONS

• RawVLA preserves task-relevant scene structure across LIBERO camera views despite baseline differences in brightness, color, and local contrast from identical RAW observations.RoboTwin examples show similar rendering variation across three cameras and a different renderer; closed-loop results measure manipulation utility.Source: E RAWVLA-BENCH RENDERING VISUALIZATIONS · p. 17

• The haze transform mixes 95% gray-white airlight (Iair = 235) into clean RGB, retaining 5% of the original signal; evaluation adds no sensor noise or blur and uses RGB rather than sensor RAW.Source: F ADDITIONAL ROBUSTNESS EVALUATION… · p. 18
• A haze-specific RawVLA instance was trained for 1,500 updates on paired LIBERO views with clean-frame reconstruction targets, which were unavailable during closed-loop evaluation.At each simulator step, RawVLA processes both views before the same frozen PI-0 policy; inference uses the standard causal procedure without future observations or clean reference frames.Source: F ADDITIONAL ROBUSTNESS EVALUATION… · pp. 18–19
• Across four equally sized suites, PI-0’s average success rate rises from 66.5% to 88.4% with RawVLA enhancement, a 21.9 percentage-point improvement.Table 7 reports closed-loop suite success rates and the equally weighted macro-average.Source: Table 7 · p. 20, F ADDITIONAL ROBUSTNESS EVALUATION… · p. 20
• On held-out image pairs, enhancement reduces mean absolute error from 0.4271 to 0.1688 and raises PSNR from 6.56 to 13.60 dB.Source: F ADDITIONAL ROBUSTNESS EVALUATION… · p. 20
• On LIBERO-10, RawVLA enables PI-0 to succeed in 230 rollouts that fail without enhancement, versus 22 rollouts showing the opposite outcome, yielding 208 additional successes (McNemar p = 7.16 × 10−45).Source: F ADDITIONAL ROBUSTNESS EVALUATION… · p. 20

G ADDITIONAL ABLATION STUDIES

• 51.32% on LIBERO and 57.12% on RoboTwin 2.0 are reached when action, brightness, and chroma objectives are combined, balancing policy alignment with adaptation under adverse illumination.Action supervision alone preserves normal-light performance but fails under severe illumination changes; photometric auxiliaries alone improve robustness without fully aligning rendered observations to the frozen policy.Source: G ADDITIONAL ABLATION STUDIES · p. 21
• Removing exposure reduces average success from 62.24% to 50.79% on LIBERO and from 57.12% to 43.35% on RoboTwin 2.0, while other operators also contribute.White balance and color correction provide complementary illumination gains; tone mapping supports robustness, and a shared achromatic tone curve nearly matches the full model.Source: G ADDITIONAL ABLATION STUDIES · pp. 21–22, Table 9 · p. 22, Table 10 · p. 22
• Longer bursts improve LIBERO denoising, with diminishing returns on RoboTwin 2.0; moderate fusion works best on both, and frequency-wise reliability and recurrent rendering support noise suppression and temporal stability.The recurrent RawVLA formulation avoids abrupt rendering changes and supports successful execution in an overexposed LIBERO trajectory, unlike frame-wise RAW-Adapter.Source: G ADDITIONAL ABLATION STUDIES · pp. 22–23, Table 11 · p. 22

H REAL-WORLD EXPERIMENTAL SETUP

• The real-world setup uses three synchronized cameras streaming 12-bit Bayer RAW at 30 FPS and collects over 200 trajectories per task; PI-0.5 is fine-tuned on demonstrations, then frozen while one ISP is trained for normal and synthetic low light.The three camera streams are photometrically calibrated to matched illumination; deployment uses the same learned ISP and normally fine-tuned policy across both lighting conditions.Source: §H.1 · p. 23, §H.2 · pp. 23–24, §H.3 · p. 24

H.4 QUALITATIVE TASK EXECUTIONS

• In four real-world tasks, the Default ISP yields nearly zero-valued low-light observations and stalls, Dark-ISP recovers some structure but stalls early, while RawVLA preserves usable views and completes every sequence.Low-light runs take longer because the policy repeatedly refines end-effector poses during precise operations.Source: §H.4 · p. 25, Figure 16 · p. 26, Figure 17 · p. 26

I WAM ILLUMINATION ROBUSTNESS ON LIBERO

• Across the evaluated WAM backbones, learned ISPs produce no substantial or consistent LIBERO robustness gain; the strongest averages differ by only 1.19 percentage points for WM4A-Wan and 0.14 for WM4A-Cosmos.WAMs are sensitive to pixel-level appearance changes, and action supervision through the VAE-based visual-generation pathway may provide weak or distorted ISP gradients.Source: I WAM ILLUMINATION ROBUSTNESS ON LIBERO · p. 28, Table 13 · p. 28

J LIMITATIONS AND FUTURE STUDIES

• The study’s scope is limited by simulated three-channel pseudo-RAW, separate ISPs per policy, and a small set of indoor tasks and lighting conditions; broader physical-sensor, cross-policy, outdoor, and adverse-weather evaluations remain open.The severe-haze experiment also uses synthesized RGB observations rather than physically captured RAW data.Source: J LIMITATIONS AND FUTURE STUDIES · p. 28

K CONTROLLED ISP PERTURBATION PROTOCOL

• The controlled protocol perturbs linear pseudo-RAW before rendering to model exposure and acquisition noise, and separately varies chromatic response, tone mapping, and bit depth; environment and backbone settings specify sweep coverage.Exposure is applied before clipping and RAW10 quantization; noise is added after capture attenuation and before recovery, while bit depth can be varied at RAW or rendered RGB stages.Source: K CONTROLLED ISP PERTURBATION PROTOCOL · p. 29, §K.1 · pp. 29–30, §K.2 · p. 31

K.3 ROBOTWIN 2.0 PERTURBATION SETTINGS

• RoboTwin 2.0 perturbations use specified exposure, chromatic, tonal, and bit-depth settings, while noise draws are deterministic from seed, frame, view, and capture exposure.The visual specifies exposure offsets of −3 to +3 EV and bit depths from 7 to 2; chromatic settings use δwp = 0.075 and s = 5.0, and quantization applies to default-ISP RGB.Source: Figure 21 · p. 32, §K.3 · p. 32

L FULL ISP SENSITIVITY RESULTS

• The LIBERO sensitivity results cover exposure, physical sensor noise, white-point, relative color, tonal response, and bit depth across six VLA checkpoints.The results are organized across LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, LIBERO-10, and their micro-average; the supplied passages identify the tables but do not provide their cell values.Source: L FULL ISP SENSITIVITY RESULTS · p. 32

L.1 ROBOTWIN 2.0 ISP PERTURBATION RESULTS

• RoboTwin 2.0 evaluates five matched ISP perturbation axes on Qwen3-OFT and FastWAM, with Easy and Hard results over 50 rollouts on each of 13 tasks.Exposure uses direct RGB inputs shared by both backbones; Original denotes unperturbed RGB observations.Source: §L.1 · p. 42

M EFFECT OF RAW & RGB NORMALIZATION

• RAW-domain exposure normalization substantially improves average success for most VLA baselines, particularly under severe shifts, though the benefit varies by architecture.Linear RAW retains more scene-referred intensity information before tone mapping and clipping, preserving recoverable shadow and highlight structure for the ISP and policy.Source: M EFFECT OF RAW & RGB NORMALIZATION · p. 44, Table 26 · p. 44