PaperlayerView exact revision on arXiv

Source-linked AI summary

POS-ISP: Pipeline Optimization at the Sequence Level for Task-aware ISP

Jiyun Won, Heemin Yang, Woohyeok Kim, Jungseul Ok, Sunghyun Cho

arXiv:2604.06938v1cs.CV

TL;DR

Task-aware ISP optimization must jointly select modular pipelines and parameters despite limitations in NAS and stepwise RL. POS-ISP predicts the complete sequence and parameters in one forward pass and evaluates it with a terminal task reward. Across object detection, instance segmentation, and image enhancement, it achieves strong task performance with substantially lower computational and memory overhead, while longer unconstrained pipelines can increase end-to-end latency.

  • Problem

    Task-aware ISP optimization must jointly optimize module sequences and parameters, but NAS has training-inference inconsistency and stepwise RL incurs unstable training and repeated computationally costly decisions.

  • Method

    POS-ISP uses sequence-level reinforcement learning with recurrent sequence and parameter predictors to generate a complete image-adaptive ISP pipeline in one forward pass and score it using a terminal task reward.

  • Results

    POS-ISP achieves state-of-the-art accuracy on object detection and instance segmentation, promising image-enhancement results, and substantially lower computational and memory overhead.

  • Takeaways & Limitations

    Sequence-level optimization provides a stable and efficient paradigm for task-aware modular ISP optimization across multiple downstream tasks.

  • Takeaways & Limitations

    Unconstrained pipeline lengths can increase end-to-end inference latency, and separate training for each downstream task limits multi-task scalability.

Abstract

from arXiv · show

Recent work has explored optimizing image signal processing (ISP) pipelines for various tasks by composing predefined modules and adapting them to task-specific objectives. However, jointly optimizing module sequences and parameters remains challenging. Existing approaches rely on neural architecture search (NAS) or step-wise reinforcement learning (RL), but NAS suffers from a training-inference mismatch, while step-wise RL leads to unstable training and high computational overhead due to stage-wise decision-making. We propose POS-ISP, a sequence-level RL framework that formulates modular ISP optimization as a global sequence prediction problem. Our method predicts the entire module sequence and its parameters in a single forward pass and optimizes the pipeline using a terminal task reward, eliminating the need for intermediate supervision and redundant executions. Experiments across multiple downstream tasks show that POS-ISP improves task performance while reducing computational cost, highlighting sequence-level optimization as a stable and efficient paradigm for task-aware ISP. The project page is available at https://w1jyun.github.io/POS-ISP

1. Introduction

Task-aware ISP optimization must jointly choose module sequences and parameters, but existing NAS and stepwise RL approaches introduce inference mismatch, unstable training, or high computational cost. POS-ISP instead predicts complete pipelines in one pass and evaluates them with a final task reward, achieving strong multi-task results with lower resource usage.

  • Motivation: Fixed ISP chains often misalign with task objectives, while manual tuning requires time-consuming adjustment of many coupled parameters.The affected objectives include visual appearance, object detection, and semantic segmentation.
  • Limitations of Existing Methods: NAS introduces training-inference inconsistency through mixture training, whereas stepwise RL requires repeated evaluations and future reward estimation.These limitations contribute to unstable training and high computational overhead in modular ISP optimization.
  • POS-ISP: POS-ISP predicts the entire ISP pipeline and its parameters in a single forward pass, directly optimizing a terminal task reward.This sequence-level formulation avoids intermediate supervision and redundant stage-wise executions.
  • POS-ISP: A recurrent sequence predictor uses preceding modules as context to model inter-module dependencies, while a parameter predictor generates image-adaptive module parameters.The two predictions jointly define the complete ISP pipeline whose output is evaluated by task performance.
  • Evaluation: POS-ISP is evaluated on object detection, instance segmentation, and image enhancement, outperforming other task-aware methods with lower computational cost and memory footprint.The reported evaluation includes both quantitative and qualitative comparisons.

2. Related Work

Prior task-aware ISP methods use differentiable proxies or stepwise reinforcement learning to optimize modular pipelines. POS-ISP replaces stage-wise decisions with sequence-level prediction in one forward pass, yielding more stable training and improved computational efficiency.

  • Task-aware ISP Optimization: End-to-end neural ISP methods learn RAW-to-RGB mappings, while modular designs retain interpretability and computational efficiency.Modular pipelines use established operations such as white balance and denoising.
  • Differentiable Search: ReconfigISP uses differentiable proxy networks to enable gradient-based optimization of modular ISP sequences and parameters.The proxy approach addresses the non-differentiability of the search problem.
  • Reinforcement Learning: DRL-ISP sequentially selects modules, while AdaptiveISP searches discrete sequences and predicts continuous module parameters within an RL framework.Both approaches formulate ISP search as sequential decision-making.
  • POS-ISP: POS-ISP performs sequence-level optimization in a single forward pass without intermediate supervision, resulting in more stable training and improved computational efficiency.This contrasts directly with stepwise RL approaches.

3. POS-ISP

POS-ISP jointly searches task-specific ISP module sequences and image-adaptive parameters, using sequence-level prediction and terminal task rewards. Its recurrent architecture models module dependencies while a parameter predictor generates parameters for selected modules.

  • 3.1. Problem Formulation: POS-ISP represents an ISP pipeline as an ordered sequence of modules with corresponding parameters, mapping RAW input to sRGB output.Each candidate module has its own parameters, and the formulation assumes each module is sampled at most once.
  • 3.1. Problem Formulation: The framework learns one task-specific module sequence while predicting parameters adapted to each input image.The selected sequence is shared across images for a task, whereas the parameter predictor produces image-specific settings.
  • 3.2. Network Architecture: The sequence predictor models a distribution over complete module sequences, including start and end tokens so pipelines can have arbitrary lengths.A recurrent GRU architecture captures sequential dependencies and conditions each prediction on preceding modules.
  • 3.2. Network Architecture: At each recurrent step, the previous module and hidden state are used to predict a probability distribution over candidate modules.The hidden state encodes preceding-module context, and an MLP decoder with softmax produces the conditional distribution.
  • 3.2. Network Architecture: Temperature-controlled sampling balances exploration and exploitation while complete pipelines are sampled until the end token.The sequence predictor is trained by evaluating the downstream task performance of sampled pipelines.
  • 3.2. Network Architecture: A lightweight CNN parameter predictor processes a downsampled 64×64 image and outputs parameters for all candidate modules, applying only those selected in the sequence.The framework jointly trains sequence and parameter predictors for downstream task optimization.
  • 3.3. ISP Search: The reward compares target-task performance between input and output images while penalizing implausible outputs.For object detection, the target loss is supplied by a pretrained detector; the penalty discourages extreme exposure shifts using bounds 0.01 and 0.9.
  • 3.3. ISP Search: REINFORCE updates the sequence predictor to increase the likelihood of sampled module sequences receiving high rewards.The policy objective weights the summed log probabilities of selected modules by the pipeline reward.

4. Experiments

POS-ISP is evaluated across object detection, instance segmentation, and image enhancement, with comparisons covering task performance, qualitative output, optimization behavior, and efficiency. The experiments show strong task results, more stable optimization, and lower computational cost than stepwise RL baselines.

  • Object detection: POS-ISP achieves the highest accuracy on object detection by using stable sequence-level optimization with accurate final rewards.Task-driven methods generally outperform the in-camera ISP, while ReconfigISP and stepwise RL methods remain limited by training-inference mismatch or unstable reward estimation.
  • Instance segmentation: Task-driven ISPs outperform the in-camera ISP on instance segmentation but remain inferior to input RAW images.Dense prediction produces high-variance pixel-level rewards; POS-ISP addresses this by optimizing directly with task-level supervision rather than unstable value estimates.
  • Image enhancement: POS-ISP more closely matches Expert C’s Adobe FiveK retouching in brightness and color tones than the compared enhancement methods.The evaluation uses Expert C retouched images as ground truth, while competing methods exhibit underexposure, oversaturation, or color and white-balance shifts.
  • Training dynamics: POS-ISP improves steadily during training, whereas AdaptiveISP fluctuates or improves only marginally early in training.Policy entropy decreases and the selected pipeline’s relative likelihood increases by approximately 20--60× across datasets.
  • Inference efficiency: POS-ISP reduces inference overhead because it predicts the complete pipeline in one forward pass, unlike RL controllers that execute repeatedly during construction.DRL-ISP adds further cost through its heavy feature extractor; the comparison uses three modules for DRL-ISP and five for AdaptiveISP.
  • Ablation on sequence predictor: A recurrent sequence predictor improves performance over a probability-table variant by modeling contextual relationships and inter-module dependencies.The probability-table variant makes each module decision independently, whereas the recurrent structure models the influence of module order on sequence prediction.

5. Conclusion

POS-ISP jointly learns task-specific ISP module sequences and parameters, using sequence-level optimization to improve accuracy with lower computational and memory overhead. Its main limitations are search-space growth with more candidate modules and separate training requirements for different downstream tasks.

  • POS-ISP jointly learns module sequences and parameters for target downstream tasks.
  • POS-ISP eliminates unstable future reward estimation and stepwise decisions while modeling inter-module dependencies.
  • POS-ISP achieves state-of-the-art accuracy on object detection and instance segmentation with substantially lower computational and memory overhead.
  • The search space grows with more candidate ISP modules, potentially requiring longer training to converge.
  • Separate training for each downstream task limits scalability in multi-task settings and increases system complexity.

S1. Additional Results

Additional experiments evaluate POS-ISP across enhancement, depth estimation, detection, and segmentation. The results report strong performance across diverse tasks, including best performance across all listed depth metrics and improved qualitative enhancement behavior.

  • Additional Results: Additional visual examples cover object detection, instance segmentation, and image enhancement.
  • Image Enhancement: Existing task-driven ISP methods improve enhancement metrics over input images but often fail to reproduce the desired style fully.
  • Image Enhancement: POS-ISP achieves the best reported image-enhancement performance among the compared methods.
  • Depth Estimation: The depth-estimation evaluation uses RMSE supervision and standard metrics including AbsRel, SqRel, RMS, RMSlog, and accuracy (δ < 1.25).
  • Depth Estimation: POS-ISP consistently achieves the best performance across all depth-estimation metrics, including RMS, relative errors, and accuracy.
  • Overall Results: Across detection, segmentation, enhancement, and depth estimation, the reported gains extend beyond recognition-based tasks.

S2. Additional Analysis

Additional analyses examine stability and runtime. POS-ISP shows reproducible gains, avoids critic-network instability, and can run efficiently on mobile hardware, although unconstrained pipeline lengths may increase end-to-end latency.

  • Optimization Stability: POS-ISP consistently outperforms AdaptiveISP across three random seeds, indicating stable optimization and reproducible gains.
  • Optimization Stability: Prior stepwise RL critic losses exhibit oscillations and spikes during training.
  • Optimization Stability: Bootstrapped critic-estimate errors accumulate and propagate through policy updates, leading to unstable optimization.
  • Optimization Stability: POS-ISP eliminates the need for an unstable critic network through sequence-level optimization.
  • Runtime Analysis: Unconstrained pipeline lengths increase end-to-end latency relative to methods with explicit length constraints.
  • Runtime Analysis: A runtime penalty produces shorter pipelines and substantially lower end-to-end latency while maintaining comparable detection performance.
  • Runtime Analysis: On a Galaxy S10 CPU, POS-ISP runs about 4× faster than AdaptiveISP without quantization or hardware-specific acceleration.

S2.3. Fixed vs. Dynamic Pipeline Length

Pipeline-length ablations show that POS-ISP’s gains persist at fixed length and improve further with dynamic length selection. A single task-level sequence with per-image parameters also supports stable behavior across diverse inputs and illumination conditions.

  • Fixed Pipeline Length: POS-ISP outperforms AdaptiveISP even with the same five-stage sequence length and candidate module set.
  • Dynamic Pipeline Length: Dynamic pipeline length yields further improvements over both AdaptiveISP and the fixed-length POS-ISP variant.
  • Fixed Pipeline Length: The fixed-length ablation attributes performance gains to sequence-level optimization and final-output rewards rather than simply longer pipelines.
  • Single Sequence per Task: POS-ISP uses one sequence per task while adapting module parameters per image.
  • Single Sequence per Task: POS-ISP achieves the lowest degradation rate P(∆< 0) and mildest worst-case drops across both datasets.
  • Single Sequence per Task: Degradation rates remain comparable between Normal and Dark subsets: LOD 0.138 vs. 0.108 and LIS 0.199 vs. 0.174.
  • Single Sequence per Task: The results indicate that fixed sequences with per-image parameter adaptation can handle diverse image conditions while maintaining stable behavior.

S2.5. Generalization Across Optimization Settings

POS-ISP remains effective across different reward backbones and training objectives. Its sequence-level optimization consistently outperforms AdaptiveISP under varied depth-supervision losses.

  • Reward backbone: ReconfigISP performs worse than the input images under the YOLOv13 reward backbone because soft training mismatches hard inference.On LOD-All, the optimized pipeline can also overfit the Dark subset and produce overly aggressive enhancement.
  • Reward backbone: POS-ISP achieves the best performance with YOLOv13, demonstrating generalization across pretrained reward models.The comparison includes ReconfigISP and other RL-based methods under a different detection backbone.
  • Training objective: POS-ISP consistently outperforms AdaptiveISP across RMSE, AbsDiff, and AbsRel training objectives.The gains remain stable across different loss functions and evaluation metrics.

S2.6. Ablation on Parameter Predictor

The ablations examine whether image-only parameter prediction is sufficient and whether task-adapted module sequences matter. Image-only conditioning is favored for stability, while sequence choice remains decisive for downstream performance.

  • Parameter predictor: Image-only parameter prediction is sufficient in the reported training setting as sequence distributions concentrate on high-reward pipelines.Early high-entropy exploration makes sequence-conditioned parameter learning unstable, while later concentration supports image-only conditioning.
  • Task-adaptive sequence: Cross-task evaluation compares module sequences from different source tasks on object detection with parameters retrained for detection.The table highlights the best metrics among the evaluated source-task sequences.
  • Parameter predictor: Sequence-conditioned prediction has higher variance and lower performance than image-only prediction across random seeds.The sequence-conditioned variant is more expressive but unstable sequences interfere with parameter learning during early training.
  • Task-adaptive sequence: Sequences optimized for image enhancement or instance segmentation yield lower object-detection performance even after parameter retraining.The object-detection sequence preserves strong performance, indicating that parameters alone cannot recover a task-mismatched sequence.

S2.8. Image-adaptive Parameter Prediction

POS-ISP predicts parameters that adapt to illumination and scene conditions. Same-domain parameter application performs best, while cross-domain transfer degrades performance and changes output appearance.

  • Cross-exposure evaluation: Same-domain parameter prediction and application achieve the highest performance, whereas cross-domain transfer consistently reduces performance.The experiment estimates parameters from LOD-Dark or LOD-Normal images and applies them to LOD-Dark inputs.
  • Parameter distributions: LOD-Dark inputs receive stronger brightness compensation than LOD-Normal inputs in the predicted exposure- and tone-related parameters.Parameter histograms show noticeable shifts between the two illumination domains.
  • Qualitative analysis: Applying LOD-Normal parameters to LOD-Dark scenes produces underexposed or locally flattened outputs.The qualitative examples visualize the consequences of cross-domain parameter swapping.

S3. Implementation Details

POS-ISP uses a fixed pool of modular ISP operations, module-specific normalized parameters, and neural predictors for sequences and continuous parameters. The implementation includes recurrent sequence decoding with step conditioning and image-feature-based parameter prediction.

  • Parameterization: Module parameters are predicted in module-specific dimensions, normalized to [0, 1], then rescaled to each module’s numerical range.For module M_i, θ_i ∈ [0, 1]^d_i, where d_i is the number of parameters.
  • ISP module pool: The implementation selects modules from a pool of 10 candidate ISP operations, including standard operations such as white balance and tone mapping.The adopted module details are summarized in Table S13.
  • Tone mapping: Tone mapping predicts θ_t ∈ [0, 1]^8, rescales it to [0.5, 2.0]^8, and combines eight basis functions into a curve.The normalized weighted basis combination uses the predicted tone-mapping coefficients.
  • Contrast: Contrast adjustment rescales θ_c to [−1, 1] and blends the original RGB vector with an S-curve-transformed version.The luminance uses channel weights 0.27, 0.67, and 0.06.
  • Saturation: Saturation enhancement converts RGB inputs to HSV, strengthens saturation using value-dependent adjustment, and blends the result with the original pixel.The saturated pixel is converted back to RGB before blending.
  • Desaturation: Desaturation linearly interpolates between the original RGB vector and a grayscale vector using θ_d.The grayscale luminance is computed as 0.27 I_R + 0.67 I_G + 0.06 I_B.
  • Sequence predictor: The recurrent sequence predictor uses a GRU to autoregressively predict module indices and an <eos> token.A learnable step embedding is transformed into FiLM parameters to modulate the hidden state.
Loading 2604.06938v1…