PaperlayerView exact revision on arXiv

Source-linked AI summary

DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models

Zhuoming Liu, Jinhong Lin, Kwan Man Cheng, Lin Zhang, Shayok Bagchi, Yin Li

arXiv:2606.05758v1cs.CVcs.AIcs.LG

TL;DR

Precise continuous outputs remain difficult for VLMs built around discrete-token interfaces, especially for temporal localization and robotic control. DRIFT combines a coarse base predictor with flow-matching residual refinement, and consistently outperforms regression- and generative-based baselines across perception and planning tasks. The framework improves benchmark performance across multiple VLM architectures, including reported gains on temporal grounding and robotic control.

  • Problem

    Discrete-token interfaces are poorly suited to precise continuous outputs such as temporal event boundaries and robotic control actions.

  • Method

    DRIFT combines a base predictor with a flow-matching refinement module that models residual errors around an initial coarse prediction.

  • Results

    DRIFT consistently outperforms direct regression and standalone flow-matching decoders across perception and planning tasks, with gains of +2.6% on Charades-STA and +2.1% on ActivityNet Captions.

  • Takeaways & Limitations

    DRIFT provides a general adaptation framework for extending pretrained VLMs beyond discrete token generation toward precise continuous decoding.

  • Takeaways & Limitations

    DRIFT assumes a sufficiently informative base predictor, and its VLA experiments are limited to simulation environments.

Abstract

from arXiv · show

Many modern vision-language models (VLMs) build on autoregressive decoding of discrete tokens. While text-based output interfaces enable scalable pretraining and strong zero-shot generalization across diverse tasks, they are poorly suited for problems that require precise continuous outputs, such as localizing temporal boundaries of events or generating robotic control actions. To address this challenge, we propose DRIFT, a general framework for adapting pretrained VLMs to continuous decoding tasks. DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching that iteratively improves the prediction. This residual formulation transforms the generative modeling problem from learning a global output distribution to modeling a localized residual distribution around a strong prior, substantially simplifying optimization. We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control. Across multiple tasks and architectures spanning MLLMs, VLAs, and WAMs, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.

1 Introduction

Modern VLMs use scalable discrete-token decoding, but precise continuous outputs require adapting that interface without losing pretrained generality. DRIFT combines coarse prediction with residual flow refinement and improves results across perception and planning tasks.

  • Motivation: Discrete tokenization introduces quantization errors, discards ordinal structure, and misses fine-grained variation in continuous targets.The challenge arises in tasks such as temporal event localization and robotic control.
  • Limitations of Prior Approaches: Existing methods use token discretization or lightweight regression heads, while flow-based decoders require more data and optimization effort.Flow-based methods can also show high prediction variance in low-data regimes and model global structure that deterministic predictors already capture.
  • DRIFT: DRIFT cascades an efficient deterministic coarse predictor with a flow-matching refinement module that models residual errors around the initial estimate.This localized residual formulation replaces global distribution learning with modeling around a strong prior.
  • Evaluation: DRIFT evaluates visual grounding and robotic control across MLLMs, VLAs, and WAMs, consistently improving over regression and standalone flow-matching baselines.The evaluation spans both perception and planning tasks.
  • Results: +2.6% on Charades-STA and +2.1% on ActivityNet Captions are reported gains over the latest methods for temporal video grounding.On VLA tasks, Qwen3-VL-2B with DRIFT achieves 97.9% average Libero accuracy and 61.5% Simpler WidowX accuracy.

2 Related Work

Prior continuous-output strategies are often task-specific or retain limitations of tokenization and direct regression. DRIFT instead provides a general adapter that preserves pretrained VLM backbones while refining continuous predictions around learned priors.

  • VLMs: VLM research spans multimodal understanding and reasoning, as well as action-oriented models that connect visual inputs to behavior.The paper distinguishes MLLMs, VLAs, and WAMs within this broader VLM landscape.
  • Continuous-Output Tasks: Visual grounding localizes spatial or temporal extents, while VLA tasks predict continuous robot action trajectories from visual observations and language instructions.These tasks represent continuous-output settings studied by DRIFT.
  • DRIFT: DRIFT extends pretrained VLMs to continuous outputs through an efficient general adapter that refines predictions around a learned prior.The approach preserves the reusable Transformer backbone across perception and planning problems.
  • Flow Matching: Flow matching learns continuous velocity fields for transporting samples between distributions and has been applied to continuous value decoding in VLA systems.The paper uses flow matching as the generative refinement basis for DRIFT.
  • Method Positioning: DRIFT combines a base predictor with flow-matching refinement and provides theoretical justification for the resulting optimization advantages.This frames the method as a general adaptation framework rather than a task-specific decoder.

3 DRIFT: Decoding Continuous Output via Residual Flow Adapter

DRIFT adapts autoregressive VLMs for continuous decoding by combining a base predictor with a residual flow refinement process. Its design uses learned priors, gated refinement, and ODE inference, with experiments and theory supporting improved distribution modeling and benchmark performance.

  • 3.1 Preliminaries: Autoregressive VLMs encode visual and textual inputs into embeddings before decoding answers through a text-token function.Continuous outputs require a separate decoder g(z) that maps VLM embeddings to continuous values.
  • 3.3 Toy Experiments and Theoretical Justification: DRIFT converges faster than flow matching and more faithfully recovers increasingly multimodal target distributions in synthetic toy experiments.The experiments attribute improved characterization to disentangling distributional modes.
  • 3.2 Design of DRIFT: DRIFT initializes flow matching from a Gaussian centered at the coarse prediction g(z), rather than from a zero-centered noise distribution.The learned scale σ controls the starting distribution's variance.
  • 3.2 Design of DRIFT: The target-prediction parameterization enables a skip connection that combines the base predictor with the refinement module through a learnable gating vector.The refinement module iteratively improves the residual between the target and coarse prediction.
  • 3.2 Design of DRIFT: When the base predictor is jointly learned, gradient stopping in the interpolant prevents the model from relying primarily on flow matching while ignoring coarse prediction.An additional MSE loss supervises an MLP base predictor when no predictor is provided by the VLM.
  • 3.2 Design of DRIFT: Inference samples noise around g(z), solves the velocity-field ODE, and can average multiple samples using typically 10-20 integration steps.Averaging multiple samples reduces prediction bias and often slightly improves accuracy.
  • 3.3 Toy Experiments and Theoretical Justification: Residual refinement strictly reduces prediction error relative to the base predictor when the bridge state contains residual information beyond g(z).The theoretical analysis assumes a fixed base predictor and compares residual refinement with direct flow matching when the predictor captures condition-dependent mean structure.

4 Experiments and Results

DRIFT is evaluated across robotic control, temporal and spatial grounding, world action models, and decoder ablations. It consistently improves continuous-output performance across model architectures and task settings.

  • VLA tasks: DRIFT improves OpenVLA results across Libero and Simpler robotic-manipulation subtasks.With OpenVLA, the Libero average rises from 75.5% to 77.7%, while Simpler rises from 7.7% to 13.5%.
  • VLA tasks: 97.9% Libero and 61.5% Simpler average success rates establish new state-of-the-art results with Qwen3-VL-2B.The results also exceed VLM4VLA with Qwen3-VL-2B, while revealing a strong effect of VLM pretraining.
  • Temporal video grounding: DRIFT outperforms ED-VTG on 7 of 8 temporal-grounding metric columns, except ActivityNet-Captions R@0.7.On Charades-STA, DRIFT adds 1.5% at R@0.3 and mIoU; on ActivityNet-Captions, it surpasses ED-VTG on R@0.3, R@0.5, and mIoU.
  • Spatial grounding: DRIFT improves Qwen3-VL-2B on all RefCOCO spatial-grounding benchmarks.This result demonstrates the framework’s versatility beyond temporal grounding and robotic control.
  • World action models: DRIFT consistently improves FastWAM on the Libero benchmark suite despite FastWAM already using a flow-matching decoder.The adaptation adds and jointly optimizes an MLP base predictor with FastWAM’s original flow decoder.
  • Ablation studies: DRIFT outperforms MLP, diffusion, and flow-matching decoders, while joint training raises Libero-Long performance from 88.2% to 93.0% and 96.4%.The 93.0% result is obtained with a co-trained base predictor, and 96.4% with the full DRIFT decoder; the best configuration uses x-prediction and a skip connection.

5 Conclusion

DRIFT adapts pretrained VLMs for continuous decoding by combining a coarse base prediction with residual flow-matching refinement. The framework improves optimization and learning efficiency while supporting continuous decoding across diverse tasks and architectures, subject to the availability of an informative base predictor.

  • Conclusion: DRIFT augments a base predictor with a flow-matching refinement module that progressively models residual errors around a strong prior.The final prediction combines the coarse estimate with learned residual refinement.
  • Conclusion: The residual formulation simplifies generative modeling and improves optimization and learning efficiency for continuous decoding.The refinement network is trained to predict the residual between the target and the base predictor output.
  • Conclusion: DRIFT demonstrates consistent improvements over strong regression- and generative-based baselines across visual grounding and robotic control tasks spanning MLLMs, VLAs, and WAMs.The reported experiments cover multiple continuous decoding settings and architectures.
  • Limitations and Future Work: The framework assumes a sufficiently informative base predictor whose underlying VLM representations are relevant to the target continuous decoding task.When such priors are weak or unavailable, the benefits of residual refinement may diminish.
  • Theoretical Assumptions: The theoretical analysis uses truncated time sampling, bounded targets and outputs, and finite function-class complexity assumptions.Truncation prevents weight divergence near t=1, while the boundedness and complexity conditions support nontrivial generalization bounds.

A.3 Advantage over Direct MLP

DRIFT’s residual refinement can outperform direct MLP prediction when the bridge state contains information about the residual. Its population and finite-sample guarantees quantify this improvement, while comparison with direct flow matching shifts the rationale from Bayes risk to residual-target geometry and variance.

  • Population-level improvement over direct MLP: The Bayes residual predictor m(X)=E[r|X] is mean-squared-error optimal for predicting the residual given the bridge state.The squared-loss decomposition shows that any predictor’s excess risk over m equals its squared distance from m.
  • Population-level improvement over direct MLP: DRIFT strictly improves over direct MLP at the population level whenever the residual signal m(X) is not identically zero.Direct MLP is the zero residual predictor, and the population risk gap equals the expected squared norm of m(X), with an analogous weighted expression.
  • Finite-sample improvement over direct MLP: Finite-sample ERM strictly outperforms direct MLP when residual signal exceeds approximation and generalization errors under the stated condition.The guarantee holds with probability at least 1−δ under the assumptions and a sufficient inequality involving residual magnitude, approximation error, and complexity terms.
  • Comparison with direct flow matching: DRIFT and direct full-target flow matching have equivalent unrestricted population optima, so residualization does not lower Bayes risk.The objectives are related by a loss-preserving deterministic shift between residual and full-target predictors.
  • Residual-target geometry: Residualization removes condition-dependent mean structure captured by the base predictor but preserves irreducible conditional uncertainty.When the base predictor is useful, the residual mean can have a smaller second moment than the direct target mean, even for normalized outputs.
  • Residual-target geometry: DRIFT’s residual bridge stays around a coarse anchor, whereas direct generative decoding transports noise toward the full target region without reducing final conditional diversity.The paper frames this as a geometric and statistical advantage in the learned generative component rather than a lower unrestricted optimum.

B.1 DRIFT Architecture

DRIFT uses a two-stage decoder: a Base Predictor first produces a coarse continuous estimate, and a Flow Refiner iteratively predicts the remaining residual to the target.

  • DRIFT Architecture: The Base Predictor maps the MLLM representation to a coarse continuous estimate ˆy.This estimate provides the anchor for subsequent residual refinement.
  • DRIFT Architecture: The Flow Refiner iteratively predicts the residual between ˆy and the ground-truth target.Its prediction is combined with the base estimate through a skip connection to form the final output.
  • DRIFT Architecture: The Flow Refiner embeds the condition representation, flow time, and bridge state before processing them with self-attention and an output head.The output head produces the flow prediction used in the residual update.

B.2 VLA Implementation Details

The VLA implementation adapts Qwen3-VL and OpenVLA with task-specific base predictors and a shared residual flow-refinement design, evaluated across Libero and Simpler WidowX.

  • Training data: Libero training uses Spatial, Object, Goal, and Long suites with 53K, 67K, 52K, and 101K samples, respectively.The corresponding training trajectories number 432, 454, 428, and 379.
  • Training data: Simpler WidowX training uses Bridge and Fractal splits containing 1.28M and 3.4M samples, respectively.Action trajectories are normalized using the 99th percentile value for the trajectories.
  • Base models: The experiments use Qwen3-VL-2B-Instruct and OpenVLA as base models, with LoRA adapters added to the vision encoder and LLM backbone.LoRA rank and α differ between Libero and Simpler settings.
  • Base Predictor: Qwen3-VL uses a learnable MLP base predictor trained with MSE, whereas OpenVLA uses its tokenizer over 256 action bins with negative log-likelihood training.For OpenVLA, an action is sampled from the predicted multinomial distribution to form ˆy.
  • Flow Refiner: The VLA Flow Refiner uses four self-attention layers with eight heads and hidden dimension 768, optimized against target velocity with MSE.Qwen3-VL predicts eight future actions for Libero and four for Simpler; OpenVLA predicts one future action.
  • Evaluation: DRIFT is evaluated on Libero-Spatial, Libero-Object, Libero-Goal, Libero-Long, and Simpler WidowX using success rate.Inference uses K=10 forward-Euler steps and one stochastic noise sample per query.

B.3 TVG Implementation Details

The TVG implementation uses ET-Chat with temporal special tokens, a coarse segment predictor, and a residual flow refiner evaluated under the ED-VTG protocol.

  • Training data: Training uses TimeLens-100K, containing approximately 20K source videos and 100K query–segment temporal-grounding annotations.The videos combine multiple video-language datasets, including CosMo-Cap, InternVid-VTime, DiDeMo, QuerYD, and HiREST.
  • Backbone adaptation: ET-Chat’s visual encoder and frame compressor are frozen, while LoRA adapters on LLM attention projections are trained jointly with the Flow Refiner.Three temporal special tokens are added, and the LLM is trained to emit them with its standard next-token objective.
  • Decoder components: The Base Predictor applies a small MLP to <VALUE> hidden states, predicts center-width segments, expands width by 1.2×, and converts to start-end coordinates.The Flow Refiner predicts velocity in residual center-width coordinates.
  • Training and inference: TVG inference normalizes timestamps to [−1,1], integrates with K=10 forward-Euler steps, then converts and clips predictions to valid video time.Training uses flow MSE on velocity with AdamW for four epochs on eight A100 GPUs.
  • Evaluation: Recall@1 at threshold τ measures the fraction of queries whose top-1 interval exceeds IoU τ with ground truth, while mIoU averages IoU across queries.These are the reported temporal-grounding metrics.

B.4 Spatial Grounding Implementation Details

The implementation details specify separate configurations for visual grounding and robotic-control experiments, including data, predictors, flow refiners, optimization, and evaluation protocols.

  • Spatial Grounding: Visual grounding uses RefCOCO, RefCOCO+, and RefCOCOg training sets containing 120K, 120K, and 80K samples, respectively.Bounding-box coordinates are normalized to [0,1] in [cx,cy,w,h] format.
  • Spatial Grounding: Qwen3-VL-2B-Instruct is adapted with rank-32 LoRA modules on both the vision encoder and LLM backbone.The additional <think> token’s final-layer hidden state provides condition c.
  • Spatial Grounding: The grounding base predictor is a learnable MLP trained with MSE, while the 8-layer flow refiner predicts target velocity using MSE.The refiner uses 8 heads and hidden dimension 768.
  • Spatial Grounding: Grounding training uses batch size 64, learning rate 4 × 10^-5, one epoch, cosine decay, 3% warmup, and AdamW on 8 A100 GPUs.The reported training duration is approximately two hours.
  • Robotic Control: For Libero control, DRIFT uses 272K samples across Spatial, Object, Goal, and Long splits, with FastWAM’s VideoDiT and ActionDiT initialization.The base predictor aggregates text and vision features through a 2-layer Q-former with 16 learnable queries before MLP prediction.
  • Robotic Control: Libero evaluation covers four suites with 50 trials per each of 10 subtasks, reports success rate, and trains on 8 H200 GPUs for 24 hours.The ActionDiT serves as the flow refiner and is optimized with velocity-prediction MSE.

C Additional Visualization

Additional TVG visualizations compare timestamp predictions and show DRIFT refining coarse temporal boundaries, while also exposing annotation-driven errors.

  • Visualization: Figure 5 compares start and end timestamps predicted by different modules on additional TVG examples.The figure is presented as an additional visualization of TVG results.
  • Visualization: DRIFT achieves more accurate temporal boundaries by iteratively refining coarse predictions with its flow refiner.This behavior is illustrated in Samples 1–4.
  • Error Analysis: Samples 5–6 show errors shared by all decoders, primarily arising from ambiguous or inaccurate annotations.These examples concern inconsistent temporal boundaries and ambiguous natural-language queries.
  • Error Analysis: A two-second sandwich-bite action is labeled as a five-second interval, while “sits on a bed” can denote sitting down or already being seated.The examples motivate better-curated annotations and improved handling of ambiguous queries.

D Additional ablation

The additional ablation tests whether DRIFT depends on a trained base predictor by replacing it with an intentionally untrained predictor.

  • Additional Ablation: Without the auxiliary base-predictor loss and with refinement gradients stopped, the predictor remains untrained and reaches 0% accuracy on Libero-Long.This creates the deliberately weak, random-base-predictor condition.
  • Additional Ablation: With the random base predictor, DRIFT effectively degenerates to flow matching while achieving comparable performance.The ablation indicates that the untrained predictor condition behaves similarly to a flow-matching model.
Loading 2606.05758v1…