Deforming Videos to Masks: Flow Matching for Referring Video Segmentation

• RVOS requires anchoring natural-language concepts to dynamic, fine-grained pixels, but instance-centric pipelines can bottleneck semantics and temporal consistency.The cited work includes query-based methods such as ReferFormer and VLM-based methods such as LISA.Source: §1 · p. 2
• The framework combines pretrained T2V models' pixel-level synthesis, text-conditioned alignment, and video-native spatio-temporal reasoning.Source: §1 · p. 2
• FlowRVS treats RVOS as a convergent flow from a high-entropy video to one mask, with the text query selecting the target.Its proposed adaptations include boundary-biased sampling, direct video injection, and task-specific VAE adaptation with start-point augmentation.Source: §1 · p. 2
• FlowRVS learns a direct, language-guided deformation from a video's spatio-temporal representation to its target mask.Source: §1 · p. 2
• The paper presents principled techniques for transferring text-to-video generative models to video understanding.Source: §1 · p. 3
• FlowRVS reports new state-of-the-art results, improving J&F by 1.6% on MeViS and by 2.7% on zero-shot DAVIS 2017.Source: §1 · p. 3

• RVOS methods often localize a text-described region before a separate framewise segmentation stage, while early approaches added temporal linking to image segmentation.Source: §2 · p. 3
• The locate-then-segment family includes query-based architectures, multimodal localization followed by mask decoders, VLM+SAM systems, and VD-IT's T2V feature extraction.The passage notes that intermediate object queries or features can bottleneck holistic video-to-mask optimization.Source: §2 · p. 3
• FlowRVS fine-tunes the full generative process to learn a direct video-to-mask deformation flow, rather than extracting features or making a static one-step prediction.It also differs from noise-to-image conditional generation by starting from the video source; the text query conditions the ODE path.Source: §2 · p. 3
• Figure 2 depicts FlowRVS learning a velocity field and using an ODE solver to deform video latents into target-mask latents.The caption characterizes this video-to-mask paradigm as superior to noise-based or one-step prediction approaches.Source: Figure 2 · p. 4

3.1 PROBLEM REFORMULATION: RVOS AS A CONTINUOUS FLOW

• RVOS is reformulated as learning a velocity field that gradually deforms a video representation into the target mask under text guidance.The ODE trajectory starts at video latent z0 and ends at mask latent z1, replacing a single global mapping with a local velocity field.Source: §3.1 · p. 4
• The video-to-mask task is a convergent transformation from a complex video latent to one correct mask, making the initial text-guided flow direction critical.The passage warns that an initial-step error is irrecoverable.Source: §3.1 · p. 4

3.2 TRANSFERRING TEXT-TO-VIDEO MODEL TO RVOS

• Boundary-Biased Sampling oversamples timestep t = 0 so training focuses on the high-influence initial text-guided velocity.The passage reports that this strategy stabilizes training and turns a failing baseline into an effective model.Source: §3.2 · p. 5
• Start-Point Augmentation stochastically encodes and normalizes the initial video latent to expose the model to nearby starting points and regularize its flow.Source: §3.2 · p. 5
• Direct Video Injection concatenates the original video latent with the evolving state at each ODE step, keeping source context available to each update.The passage says this helps prevent trajectory drift and improves fine-grained accuracy with negligible computational overhead.Source: §3.2 · p. 5

3.3 ANALYSIS OF ALTERNATIVE PARADIGMS

• Direct mask prediction is described as an ill-posed information collapse, while noise-to-mask flow demotes the video to a conditional signal rather than refining its structure.Source: §3.3 · p. 6
• One-step velocity prediction outperforms direct mask prediction and noise-to-mask flow, but is limited by computing the full deformation in one pass.The authors attribute its limitation to missing the gradual iterative refinements of a multi-step process.Source: §3.3 · p. 6
• The analysis concludes that multi-step video-to-mask flow is the most effective tested paradigm when paired with start-point-focused adaptations.Source: §3.3 · p. 6

4.1 BENCHMARK AND METRICS

• The evaluation covers MeViS, Ref-YouTube-VOS, and Ref-DAVIS17, with J, F, and their average J &F as primary metrics.MeViS emphasizes complex motion; Ref-YouTube-VOS tests generalization across objects and scenes; Ref-DAVIS17 emphasizes precision and temporal consistency.Source: §4.1 · p. 6

4.2 IMPLEMENTATION DETAILS

• FlowRVS uses Wan 2.1, freezing its text encoder and VAE encoder while fine-tuning the DiT; the VAE decoder is tuned separately for masks.Training differs by dataset: Ref-YouTube-VOS uses static-image pretraining followed by video fine-tuning, whereas MeViS training uses no static-image pretraining.Source: §4.2 · pp. 6–7

4.3 MAIN RESULTS

• FlowRVS exceeds VISA-13B by 7.0 points and ReferDINO by 1.4 points on MeViS, and qualitative examples show stronger temporal and language grounding.The figure contrasts FlowRVS with VD-IT's unstable masks and ReferDINO's difficulty with long-range descriptions.Source: §4.3 · p. 7, Figure 4 · p. 8
• 73.3 J &F on Ref-DAVIS17 is reported zero-shot, while FlowRVS scores 50.7 J &F on MeViS, 1.2 points above SAMWISE.The Ref-DAVIS17 model was trained on Ref-YouTube-VOS without DAVIS fine-tuning.Source: §4.3 · pp. 7–8

4.4 ABLATION STUDIES

• Noise-to-Mask Flow performs poorly at 32.3 J &F, while Onestep Mask Prediction reaches 36.2 J &F; shifting from state to velocity prediction yields a +14.6 J &F gain.Source: §4.4 · pp. 8–9
• Boundary-Biased Sampling at p = 0.5 gains 10.0 J &F over the base flow, while Direct Video Injection adds 2.0 J &F.A stronger p = 0.75 bias scores 56.5 J &F, still above the baseline; the authors describe DVI as a persistent context anchor that prevents drift.Source: §4.4 · p. 9
• Removing Wan weight initialization collapses performance to 21.1 J &F, underscoring the role of pretrained T2V weights in this framework.Source: §4.4 · p. 9
• With a fixed flow model, fine-tuning the VAE decoder improves performance by 0.9 J &F over the frozen decoder's 60.0 J &F.The adaptation is intended to bridge the domain gap between continuous video latents and binary masks.Source: §4.4 · p. 9, Table 3 · p. 9

5 CONCLUSION AND FUTURE WORK

• FlowRVS combines a continuous, text-conditioned video-to-mask flow with BBS, SPA, and DVI adaptations, and reports state-of-the-art performance.The conclusion describes the adaptations as start-point focused and says they align T2V generative strengths with the task’s discriminative demands.Source: §5 · p. 9
• The paper frames principled adaptation of T2V models as a way to bridge generative processes and discriminative objectives.The conclusion specifically points to fortifying the flow’s structured starting point.Source: §5 · p. 9
• The authors identify conditional deformation processes as a potentially useful paradigm beyond RVOS.They suggest start-point stabilization may inform future applications of larger foundation models to video-understanding tasks.Source: §5 · p. 10

ETHICS STATEMENT

• The manuscript does not cover ethics-related topics or experiments.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Implementation details and hyperparameter settings are provided; code is planned for release depending on acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

A HYPERPARAMETERS SETTINGS

• AdamW is used for all models, with key hyperparameters for Ref-YouTube-VOS and MeViS main experiments listed in Table 4.Source: A HYPERPARAMETERS SETTINGS · p. 13
• Table 4 covers settings for 2D pre-training, video-dataset DiT fine-tuning, and separate VAE decoder adaptation.Source: Table 4 · p. 13

B MORE QUALITATIVE RESULTS

• FlowRVS handles challenging video-text pairs, particularly scenes with significant occlusion and nuanced textual descriptions.These examples are presented as additional qualitative results.Source: B MORE QUALITATIVE RESULTS · p. 13

C COMPREHENSIVE RESULTS

• For fine-grained relational actions, FlowRVS may identify the subject but fail to isolate which horse performs the described interaction.The example involves the phrase “swings its tail and strikes the head.”Source: C COMPREHENSIVE RESULTS · p. 13
• Among similar objects with temporal ordering cues such as “the second bird,” FlowRVS may segment the first bird that moves.The passage describes this as a difficulty with counting and ordering logic in dense scenes.Source: C COMPREHENSIVE RESULTS · p. 13
• FlowRVS segments the unseen phrase “the dog somersaulting” throughout complex nonrigid motion.The phrase was absent from training data.Source: C COMPREHENSIVE RESULTS · p. 13
• Figures 5 and 6 show FlowRVS visualization examples on MeViS-Valid-u and MeViS-Valid, respectively.Source: Figure 5 · p. 14, Figure 6 · p. 14

D.1 BOUNDARY-BIASED SAMPLING (BBS)

• BBS mixes a point mass at t=0 with uniform timestep sampling and uses p=0.5 to emphasize the flow’s initial step.Source: §D.1 · pp. 14–15
• Figures 7–9 cover Ref-YouTube-VOS-Valid examples, failure cases, and novel action phrases outside MeViS training.Source: Figure 7 · p. 15, Figure 8 · p. 15, Figure 9 · p. 16

D.2 START-POINT AUGMENTATION (SPA)

• SPA samples and normalizes a start latent from the video VAE posterior, exposing training to nearby latents to learn a smoother, more generalizable velocity field.Source: §D.2 · p. 15

D.3 DIRECT VIDEO INJECTION (DVI)

• DVI concatenates the current state with the initial video latent, doubling the DiT’s input channels while leaving subsequent layers unchanged.Source: §D.3 · pp. 15–16