Moving Regions, Not Moving Objects: Segmenting What Actually Moves under Camera Motion

• Existing benchmarks often label an object as moving throughout a sequence once it moves at any time, although it may be stationary in a particular frame pair.GMOS annotates moving and stationary intervals but uses an object-level motion head.Source: §1 · p. 2
• MoRSe defines motion for an ordered frame pair, so a source region can be labeled differently as the target changes.This distinguishes independently moving scene points from displacement caused by the camera.Source: §1 · p. 2
• MoRSe uses target-conditioned attention to predict each source coordinate’s moving-region mask without flow, point tracks, or a segmentation foundation model at inference.A target window aggregates pairwise predictions, while a promptable segmenter can recover whole moving objects.Source: §1 · p. 2
• MoRSe-5K comprises 42 sequences and 5,431 frames across Static, Dynamic-Static, and Dynamic settings; its all-empty Static ground truth directly measures ego-motion false positives.The benchmark separates temporal sensitivity, spatial localization, and robustness to camera motion.Source: §1 · p. 2
• MoRSe performs best on MoRSe-5K moving-region segmentation and remains competitive on conventional sequence-level MOS benchmarks outside its target formulation.Source: §1 · p. 2

2.1 FLOW- AND TRACK-BASED SEGMENTATION

• Flow- and track-based systems combine motion proposals with segmentation, while MoRSe jointly learns pair-conditioned evidence and mask prediction.The staged pipelines can accumulate errors across correspondence, prompting, and mask propagation.Source: §2.1 · p. 3

2.2 GEOMETRIC MOTION REASONING

• Geometric methods use camera-consistency cues or reconstruction features, but often require fitting, a separate mask statistic, or masks defined per frame or sequence.MoRSe predicts camera parameters only as an auxiliary task and does not use them to compensate mask features.Source: §2.2 · p. 3

2.3 SCENE FLOW FROM 4D RECONSTRUCTION

• Feed-forward 4D models predict dense world-coordinate geometry and motion that can be thresholded into masks, whereas MoRSe directly supervises a pair-conditioned mask.Thresholding is separate from the learned output in those 4D models.Source: §2.3 · p. 3

3.1 PROBLEM FORMULATION AND OVERVIEW

• A source coordinate is labeled moving when its scene point moves independently of the camera between the ordered frames; camera-induced displacement alone is negative.Changing the target can change the label while the source remains fixed, though real-video pair labels inherit object-level annotation granularity.Source: §3.1 · p. 3

3.2 JOINT ENCODER AND CAMERA PREDICTION

• The joint encoder processes both frames through a ViT and a shallow convolutional branch, then fuses and splits features by view; camera pose and field of view are auxiliary outputs outside the mask path.The convolutional branch retains high-resolution detail that patch-tokenized ViT features discard.Source: Figure 2 · p. 4, §3.2 · p. 4

3.4 SPARSE QUERY DECODER

• Four cross-attention blocks refine each source-coordinate query from target keys and values before a shared MLP predicts its motion logit.Each coordinate is decoded independently of the other queried coordinates.Source: Figure 2 · p. 4, §3.4 · p. 5

3.3 TARGET MEMORY

• Target features form an explicit memory that source queries read individually, while sampled source feature fields are fused into each query’s initial state.Query-resolution reads matter when a small moving part and nearby static region share a coarse feature cell.Source: §3.3 · p. 5

3.5 SAMPLING AND OBJECTIVES

• Training supervises directed non-identity frame pairs in both temporal directions, using tracked-point motion labels, real-video annotations, and all-negative static clips.The motion loss combines point-wise binary cross-entropy and soft Dice; the camera objective is zero-weighted when pose labels are untrustworthy.Source: §3.5 · pp. 5–6
• Each pair is supervised on a bank of 1,024 source coordinates, with sampling changing supervised coordinates but not the prediction function.Source: §3.5 · p. 5

4.1 TRAINING-DATA CURATION

• The training set combines trajectory supervision, curated mask sources, and static-scene data; conflicting whole-object annotations led to retaining 392 of 5,231 candidate clips.The 28 source entries comprise 19 trajectory sources, seven mask-source entries, and two static-scene sources.Source: §4.1 · p. 6

4.2 THE MORSE-5K BENCHMARK

• MoRSe-5K contains 42 sequences and 5,431 frames across Static, Dynamic-Static, and Dynamic settings that separate camera and scene-motion conditions.The benchmark includes 21 rendered sequences and 21 real sequences.Source: §4.2 · p. 6

4.3 DATASETS, METRICS, AND BASELINES

• Static frames are scored with pixel and frame error rates, while Dynamic-Static and Dynamic use DAVIS region similarity, contour accuracy, and their mean against moving-region and moving-object labels.The region variant is primary because the shared SAM 3 readout can expand false positives into whole instances; scene-flow baselines use frozen thresholds.Source: §4.3 · pp. 6–7

4.4 MOVING-REGION SEGMENTATION ON MORSE-5K

• MoRSe gives the strongest moving-region scores, particularly on boundaries, while largely suppressing false positives on Static sequences where most baselines respond to camera motion.Its advantage is smaller for moving-object evaluation, where completing an instance from partial motion evidence favors GMOS-S on one component.Source: Table 1 · p. 7, §4.4 · p. 7

4.5 WINDOW AGGREGATION

• Unioning predictions across target frames raises recall with a modest precision cost, while mean J &F stays within a point because false positives are largely shared across targets.The comparison changes which cached source-target predictions are unioned.Source: Table 2 · p. 7, §4.5 · p. 7

4.6 CONVENTIONAL MOS BENCHMARKS

• Unioning masks over adjacent targets transfers MoRSe to sequence-level MOS without retraining, but GMOS, which predicts instances natively, leads all three benchmarks.Expanding region outputs with the shared SAM 3 readout gains 4.2 points on DAVIS16-M and roughly doubles the MoCA score, but costs 4.5 points on FBMS-59.Source: Table 3 · p. 8, §4.6 · p. 8

4.7 ABLATIONS

• Ablations favor retaining the low-level branch and auxiliary camera objective; geometry-pretrained initialization outperforms DINO variants, while fusion variants remain close to the default.Removing the low-level branch degrades region quality and static-scene suppression; removing the camera objective harms region accuracy and background suppression.Source: §4.7 · p. 9

Boundary quality

• Masks can have softer boundaries around hair or clothing when their colors resemble the surroundings.Unlike masks from a promptable segmenter, MoRSe predicts its boundaries directly.Source: §5 · p. 9

Uniform and repetitive surfaces

• On uniform or repetitive surfaces, the decoder may rely on outer edges and fill interiors inconsistently.Blank walls and tiled floors can look alike at matched and mismatched positions.Source: §5 · p. 9

Small-displacement motion

• Very small displacement between source and target frames may go undetected, yielding missed detections.Source: §5 · p. 9

Scarcity of real-scene region-level labels

• Real-scene region-level labels are scarce, and transfer from the mostly rendered training data is untested at scale.Existing benchmarks outline whole instances, while synthetic corpora can derive motion masks from per-point 3D trajectories.Source: §5 · p. 9

6 CONCLUSION

• Experiments show improved region accuracy and background suppression over specialized motion-segmentation and scene-flow baselines.Source: §6 · p. 9
• MoRSe predicts each source-coordinate mask by attending to the target, without optical flow, point tracks, or a separate segmentation foundation model at inference.Source: §6 · p. 9
• Real-scene supervision and small displacements remain open issues for the approach.Source: §6 · p. 9

AI USE STATEMENT

• Generative AI assisted with literature organization, language polishing, and manuscript editing.Source: AI USE STATEMENT · p. 10
• For the research, it supported brainstorming, architectural discussions, refactoring, and implementation and debugging of training and evaluation code.Source: AI USE STATEMENT · p. 10
• Authors made the design decisions, reviewed suggestions, verified claims and citations, and state that AI did not generate data or experimental results.The authors take full responsibility for the work.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The planned release includes code, trained weights, pair-index files, and query banks used for training.Source: REPRODUCIBILITY STATEMENT · p. 10
• MoRSe-5K materials include 42 sequences, Blender scene configurations for the synthetic split, frame-wise annotations, and evaluation scripts.The materials are intended to support reproduction of the three camera-motion settings.Source: REPRODUCIBILITY STATEMENT · p. 10
• Dataset construction and evaluation are detailed in Section 4, while Appendix C tabulates training objectives, query sampling, and their numeric constants.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

• The work uses publicly available video segmentation benchmarks.Source: ETHICS STATEMENT · p. 10
• No new data were collected from human participants.Source: ETHICS STATEMENT · p. 10
• The authors report no issues involving human subjects, private data, or harmful insights.Source: ETHICS STATEMENT · p. 10

A TRAINING-DATA CONSTRUCTION AND SPLIT AUDIT

• Training combines 28 source entries: 19 trajectory sources, seven mask sources, and two static-scene sources.Trajectory labels are generated online from ground-truth 3D displacement using a 7 mm threshold; the three families carry 92.3%, 6.0%, and 1.7% of sampling mass, respectively.Source: A TRAINING-DATA CONSTRUCTION AND SPLIT… · p. 14
• The mask-source funnel retains 392 of 5,231 candidate clips to reduce mismatches between object-level annotations and moving-region labels.It targets unlabelled movers counted as negatives and mostly stationary outlined instances counted as positives, using geometric or interval-level motion checks followed by manual review.Source: A TRAINING-DATA CONSTRUCTION AND SPLIT… · p. 14
• Retained mask annotations seed source-frame query banks, not fine-grained labels of independently moving subregions.Source: A TRAINING-DATA CONSTRUCTION AND SPLIT… · pp. 14–15
• The split audit checks retained sequences against official training splits and evaluation manifests before frame or pair sampling, and removes implicated clips.The released manifest records each candidate’s source, split, sequence identifier, filtering outcome, and no-leak status.Source: A TRAINING-DATA CONSTRUCTION AND SPLIT… · p. 15

B COMPUTATIONAL STRUCTURE

• Decoder cost is O(BQNtC), so concentrating supervision on informative query locations makes query-level interaction more affordable to train.The joint encoder and target-memory construction are paid regardless of decoded coordinates and may dominate runtime for a large initialization.Source: B COMPUTATIONAL STRUCTURE · p. 15

C IMPLEMENTATION DETAILS

• Table 6 collects query-sampling, loss, camera-head, memory, decoder, empty-mask, and cleanup constants used in the main text.Source: C IMPLEMENTATION DETAILS · p. 15, Table 6 · p. 16
• The training recipe fine-tunes UniQuery4R-initialized weights with a randomly initialized motion-mask output head for 20,000 iterations.It uses AdamW, batch size one clip per GPU, warm-up and cosine decay, gradient clipping, and bf16 arithmetic.Source: C IMPLEMENTATION DETAILS · p. 15
• The sampler mixes mask, trajectory, and pseudo-static datasets using configured probabilities, while an explicit dynamic-dataset list gates the motion branch.Source: C IMPLEMENTATION DETAILS · p. 15
• Trajectory labels use a 7 mm 3D-displacement threshold; mask banks distinguish moving, stationary-object, and background pixels, retaining all-negative frames.The current recipe uses 0.5 BCEWithLogits + 0.5 Dice scaled by λmask = 3.0; inference thresholds the raw logit at 0.Source: C IMPLEMENTATION DETAILS · pp. 15–16

D EVALUATION PROTOCOL AND BASELINE REPRODUCTION

• Scene-flow masks threshold per-pixel world-coordinate displacement, then receive morphological cleanup and removal of components below 300 px.For Trace Anything, the displacement is computed from its trajectory field at the source and target timestamps.Source: D EVALUATION PROTOCOL AND BASELINE… · p. 16

E UNION OVER TARGETS

• False positives grow more slowly than an independent-error model predicts as targets are unioned, because different targets tend to fail on the same pixels.The measured error saturates, making a maximum a usable window operator.Source: E UNION OVER TARGETS · p. 17
• The first target-set expansion recovers useful motion, while later targets add progressively more false positives; mean quality peaks at a modest window.The evaluation uses 115 sequences from five conventional MOS benchmarks, with shared pair logits assembled into target sets offline; the long-sequence subset’s absolute values are not comparable to Table 3.Source: E UNION OVER TARGETS · pp. 17–18

F ABLATION DETAILS

• Removing the convolutional pyramid causes the clearest architectural-control degradation, while alternative fusion rules remain within the default seed spread.This supports retaining a separate low-level branch while keeping fusion simple.Source: F ABLATION DETAILS · p. 18

G EFFICIENCY

• Decoder cost grows with query count, and larger encoding windows increase per-pair cost as queries read larger target memories.The sweep exposes a smooth accuracy–latency trade-off; encoder amortization partly offsets the larger-window cost, and mixed precision is effectively accuracy-neutral.Source: G EFFICIENCY · p. 19

H DIAGNOSTICS AND FAILURE CASES

• Single-pair motion reads an object that returns to its start within the interval as not moving, and coordinate-wise decoding is most exposed on thin structures and low-texture support.MoRSe also does not reliably predict regions moving only a few pixels; examples include partial recovery of a swaying branch and a missed small animal behind a rock.Source: H DIAGNOSTICS AND FAILURE CASES · p. 19

I MORE QUALITATIVE RESULTS

• On conventional MOS benchmarks, MoRSe recovers moving regions, while SAM 3 completes them to the instance masks those benchmarks expect.The qualitative MoRSe-5K examples compare shared-source groups as the target changes.Source: I MORE QUALITATIVE RESULTS · pp. 19–20, Figure 6 · p. 20