Source-linked AI summary
Mover360: Controllable Object Manipulation in 360° Panoramic Images
Haoyi Zhong, Fang-Lue Zhang, Andrew Chalmers, Taehyun Rhee
TL;DR
Object-level editing in ERP panoramas is difficult because wrap-around, distortion, and global scene continuity challenge perspective editors. Mover360 addresses this with a unified ERP-aware framework and paired UE5 supervision, achieving the strongest overall results across both benchmark domains and evaluation protocols. Its scope remains bounded by fixed prompts, ambiguous point guidance, imperfect depth, and difficult scene-wide lighting or interactions.
Problem
ERP panoramas make controllable object editing difficult because horizontal wrap-around, latitude-dependent distortion, and global scene continuity complicate perspective-based editing.
Method
Mover360 uses a translation-centered unified model with ERP-aligned point, bbox, and mask instruction maps, auxiliary depth, and UE5-generated paired data for Translation, Remove, and Insert.
Results
Across both synthetic and real test domains and two evaluation protocols, Mover360 achieves the strongest overall results among evaluated perspective-editing, insertion, and inpainting baselines.
Takeaways & Limitations
The framework provides a single-click default interaction while retaining bbox and mask controls for more explicit object-size and silhouette guidance.
Takeaways & Limitations
The method uses three fixed prompts, can face ambiguous point guidance and depth failures, and may struggle with complex cast shadows, reflections, object interactions, and large-area lighting changes.
Abstract
from arXiv · showhide
We present Mover360, a controllable object manipulation framework for 360° images. Unlike perspective images, 360° images in equirectangular projection (ERP) exhibit horizontal wrap-around, latitude-dependent distortion, and global scene continuity, which makes object-level edits difficult for existing perspective editors to produce and for users to specify. To address this, Mover360 centers on object Translation (relocating a specified object within an existing panorama) while supporting reference-guided Insert and Remove as auxiliary tasks. Its interface unifies point-, bbox-, and mask-guided control by encoding each task into a fixed prompt and a compact, ERP-aligned instruction map. In the default point mode, a single click relocates an object, allowing the model to infer a plausible size, support, and illumination using panoramic context and an auxiliary depth condition. Structurally, Mover360 is a lightweight adaptation of a pretrained diffusion transformer. To generate paired supervision, we construct a UE5 data-generation pipeline with surface-aware object placement and randomized illumination, yielding large-scale paired data and a dual-domain benchmark of synthetic and real panoramas with ground truth for all three tasks. Across both test domains and two evaluation protocols, Mover360 outperforms strong baselines for perspective editing, insertion, and inpainting in reconstruction fidelity, semantic consistency, and distributional quality. Code and our benchmark dataset are available at https://zhonghaoyi.github.io/Mover360/.
1 INTRODUCTION
Mover360 addresses the geometric and contextual challenges of object editing in ERP panoramas with a unified, translation-centered framework. It combines ERP-aware interaction, paired supervision, and evaluation protocols designed for panoramic editing.
- Motivation: ERP panoramas introduce wrap-around, latitude-dependent distortion, and global scene continuity that make direct perspective editing unreliable.Such edits can produce seam artifacts, distorted geometry, unstable scale, and inaccurate spatial control.
- Motivation: Perspective re-projection can miss out-of-view effects such as long cast shadows and leave discontinuities at frustum boundaries.The editor also sees only the viewport rather than the surrounding sphere.
- Framework: Mover360 centers on Translation while exposing Remove and Insert through the same unified model and interface.Translation jointly completes the source background and places the object plausibly at the target with consistent scale, support, occlusion, and illumination.
- Framework: A three-channel ERP-aligned instruction map supports point, bbox, and mask guidance, with a single click sufficient in the default point mode.An auxiliary depth condition helps infer plausible size, support, and illumination from panoramic context.
- Data and evaluation: A UE5 pipeline provides paired camera–object data and a dual-domain benchmark with ground truth for all three tasks.The pipeline yields 12,150 paired camera–object sequences and a benchmark containing 210 synthetic and 50 real captured tuples.
- Data and evaluation: Across two evaluation protocols and both test domains, Mover360 achieves the strongest overall results among evaluated panorama-editing baselines.The protocols include direct ERP application and a favorable perspective re-projection.
2 RELATED WORK
Related work provides foundations in controllable image editing, panorama generation, and synthetic paired-data construction. However, existing methods do not directly provide the required combination of object-level spatial control and ERP panorama editing.
- Controllable image editing: Recent editors combine language with visual exemplars, masks, points, and structural controls for instruction following and spatial manipulation.ControlNet and drag-based methods strengthen local spatial guidance, while object editing also requires identity preservation and plausible support.
- Panorama generation: Panorama research has adapted generative models to wide-field-of-view imagery through ERP-aware, multi-view, and spherical-processing strategies.These methods address panorama generation and translation rather than the full object-manipulation setting.
- Gap: Existing panorama systems cannot serve as baselines here when they lack object-level spatial control, rely only on global text, or do not release relevant weights.This limitation separates panorama generation methods from the evaluated controllable-editing setting.
- Paired supervision: Because real before-and-after scenes are difficult to collect, prior editors use synthetic instructions, generated pairs, game-engine rendering, or automatically constructed panorama data.Mover360 follows this need for controlled supervision while targeting object trajectories in UE5.
3 MOVER360 DATASET AND BENCHMARK
Mover360 constructs paired panorama-editing data in UE5 and evaluates it with synthetic and real benchmarks. The pipeline varies object motion, viewpoint, illumination, and references to support the three editing tasks.
- Data generation: The UE5 corpus uses five scenes, 608 unique objects, 60 motion paths, three panoramic cameras per path, and 10-frame trajectories.Five frames per trajectory are retained after stratified sampling and degenerate-frame filtering.
- Data generation: Motion paths are object-size-dependent and use surface raycasting to avoid penetration and maintain contact with floors and other surfaces.This avoids unrealistic motion for objects whose size or available space is incompatible with a path.
- Multi-view supervision: Multiple panoramic cameras expose the same motion with different poses, locations, and ERP distortions.Object sequences provide Translation pairs, while background frames support Remove and object-absent inputs for Insert.
- Lighting and reference: Global, local, and nearby-lamp illumination parameters are randomized during rendering.The procedure varies main and sky lights as well as lamp position, intensity, temperature, and beam or scattering angle.
- Lighting and reference: Reference-guided insertion selects among eight perspective views by retaining the three most similar DINOv3 views and sampling one.Similarity is computed against the target object crop in the panorama.
- Benchmark: The benchmark contains 210 held-out UE5 tuples and 50 real captured tuples, with ground truth for Translation, Remove, and Insert.The real set photographs panoramas before and after physical relocation, an object-absent background, and a perspective reference.
4 METHOD
Mover360 formulates panorama editing as a unified three-task system using ERP-aligned controls, depth conditioning, and a pretrained diffusion transformer. Its point-guided interface reduces specification burden while retaining bbox and mask controls, and circular processing addresses panorama periodicity.
- Model architecture: The model predicts an edited panorama from the input ERP image, fixed task prompt, instruction map, auxiliary DA2 depth, and optional insertion reference.The target latent is generated by the diffusion transformer and decoded by the VAE.
- Task formulation: Mover360 uses one backbone for Translation, Remove, and Insert, with task-specific prompts and instruction-map channels.Translation combines source removal and target placement; Remove and Insert operate as auxiliary tasks sharing the same interface.
- ERP-aligned instruction map: The instruction map encodes source regions, target regions, and target points as ERP-aligned channels supporting mask, bbox, and point guidance.Target points use Gaussian heatmaps, while source and target regions can use full object masks or rectangular box masks.
- Control granularity: Point guidance asks only for a target contact location, allowing the model to infer object extent, scale, and support from the panorama and auxiliary depth.BBox guidance provides stronger size and area control and can improve quantitative reconstruction metrics.
- Auxiliary depth conditioning: Depth is estimated consistently during training and inference, then used as a geometric condition for support, scene layout, and scale reasoning.The depth condition is randomly dropped with probability 0.1 to improve robustness when the cue is unreliable or unavailable.
4.5 Reference Branch for Insertion
The insertion reference branch supplies object appearance through a single perspective image, while the shared latent pipeline jointly processes text and image conditions. It is enabled only for Insert and disabled for Translation and Remove.
- Reference branch: The reference branch is activated only for Insert, where a single perspective reference image specifies the appearance of the object to add.For Translation and Remove, the branch is directly disabled rather than replaced with a null reference.
- Reference selection: Training selects one of the three most DINOv3-similar reference views from eight candidates to balance appearance consistency and reference-view diversity.Similarity is computed between each candidate view and the target object crop in the panorama.
- Latent encoding: Reference images are encoded by the Flux2-Klein VAE and fused with the input panorama, instruction map, and depth conditions.Condition latents are concatenated and patchified before entering the diffusion transformer.
- Multi-condition tokenization: Native 4D RoPE coordinates distinguish text tokens, noisy panorama latents, and conditioning-image slots while preserving each image condition’s spatial layout.Different conditioning slots receive distinct T coordinates, and the resulting embeddings are concatenated in transformer token order.
- ERP periodicity: Mover360 uses horizontal circular padding at the VAE boundary to handle panorama continuity while retaining the native latent-grid RoPE formulation.The method does not introduce a new spherical RoPE.
4.8 Panorama-Aware VAE Circular Padding
Mover360 preserves ERP horizontal periodicity with circular padding and adapts a pretrained diffusion transformer using parameter-efficient LoRA. Its training follows a flow-matching objective without additional reconstruction, perceptual, seam, prompt, or depth losses.
- Panorama-Aware VAE Circular Padding: Circularly padding the panorama by 64 pixels per side preserves horizontal periodicity before VAE encoding and resizing.The method adopts circular padding as a standard panorama-processing remedy rather than presenting it as a contribution.
- Parameter-Efficient Adaptation: LoRA adapts Flux2-Klein-4B-Base through attention and projection layers while leaving most pretrained parameters unchanged.The configuration uses rank 32, alpha 32, dropout 0, and 37,355,520 trainable parameters.
- Flow-Matching Objective: Mover360 follows the Flux2-Klein backbone’s flow-matching objective for training.The supplied method description specifies sigmoid-normal timestep sampling and latent interpolation, while the optimization uses only this objective.
- Flow-Matching Objective: The optimization excludes additional RGB reconstruction, perceptual, seam, prompt, and auxiliary depth objectives.
5 IMPLEMENTATION DETAILS
Mover360 is trained at 1024 × 512 ERP resolution with AdamW, a cosine-annealing schedule, and distributed training across eight GPUs. The reported run lasts eight epochs, approximately 30K iterations, and about 20 hours.
- Training Configuration: Training uses 1024 × 512 ERP resolution, AdamW, and an initial learning rate of 2×10^-4.The learning rate follows a step-wise CosineAnnealingLR schedule.
- Training Configuration: The model trains on 8 RTX Pro6000 GPUs with a per-GPU batch size of 2.
- Training Duration: The reported training run lasts 8 epochs, approximately 30K iterations, and about 20 hours.
6 EXPERIMENTS
Mover360 is evaluated against perspective editing baselines on dual-domain panorama benchmarks using ERP and favorable perspective protocols. It achieves the strongest overall results while depth and finer spatial guidance improve performance, with limitations for unusually large edits and open-ended control.
- Evaluation Setup: The benchmark contains 210 synthetic UE5 tuples and 50 real captured tuples per task, each with paired ground truth.The UE5 set uses controlled renders, while the real set tests transfer to captured panoramas with real optics, lighting, and scene statistics.
- Baselines: Mover360 is compared with publicly available perspective object-editing baselines because existing panorama methods lack the required controllable object-level interfaces or released weights.Baselines are adapted to their native interfaces for Translation, Insert, and Remove.
- ERP Protocol: On the ERP protocol, Mover360 (bbox) achieves the best value on all six Translation metrics on UE5, including FID 95.5 versus 108.9 for the strongest baseline.The point-guided Translation variant also surpasses every baseline on UE5 FAED, SSIM, LPIPS, and FID.
- ERP Protocol: On the real ERP set, Mover360 (bbox) leads Translation on PSNR, SSIM, LPIPS, and FID and leads Insert on FAED, PSNR, SSIM, LPIPS, and FID.Insert-Anything retains the best real FAED for Translation at 0.229 versus 0.356 for Mover360.
- Perspective Protocol: Even under the favorable perspective protocol, Mover360 remains strongest on UE5 across nearly all metrics and leads all six Remove metrics on the real set.Insert-Anything remains ahead on selected DINOv3 and other real Insert metrics, while Mover360 retains the best real Insert FAED, SSIM, and LPIPS.
- Limitations: Perspective re-projection cannot frame very large or near-field objects, with about 5% of samples exceeding the 150° limit and falling back to the ERP path.The model also has limitations for open-ended language control, unusual target scales, difficult depth regions, complex illumination interactions, and edits far outside its UE5 training distribution.
- Ablations: Depth improves most evaluated metrics, while guidance improves Translation and Insert monotonically from point to bbox to mask.Real Insert FAED improves from 0.454 with point guidance to 0.354 with bbox and 0.226 with mask; point guidance remains the default lightweight interface.
7 CONCLUSION
Mover360 provides translation-centered object manipulation for 360° ERP panoramas, with Remove and reference-guided Insert supported by the same model. Its interaction design, paired data, ERP-aware adaptation, and depth conditioning support strong quality across domains and guidance modes.
- Mover360 centers on Translation while supporting Remove and reference-guided Insert through the same model.
- A three-channel ERP-aligned instruction map unifies point, bbox, and mask guidance, enabling object relocation from a single click.
- A UE5 pipeline renders surface-aware object trajectories at scale and supports a dual-domain benchmark with real captured ground truth for all three tasks.
- The model adapts a pretrained rectified-flow transformer using LoRA, ERP-aligned condition latents, and auxiliary depth conditioning without changing the backbone.
- Across both test domains and two evaluation protocols, Mover360 achieves strong reconstruction fidelity, semantic consistency, and distributional quality.
Supplementary Material for Mover360: Controllable Object Manipulation in 360◦Panoramic Images
The supplementary benchmark protocols give perspective-editing baselines known-size target boxes or masks, while Mover360 is evaluated by default with only a target point.
- All perspective-editing baselines receive the required known-size target bbox or mask during ERP evaluation.
- Mover360 is evaluated by default with only a target point rather than a known-size target bbox or mask.
S1 BASELINE ADAPTATION PROTOCOLS
Perspective object-editing baselines are adapted to ERP panoramas through their native interfaces, with the per-method protocols summarized in Table S1.
- Each perspective-image editing baseline is adapted to ERP panoramas using its native interface.
- Table S1 summarizes the per-method adaptation protocols used for evaluation.
S2 ADDITIONAL QUALITATIVE RESULTS
Supplementary figures provide additional qualitative results for Translation, Insert, and Remove, plus a depth ablation showing improved target-size inference with depth conditioning.
- Additional qualitative results: Figures S1–S3 show additional Mover360 results for Translation, Insert, and Remove with overlaid instructions and magnified edited regions.
- Depth ablation: Without depth, the relocated object appears implausibly large, whereas depth conditioning produces a size consistent with local scene depth.
- Translation: Figure S1 presents Translation pairs with source regions, motion arrows, target points, and magnified target edits.
- Insert: Figure S2 presents Insert pairs with reference objects, target points, and magnified inserted objects.
- Remove: Figure S3 presents Remove pairs with marked source objects and magnified completed backgrounds.