PaperlayerView exact revision on arXiv

Source-linked AI summary

ALIVE: Interaction-Aligned Object Insertion for First-Frame-Guided Video Editing

Zhenghong Zhou, Zhe Lin, Jiebo Luo, Yuqian Zhou

arXiv:2610.08779v1cs.CV

TL;DR

Inserted objects can be visually convincing yet fail to respond coherently to source actions, and constructing pairs that preserve the surrounding action is challenging. ALIVE combines interaction-focused editing data, a first-frame-guided editor, and VLM-predicted temporal prompts; it improves Overall over the strongest evaluated baseline on both interaction and general insertion benchmarks.

  • Problem

    Video editors can insert objects but may not make them respond coherently to source actions, while constructing interaction pairs requires changing object presence and preserving the surrounding scene and action.

    Source: §1 · p. 2

  • Method

    ALIVE trains a first-frame-guided diffusion editor on 35,800 pairs and uses a VLM to predict temporally localized interaction prompts from the source video, edited first frame, and object-only instruction.

    Source: §1 · p. 2

  • Results

    43.9% and 4.4%: without VLM guidance, ALIVE improves Overall scores over the strongest evaluated baseline on the two benchmarks, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ALIVE evaluates both interaction-focused and general insertion quality, and VLM-predicted guidance further improves interaction editing without additional user input.

    Source: §6 · p. 9

    The inputs do not fully specify geometry, contact conditions, or forces, leaving multiple plausible object responses; VLM guidance errors can cause incorrect behavior, and complex cloth deformation and rapid large-angle rotations remain difficult.

    Source: §5.3.2 · p. 9

Abstract

from arXiv · show

Current video editors can insert objects but often struggle to make them participate in interactions such as being picked up or manipulated. We introduce ALIVE, a framework that makes inserted objects "alive" through coherent interactions with the source video's contents, using an edited first frame and an instruction naming only the added object. We curate 35,800 editing pairs combining 3D-rendered, model-generated, and real-world videos with general editing pairs from ROSE. Each pair differs in the target object's presence while preserving the surrounding action, teaching editors coordinated object behavior and source preservation. We further train a vision-language model (VLM) to predict interaction guidance from the same inputs. We introduce the ALIVE-interaction benchmark to assess interaction fidelity, source preservation, and visual coherence using a unified VLM-based protocol, and evaluate on the general video object insertion benchmark. Without VLM guidance, ALIVE improves Overall over the strongest evaluated baseline by 43.9% and 4.4% on the two benchmarks, respectively. VLM-predicted guidance further improves the ALIVE-interaction score by 0.95 points without additional user inputs.

1 INTRODUCTION

ALIVE targets object insertion that responds coherently to source actions, combining interaction-focused data, first-frame-guided editing, and VLM-predicted temporal prompts. It evaluates interaction and general insertion, reporting higher Overall scores than the strongest evaluated baseline on both benchmarks.

1 INTRODUCTION

  • ALIVE combines 35,800 editing pairs from rendered, model-generated, real-world, and ROSE sources with quality checks and multi-level interaction prompts.The VLM predicts temporally localized prompts from the same inputs without user-provided interaction descriptions, masks, or trajectories.Source: §1 · p. 2
  • The editor uses a source video, edited first frame, and object-only instruction so the frame fixes initial appearance and placement while later behavior follows source actions.The VLM predicts temporally localized interaction prompts from those inputs.Source: §1 · p. 2
  • 43.9% and 4.4%: without VLM guidance, ALIVE improves Overall scores over the strongest evaluated baseline on the two benchmarks, respectively.Predicted temporal prompt guidance further improves interaction performance.Source: §1 · p. 2
  • ALIVE-interaction assesses coherent participation in source actions and source preservation, while a general insertion benchmark assesses broader editing quality.Ablations examine data composition and interaction guidance.Source: §1 · p. 2

2 RELATED WORK

Related work covers general video editing and paired datasets, first-frame-guided editing, interactive editing, and VLM-supported video editing. ALIVE is positioned as first-frame-guided insertion in which objects respond to source actions without user-provided interaction descriptions.

Video editing and paired video

  • Shared video-editing systems unify tasks, while instruction-guided, reference-guided, first-frame-guided, and sparse-keyframe methods offer different conditioning interfaces.First-frame-guided approaches propagate initial visual edits through source videos.Source: §2 · p. 2
  • Paired datasets cover broad editing and object removal or insertion, while ALIVE focuses on first-frame-guided insertion with coherent responses to source actions.ALIVE pairs are supported by interaction-focused data and multi-level prompt annotations.Source: §2 · p. 2

Interactive video editing

  • Interactive editing includes streaming workflows and within-scene physical interactions; ALIVE studies insertion responding to source actions without interaction descriptions from users.The cited within-scene systems address downstream interactions after removal or edit actions and dynamics through text.Source: §2 · p. 3

VLM guidance for video editing

  • Prior VLM-assisted editing systems use multimodal understanding for scene descriptions, edit planning, or affected-region identification; ALIVE predicts temporally localized prompts for inserted-object responses.Those prompts guide corresponding video intervals.Source: §2 · p. 3

3 ALIVE DATASET AND BENCHMARK

ALIVE combines interaction-focused editing pairs, multi-level prompt annotations, and benchmarks that evaluate object responses to source actions as well as general insertion quality.

3.1 INTERACTION PAIR CONSTRUCTION

  • Pair construction varies target-object presence while preserving surrounding scene and action, using rendered, model-generated, and real-world interaction videos plus ROSE general editing pairs.Rendered scenes provide aligned views and masks; model-generated and real-world sources use segmentation or geometry followed by object removal.Source: §3.1 · p. 3
  • Quality checks exclude interaction clips without visible interaction or adequate visual quality, and pairs with incomplete or visibly incorrect object removal.The rendered, model-generated, real-world, and ROSE sources contribute complementary interaction and non-interaction cases.Source: §3.1 · p. 3
  • 35,800 unique pairs combine 14,362 3D-rendered, 5,896 model-generated, 8,542 real-world, and 7,000 ROSE general editing examples.The sources provide aligned pairs, common interactions, real-world examples, and cases without object interaction.Source: §3.1 · p. 3

3.2 MULTI-LEVEL INTERACTION ANNOTATION

  • P1 names the added object, while P2–P4 add interaction descriptions ranging from the principal interaction to detailed sequences and temporally localized chunks.P0 has no text; annotations support studying guidance levels and VLM prediction from the source, edited first frame, and P1.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 EVALUATION BENCHMARKS

  • The interaction benchmark has 128 cases, and the general insertion benchmark has 103; a common protocol evaluates editing fidelity, source preservation, and visual coherence.The interaction set includes held-out and out-of-domain subsets; general insertion cases come from ROSE and EffectErase.Source: §3.3 · p. 4

4 METHOD

ALIVE generates an edited video from the source, an edited first frame, and an object-identity instruction, and uses temporally aligned prompts to guide interactions. A VLM predicts those prompts from the same inputs.

4.1 TASK DEFINITION

  • The task is to generate an edited video whose inserted object participates in source actions while unrelated scene content is preserved, using the source video, edited first frame, and identity-only instruction.The edited first frame specifies appearance and initial placement, while the source supplies action context; the target video is training supervision unavailable at inference.Source: §4.1 · p. 5

4.2 VIDEO EDITING MODEL

  • The editor conditions on aligned source and target latents, concatenating clean source latents with noisy target latents while keeping the edited first-frame tokens clean during denoising.This supplies source context without increasing the number of video tokens.Source: §4.2 · p. 5
  • P4 aligns interaction descriptions with temporal intervals and routes each description to video tokens according to temporal overlap; P0–P3 use clip-level text conditioning.Zero overlap masks a context, and normalization prevents longer descriptions from gaining prior attention mass solely through length.Source: §4.2 · p. 5
  • A single editor is LoRA-trained on ALIVE pairs using mixed prompt levels and a flow-matching objective over target tokens not conditioned on the edited first frame.For VLM-guided editing, it receives P4 prompts predicted from the original inputs.Source: §4.2 · p. 6

4.3 VLM-BASED INTERACTION GUIDANCE

  • A LoRA-adapted VLM predicts chunk descriptions and intervals from ordered source frames, the edited first frame, and P1, then supplies P4 guidance to the editor.Prediction uses these inputs during training and inference, although annotation may use the target video.Source: §4.3 · p. 6

5 EXPERIMENTS

ALIVE improves interaction-focused insertion while retaining strong general insertion quality, and the experiments examine editor comparisons, training-data composition, and interaction guidance. They also show that guidance errors and complex motion remain limitations.

5.1 EXPERIMENTAL SETUP

  • The evaluation uses a reference-aware VLM judge to assess interaction fidelity, source preservation, and visual coherence across both benchmarks.Overall aggregates six dimensions on a 0–100 scale; Table 1 also reports task fidelity and motion/propagation.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 COMPARISON WITH EXISTING VIDEO EDITORS

  • On the ALIVE-interaction benchmark, ALIVE scores 86.71 versus 60.24, a 26.46-point difference; on general insertion, it scores 92.28 versus 88.39, a 3.89-point difference.Source: §5.2 · p. 7

5.3.1 DATA SOURCES

  • Model-generated pairs perform best on interaction, while ROSE leads individual sources on general insertion; the mixture scores 82.35 versus 60.12 for ROSE-only on ALIVE-interaction.The mixture stays within 0.46 points of model-generated-only on interaction and improves general insertion by 7.61 points, with the highest general-insertion and pooled scores.Source: §5.3.1 · p. 7, Table 2 · p. 8

5.3.2 PROMPT LEVELS AND VLM GUIDANCE

  • Annotated P3 and temporally localized P4 raise interaction Overall to 87.37 and 88.69, compared with 85.38 for P0 and 86.71 for P1.P2 adds little benefit; Figure 5 illustrates how P4’s specified placement sequence relates to fewer strap artifacts than P1.Source: §5.3.2 · p. 8, Figure 5 · p. 9
  • Predicted P4 raises P1’s interaction Overall from 86.71 to 87.66 using the original editing inputs, with gains in temporal consistency, background preservation, and motion/propagation.For the seedling example, the predicted prompt describes lowering, releasing, and firming soil, and the plant is better seated as hands press around it.Source: §5.3.2 · pp. 8–9
  • ALIVE remains limited by underspecified geometry, contact conditions, and forces; guidance errors and complex cloth deformations or rapid large-angle rotations also challenge it.The source video and edited first frame can leave multiple plausible object responses.Source: §5.3.2 · p. 9

6 CONCLUSION

ALIVE inserts objects that participate coherently in source-video interactions, using complementary data and benchmarks for interaction-focused and general insertion. The conclusion reports benefits from interaction-focused training and prompt guidance, including VLM-predicted guidance without additional user input.

6 CONCLUSION

  • ALIVE combines complementary data sources and benchmarks to support coherent object interactions, and VLM-predicted guidance further improves interaction editing without additional user input.P4 performs best among the evaluated prediction formats.Source: §6 · p. 9

A.1 TRAINING DETAILS

The appendix describes separate training configurations for ALIVE’s diffusion editor and its VLM prompt predictors.

A.1 TRAINING DETAILS

  • The diffusion editor adapts LTX-2.5 with rank-128 LoRA on 35,800 pairs, using 81-frame clips and 20,000 updates.Training samples prompt levels P0–P4 with probabilities 0.1/0.3/0.2/0.2/0.2; inference generates 81 frames with 30 denoising steps.Source: §A.1 · p. 14
  • Separate P2, P3, and P4 predictors adapt Qwen3.6-27B with rank-64 LoRA, using source frames, the edited first frame, and P1 as inputs.Each predictor uses a global batch size of 32 and the 2,048-update checkpoint.Source: §A.1 · p. 14

A.2 INFERENCE SETTINGS AND EFFICIENCY

The appendix profiles editor inference costs on six benchmark videos and separately measures VLM prompt prediction.

A.2 INFERENCE SETTINGS AND EFFICIENCY

  • Table 4 reports mean wall-clock time and the largest peak allocated GPU memory across six videos, using one A100 80GB per video.Output dimensions and frame counts are listed separately from denoising iterations; peak memory is measured in GiB.Source: Table 4 · p. 14, §A.2 · p. 14
  • The measured 207.25 seconds per edit sums the separately profiled editor and VLM stages, excluding model-switching overhead.Timing includes conditioning, required inversion or per-video adaptation, sampling, decoding, and export; offline preparation and one-time setup are excluded.Source: §A.2 · p. 14

A.3.1 TRAINING LENGTH

Longer P1-only LTX-2.5 training improves Overall scores on both benchmarks; the appendix also compares backbones and details the shared evaluation protocol.

A.3.1 TRAINING LENGTH

  • 82.35 to 86.46 on ALIVE-interaction and 86.93 to 89.20 on general insertion are the Overall scores after 5k versus 20k training steps.The comparison uses P1-only LTX-2.5 editors trained on the same 35,800 pairs with fixed evaluation cases and generation seeds.Source: §A.3.1 · pp. 14–15, Table 5 · p. 15

A.3.2 BASE MODELS AND MODEL SIZE

  • Wan 14B scores higher in the 5k-step comparison, while LTX-2.5 trains substantially faster and serves as the main experimental backbone.For 5k updates, reported training time is about 63 hours for Wan 14B versus 8.1 hours for LTX-2.5.Source: §A.3.2 · p. 15, Table 6 · p. 15

A.4 EVALUATION PROTOCOL

  • The evaluation judge receives source, reference, and candidate frames, the edited first frame, and a shared object-addition instruction; only the candidate is scored.ALIVE-interaction uses P1, while the general insertion benchmark uses the instruction to add the object or person shown in the edited first frame.Source: §A.4 · p. 15, Table 7 · p. 16
  • Aligned evaluation samples 16 native output frames, including both endpoints, and maps them to the source frames selected by each generator; I2VEdit uses all 14 native frames.Spatial alignment follows each generator’s documented aspect-preserving resize and center crop.Source: §A.4 · p. 15
  • The judge scores plausible interactions, source preservation, and visual coherence on six weighted dimensions, with each dimension scored from 0 to 4 and Overall scaled by 25.The rubric allows reasonable spatial and temporal differences from the reference but treats missing interactions, broken contact, impossible motion, and source corruption as errors.Source: §A.4 · pp. 15–16

Related papers

Loading 2610.08779v1…