PaperlayerView exact revision on arXiv

Source-linked AI summary

KINESYS: Test-Time Adaptation for Reference-Based Effect Editing in Images

Noah Frahm, Luchao Qi, Haoze Li, Dan B Goldman, Sambit Bhattacharya, Roni Sengupta

arXiv:2610.33292v1cs.CV

TL;DR

Effect-reference editing asks an image editor to realize selected properties of a visual reference without changing unrelated source content. KINESYS adapts a generator using reference-aware and source-aware feedback, with optimization across candidates and independent runs. It achieves strong source preservation while remaining competitive with Nano Banana 2 on effect transfer.

  • Problem

    Effect-reference editing must transfer specified visual properties while preserving the source, although instructions alone may not specify the desired effect’s appearance and interaction details.

    Source: §1 · p. 2

  • Method

    KINESYS adapts a generator for each request using reference-grounded VQA, source-aware EditReward, and optimization across fixed-noise candidates and independent runs.

    Source: §1 · p. 2

  • Results

    LPIPS distance is 0.377 versus Nano Banana 2’s 0.429, and TPIPS similarity is 0.731 versus 0.696; KINESYS leads evaluated open-weight editors while remaining competitive with Nano Banana 2 on effect transfer.

    Source: §1 · p. 3, §4.2 · pp. 7–8

  • Takeaways & Limitations

    Fresh noise samples yield multiple source-preserving versions of the referenced effect, with variation in texture, shape, density, and placement.

    Source: §1 · p. 2

Abstract

from arXiv · show

Visual effects are often easier to demonstrate via reference than to describe in words. We study effect-reference editing, where a reference image shows a desired visual effect and language identifies which properties should guide the edit. Existing single-pass editing techniques often fail to make fine-grained edits, resulting in unintended changes to source content or deviations from the reference effect. We introduce KINESYS, a test-time adaptation framework for effect-reference editing that combines reference-aware and source-aware feedback to transfer the desired effect while preserving the composition and content of the original shot. To ensure KINESYS can produce a diverse set of edits that preserve the source, we develop a robust optimization strategy in which the generative model is fine-tuned across multiple noise samples and optimization runs. Evaluation with VLM judges, source preservation metrics (LPIPS, TPIPS), and a user study shows that KINESYS outperforms state-of-the-art image editors in source preservation; on effect transfer, it remains competitive with Nano Banana 2 and outperforms open-weight editors.

1 INTRODUCTION

KINESYS addresses effect-reference editing, where an editor must transfer selected visual properties from a reference without changing unrelated source content. It adapts the generator using complementary feedback and optimization across multiple candidate edits and runs.

1 INTRODUCTION

  • Effect-reference editing requires transferring specified visual properties while preserving the source subject and scene structure.A text instruction may name an effect but leave its texture, color, scale, density, placement, and interactions unspecified.Source: §1 · p. 2
  • KINESYS fine-tunes a generator using reference-grounded VQA and source-aware EditReward to assess effect properties and source-preserving instruction following.The VQA judge sees the reference but not the source; EditReward sees the source but not the reference.Source: §1 · p. 2
  • To reduce dependence on an unlucky initial candidate set, KINESYS uses fixed-noise candidates across independent fine-tuning runs, then prunes runs and selects a checkpoint.Its optimization strategy uses LoRA and an early diversity penalty to discourage candidate collapse.Source: §1 · p. 2
  • After request-specific adaptation, unseen noise samples produce multiple source-preserving versions whose texture, shape, density, and placement can vary.The benchmark covers 81 editing requests spanning eight visual effects.Source: §1 · p. 2
  • KINESYS achieves the strongest source preservation, remains competitive with Nano Banana 2 on effect transfer, and outperforms evaluated open-weight editors on both axes.The blinded user study also favored KINESYS for source preservation against Nano Banana 2 and Step1X-Edit.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work covers learned image editors, reference-guided editing, visual-effects generation, feedback-driven adaptation, and inference-time search. KINESYS adapts a generator to one effect-reference request using both the source and effect reference.

2 RELATED WORK

  • Dataset-trained editors and reference-guided methods include paired-data learning such as InstructPix2Pix (Brooks et al., 2023), reference transfer, mask-free editing, and operation inference from before-and-after pairs.KINESYS adapts to one request without an effect-specific training set or target edited image.Source: §2 · p. 3
  • Visual-effects methods include dataset-trained reference-based generation and test-time prompt refinement, whereas KINESYS adapts an editor for one image using a source and effect reference.The reference defines the effect; the source defines what stays unchanged.Source: §2 · p. 3
  • Request-specific adaptation includes fine-tuning for a source and text, reward optimization, feedback-trained shared editors, and concept-specific adaptation from reference images.Dual-Process (Luo et al., 2025) most closely relates through VLM-guided request-specific fine-tuning, which KINESYS extends to effect-reference editing with source constraints.Source: §2 · p. 3
  • Editing evaluators range from source-and-instruction feedback to reference-aware judgments; KINESYS combines both feedback roles within request-specific fine-tuning.Prior reference-aware examples include ICE-Bench, EditCLIP, and dataset-level post-training with EVR.Source: §2 · p. 3
  • Unlike inference-time search and iterative revision methods that hold the generator fixed or optimize controls, KINESYS allocates compute across independent fine-tuning runs.It returns an editor that can produce several source-preserving versions of the referenced effect.Source: §2 · p. 3

3 METHOD

KINESYS adapts an image editor to one effect-reference request using reference-aware and source-aware feedback. Its optimization learns from multiple candidates, maintains diversity early, and searches independent runs before selecting an adapted editor.

3 METHOD

  • Given a source, effect reference, instruction, reference-grounded questions, and optional mask, KINESYS adapts an editor to transfer selected effect properties while preserving source content.Without a mask, the entire source image may be edited.Source: §3 · pp. 3–4

3.1 COMPLEMENTARY EDITING FEEDBACK

  • Reference-grounded VQA compares candidates with the effect reference and scores whether they realize the selected appearance, placement, or interaction properties.Its loss averages the negative log-likelihood of the desired Yes answers.Source: §3.1 · p. 4
  • Source-aware EditReward scores instruction following and visual quality, complementing VQA, which cannot directly assess unintended changes to the source.The objectives are intended to balance effect transfer with source preservation and artifact control.Source: §3.1 · pp. 4–5

3.2 ROBUST OPTIMIZATION

  • Each run reuses N fixed noise samples to render candidates across optimization steps, making successive feedback comparable as the editor changes.Candidate EditReward scores are smoothed, standardized, averaged, and converted through softmax into weights that reduce—but do not eliminate—the influence of poorly scored candidates.Source: §3.2 · p. 5
  • A diversity term discourages early convergence among candidates by pushing current latents away from other candidates’ recent states, then decays to zero.For N = 1, the diversity term is set to zero.Source: §3.2 · p. 5
  • KINESYS runs independent LoRAs with separate noise sets, prunes lower-ranked runs at scheduled checkpoints, and selects the saved checkpoint with the highest EditReward on held-out noise.The selected editor can produce further effect variations from fresh noise while retaining the source scene.Source: §3.2 · pp. 5–6

4 EXPERIMENTS

The evaluation tests effect transfer and source preservation across effect-reference requests, comparing KINESYS with open-weight and proprietary editors and probing feedback and sampling choices. Results favor KINESYS on preservation, with competitive transfer and multiple source-consistent variants.

4.1 EXPERIMENTAL SETUP

  • Evaluation spans 81 requests across eight effects, comparing direct editors on shared two-seed cases using Gemini quality judgments and LPIPS and TPIPS as source-preservation proxies.The 81 requests cover eight categories; similarity measures do not directly assess camera pose, object layout, or subject identity.Source: §4.1 · pp. 6–7

4.2 COMPARISON WITH EXISTING EDITORS

  • LPIPS distance 0.377 versus Nano Banana 2’s 0.429 and TPIPS similarity 0.731 versus 0.696; KINESYS also leads open-weight editors on Gemini and EditScore-72B.Table 1 compares methods on 156 shared cases from 81 requests and two seeds.Source: Table 1 · p. 7, §4.2 · pp. 7–8
  • 45% versus 25% source-preservation preference over Nano Banana 2 and 54% versus 21% over Step1X-Edit; effect-transfer preference was close to Nano Banana 2 (48% versus 42%).The blinded paired-comparison study covered 16 edits; ties were allowed.Source: Figure 3 · p. 7, §4.1 · p. 7, §4.2 · p. 8
  • In the snow example, KINESYS retains the source house and viewpoint while Nano Banana 2 changes both; for fire, KINESYS avoids Step1X-Edit’s scene-wide blaze.Figure 4 compares the reference, source, KINESYS, Nano Banana 2, and Step1X-Edit v1.2 across six effects.Source: Figure 4 · p. 8

4.3 ABLATION

  • Without EditReward, unrelated source changes appear; without VQA, explosion and wake forms depart from the reference, while full KINESYS balances transfer and preservation.The ablations illustrate these differences on explosion and wake edits.Source: §4.3 · p. 8, Figure 5 · p. 9
  • Three candidates balance editing quality and per-request compute; increasing N from one to three raises mean Gemini from 8.59 to 9.00 and mean worst-seed score from 8.11 to 8.73.At N = 5, both reported scores fall from their N = 3 values.Source: §4.3 · p. 8, Table 2 · p. 9

N Mean↑ Worst↑

  • Five runs raise mean Gemini from 7.31 to 9.00 and mean worst-seed score from 6.45 to 8.73; eight runs score higher but cost more compute.The authors use five runs to favor worst-seed quality without the cost of eight.Source: Table 2 · p. 9, N Mean↑ Worst↑ · p. 9

4.4 APPLICATIONS

  • Masks localize edits; fresh noise varies effect details while source scenes stay consistent, with mean EditScore-72B 8.91 versus 8.93 for selected outputs across five unseen draws.The fresh-noise evaluation uses all 162 selected checkpoints, one per request and seed.Source: §4.4 · p. 9, Figure 6 · p. 9, Figure 7 · p. 10

5 CONCLUSION

KINESYS adapts a generator to transfer reference-demonstrated effects while preserving source composition and content, with complementary feedback and independent runs supporting that balance. The conclusion notes limitations in rendering, recognition, starting noise, and latency, then previews the appendices.

5 CONCLUSION

  • KINESYS transfers reference-demonstrated effects while preserving source composition and content, using complementary VLM feedback and independent fine-tuning runs.Fresh-noise samples offer multiple edit variations; the conclusion reports better LPIPS and TPIPS scores than Nano Banana 2 on shared cases.Source: §5 · p. 10
  • The approach depends on what its generator can render and its VLM judges can recognize; poor starting noise may yield weak edits, and per-request fine-tuning adds latency.An unfamiliar creature paired with a rare stylized electrical effect is given as an example of a difficult edit.Source: §5 · p. 10

APPENDIX

  • The appendices cover visual comparisons, evaluation-data construction, method settings, model-facing prompts, and blinded-survey preferences.Appendix A compares methods, B describes the 81 requests, C documents implementation settings, D records prompts, and E reports survey preferences by effect family.Source: APPENDIX · p. 15

A ADDITIONAL VISUAL RESULTS

The additional visual results present 20 effect-reference comparisons across eight methods. Each row holds the source and reference constant across methods, and the figures split the grid between rows 10 and 11.

A ADDITIONAL VISUAL RESULTS

B EVALUATION DATASET CONSTRUCTION

The benchmark comprises 81 effect-editing requests across eight families, paired with source and reference photographs but no target edited image. Requests use fixed family-level instructions and questions, with whole-image edit regions in the automated benchmark.

B EVALUATION DATASET CONSTRUCTION

  • The evaluation covers 81 requests across eight effect families, using 78 distinct source photographs and 63 distinct effect references.Three car photographs appear in both the dent and fire families; 13 references serve two to four requests within their family.Source: B EVALUATION DATASET CONSTRUCTION · p. 17
  • Each request pairs a source photograph with a reference photograph showing the desired effect on different content, and no target edited image is provided.References were selected by reviewing source photographs alongside references from the same effect family, without using generated target edits.Source: B EVALUATION DATASET CONSTRUCTION · p. 17
  • Effect-family request counts range from eight to 13, and reported automated scores weight retained results from each request and seed equally.Figure B.1 counts requests rather than unique photographs; Table B.1 lists family counts and properties selected for verification.Source: B EVALUATION DATASET CONSTRUCTION · p. 17, Figure B.1 · p. 18, Table B.1 · p. 18
  • The benchmark uses one instruction template and three fixed yes/no questions per effect family to assess effect location, realism, and a defining reference-matched property.The desired answer to each question is “Yes,” and questions remain fixed during fine-tuning.Source: B EVALUATION DATASET CONSTRUCTION · p. 17
  • Source and reference photographs were downscaled to a longest side of at most 768 pixels, while the adapted editor's reference-conditioning cap was 384 pixels.The automated benchmark specifies a whole-image edit region and supplies mask-capable editors with a full-image mask.Source: B EVALUATION DATASET CONSTRUCTION · p. 17

C.1 DIVERSITY TERM AND DECAY

The appendix specifies how diversity is encouraged early in optimization, how candidate rewards are smoothed and fused, and how runs and checkpoints are allocated and selected.

C.1 DIVERSITY TERM AND DECAY

  • Each fine-tuning run keeps its own bank of previous-step candidate latents, replaced after each step; the bank starts empty, so the diversity penalty is initially zero.For N = 1, there are no candidate pairs and the diversity term is set to zero.Source: §C.1 · p. 19
  • The diversity weight starts at 1 and decays during the first 60% of optimization, so the term contributes only during early steps.The previous-step latents and bandwidth are held fixed during gradient computation; gradients flow through the current candidate latent.Source: §C.1 · p. 19

C.2 CANDIDATE GATE

  • The candidate gate smooths each candidate's reward per head, standardizes rewards within the candidate population, and fuses the resulting advantages to weight task gradients.When reward spread falls below the noise floor, the standardized advantages shrink and the gate approaches uniform candidate weights; the diversity gradient remains separate.Source: §C.2 · p. 19

C.3 FINE-TUNING RUN ALLOCATION AND CHECKPOINT SELECTION

  • Run allocation summarizes VQA desired-answer probabilities and higher-is-better EditReward scores across a trailing 15-step window, then standardizes and blends those summaries across active runs.The VQA probabilities average over candidates and questions; the EditReward score is averaged over the same candidates and steps.Source: §C.3 · p. 20
  • The search starts with five runs and retains three, then two, at checkpoint labels 15 and 30; surviving runs may stop early after reward drawdown, but not before step 50.Early stopping requires the smoothed reward to remain more than 0.3 below its running peak for three consecutive steps; the run restores its best saved LoRA state from a chunk boundary.Source: §C.3 · p. 20
  • Final checkpoint selection uses EditReward on a dedicated fixed-noise evaluation image, separate from the fine-tuning candidates; fresh-noise evaluation later uses five additional unseen samples.The evaluation image contributes neither gradients nor allocation feedback, and VQA is not used for cross-checkpoint selection.Source: §C.3 · p. 20

C.4 IMPLEMENTATION DETAILS

KINESYS adapts FLUX.2 Klein 4B with LoRA updates using reference-grounded VQA and EditReward feedback, while searching across candidate edits and fine-tuning runs.

C.4 IMPLEMENTATION DETAILS

  • KINESYS updates only rank-16 LoRA matrices, keeping the base generator and text encoder frozen.Each run uses up to 100 optimization steps, with checkpoints after 16, 31, 46, 61, 76, 91, and 100 updates.Source: §C.4 · p. 21
  • Each run evaluates three fixed-noise candidates with Gemma VQA and EditReward, while a five-run search prunes to three and then two runs.The diversity weight decays to zero after 60% of fine-tuning; the task objective weights VQA at 1 and EditReward at 2.Source: §C.4 · p. 21

C.5 SEARCH CONFIGURATION STUDY

The appendix reports the search-configuration analysis and specifies the generation inputs, optimization prompts, and frozen Gemini judge used in canonical experiments.

C.5 SEARCH CONFIGURATION STUDY

  • The search-configuration analysis covers 16 requests across eight effect families, each evaluated with two seeds for 32 results.The accompanying table lists the requests evaluated under both seeds.Source: §C.5 · p. 21, Table C.1 · p. 21
  • For K values of 1, 3, 5, and 8, runs are pruned at checkpoint labels 15 and 30 to retained counts of (1, 1), (2, 2), (3, 2), and (4, 2).The study fixes K = 5 when varying N and N = 3 when varying K.Source: §C.5 · p. 21

D OPTIMIZATION AND EVALUATION PROMPTS

  • The VQA and EditReward prompts affect fine-tuning gradients, while Gemini prompts are evaluation-only and do not affect fine-tuning or checkpoint selection.EditReward also selects the final checkpoint.Source: D OPTIMIZATION AND EVALUATION PROMPTS · p. 22

D.1 GENERATION INPUTS

  • The adapted generator receives both the source and effect-reference images, while editors receive the same fixed instruction and mask-capable editors also receive the full-image mask.The request's source and reference are supplied to each editor where supported.Source: §D.1 · p. 22

D.2 REFERENCE-GROUNDED VQA

  • The VQA loss averages negative log-likelihood across three fixed reference-grounded questions, with the reference presented before the candidate and Yes as the target answer.One snow criterion asks whether snow covers rooftops and lawns as it does in the reference.Source: §D.2 · p. 22

D.3 FROZEN GEMINI 2.5 FLASH SCORE JUDGE

  • The Gemini 2.5 Flash evaluator rates perceptual quality, source consistency, and edit adherence from 0 to 10; Table 1 reports the mean of the first two, excluding edit adherence.It receives the source, edited candidate, and effect reference in that order. The reference is an exemplar rather than a pixel-exact target.Source: §D.3 · p. 23

E USER STUDY

The user study used blinded pairwise comparisons to assess source preservation and effect transfer against Nano Banana 2 and Step1X-Edit v1.2. Its reported analysis covers two cases per effect family across eight families.

E USER STUDY

  • The study compares methods in blinded 2 × 2 videos, with the source and reference static above anonymous outputs and synchronized sliders sweeping between source and result.Method assignment was counterbalanced, item order shuffled, and model names hidden.Source: E USER STUDY · p. 24
  • The analysis excludes two rain cases and reports 16 cases, two per family, compared separately with Nano Banana 2 and Step1X-Edit v1.2.Participants judged source preservation and effect transfer; most were not VFX experts.Source: E USER STUDY · p. 24
  • Figures E.1 and E.2 show preferences by effect family, with purple favoring KINESYS, gray indicating a tie or unclear choice, and red favoring the baseline.They contain 41 completed response sets for Nano Banana 2 and 40 for Step1X-Edit v1.2; the boat-wake example identifies the first question as source preservation.Source: Figure E.1 · p. 24, E USER STUDY · p. 24, Figure E.2 · p. 25

Related papers

Loading 2610.33292v1…