PaperlayerView exact revision on arXiv

Source-linked AI summary

PredErase: Training-Free Object-and-Effect Removal with Predictive Latent Guidance

Waikit Xiu, Qiang Lu, Junbiao Chen, Xiying Li

arXiv:2609.00956v1cs.CV

TL;DR

Object removal under instance-only masks must address cast shadows and contact shading outside the mask, which mask-local frozen Fill editing can leave behind. PredErase expands the editable support with a contact band and steers frozen FLUX.2 using an I-JEPA hole target, improving the native backbone across the reported benchmarks while remaining weaker than supervised removers on some appearance metrics. Its supported scope is training-free object-and-effect editing of frozen Fill, with limitations for near-full-image masks and non-upright or detached effects.

  • Problem

    Object removal must erase cast shadows and contact shading outside Mobj, while training-free editors commonly leave such unmasked residuals untouched.

  • Method

    PredErase separates editable support from hole structure by expanding Mobj to Mflux and using frozen I-JEPA representations to guide frozen FLUX.2 sampling.

  • Results

    PredErase improves the native FLUX.2 backbone under instance-only masks across RemovalBench, RORD-Val, and DEFACTO-Val, although supervised removers remain stronger on several full-image appearance metrics.

  • Takeaways & Limitations

    The method supports training-free object-and-effect editing of frozen Fill without paired clean plates or weight updates, rather than replacing supervised paired-data erasers.

  • Takeaways & Limitations

    PredErase is less reliable for near-full-image targets and assumes upright ground contact; side lighting, detached shadows, and reflections fall outside its geometry.

Abstract

from arXiv · show

Removing an object is not the same as filling its mask. Cast shadows and contact shading usually lie outside the user-provided instance mask M_obj, so a frozen Fill model that edits only that mask leaves the object's photometric footprint on nearby surfaces. Supervised removers learn this joint erasure from paired clean plates. Training-free editors freeze pretrained weights, yet most still treat M_obj as the entire editable support and steer sampling with CLIP or DINO energies that do not predict the occluded scene. We present PredErase, a training-free inference procedure on frozen FLUX.2 and I-JEPA. The method separates where Fill may rewrite pixels from what structure should occupy the hole. A contact-band expansion M_flux of M_obj exposes local residuals on the supporting plane. I-JEPA, pretrained for masked token prediction, supplies a context-conditioned hole target in representation space; sparse projected gradients align decoded Fill completions with that target inside the instance, while coordinates outside the packed support stay locked. Under instance-only masks on RemovalBench, RORD-Val, and DEFACTO-Val, PredErase improves the native FLUX.2 backbone. Supervised removers remain stronger on several full-image appearance metrics; the supported claim is training-free object-and-effect editing of frozen Fill, not replacement of paired-data erasers.

1. Introduction

PredErase frames object-and-effect removal as editing both the instance and geometry-coupled residuals under instance-only masks. It combines an expanded Fill support with an I-JEPA hole target to steer frozen FLUX.2 without paired clean plates or weight updates.

  • Motivation: Object removal must erase cast shadows and contact shading outside the instance mask while preserving unrelated content.Mirror reflections, refraction, and global illumination are outside the stated setting.
  • Motivation: A frozen Fill model restricted to Mobj can complete the hole while leaving photometric traces unchanged, which full-image clean-plate metrics detect.RemovalBench and RORD-Val evaluate full-image agreement rather than only pixels inside Mobj.
  • Gap: Training-free editors commonly keep Mobj as the editable support, so unmasked shadows and reflections remain untouched unless users dilate the mask manually.Supervised effect-aware removers address this with paired object–clean-plate data, often with backbone-specific adaptation.
  • Approach: PredErase separates where Fill may rewrite pixels from what structure should occupy the hole by expanding support to Mflux and using I-JEPA prediction.Projected decoder-side updates are confined to the packed latent support.
  • Results: PredErase improves native FLUX.2 across reported metrics on RemovalBench, RORD-Val, and DEFACTO-Val, while supervised OmniEraser remains stronger on some appearance metrics.The reported contribution is training-free object-and-effect editing of frozen Fill, not replacement of paired-data erasers.

2. Related Work

Prior work either learns effect-aware removal from paired clean plates or performs training-free mask-local editing with semantic guidance. PredErase instead uses masked predictive representations to infer what belongs in the hole from visible context.

  • Inpainting and supervised removal: Inpainting models complete content inside object masks, but cast shadows and contact shading often remain outside those masks.Clean-plate benchmarks make this outside-mask failure measurable, while supervised removers learn joint erasure from paired data.
  • Training-free editing: Training-free editors update frozen diffusion or flow latents, commonly using CLIP or DINO energies while retaining the object mask as editable support.Unmasked shadows and reflections persist unless they already lie inside the mask.
  • Predictive representations: I-JEPA predicts masked-region tokens from visible context without decoding pixels, matching the hole-prediction interface needed for object removal.PredErase uses the frozen predictor as a representation-space hole prior.

3. Methodology

PredErase steers a frozen Fill trajectory by separately defining an effect-aware editable support and a context-conditioned representation target for the object hole. Sparse projected guidance changes only packed-support latents while Fill supplies appearance.

  • 3.1. Problem Setup: The task seeks an output where the masked instance and associated outside-mask effects are absent.The input consists of an RGB image I and binary object mask Mobj.
  • 3.1. Problem Setup: With M=Mobj, local completion can mismatch broader scene structure and residuals outside the mask remain fixed as conditioning evidence.These are the two coupled limitations motivating PredErase.
  • 3.2. PredErase: PredErase freezes both the Fill generator and I-JEPA predictor, redirecting only the Fill sampling trajectory at inference time.Fill remains responsible for appearance, including residual illumination inside Mflux.
  • 3.2. PredErase: Gray-filled Mobj produces a cached I-JEPA target, while expanding Mobj into Mflux determines where Fill may edit.The predictive branch specifies replacement structure; the generative branch specifies editable support.
  • 3.3. Predictive Representation Prior: I-JEPA alignment compares current and predicted hole tokens, then sparse projected-gradient updates steer decoded Fill completions toward context-predictable structure inside Imask.Fill remains responsible for appearance and can clean residuals over the larger Mflux.
  • 3.3. Effect-aware edit region: A contact-band prior expands the object mask to expose local residuals on the supporting plane, assuming upright, ground-contacted objects.The construction uses a contact segment and orientation-dependent band geometry; non-upright inputs need externally supplied orientation.
  • 3.4. Latent Update: Guidance is applied only at t∈{4,2} for T=14, leaving the remaining flow-matching steps unchanged.The sparse schedule adds two late corrections after a reasonably formed decoded preview is available.
  • 3.4. Latent Update: Projection permits movement only on packed-support coordinates, restoring all other coordinates to the native Fill state.This confines the additional guidance update outside the editable latent support.

4. Experiments

PredErase is evaluated as a training-free, frozen-backbone editor across three public removal benchmarks and official protocol metrics. It improves native FLUX.2 broadly, while supervised removers remain stronger on selected full-image metrics and the method is limited when visible context or contact-band assumptions fail.

  • Experimental setup: PredErase is compared with classical and diffusion inpainters, training-free editors, supervised removers, and native frozen FLUX.2 on official benchmark splits.Evaluations cover RemovalBench and RORD-Val under OmniEraser, and DEFACTO-Val under SmartEraser.
  • Main results: 0.108 versus 0.496 CMMD and 24.36 versus 22.70 dB PSNR show PredErase improving native FLUX.2 on RemovalBench.The comparison uses full-image metrics under the OmniEraser protocol.
  • Main results: 55.59 versus 149.02 FID and 0.305 versus 0.644 CMMD show improvement over native FLUX.2 on RORD-Val.PredErase also has the best listed LPIPS and PSNR there, while OmniEraser leads FID and CMMD.
  • Main results: On DEFACTO-Val, PredErase leads listed ReMOVE, LPIPS, and PSNR, whereas supervised SmartEraser remains strongest on CMMD and SSIM.Every reported axis improves over native FLUX.2 in this evaluation.
  • Qualitative results: Qualitative comparisons show native FLUX.2 leaving smears, broken grout, silhouettes, and contact shading on supporting surfaces.PredErase rewrites the instance and nearby residuals within M_flux, producing more coherent desk tone and grout continuation.
  • Ablation study: Full ablations outperform native FLUX.2 and text-only expanded-support controls, while matched CLIP and DINOv2 prior swaps trail I-JEPA.The ablations isolate I-JEPA under fixed M_flux; the prior comparison uses the same schedule and operating point.
  • Limitations: PredErase is less reliable for near full-image targets, assumes upright ground contact for its contact band, and inherits Fill-backbone appearance biases.Detached shadows, side lighting, reflections, and residuals detached from the contact contour fall outside the stated geometry.

5. Conclusion

PredErase frames training-free object-and-effect removal as jointly deciding which pixels may change and what should fill the object hole. It expands editable support for local residuals, uses I-JEPA for hole structure, and improves frozen FLUX.2 under instance-only masks.

  • PredErase treats object-and-effect removal as two coupled questions: editable support and hole content.
  • A contact-band expansion exposes local residuals on the supporting plane to frozen Fill.
  • I-JEPA supplies a representation-space hole target, while sparse projected updates align decoded completions inside the instance.
  • On standard clean-plate protocols, PredErase improves a frozen FLUX.2 backbone under instance-only masks.

Supplementary Material

The supplementary material formalizes the gated latent operators used by PredErase. It defines the frozen Fill and I-JEPA targets, separates editable and background subspaces, and describes the corresponding projection geometry.

  • The frozen Fill model defines a conditional law on the gated support, while I-JEPA provides a deterministic masked-token prediction and cached hole target.
  • Packed latent space is partitioned by the binary gate into editable and background subspaces.
  • Every latent has a unique gated decomposition, and guidance may change only the editable component.
  • The gated restoration map is geometrically an affine projector onto the cached latent plus the editable subspace.

A.2. Design rationale and analysis scope

The design rationale aligns I-JEPA with masked hole prediction while enlarging the editable support for outside-mask residuals. Formal results certify local operator behavior, not global removal quality or metric gains.

  • Design rationale: I-JEPA predicts hole representations from visible context, providing a context-conditioned completion target without clean plates.
  • Design rationale: Unlike CLIP and DINO alternatives, I-JEPA directly supplies a mask-indexed, context-only prediction for the missing region.
  • Analysis scope: The formal propositions are local operator checks and establish neither global convergence nor improved clean-plate metrics.
  • Editable support: The enlarged support makes additional latent coordinates available for revising residuals outside the object mask, but does not guarantee clean-plate quality.
  • Gated updates: Projected guidance preserves background coordinates, so every coordinate outside the gate remains unchanged.
  • Gated updates: Under an L-smoothness assumption and 0 < η ≤ 1/L, each guided micro-step locally decreases the alignment objective.

B.1. Implementation details

The implementation freezes FLUX.2 and I-JEPA, constructs packed latent gates from the expanded support, and uses fixed preprocessing, prompts, and evaluation settings. Hyperparameters are selected once rather than retuned per benchmark split.

  • Prompts and conditioning: The default implementation uses a shared shadow-aware erasure prompt and a generic prompt ablation while keeping other modules fixed.
  • Prefill: Source-prefill preserves visible context outside the editable support, whereas the ablation gray-fills that support before encoding.
  • Latent geometry: Mflux is downsampled to the VAE grid and packed into FLUX latent-token layout to construct the editable gate.
  • Defaults: PredErase freezes Fill and I-JEPA weights, uses no paired removal training, and caches the I-JEPA target once per image.
  • Hyperparameters: The reported operating point is selected by a held-out visual check and RemovalBench clean-plate CMMD/PSNR rather than per-benchmark retuning.
  • Compute and evaluation: Timing runs use a single NVIDIA A100-40GB GPU and exclude model loading and metric computation.

B.2. Benchmark splits

The evaluation spans two real clean-plate benchmarks under the OmniEraser protocol and one synthetic benchmark under the SmartEraser protocol.

  • RemovalBench contains 69 real 1024×1024 object–clean-plate pairs with instance-only masks under the OmniEraser protocol.It is the primary split for testing outside-mask cast shadows and contact shading.
  • RORD-Val is a larger clean-plate validation set with broader scene scales and content variation, evaluated using the OmniEraser metrics.
  • DEFACTO-Val evaluates synthetic 1024×1024 removals under the SmartEraser protocol using a separate metric suite.

B.3. Metrics and evaluation protocol

The evaluation uses protocol-aligned splits, masks, ground truth, and metric implementations, with full-image metrics designed to capture residual effects outside the instance mask.

  • Protocol alignment means sharing splits, masks, ground truth, and metric implementations with OmniEraser or SmartEraser, not retraining baselines on FLUX.2.
  • RemovalBench and RORD-Val use the OmniEraser layout, while DEFACTO-Val follows the SmartEraser baseline layout and public metric code.
  • PSNR measures full-image pixel fidelity, CMMD measures CLIP-space distributional discrepancy, and LPIPS measures feature-space perceptual distance.Residual shadows and lighting differences outside M_obj can lower full-image scores.

B.4. Ablation details

Ablations compare module removal, guidance priors, and stochastic variability while holding evaluation settings aligned across the reported splits.

  • Module ablation: Full PredErase qualitatively reconstructs support and background structure more closely than Pure FLUX.2 and the expanded-support text-only control.The w/o JEPA control reduces residuals but remains structurally incomplete.
  • Guidance prior comparison: The prior comparison keeps M_flux, prefill, prompting, schedule, projected locking, and five-seed aggregation fixed while changing only the feature objective.
  • Guidance prior comparison: Under a matched Full stack, JEPA guidance outperforms CLIP and DINOv2 guidance on RemovalBench, with FID 52.69 versus 107.6/110.0 and CMMD 0.108 versus 0.150/0.148.The visual comparison likewise shows cleaner support-plane reconstruction with JEPA.
  • Statistical evaluation: Full PredErase has RemovalBench split-mean standard deviations of 0.002 for LPIPS and 0.07 dB for PSNR across seeds.Tables report seed-averaged point estimates.
  • Statistical evaluation: Full PredErase versus native FLUX.2 yields median paired improvements of 0.009 LPIPS and 1.65 dB PSNR on RemovalBench, both with p≪0.05.All four matched ablation contrasts are significant on LPIPS and PSNR at α=0.05, including after Bonferroni correction.
  • Failure cases: Large targets can leave JEPA-guided Fill under-constrained, producing streaky or smeared silhouettes when visible context is thin.

C.1. Limitations

PredErase is scoped to benchmark-aligned upright scenes and remains vulnerable when geometry, visible context, or the training-free operating point falls outside that setting.

  • The contact-band construction targets cast shadows and contact shading on supporting surfaces, not reflections, floating objects, or arbitrary illumination effects.
  • Without orientation metadata, PredErase does not infer non-horizontal supports because its default construction assumes a downward image axis as the contact normal.
  • When the instance covers most of the frame, thin visible context can produce streaky or incomplete completions, and the method has no dedicated fallback.
  • Supervised baselines retain advantages on several reported axes because PredErase keeps Fill and I-JEPA frozen and adapts only through test-time latent updates.
  • The formal operator properties hold under stated smoothness and gating assumptions, while module and guidance-prior claims are empirical.
  • Proposed extensions include instance-shadow detectors, hierarchical synthesis for large targets, broader frozen-backbone evaluation, video removal, and richer lighting models.
Loading 2609.00956v1…