PaperlayerView exact revision on arXiv

Source-linked AI summary

SREdit: Scene Response Guided Appearance Editing for Projection Mapping

Yuqiao Xia, Yuchen Deng, Ziheng Chang, Bingyao Huang

arXiv:2610.32792v1cs.CVcs.MM

TL;DR

Projection-based appearance editing needs a desired image, but image editors lack scene geometry and response, leaving misalignment, shading, or out-of-range changes that compensation cannot correct. SREdit uses reconstructed-scene response to guide early sampling, improving projected appearance and reproduction fidelity across two editors with little overhead.

  • Problem

    Image editors lack accurate scene geometry and response, so edits may contain misalignment, inconsistent shading, or out-of-range changes that downstream compensation cannot correct.

    Source: §1 · pp. 1–2

  • Method

    SREdit estimates a scene response map and uses it with source appearance during early sampling to adapt edits while preserving registration with the physical object.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    72.4% of user-study comparisons preferred SREdit, and experiments across two pretrained image editors and six physical setups showed consistent improvements in projected appearance quality and reproduction fidelity.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Scene response can guide pretrained image editing before projector compensation with less than one second of added generation time and negligible VRAM overhead.

    Source: §6 · p. 10

    A single parameter pair cannot provide the preferred balance between physical reproduction and edit richness across all physical setups.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

Projection-based appearance editing changes selected visual attributes of a physical object through projection mapping (PM). To display a desired appearance, PM typically uses projector compensation, which computes a projector input that counteracts the geometric and photometric distortions of the scene. Compensation can only approximate this appearance within the available projection range, and it cannot revise the appearance itself. Pretrained LLM-based image editors provide a direct way to produce the desired appearance from a text prompt or reference image; projector compensation can then attempt to reproduce the edited result through projection. However, these editors do not have an accurate measure of the scene geometry or scene response. Therefore, their outputs may contain geometric misalignment, inconsistent shading, or changes outside the available projection range, which downstream projector compensation cannot correct. To address this problem, we introduce SREdit, an efficient response-guided framework for projection-based appearance editing. SREdit estimates a scene response map from the reconstructed scene that reflects how strongly projected illumination can affect different surface regions. SREdit uses the source appearance to constrain the early sampling process and preserve registration with the fixed physical object, while the scene response map adapts the amount of editing across surface regions. Experiments across two pretrained image editors show that SREdit improves projected appearance quality and reproduction fidelity. It adds less than one second per image with negligible VRAM overhead. Participants also consistently preferred the results produced with SREdit in our user study.

1 INTRODUCTION

Projection-based appearance editing must account for scene geometry and response because projector compensation cannot revise an unsuitable edited target. SREdit guides pretrained image editing with scene response and source appearance, improving projected results across multiple editors and setups.

1 INTRODUCTION

  • Projector compensation can only approximate a desired appearance within the available projection range and cannot revise the target itself.Image editors without accurate scene geometry or response may produce misalignment, inconsistent shading, or changes outside that range.Source: §1 · pp. 1–2
  • SREdit estimates a scene response map and uses source appearance during early sampling to preserve registration while adapting edits to regional response.Lower-response regions are preserved more strongly, while higher-response regions allow greater change.Source: §1 · p. 2
  • SREdit incorporates scene response into pretrained image editing while separating appearance editing from downstream projector compensation.It requires no additional training or editor control module and does not backpropagate through the editor during generation.Source: §1 · p. 2
  • Across two pretrained image editors and six physical setups, SREdit improves projected appearance quality and reproduction fidelity with little additional computation and memory.Participants also preferred SREdit's results in the user study.Source: §1 · p. 2

2 RELATED WORK

Prior projection-mapping work addresses compensation, relighting, and generated projection content, but generally assumes the desired appearance is already specified or externally supplied. Image-editing controls preserve image content or impose user-defined regional changes, whereas SREdit uses scene response to guide edits for a fixed physical setup.

2.1 Projection mapping and appearance editing

  • Projection mapping systems support varied physical appearance effects, but projected content or desired effects are usually pre-authored or produced by application-specific pipelines.Compensation solves the inverse problem of finding projector input for a prescribed camera-view appearance; ProCams models predict camera images from projector input and scene.Source: §2.1 · p. 2
  • Generative projection methods reduce manual content preparation, yet prior work mainly targets content generation, surface adaptation, or projector-input optimization.SREdit instead starts from the observed scene and edits its appearance under the current setup while preserving registration with the physical object.Source: §2.1 · pp. 2–3

2.2 Image editing and conditional control

  • Image-editing methods preserve source content using image, text, mask, or model-feature controls, but do not directly adapt edits to spatial scene response.Editing reconstructed NeRF or Gaussian scenes can maintain view consistency, but requires updating and optimizing the reconstruction for each edit.Source: §2.2 · p. 3
  • Prior conditional-control methods use trained modules, inference-time gradients, or user-specified regional edit strengths; SREdit instead applies response-guided sampling across two editors without those additions.Its response map represents the relative effect of direct projector illumination at each surface location.Source: §2.2 · p. 3

3 METHOD

SREdit separates camera-space appearance editing from downstream projector compensation and uses the reconstructed scene to guide the editor. A spatial response map and source anchoring constrain early sampling, while later steps remain free to refine the edit.

3.1 Problem formulation

  • The task first edits the observed object into a camera-space target, then separately refines a projector input to reproduce that target in the fixed physical scene.The editing condition may include a text prompt and, when supported, a reference image.Source: §3.1 · p. 3

3.2 Scene response map

  • The scene response map averages channelwise direct-response factors and normalizes the scalar response over the reconstructed-object support, with larger values indicating stronger local response.The map is applied to the latent update, with values outside the reconstructed-object support set to zero.Source: §3.2 · pp. 3–4

3.3 Response-guided sampling

  • During early sampling, SREdit corrects the post-Euler latent toward a source anchor at the same noise level, applying stronger correction in lower-response regions before leaving later steps free for edit refinement.The correction is restricted to early denoising progress so repeated anchoring does not suppress the requested edit.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

The experiments compare SREdit with projection-mapping baselines and Native image-editor pipelines using physical projection, then evaluate reference-guided editing and ablations. Across these tests, SREdit improves projected results and receives higher user preference, with small added runtime and negligible VRAM overhead.

4.1.1 Physical setup

  • Experiments use fixed projector-camera poses and settings; SREdit is compared with Nepmap and LAPIG and with Native versions using the same editor and compensation.The renderer remains frozen during editing, and compensation optimizes only the projector input; the reported SREdit settings were selected on a separate setup and held fixed in the main comparison and scene-map ablation.Source: §4.1.1 · p. 5, §4.1.2 · p. 5, §4 · p. 5

4.1.3 Evaluation metrics

  • Evaluation separates reproduction fidelity from agreement with the requested appearance and also reports generation time and peak VRAM.PSNR, SSIM, LPIPS, and ∆E compare each method’s intended appearance with its captured result; CLIP and BLIP-ITM measure agreement with the requested appearance.Source: §4.1.3 · p. 5

4.2 Comparison with baselines

  • Nepmap’s high SSIM and PSNR accompany smooth, low-frequency edits, while LAPIG misaligns on 3D objects and Native outputs can clip changes or misalign.Nepmap’s SSIM is 0.5962 and PSNR is 19.1512 dB; Native-LongCat’s registration mismatch persists after projection, unlike the corresponding SREdit-LongCat results shown.Source: §4.2 · p. 6, Figure 5 · p. 6
  • SREdit improves all six quality metrics over the corresponding Native variants for both image editors.For LongCat, PSNR increases by 5.9392 dB, LPIPS decreases by 0.1053, and CLIP rises by 1.88; for Qwen, PSNR increases by 4.8027 dB, ∆E decreases by 5.6617, and BLIP-ITM rises by 6.16. Within each same-editor comparison, runtime changes by less than one second with negligible VRAM overhead.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.2.1 User study

  • Across 720 judgments, SREdit received 72.4% overall preference, and 96.5% of its results were judged free of visible misalignment versus 80.4% for Native.The preference remained significantly above chance after aggregating repeated judgments by participant (two-sided sign test with Holm correction, p < 0.001).Source: §4.2.1 · p. 7, Table 2 · p. 7

4.3 Reference-guided editing

  • In 12 reference-guided Qwen cases, SREdit retained more lower-response surface color and shading and improved all four reproduction metrics.PSNR increased from 18.5257 dB to 21.7335 dB; captured results also showed more consistent registration and fewer color and shading artifacts.Source: §4.3 · pp. 7–8, Table 3 · p. 7, Figure 6 · p. 8

4.4 Ablation studies

  • The scene response map improved CLIP and BLIP-ITM over Uniform, which favored conservative edits and higher reproduction scores.In the shown captures, Uniform left source-colored unedited patches, while the response map produced fewer such patches.Source: Table 4 · p. 7, §4.4 · p. 9, Figure 8 · p. 9
  • Control strength S sets correction magnitude, while r distributes its normalized temporal weight; stronger S improves reproduction metrics, but large r with strong S can suppress detail.The four reproduction metrics improve monotonically with S at every r and are best at r = 0.5, S = 3.0, which also gives the grid’s lowest CLIP score; the authors recommend r around 0.2 and adjusting S.Source: Figure 7 · p. 8, §4.4 · p. 9

5 DISCUSSION

The discussion describes a trade-off between physical reproduction and edit richness: the preferred balance varies by setup and user priorities. It also identifies computational and soft-constraint limits for SREdit.

5.1 Limitations and future work

  • Increasing control tends to preserve registration and observed surface appearance but may weaken requested texture; weaker or shorter control may make edits harder to reproduce through projection.A single parameter pair cannot provide the preferred balance across physical setups.Source: §5.1 · p. 9
  • The pretrained image editors still dominate the pipeline’s runtime and memory requirements, despite SREdit avoiding editor training, added control modules, and generation-time backpropagation.The authors propose examining response-guided sampling with lighter image editors.Source: §5.1 · p. 9
  • Response-guided sampling is a soft constraint, so extreme surface colors, large appearance differences, or overly strong control may suppress the requested texture completely.The authors propose an efficient hard constraint that avoids generation-time gradients or an additional model.Source: §5.1 · p. 9
  • Under one fixed setting, registration remains, but requested wood texture is weakened or spatially uneven in the intended and camera-captured appearances.The example uses r = 0.2 and S = 2.5.Source: Figure 9 · p. 9, §5.1 · p. 9

6 CONCLUSION

SREdit incorporates reconstructed-scene response into pretrained image editing for projection mapping while leaving downstream projector compensation separate. Across two editors and six physical setups, it improves projected appearance quality and reproduction fidelity with little added computation and memory.

6 CONCLUSION

  • SREdit estimates a scene response map from the reconstructed scene to represent how strongly projected illumination affects different surface regions.During early sampling, source appearance helps preserve registration while scene response adapts the amount of editing.Source: §6 · p. 9
  • SREdit incorporates scene response into pretrained image editing while separating appearance editing from downstream projector compensation.It requires no additional training or control module, does not backpropagate through the editor during generation, and leaves compensation unchanged.Source: §6 · pp. 9–10
  • 72.4% of user-study comparisons preferred SREdit; across two pretrained editors and six physical setups, it consistently improved projected appearance quality and reproduction fidelity.Response-guided sampling adds less than one second to image generation with negligible VRAM overhead.Source: §6 · p. 10

A INTRODUCTION

The supplementary material adds qualitative comparisons and parameter ablations across six physical setups and diverse prompts, including generated appearances and camera-captured projection results.

A INTRODUCTION

  • The supplementary material presents qualitative comparisons and parameter ablations across six physical setups and diverse text prompts.Source: A INTRODUCTION · p. 12
  • Figures S1–S20 include both generated appearances and camera-captured projection results for evaluating SREdit under real projection.Source: A INTRODUCTION · p. 12
  • The supplement provides the complete user-study protocol and all 36 comparison trials.Source: A INTRODUCTION · p. 12

B EXPERIMENTAL CONFIGURATION

The experiments use two pretrained image editors across six physical setups, with physically projected and camera-captured outcomes. The configuration describes the scene-response map, parameter sweep, and evaluation measures.

B EXPERIMENTAL CONFIGURATION

  • Qwen and LongCat use 40 and 50 sampling steps, respectively; their main settings are r = 0.2 with S = 2.5 and S = 3.0, respectively.The parameter sweep tests four r values and six S values.Source: B EXPERIMENTAL CONFIGURATION · p. 12
  • Evaluation uses six physical setups and 18 text prompts, measuring PSNR, SSIM, LPIPS, and ∆E within the object region, plus CLIP and BLIP-ITM agreement with target descriptions.Source: B EXPERIMENTAL CONFIGURATION · p. 12
  • All projection results are physically projected and camera-captured; Native and SREdit versions share the source, request, sampling-step count, and compensation procedure for each editor.Qwen and LongCat receive the same request per case; Nepmap and LAPIG requests are adapted to their input formats while retaining the target appearance.Source: B EXPERIMENTAL CONFIGURATION · p. 12
  • The scene response map averages channelwise direct-response factors and normalizes them over the reconstructed-object support; it is computed once per setup and held fixed across requests.The uniform-map ablation assigns the mean response to every image location while keeping the editor, sampling schedule, control parameters, and compensation procedure unchanged.Source: B EXPERIMENTAL CONFIGURATION · p. 12

C USER STUDY CONFIGURATION

The study used anonymous, within-subject comparisons of camera-captured SREdit and Native results, with randomized A/B placement across 36 trials.

C USER STUDY CONFIGURATION

  • Twenty participants aged 18–24 took part; nine reported experience in image processing, computer graphics, or computer vision.The group included 10 female and 10 male participants.Source: C USER STUDY CONFIGURATION · p. 12
  • Each participant evaluated 36 pairs—18 Qwen and 18 LongCat comparisons—with A/B positions randomized and recorded for every trial.The results were displayed anonymously, and participation was voluntary with informed consent.Source: C USER STUDY CONFIGURATION · p. 12
  • Participants judged text alignment, artifacts, lighting and surface realism, visible misalignment, and overall preference.Comparative responses used a four-level forced-choice scale; visible misalignment was judged separately as present or absent.Source: C USER STUDY CONFIGURATION · p. 12

D REPRESENTATIVE MAIN AND ABLATION RESULTS

This appendix presents representative generated appearances and physical projections, parameter ablations, and the prompts used for the comparisons.

D REPRESENTATIVE MAIN AND ABLATION RESULTS

E PROMPTS FOR REPRESENTATIVE COMPARISONS

  • Table S1 records prompts for Figure 1, the two Figure 5 comparisons, and supplementary Cases 1–10; the reported wording, capitalization, and punctuation match each method’s prompts.The table includes the full prompts corresponding to abbreviated labels in Figure 1.Source: E PROMPTS FOR REPRESENTATIVE COMPARISONS · p. 23, Table S1 · p. 23

F USER STUDY QUESTIONNAIRE

The questionnaire asked participants to compare anonymous projected results against the source object and editing instruction. Its criteria covered appearance matching, artifacts, realism, alignment, and overall preference.

F.1 Instructions

  • Participants were instructed to assess each projected result in relation to the editing request and source object, with anonymous methods and no correct answer.The study information was reviewed and informed consent was obtained before participation.Source: §F.1 · p. 24

F.2 Interface and questions

  • Each trial displayed the source object, instruction, and two anonymous results labeled A and B, with SREdit and Native assignments randomized.The A/B assignment was recorded for each trial.Source: §F.2 · p. 24
  • The questionnaire compared text alignment, visible artifacts, lighting and surface realism, visible misalignment, and overall preference.The specific questions asked which result better matched the request, had fewer artifacts, looked more realistic, and was preferable overall, and whether either showed misalignment.Source: §F.2 · p. 24
  • Alignment, artifact, realism, and preference judgments used a four-level forced-choice scale; misalignment was marked present or absent for each result.Responses were mapped back to the method shown before aggregation, and practice trials were excluded from analysis.Source: §F.2 · p. 24

F.3 All main trials

  • The 36 formal-trial panels preserve the study’s A/B order and show the source object alongside both anonymous results, with the instruction in each caption.The appendix reproduces the trials in presentation order.Source: §F.3 · p. 25
Loading 2610.32792v1…