Source-linked AI summary
MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenes
Mingzhe Du, Thong Thanh Nguyen, Nguyen Tran Cong Duy, See-Kiong Ng, Luu Anh Tuan
TL;DR
Material replacement lacks a fair benchmark because valid edits are one-to-many and systems receive different conditioning signals. MatReplace introduces a reference-free, four-dimension benchmark with aligned instruction, mask, and reference-image tracks; strongest closed editors perform well when materials are named, while pixel-based grounding degrades across families.
Problem
Material replacement has no public benchmark isolating the task, while reference-based metrics penalize valid diversity and favor a reference generator's style.
Method
MatReplace evaluates local material correctness, lighting harmony, outside preservation, and inside structure across three conditioning tracks that vary one input signal at a time.
Results
Reference-image conditioning degrades every family, whereas leading closed instruction editors reach exemplar-level rendering and surpass the exemplar anchor under the primary aggregate.
Takeaways & Limitations
Named-material rendering is largely solved by the strongest closed editors on this distribution, but grounding materials from pixels remains open.
Takeaways & Limitations
The benchmark is fully synthetic and single-source, and transfer to real photographs is unverified.
Abstract
from arXiv · showhide
Material replacement is a common interior-design operation: changing the material of a selected surface while preserving its geometry, surroundings, and illumination. Despite its commercial relevance, no public benchmark isolates this task, and evaluating it is challenging. Reference-based metrics penalize valid outputs in this inherently one-to-many setting, favor the style of the reference generator, and cannot fairly compare editors that receive different forms of guidance. We introduce MatReplace, a reference-free benchmark that evaluates edits along four verifiable dimensions: local material correctness, global lighting harmony, outside preservation, and inside structure. It defines three tracks that vary one conditioning signal at a time: (A) instruction only, (B) instruction plus region mask, and (C) material reference image instead of instruction. Our results reveal a clear divide between naming and visually grounding materials. In Track A, leading closed-source editors achieve exemplar-level material rendering and surpass the exemplar anchor under our primary aggregate. In Track B, masks help only mask-compatible models with weak scene preservation, with task-paired, single-seed effects ranging from +0.137 to -0.090 across aligned model families. In Track C, reference-image conditioning degrades every family under both aggregates, by -0.031 to -0.508; in the worst cases, models repaint the reference image itself and perform worse than returning the input unchanged. Thus, named-material rendering is largely solved by the strongest closed editors on this distribution, but grounding materials from pixels remains an open challenge. Expert ratings validate our ranking (Kendall's tau = 0.68) and align with our aggregates more closely than GT-referenced or CLIP-based baselines.
1 Introduction
MatReplace isolates material replacement as a surgical interior edit and evaluates it without relying on a stored reference image. It also separates instruction, mask, and reference-image conditioning so model quality is not conflated with input signals.
- Material replacement changes one marked surface while preserving its geometry, the surrounding scene, and illumination.
- Existing instruction-editing benchmarks fold material replacement into broad categories and commonly score against one stored reference image.
- Reference-image scoring penalizes valid one-to-many outputs and can reward the rendering style of the reference generator.
- Tracks A, B, and C vary conditioning across instruction, instruction plus mask, and mask plus reference, respectively, with aligned model families.
- The benchmark evaluates local material correctness, lighting harmony, outside preservation, and inside structure as independently verifiable dimensions.
2 Related Work
Prior work spans instruction editing, reference-guided appearance transfer, material editing, and material understanding, but lacks a shared benchmark for interior material replacement. MatReplace combines these concerns in an auditable reference-free evaluation.
- Instruction-based editing and its benchmarks: Instruction-editing benchmarks primarily use GT-referenced distance and CLIP text alignment, inheriting reference-image biases.
- Reference-guided appearance transfer: Track C treats reference-guided appearance transfer as material appearance re-rendering under scene geometry and illumination rather than reference-image pasting.
- Material editing and design-domain data: Prior material-editing methods address control, exemplar transfer, light-aware transfer, or material-map extraction on separate small datasets.
- Material editing and design-domain data: No shared suite compares these material-editing approaches with general editors or isolates an interior editing operation.
- Material understanding and intrinsics: MatReplace connects material recognition and intrinsic-image analysis to editing through material correctness, appearance similarity, and shading invariance.
- Reference-free evaluation: Its reference-free protocol decomposes the editing contract into deterministic, auditable dimensions built from established components.
3 The MatReplace Benchmark
MatReplace defines a single-surface task, constructs a curated synthetic benchmark, and evaluates three conditioning tracks with released inputs and rejected tasks reserved for auditing. Its scope deliberately excludes composite and geometry-changing edits.
- Task definition: Each task contains a scene, binary surface mask, target-material reference, instruction, and 10-way material label; submissions are edited images at input resolution.
- Task definition: The contract requires target material inside the mask, unchanged content outside it, preserved interior geometry, and shading under scene illumination.
- Sources and curation: 2,017 candidate sessions were reduced to 1,421 frozen tasks through algorithmic validity checks and semantic auditing.
- Sources and curation: The semantic audit reported 70.8% mask-object alignment, 96.4% edit correctness, 83.5% outside preservation, 93.3% reference-edit consistency, and 99.7% watermark absence.
- What the rejects look like: Rejected tasks most often had masks that spilled or missed, followed by edits leaking outside the mask, geometry resculpting, exemplar-reference mismatch, or scene redraw.
- Three conditioning tracks: Tracks A, B, and C provide x+T, x+m+T, and x+m+r, with material named only by the image in Track C.
- Sources and curation: The 596 rejected sessions form a disjoint pool for auditing and are never used for evaluation.
- Scope and non-goals: The benchmark isolates single-surface replacement and excludes insertion, removal, geometry edits, multi-surface restyling, and global style transfer.
4 Reference-Free Evaluation Protocol
MatReplace evaluates material edits without scoring against stored outputs, using four auditable dimensions and an unweighted aggregate. Anchor and robustness checks test whether these dimensions distinguish targeted failures and preserve ranking quality.
- Scoring dimensions: The protocol computes material correctness, outside preservation, inside structure, and lighting harmony from task inputs and the submission, never a stored solution.A 7-pixel boundary band is excluded from outside comparisons.
- Scoring dimensions: Material correctness combines family-level material classification with DINOv2 appearance similarity to the material reference.Family grouping avoids penalizing distinctions that current vision encoders legitimately confuse.
- Scoring dimensions: Outside preservation uses masked LPIPS, while inside structure uses scale-and-shift-aligned monocular depth error.PSNR and SSIM are reported as preservation diagnostics.
- Scoring dimensions: Lighting harmony correlates low-pass luminance patterns inside the mask and penalizes gradient-energy seams outside its boundary.The luminance correlation is invariant to albedo scale, while the seam term targets halos and glow.
- Validation and robustness: 0.833: the exemplar anchor’s clean-subset aggregate, versus 0.829 across all tasks; identity fails material while paste is caught by structure and harmony.Identity scores 1.000/0.999/1.000 on preservation, structure, and harmony, but 0.242 on material; paste scores 0.673 material, 0.548 structure, and 0.697 harmony.
- Validation and robustness: τ = 0.91: family-only material scoring barely changes the ranking, while the gate has mild re-ranking at τ = 0.77 and preserves the Track-C collapse.The gate is reported as the primary aggregate alongside raw overall.
5 Experiments
Experiments compare 11 systems across 23 track entries using fixed, minimally templated protocols. Closed editors lead instruction-only editing, masks selectively help compatible pipelines, and reference conditioning causes severe failures for some entries.
- Systems and protocol: 11 systems produce 23 track entries, with six families completing all three tracks from one checkpoint or pipeline family.The evaluated set includes closed APIs and open cross-track editors such as Qwen-Image-Edit, OmniGen2, BAGEL-7B, and FLUX.1-Kontext.
- Systems and protocol: All open models use native or recommended resolution, default guidance, 50 denoising steps, fixed seeds, and identical minimal track prompt templates.Outputs are resized to input resolution before scoring; per-model prompt tuning is avoided to prevent input-privilege confounds.
- Track A: 0.760/0.739 vs. 0.729: the two closed editors lead Track A on both aggregates and clear the exemplar anchor under the gate.Nano-Banana-2-Lite ranks first across providers; GPT-Image-2’s result also reflects self-consistency with its generated exemplars.
- Track B: 0.680/0.678: Qwen and FLUX-Kontext inpainting lead Track B, while the closed-model order is preserved.The result supports selective benefit from masks for mask-specialized pipelines.
- Track C: 0.204 and 0.062: OmniGen2-ref and BAGEL-7B fall below the 0.236 do-nothing floor in Track C after every family drops.Reference-image conditioning can therefore underperform returning the input unchanged in these entries.
6 Analysis
The analysis shows that conditioning benefits depend on model compatibility, while reference images consistently worsen performance. The decomposed evaluation also exposes distinct failure modes and stratified difficulty.
- Conditioning effects: +0.137 to −0.090: masks help only models that are weak at preservation and can consume masks.FLUX-Kontext gains +0.137 under the gate, while OmniGen2 loses −0.090.
- Conditioning effects: Reference conditioning degrades every family, with failures ranging from material-identification collapse to repainting the reference instead of the scene.BAGEL’s role-binding failure collapses preservation and structure, while Qwen and OmniGen2 lose material-family accuracy.
- Leaderboard patterns: Closed editors lead all three tracks, while every Track C family loses ground and two entries fall below the gated do-nothing floor.Table 3 orders systems by the gated aggregate and marks entries below the 0.236 floor.
- Leaderboard patterns: Overall ties conceal opposite errors: one model may render the material while damaging geometry, whereas another preserves geometry but misses the material.The same pattern appears for OmniGen2 versus InstructPix2Pix on Track A and SDXL versus SD3.5 on Track B.
- Leaderboard patterns: The closed–open gap concentrates in preservation and structure, not material recognition.Track A preservation is 0.95–0.96 for closed versus 0.77–0.90 for open models, while structure is 0.82–0.85 versus 0.56–0.71.
- Stratified difficulty: Large masks reduce every model’s score by 0.05–0.10, while cluttered other surfaces are hardest and flat countertops easiest.GPT-Image-2 scores 0.755 on large masks versus 0.840 on medium masks; cluttered other surfaces score 0.698 and countertops 0.868.
7 Human Expert Calibration
Human calibration broadly supports MatReplace’s leaderboard and shows stronger agreement than established reference-based and CLIP-based metrics. Remaining divergences reveal a trade-off between edited-region quality and outside-region fidelity, alongside limited pairwise evidence.
- Protocol: The calibration used 425 scoring judgments from two expert raters, with ties treated as half-wins in Bradley–Terry fitting.Judgments covered 422 unique tasks, while 25 known-winner pairings served as attention checks and 24 passed.
- Human agreement: Kendall τ = 0.68 and Spearman ρ = 0.87: human Bradley–Terry strengths correlate strongly with the leaderboard.On Track A, Kendall τ rises from 0.62 to 0.87 after excluding one outlier, reaching 1.00 under the gate.
- Human agreement: The 28% tie rate falls on pairings that the leaderboard also refuses to separate.Human near-ties include GPT-Image-2 Track A versus B and OmniGen2 versus InstructPix2Pix.
- Metric comparison: Every MatReplace aggregate correlates more strongly with raters than LPIPS, CLIPScore, and CLIP-I baselines.The advantage is significant over LPIPS and CLIPScore and marginal over CLIP-I, with Δτ = +0.14 and CI [−0.01, 0.28].
- Metric comparison: The GT-referenced metric scores the exemplar perfectly by construction, although raters place it fifth among 20 systems.This demonstrates a mismatch between reference proximity and human ranking.
- Divergences: Raters emphasize material identity and shading realism more than outside-region fidelity, producing divergences from the metric ranking.BAGEL ranks fourth by the metric but last with raters, while FLUX-Kontext’s measured mask gain is not visible to them.
8 Limitations
The benchmark’s conclusions are bounded by synthetic, single-source data and several documented selection, curation, taxonomy, and scoring constraints.
- Transfer to real photographs remains unverified because all benchmark images are synthetic and sourced from GPT-Image-2.Admission also conditions on the generator drawing an aligned mask, introducing selection bias.
- Human validation uses two raters with non-overlapping judgments, while anchor checks substitute for an inter-rater statistic.
- Mask quality caps yield at 70.8%, limiting attainable performance.
- Furniture accounts for 825 of 1,421 tasks, so wall and countertop conclusions rely on smaller strata.
- Raw overall rewards favor conservatism, with identity scoring 0.810; the gate view mitigates this effect in Table 2.
9 Reproducibility
MatReplace releases the benchmark assets and evaluation materials, while pinning model, sampler, provider, and library versions for reproducibility.
- The release includes task specifications, the benchmark harness, roughly 33k model generations with per-task scores, and the full human study.
- Closed models are pinned by provider ID and date, open models by checkpoint hash and sampler, and libraries by a lockfile.
10 Conclusion
MatReplace provides reference-free material-replacement evaluation across aligned conditioning tracks and finds that named-material rendering is strong while pixel-based grounding remains unresolved.
- MatReplace isolates material replacement, scores it without a ground-truth reference, and aligns model families across three conditioning tracks.
- Closed editors match the data-generating process without spatial guidance, while mask benefits disappear with model strength and can reverse.
- Reference-driven specification degrades every family, indicating that grounding materials from pixels remains an open challenge.
- A two-rater expert study reproduces the ranking and prefers the proposed aggregates to GT-referenced and CLIP baselines.
- Aligned tracks resist gaming by imitation.