PaperlayerView exact revision on arXiv

Source-linked AI summary

Texture Interpolation as Texton Matching

Ido Madar, Gal Oren, Shai Avidan

arXiv:2610.18655v1cs.CVcs.GRcs.LG

TL;DR

Texture interpolation often pairs pixels or features by spatial location, although corresponding locations in unrelated textures may have unrelated appearances. This paper changes that pairing by matching local appearance vectors inside a fixed encoder-decoder. Appearance-aware matching improves midpoint statistics, with soft and one-to-one rules favoring different transition properties.

  • Problem

    For unrelated textures, spatially corresponding elements may have no meaningful relationship, leaving the choice of which elements to mix an open interpolation question.

    Source: §1 · p. 1

  • Method

    The paper replaces positional pairing with appearance-aware texton matching while keeping the encoder and decoder fixed.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Results

    Appearance-aware matching improves midpoint statistics in CNT and lowers KID when substituted into Texture Mixer without retraining; soft matching favors ensemble statistics, while one-to-one matching favors balance.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Texture interpolation is a matching problem before it is a blending problem, and the choice of partners affects the transition’s midpoint statistics and balance.

    Source: §6 · p. 9

    No single metric establishes whether both endpoints are perceptually represented, and the evaluation includes no human preference study.

    Source: §5 · p. 9

Abstract

from arXiv · show

Texture interpolation asks what should lie between two textures. Standard methods typically blend pixels or features at corresponding spatial locations, implicitly assuming that co-located elements should be mixed. For unrelated textures, however, this correspondence is arbitrary: a crack at one location has no particular relationship to a fiber at the same location in another image. We study whether this hidden pairing choice is itself a major determinant of interpolation quality. We represent textures as collections of local appearance vectors, or textons, and replace positional pairing with appearance-aware matching while keeping the encoder, representation, and decoder fixed. This isolates the effect of correspondence from changes in model capacity or training. Across texture pairs, appearance-aware matching consistently improves the distributional quality of interpolated midpoints over positional feature blending. Soft matching produces the strongest midpoint statistics, whereas one-to-one assignments yield more balanced transitions and closer agreement with the target endpoint. The same intervention also improves a second pretrained texture interpolator without retraining. These results identify correspondence as a first-class design choice in texture interpolation: meaningful transitions depend not only on how strongly two textures are mixed, but also on which local appearances are mixed with one another.

1 INTRODUCTION

The paper treats texture interpolation as a choice of which local appearances to pair, since spatially corresponding elements in unrelated textures need not match. It tests appearance-aware pairing inside a fixed encoder and decoder and reports improved midpoint statistics.

1 INTRODUCTION

  • Changing the partner map inside a fixed encoder and decoder isolates correspondence from changes to the representation or renderer.The method associates local appearances rather than pairing elements by spatial index.Source: §1 · p. 1
  • Appearance-aware matching improves midpoint statistics over positional blending in CNT and lowers KID when substituted into Texture Mixer without retraining.The paper also reports that one-to-one matching raises KID in Texture Mixer.Source: Figure 1 · p. 2, §1 · p. 2
  • Soft matching produces the strongest midpoint statistics, while one-to-one matching improves midpoint balance and target agreement.The paper evaluates matching rules across midpoint statistics, endpoint balance, and path behavior.Source: Figure 1 · p. 2, §1 · p. 3
  • GPT-Image-1.5 scores better at midpoints, but does not provide a controllable path and costs more computation.The prompted comparator synthesizes a single intermediate from the source images and a description.Source: §1 · p. 2, Figure 1 · p. 2
  • The study examines barycentric contraction, target-end behavior, and when positional correspondence remains useful.It also reports that the matching intervention transfers to a second pretrained decoder.Source: §1 · p. 3

2 RELATED WORK

Prior texture methods represent or mix local texture elements in several ways, including feature-statistic matching, patch correspondence, and learned latent spaces. This paper’s distinction is to vary appearance association inside an existing encoder-decoder.

2 RELATED WORK

  • Textons connect local texture elements and their statistics to later parametric and deep feature-statistic models.CNT represents them explicitly as localized Gaussian primitives with appearance vectors.Source: §2 · p. 3
  • Texture-transition approaches include patch correspondence, image melding, stationary Gaussian models, neural statistics, learned latent spaces, and priority maps.Texture Mixer learns local and global latent spaces with interpolation-specific objectives.Source: §2 · p. 3
  • Transport has been used for texture synthesis or mixing, while Risser (2020) moves autoencoder features toward optimal-transport targets before interpolation.This paper instead isolates the pairing rule inside a fixed encoder and decoder.Source: §2 · p. 3
  • Unlike CNT’s primitive-level matching and movement, this study couples dense per-cell splat-field vectors while retaining the source layout.The matching object and the hardness of the matching rule are thus treated separately.Source: §2 · p. 3

3 INTERPOLATION WITH AN EXPLICIT PARTNER MAP

The method defines an explicit, source-indexed path between textures and varies the map that assigns target appearances to source textons. It constructs dense appearance fields, compares several partner rules, and analyzes how barycentric projection affects the path and its endpoints.

3 INTERPOLATION WITH AN EXPLICIT PARTNER MAP

  • The explicit parametric route moves along a representation under coefficient t, unlike a generative model that describes and samples a midpoint.The paper focuses on the parametric route and compares it with the generative alternative separately.Source: §3 · p. 3

3.1 FROM GAUSSIAN PRIMITIVES TO A DENSE APPEARANCE FIELD

  • CNT’s finest splatted field supplies local appearance vectors, or textons; per-cell matching weights appearance by spatial coverage while retaining source layout and decoder conditioning.A field cell can mix features from several Gaussian primitives, and transport costs omit coordinates.Source: §3.1 · p. 4

3.2 ONE SOURCE-INDEXED PATH, SEVERAL CHOICES OF PARTNER

  • The source-indexed map T assigns each source texton a target appearance, while the encoder and decoder stay fixed across positional and appearance-based partner choices.Gaussian affine maps are also allowed, even when their outputs are not convex combinations of observed target samples.Source: §3.2 · p. 4

3.3 BALANCED ENTROPIC TRANSPORT

  • Balanced entropic transport uses squared feature distance and uniform sampled measures; Sinkhorn solves the plan, whose barycentric partners are reused for every t.The resulting path is called soft texton matching; exact OT instead uses an unregularized assignment.Source: §3.3 · p. 4

3.4 WHAT THE MATCHING CONTROLS

  • Controls include positional pairing, random permutation, target mean, nearest neighbor, Gaussian W2, entropic transport, and exact one-to-one OT under shared encoded inputs and decoder.Random permutation is an appearance-blind negative control with exact sample coverage; the primitive-linear baseline is a distinct representation-level reference.Source: §3.4 · p. 4

3.5 PROJECTION, ENDPOINTS, AND DIRECTION

  • 38% of target variance is retained by the measured default diffuse plan; projection’s variance effect scales as t^2, and t = 1 need not reconstruct the target.Exact OT removes conditional variance on its sampled cloud, but extension and source conditioning still affect the decoded endpoint; the two transition directions can differ. The exact quadratic-cost joint displacement path is a W2 geodesic, while perceptual linearity of the decoded path remains empirical.Source: §3.5 · p. 5

4 EXPERIMENTS

Across controlled experiments, changing texton pairing within fixed models changes midpoint distributional quality; soft matching improves the primary CNT comparison and the intervention also improves Texture Mixer without retraining.

4.1 PROTOCOL AND PRIMARY COMPARISON

  • The primary comparison uses 1,000 stratified cross-category DTD pairs, with KID comparisons against 2,819 DTD references.FID is described as the community standard but biased upward at small sample sizes; comparisons therefore rest on KID.Source: §4.1 · p. 5
  • Soft texton matching improves CNT FID/KID from 108.0/0.030 for field-linear to 94.7/0.021; Gaussian alignment reaches 97.9/0.024.The encoder, representation, and decoder are held fixed while pairing changes.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 WHICH PROPERTIES OF THE PARTNER MAP MATTER?

  • Soft OT lowers KID 38% versus positional pairing, while exact OT shows no statistically resolved gain; random permutation has about four times either entropic plan’s KID.One-to-one assignment gives better balance and a smaller target gap than soft variants: 0.043 versus 0.072 balance error and 0.81 versus 0.87 target gap.Source: §4.2 · pp. 6–7, Table 2 · p. 7
  • With cyclic shifts of the same texture, positional blending’s deviation rises from 0 at 0 px to 23.9 at 64 px, while matching rules stay between 18.2 and 22.4.The regimes cross between 16 and 64 px; the authors recommend positional correspondence when reliable and texton matching otherwise.Source: §4.2 · p. 7

4.3 DOES THE EFFECT EXTEND BEYOND CNT?

  • In Texture Mixer, replacing only local-latent positional pairing lowers KID without retraining; one-to-one matching instead raises KID and gives the best balance.Target-mean matching lowers KID by 0.0031 (26%) and soft matching by 0.0028 on 1,000 pairs; both improve per-image realism.Source: §4.3 · p. 7

4.4 HOW DOES A PROMPTED MIDPOINT COMPARE?

  • Directed GPT prompting achieves KID 0.0051 versus Texture Mixer’s 0.0119 and soft matching’s 0.0190, but prompts each intermediate frame independently.Texton matching lowers KID against field-linear by 0.0114 on the same 120 pairs.Source: §4.4 · pp. 7–8, Table 3 · p. 8

4.5 WHAT IS PAID ONCE, AND WHAT IS PAID PER FRAME?

  • Encoding and matching are paid once per pair, each further coefficient needs one renderer pass, and texton matching needs no interpolation-specific training or fixed output size.At 128 px, Texture Mixer’s generator takes 0.003 s per t versus 0.02 s for CNT’s decoder.Source: §4.5 · p. 8, §4.4 · p. 8

4.6 APPLICATIONS

  • Applications include anchor-conditioned blends of several textures and bidirectional appearance transfer, whose outputs retain the conditioning texture’s arrangement.The anchor-based mixture is anchor-dependent and is not an exact multi-distribution barycenter.Source: §4.6 · pp. 8–9, Figure 5 · p. 9, Figure 4 · p. 9

5 DISCUSSION AND LIMITATIONS

The discussion frames interpolation as a directed transition under retained source conditioning and identifies limits in endpoint preservation, texture statistics, and evaluation.

5 DISCUSSION AND LIMITATIONS

  • Retaining source layout does not guarantee pixelwise preservation, and interpolating a texture with itself does not return it unchanged.The task is a directed transition under retained source conditioning.Source: §5 · p. 9
  • Diffuse barycentric maps do not reproduce the target distribution exactly, while local marginals omit phase and long-range organization, making periodic textures difficult.CNT reconstruction error also bounds the outputs.Source: §5 · p. 9
  • No single metric establishes perceptual representation of both endpoints; endpoint balance can reward superposition, and no human preference study is included.The generative comparator uses single outputs from fixed prompts.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents texture interpolation as a matching problem: which local appearances are paired matters alongside how strongly textures are mixed.

6 CONCLUSION

  • Texture interpolation is framed as a matching problem before a blending problem.The pairing decision is made inside a fixed encoder and decoder.Source: §6 · p. 9
  • Changing which textons combine improves interpolated midpoints in CNT and in a trained texture interpolator.The intervention is applied without changing the encoder and decoder.Source: §6 · p. 9
  • Soft matching favors ensemble statistics, whereas one-to-one matching favors balance.Source: §6 · p. 9

AI USE STATEMENT

The authors report AI assistance across manuscript and research activities while distinguishing that assistance from the experimental results and image assets.

AI USE STATEMENT

  • Generative AI tools assisted with manuscript organization and drafting, literature lookup, analysis, mathematical exposition, figure layout, and build and validation scripts.They also assisted with research ideation and experiment execution, including evaluation code.Source: AI USE STATEMENT · p. 10
  • GPT-Image-1.5 outputs are explicitly evaluated as a generative baseline, while the experimental results and image assets are the authors’ own.Editorial assistance did not generate replacement measurements.Source: AI USE STATEMENT · p. 10
  • Every number in the main-text tables was recomputed from saved outputs using the released scoring code.The authors state that they are responsible for the submitted content.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement points to an appendix and package containing specifications, per-pair records, prompts, and materials for reproducing the pipeline.

REPRODUCIBILITY STATEMENT

  • The appendix specifies the transport objective, numerical settings, Gaussian map, projection identities, and evaluation protocols.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The package includes per-pair scores, seeds, solver diagnostics, and pair manifests for several ablations and studies, plus the reference image list.Older evaluations retain aggregate values only.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The repository contains prompts and a call ledger, code, cached features, installation scripts, and two Colab notebooks for end-to-end pipeline reproduction.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

The paper reports no human-subject experiment and notes that texture edits may be mistaken for captured imagery. Its evaluation protocol specifies benchmark data, baselines, metrics, and implementation conditions, with some reference scores sensitive to sample count and protocol.

ETHICS STATEMENT

  • Texture editing can alter material appearance in ways viewers may mistake for captured imagery; identify edited outputs as synthetic when that distinction matters.Source: ETHICS STATEMENT · p. 10

A PROTOCOL AND BASELINE DEFINITIONS

  • The primary benchmark uses 1,000 stratified cross-category DTD pairs and compares midpoint sets with a 2,819-image reference pool using FID and KID.Images are encoded after upsampling to 256 × 256, and outputs are resized to 128 px before feature extraction; absolute FID and KID vary with sample count and reference protocol.Source: A PROTOCOL AND BASELINE DEFINITIONS · p. 13
  • The protocol distinguishes field-linear, primitive-linear, and RGB cross-fade baselines; primitive-linear is not CNT’s published Gaussian-matching interpolation.Source: A PROTOCOL AND BASELINE DEFINITIONS · p. 13
  • Texture Mixer uses its released earth-texture checkpoint outside its specific training domain, while Vacher variants run 30 rather than the published 150 iterations.GPT-Image-1.5 is evaluated through its image-editing interface; timing is remeasured under one protocol across methods.Source: A PROTOCOL AND BASELINE DEFINITIONS · p. 13
  • Manifold realism measures proximity to reference-image neighborhoods without conditioning on the selected texture pair, while DINO balance measures endpoint-distance-ratio displacement from one half.Realism medians use five reference-manifold draws; the reported observer-experiment resolution is not a confidence interval for FID, KID, or fidelity.Source: A PROTOCOL AND BASELINE DEFINITIONS · p. 13

B OPERATOR DERIVATIONS

The derivations distinguish joint transport from its deterministic barycentric projection: projecting to a conditional mean removes conditional variance. They also clarify what quadratic-cost matching rewards and how Gaussian and one-to-one variants differ in scope and behavior.

B.1 JOINT TRANSPORT AND DETERMINISTIC PROJECTION

  • Cov(Y) = Cov(T(X)) + E[Cov(Y | X)], so the barycentric map’s covariance is no greater than the target covariance, with equality only when conditional variance vanishes in every direction.Replacing the conditional distribution by its mean before decoding removes covariance t^2E[Cov(Y | X)] along the interpolated path.Source: §B.1 · p. 14

B.2 WHAT QUADRATIC COST REWARDS

  • Quadratic transport maximizes feature cross-alignment, while entropic transport trades alignment against plan entropy; random permutation preserves marginals without optimizing alignment.Feature alignment and sample coverage are separate controls, and perceptual quality need not follow feature alignment.Source: §B.2 · p. 14

B.3 GAUSSIAN TRANSPORT

  • Gaussian transport is a moment-alignment baseline using regularized Gaussian models, but CNT field vectors need not be Gaussian and decoding retains source conditioning.Covariance regularization can affect the affine map because field vectors mix a much smaller number of primitive features.Source: §B.3 · p. 14

B.4 EXACT OT (ONE-TO-ONE)

  • Exact OT is a permutation on the 1,024 sampled vectors, but nearest-neighbor extension can repeat targets; its target gap is 0.80 on the canonical benchmark.The matching-rules study reports target gaps of 0.81 for exact OT and 0.87 for the default soft plan; the symmetric variant reduced order dependence but lowered recall and set-level statistics.Source: §B.4 · p. 14

C GENERATIVE COMPARISON

The generative comparison evaluates prompted GPT-Image-1.5 midpoint images against texture interpolation methods, while optimization and primitive-matching ablations examine runtime and pairing choices. The reported comparisons highlight trade-offs in realism, balance, computational cost, and primitive layout.

C GENERATIVE COMPARISON

  • The directed GPT prompt preserves source layout while moving local appearance toward the target, whereas the symmetric prompt requests an order-independent midpoint.The 120-pair comparison uses a category-stratified subset and produces one GPT output per pair and prompt.Source: C GENERATIVE COMPARISON · p. 15
  • The directed prompt’s median distance-ratio placement is 0.456, and paired DINO realism differences from CNT-based variants range from +0.035 to +0.074.The DINO realism difference from Texture Mixer is -0.021 and not significant; the placement value is distinct from the table’s absolute balance error.Source: C GENERATIVE COMPARISON · p. 15
  • GPT-Image-1.5 has approximately 38 s API latency per output, versus 0.21 s for a local soft texton-matching midpoint; prompted frames do not share an explicit path structure.Soft matching reuses a fixed source-indexed map with affine dependence on t before decoding, while input-order dependence remains.Source: C GENERATIVE COMPARISON · p. 16

D.1 OPTIMIZATION-BASED COMPARISON

  • The optimization-based comparison finds no significant paired difference in per-sample realism or midpoint balance (all Wilcoxon p > .25), which does not establish statistical equivalence.The Wasserstein implementation takes 52.1, 52.4, and 56.5 seconds at 128, 256, and 512 pixels, respectively.Source: §D.1 · p. 18, Table 5 · p. 18

D.2 MATCHING ON CNT’S GAUSSIAN PRIMITIVES

  • With moving primitive layouts, appearance matching yields 0.080 KID versus 0.036 for position matching; soft primitive matching reaches 0.016 KID versus 0.019 for the dense operator.With frozen layouts, position, appearance, and mixed costs yield KID 0.030 / 0.030 / 0.027; the comparison indicates layout anchoring and softness distinguish coherent intermediates from superposition.Source: §D.2 · p. 18, Table 6 · p. 18

D.3 MATCHING RULES UNDER THE ORIGINAL RENDERING PROTOCOL

  • The 200-pair matching-rule study reports results under the original rendering protocol.Source: §D.3 · p. 18

D.4 EQUAL-SAMPLE COMPARISON OF HARD AND SOFT MATCHINGS

The comparisons separate matching quality from rendering and model changes, then test the intervention across a second decoder, repeated applications, and exploratory multi-texture constructions. Results show that matching rule, endpoint behavior, and the choice of conditioning or anchor all shape the resulting mixtures.

D.4 EQUAL-SAMPLE COMPARISON OF HARD AND SOFT MATCHINGS

  • Against field-linear blending on the same pipeline (KID 0.0322), the default plan gives ΔKID −0.0107 (95% CI [−0.0154,−0.0059]), while exact OT gives −0.0022 (95% CI [−0.0066,+0.0022]).The default plan retains 0.38 of target variance versus 0.88 for the sharp plan; appearance-aware matching also improves LPIPS midpoint placement on 37 of 40 pairs in a separate DINO-feature experiment.Source: §D.4 · p. 19, Table 8 · p. 19

D.5 MATCHING IN TEXTURE MIXER’S LOCAL LATENT

  • In Texture Mixer, replacing only the local-latent pairing with soft matching lowers KID without retraining, while one-to-one assignment raises it.At the reported setting, soft matching differs from the target-mean rule by +0.0003 (95% CI [−0.0006,+0.0012]); each interpolation coefficient requires one renderer pass.Source: §D.5 · pp. 19–20, Table 11 · p. 22

D.6 ENCODING CEILING; D.7 OBSERVER DEPENDENCE AND NEGATIVE CONTROLS

  • Soft matching midpoints reach KID 0.023, below the 0.025 double-reconstruction level, but reconstruction error does not predict midpoint quality (Spearman 0.08).In a separate observer protocol, an LPIPS-optimized control reaches a .50 midpoint ratio despite producing a degenerate image; observer measurements for Texture Mixer and soft matching are reported separately from the canonical rerun.Source: §D.6 · p. 20, §D.7 · p. 20, Table 9 · p. 20

D.8 ENDPOINT DIRECTION; E QUALITATIVE COMPARISONS AND APPLICATIONS

  • The operator starts from the source field and ends with target-aligned appearance under source conditioning; reversing inputs reverses those roles.Spatially varying coefficients can create local transitions without another transport solve, while qualitative examples show positional blending becomes more sensitive to displacement than appearance-aware rules.Source: §D.8 · p. 20, Figure 8 · p. 23, Figure 9 · p. 24

D.9 REGULARIZATION AND REPEATED APPLICATION

  • As entropic regularization rises from ε = .005 to .5, mean row entropy rises from 7.09 to 8.31 nats while projected target variance falls from 88% to 1%.Projection effects grow toward the target end; repeated self-loops can increase realism while drifting from the original image toward smoother, typical textures.Source: §D.9 · pp. 20–21, Figure 7 · p. 20, Table 10 · p. 21

F.1 ANCHOR-BASED MIXTURES

  • Four-texture mixtures use one texture as an anchor and its conditioning, so changing the anchor changes outputs by 46% and 38% of mean source separation in the two sets.These constructions are anchor-dependent and are not exact multi-distribution Wasserstein barycenters.Source: §F.1 · p. 26, Figure 11 · p. 26, Figure 12 · p. 26

F.2 SPARSE APPEARANCE CODING

  • Among 83 reconstructable targets, sparse mixtures improve on the best single dictionary atom in 82% of cases, with a median margin of 22%.Sixteen of 99 targets were excluded because CNT could not reconstruct them; periodic structure recurs as a failure, and the coding extension remains exploratory because subset-search settings and raw outputs were not retained.Source: §F.2 · p. 27, Figure 14 · p. 28

F.3 PLAN ALIGNMENT

  • A sharp 1,024-point assignment yields alignment scores of 0.977 ± 0.021 for 929 pairs and 0.52 ± 0.08 for 71, but the threshold was selected on the same data.The separated low-alignment group appears under sharp and hard plans; prospective failure detection has not been validated.Source: §F.3 · p. 27, Figure 15 · p. 28

Related papers

Loading 2610.18655v1…