Source-linked AI summary
Large-size multivariate scale-free texture generation from small-size learning sets with self-attention Wasserstein-GAN
Jhonatan Ancco Cerda, Kinan Abbas, Stéphane G. Roux, Patrice Abry
TL;DR
Generating multivariate scale-free textures beyond the size of available training images requires attention to multiscale statistics, which prior approaches often do not explicitly assess. The paper evaluates convolutional Wasserstein-GANs with self-attention and reports accurate statistics through eight times the training dimensions, with critic-only attention most consistent across output sizes.
Problem
Generating multivariate textures at sizes larger than the available training images is challenging, and prior approaches often do not explicitly assess whether multiscale or scale-free statistics are reproduced.
Source: §1 · p. 1Method
The paper evaluates fully convolutional Wasserstein-GAN architectures with self-attention in the generator, critic, or both, using multifractal textures as reference models without incorporating their statistics into the training loss.
Source: §1 · p. 1Results
A WGAN trained on 128 × 128 textures reproduced targeted multiscale and cross-statistical properties up to 1024 × 1024, and critic-only attention was most consistent across output sizes.
Source: §5 · p. 4Takeaways & Limitations
Critic-only attention provides the most consistent performance among the evaluated placements without adding generation-time cost.
Source: §5 · p. 4Performance degradation at 2048 × 2048 indicates the current extrapolation limit.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Multivariate scale-free texture images are nowadays central in numerous modern applications very different in nature. A common challenging issue consists of being able to generate such multivariate textures at any arbitrary size, from a learning set of (possibly much) smaller size. Parametric models often turn out to be too restrictive for the generation of realistic textures, thus prompting neural network training-based generation. While several neural network approaches propose generation at sizes larger than training, they do not explicitly assess the quality of the generation by quantifying how well multiscale or scale-free statistics in textures are reproduced and rather rely on perceptual assessment. The present work combines convolutional Wasserstein-GAN architectures with self-attention layers to generate textures of arbitrary size while focusing on reproducing multiscale statistics, without coding a model into the loss function. The relevance of using a self-attention layer is studied. Quantitative assessment conducted on multifractal textures, representative of a broad class of multiscale textures, over ten independent trainings shows that the proposed architecture is able to generate textures with accurate statistics for sizes $8 \times 8$ larger than the originals. The results show that placing self-attention in the critic provides the most consistent performance across output sizes.
1. INTRODUCTION
The paper targets generation of multivariate scale-free textures from smaller training images, where realistic texture synthesis and explicit statistical fidelity remain challenging. It evaluates convolutional Wasserstein-GANs and self-attention as approaches to preserving multiscale and cross-component statistics beyond the training size.
Context and related work
- Parametric texture models can be too restrictive, while neural approaches often rely on perceptual assessment rather than explicitly reproducing multiscale or scale-free statistics.Earlier convolutional GAN approaches commonly rely on local or fixed-scale statistics, which are not well suited to reproducing multiscale statistics.Source: §1 · p. 1
Goals and contributions
- The work assesses whether fully convolutional Wasserstein-GANs can generate multivariate scale-free textures at arbitrary sizes while preserving multiscale and cross-component statistical properties.It studies self-attention in the generator, critic, or both networks.Source: §1 · p. 1
Assessment scope
- Multivariate multifractal textures serve as reference models because they have strongly non-Gaussian statistics, rich component-wise distributions, and complex inter-component dependencies.Their statistical properties are not assumed known or used in the training loss.Source: §1 · p. 1
2. SELF-ATTENTIVE WGAN
The architecture replaces a fixed-size generator input with convolutional processing so input support can vary at inference. It combines a convolutional Wasserstein-GAN with self-attention placements evaluated in the generator and critic.
Wasserstein GAN
- The WGAN replaces the discriminator with a critic, trains the critic with a gradient penalty enforcing the Lipschitz constraint, and trains the generator to maximize the critic's score for generated samples.The penalty uses interpolations between real and generated samples.Source: §2 · pp. 1–2
Spatial-support extrapolation
- A fully connected first generator layer restricts a classical WGAN to the training-set image size, whereas a convolutional input layer permits larger spatial supports at inference.The convolutional approach extends a principle previously proposed for univariate one-dimensional time series to two-dimensional multivariate textures.Source: §2 · p. 2
Self-attention layer
- Self-attention combines learned features from different spatial locations, complementing convolutional layers that combine information within local neighborhoods.Three learned 1 × 1 convolutions produce the query, key, and value matrices; d is the query/key dimension and α is a learned scalar.Source: §2 · p. 2
- The generator attention block follows its first convolution, while the critic block sits between its third and fourth convolutions.These placements limit attention's quadratic cost while allowing nonlocal information to be processed by subsequent layers.Source: §2 · p. 2, Figure 1 · p. 2
Proposed architecture
- The common backbone uses a convolutional generator and critic, and the study assesses self-attention separately or jointly in those networks.The generator uses upsampling blocks; the critic uses stride-(2, 2) convolutions for spatial downsampling.Source: §2 · p. 2
3. MULTIVARIATE MULTISCALE TEXTURES
The section uses M-variate multifractal random walks as rich multiscale texture models and describes wavelet-based statistics for characterizing their scale-free and cross-component behavior. Wavelet leaders and covariance-based measures provide comparisons of univariate and cross-statistics across scales.
Multivariate scale-free texture models
- M-variate multifractal random walks (MMRW) serve as rich multiscale, multivariate stochastic models for textures.Their component fields combine jointly Gaussian structure with spatially structured modulations that produce departures from joint Gaussianity.Source: §3 · p. 2
Multiscale (wavelet-based) analysis
- Wavelet coefficients are inner products of each texture component with wavelet templates dilated and translated across scales and locations.Wavelet leaders take local suprema of coefficients over finer scales and a spatial neighborhood.Source: §3 · pp. 2–3
- First-order cumulants of log leaders characterize each component’s univariate scale-free statistics through linear behavior across scales.Higher-order cumulants of log leaders reflect cross-higher-order statistics; second-order cumulants characterize scale-free cross-statistics through linear scale behavior.Source: §3 · p. 3
- Wavelet covariance provides a cross-statistics measure, while its multifractal counterpart can compare dependencies beyond second-order cross-correlations.The multifractal correlation function should converge to ρ_m,m′ at fine scales, although that asymptotic limit is not reachable in practice.Source: §3 · p. 3
4. PERFORMANCE ASSESSMENT
The evaluation compares four attention configurations on bivariate multifractal textures across output sizes, using ten independent trainings. Critic-only attention performs most consistently, with good statistical agreement through 1024 × 1024 and degradation at 2048 × 2048.
4.1. Experimental setup
- The experiment trains on 128 × 128 bivariate MMRW textures and evaluates outputs from 128 to 2048 across four attention configurations and ten independent trainings.The two components have different multifractal properties, with negative second-order correlation and positive cross-multifractal dependence.Source: §4.1 · p. 3
4.2. Results and Discussion
- Generated textures show nearly identical maximum-correlation distributions to validation textures, indicating no memorization.The assessment compares each generated texture with translated and rotated training textures and retains the largest absolute Pearson correlation.Source: §4.2 · p. 3, Figure 2 · p. 3
- SA-C has the lowest RMSE in 15 of 30 size–descriptor combinations and ties for the lowest in one more; at N = 1024, it leads on five of six descriptors.Across configurations, textures up to N = 1024 show good overall agreement with the evaluated descriptors.Source: §4.2 · p. 3, Table 1 · p. 4
- At N = 2048, RMSE increases for first-order scaling coefficients and wavelet correlation in every configuration.The scale-wise comparison shows deviations at N = 2048, whereas W and SA-C reproduce approximately linear C1(j) and C2(j) through N = 1024.Source: §4.2 · pp. 3–4, Figure 3 · p. 4
- SA-G is more variable across trainings and degrades scaling and cross-correlation statistics as size increases, while critic attention is used only during fixed-size training.The authors suggest that applying generator attention, trained on 16 × 16 feature maps, to larger maps during extrapolation may distort scaling and cross-component correlations.Source: §4.2 · p. 4
- The SA-C generator trained at N_T = 128 produces visually consistent textures at N = 128, 256, 512, and 1024.The displayed components have H = 0.8 and H = 0.6.Source: §4.2 · p. 4, Figure 4 · p. 4
5. CONCLUSIONS AND PERSPECTIVES
The work evaluates fully convolutional WGANs for generating multivariate multifractal scale-free textures beyond their training support. The approach reproduces targeted statistics up to 1024 × 1024, with critic-only attention the most consistent placement; performance degrades at 2048 × 2048.
5. CONCLUSIONS AND PERSPECTIVES
- A fully convolutional WGAN trained on 128 × 128 textures reproduces targeted multiscale and cross-statistical properties up to 1024 × 1024 without encoding them in the training loss.The textures are multivariate multifractal scale-free textures.Source: §5 · p. 4
- Among the evaluated self-attention placements, critic-only attention provides the most consistent performance and adds no cost during generation.Source: §5 · p. 4
- Performance degradation at 2048 × 2048 indicates the current extrapolation limit; future work will consider higher-dimensional textures and statistically informed learning objectives.Source: §5 · p. 4