PaperlayerView exact revision on arXiv

Source-linked AI summary

MatFormer: A Generative Model for Procedural Materials

Paul Guerrero, Miloš Hašan, Kalyan Sunkavalli, Radomír Měch, Tamy Boubekeur, Niloy J. Mitra

arXiv:2207.01044v2cs.GR

TL;DR

Procedural material graphs are useful, controllable representations, but authoring them takes expertise and public libraries contain few examples. MatFormer generates these graphs through staged transformer models and supports completion of partial graphs. The authors report that it outperforms alternatives in generated graph quality and sampled-material plausibility and diversity.

  • Problem

    Procedural material graphs are difficult to author and public libraries contain only a few thousand, while their irregular, heterogeneous nodes and constrained long-range connections complicate generation.

    Source: §1 · pp. 1–2

  • Method

    MatFormer uses staged transformers to generate material graph nodes, parameters, and edges, and can complete partial graphs for exploration.

    Source: §1 · p. 2, §3.3 · p. 5, §3.5 · p. 6

  • Results

    MatFormer outperforms alternatives in generated graph quality and in the plausibility and diversity of materials sampled from those graphs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MatFormer offers free graph generation and partial-graph auto-completion as workflows for procedural material authoring.

    Source: §1 · p. 2, §5 · p. 10

    MatFormer’s current form does not condition generation on a target material image, and its performance is constrained by the diversity of its training graphs.

    Source: §5 · p. 10

Abstract

from arXiv · show

Procedural material graphs are a compact, parameteric, and resolution-independent representation that are a popular choice for material authoring. However, designing procedural materials requires significant expertise and publicly accessible libraries contain only a few thousand such graphs. We present MatFormer, a generative model that can produce a diverse set of high-quality procedural materials with complex spatial patterns and appearance. While procedural materials can be modeled as directed (operation) graphs, they contain arbitrary numbers of heterogeneous nodes with unstructured, often long-range node connections, and functional constraints on node parameters and connections. MatFormer addresses these challenges with a multi-stage transformer-based model that sequentially generates nodes, node parameters, and edges, while ensuring the semantic validity of the graph. In addition to generation, MatFormer can be used for the auto-completion and exploration of partial material graphs. We qualitatively and quantitatively demonstrate that our method outperforms alternative approaches, in both generated graph and material quality.

1 INTRODUCTION

Procedural material graphs offer controllable, resolution-independent materials, but authoring complex graphs requires expertise and public libraries are small. MatFormer generates graph nodes, parameters, and edges in stages, and supports both material generation and partial-graph completion.

1 INTRODUCTION

  • Procedural graphs provide controllable, resolution-independent material representations used with geometry to create virtual assets.Their output maps include roughness, normal, and diffuse maps, and complex definitions can require 50+ nodes.Source: §1 · p. 1
  • Material graphs challenge generative modeling with arbitrary sizes, heterogeneous node types and parameters, functional constraints, and long-range connections.Some distant-node connections are critical to material appearance.Source: §1 · pp. 1–2
  • MatFormer uses dedicated transformer stages to generate node sequences, node parameters, and directed edges.Transformers were chosen for handling sparse, long-distance connections between graph nodes.Source: §1 · p. 2
  • MatFormer generates diverse, high-quality material graphs and outperforms alternatives in graph quality and in the plausibility and diversity of sampled materials.It was trained on 2,816 procedural graphs.Source: §1 · p. 2
  • Users can provide selected noise generators and optional connected starting nodes, then iteratively explore and refine completed graphs.The authors describe this guided workflow as simpler and more intuitive than current procedural material design tools.Source: §1 · p. 2

2 RELATED WORK

Related work spans procedural texture authoring and inverse methods that fit or reconstruct materials from images, alongside generative models for images, shapes, and programs. MatFormer instead generates procedural material graphs directly, addressing their irregular structure and constraints.

2 RELATED WORK

  • Procedural material graphs combine filter nodes to process noise and patterns, but designing them requires substantial time and expertise.Procedural texture methods define functions mapping spatial locations to pixel values and expose parameters for user control.Source: §2 · p. 2
  • Earlier inverse methods fit procedural-function parameters to exemplar textures, while Hu et al. [2019] predict procedural-graph parameters from captured images and Hu et al. [2022] reconstruct SVBRDF maps using procedural masks and noise.MATch converts procedural node graphs into differentiable programs and uses stochastic gradient descent to fit graph parameters to captured images.Source: §2 · p. 2
  • Unlike methods that assume a procedural graph is given, MatFormer generates new graphs from scratch and supports free generation and partial-graph auto-completion.The cited public libraries contain 1,500–5,000 assets.Source: §2 · p. 2
  • Image-material generators output resolution-limited, non-parameterized maps, whereas MatFormer generates adjustable procedural graphs.Adjusting graph parameters can produce further material variations.Source: §2 · p. 2
  • Prior graph and program generators target domains including shapes, CAD, and meshes; MatFormer is designed for material graphs’ arbitrary nodes, edge slots, and functional constraints.The related work describes recursive networks, GRU-based program generation, transformers, genetic programming, and language generators.Source: §2 · p. 3

3 METHOD

MatFormer represents procedural materials as directed acyclic graphs and generates them in separate Transformer-based stages for node types, parameters, and edges. The design accommodates long-range dependencies and heterogeneous graph components while enforcing semantic validity.

3.1 Node Graphs

  • Node graphs are directed acyclic multigraphs of image operators whose edges carry images between slots to produce material channels.Nodes may have heterogeneous parameters, and unconnected input slots use default values.Source: §3.1 · p. 4

3.2 Transformers

  • Transformers generate graph components token by token, with invalid token choices masked and probabilities renormalized at inference.Parameter and edge generation are conditioned on the generated node sequence.Source: §3.2 · p. 4

3.3 Node Generation

  • Node ordering is selected by application: πr supports unconditional generation, while πrr supports graph autocompletion.The node generator also predicts graph depth because depth cannot be recovered from the node-type sequence alone.Source: §3.3 · p. 5

3.4 Parameter Generation

  • Parameter sequences omit default-valued parameters and flatten heterogeneous scalar, vector, or array values into scalar tokens.Node type and parameter indices allow reconstruction; continuous values are quantized into 32 levels.Source: §3.4 · p. 5

3.5 Edge Generation

  • Edge generation uses Pointer Networks to select input and output slots from learned embeddings, rather than predicting node connections directly.The embeddings encode each slot’s associated operator, node index, depth, and within-node slot index.Source: §3.5 · p. 6

3.6 Training Setup

  • The model is trained with binary cross-entropy and teacher forcing, with each generation step trained separately using ground-truth inputs.Training stops at minimum validation loss to avoid overfitting.Source: §3.6 · p. 6

4 RESULTS

The evaluation considers unconstrained material generation and guided authoring, using a filtered subset of Substance Source graphs and parameter-perturbed training examples.

4 RESULTS

  • MatFormer is evaluated on unconstrained material generation and guided authoring with suggestions for completing partial graphs.The passage describes the applications but does not report their results.Source: §4 · p. 6
  • Filtering the Substance Source subset leaves approximately 2.8k graphs after removing complex parameterizations and graphs exceeding size or slot limits.The initial subset contained approximately 4k graphs; filtering complex parameterizations left approximately 3.5k.Source: §4 · p. 6
  • Training data are augmented with 100 random parameter perturbations per original graph, producing approximately 280k graphs.Each parameter is perturbed uniformly within [0.8v, 1.2v], and the validation set contains 500 perturbed graphs from five original graphs.Source: §4 · p. 6

4.1 Unconstrained Generation

MatFormer is evaluated for unconstrained generation against baselines and parameter ablations, using qualitative examples and graph- and material-level metrics. The full method’s back-to-front ordering leads the four orderings and outperforms the baselines across all metrics; generated materials are editable, though artist feedback notes quality issues in some outputs.

4.1 Unconstrained Generation

  • The method generated ∼1k samples to demonstrate diverse, realistic materials, comparing them with several baselines both qualitatively and quantitatively; Fig. 6 shows varied spatial patterns, bumps, and specular highlights.Fig. 6 includes material graphs for two examples.Source: §4.1 · p. 6, Figure 6 · p. 8

4.1.3 Metrics.

  • Evaluation uses graph-statistic EMD, material-image FID, and normalized nearest-neighbor style and graph-edit distances; values near 1.0 are ideal for the latter two.Graph-edit distance is evaluated only for generated graphs with at least 50 nodes.Source: §4.1.3 · p. 7, Table 1 · p. 9

4.1.1 Baselines.

  • The single-macro-node baseline is unsatisfactory; pairwise connectivity statistics perform better but can produce unrealistic outputs without long-distance relationships.Macro-node outputs may need additional processing, while pairwise outputs can show a blue tint associated with incorrect normal conversion.Source: §4.1.1 · p. 6, §4.1.4 · p. 8

4.1.2 Ablations.

  • Default parameters significantly raise FID; dataset-average parameters plus perturbation improve it slightly but remain worse than learned parameter generation.The default-parameter ablation has graph-structure metrics comparable to the full method.Source: §4.1.4 · pp. 8–9

4.1.5 Editing materials.

  • Generated procedural materials are editable: four materials received two edits each, in under 10 minutes per edit and with at most 10 parameter changes.Edits include weathering, color, and structural changes such as brick size or wood-grain structure.Source: Figure 7 · p. 9, §4.1.5 · p. 9

4.1.4 Evaluation and Discussion.

  • MatFormer’s back-to-front ordering πr is the clear winner among four orders, and MatFormer has a clear advantage over baselines across all metrics.The comparison covers graph statistics, FID, and normalized nearest-neighbor style and graph-edit distances.Source: Table 1 · p. 9, §4.1.4 · p. 9

4.1.6 User Study.

  • Artist feedback described some materials as interesting, while others were hard to understand, oversaturated, or had unrealistic normals.Source: §4.1.6 · p. 10

4.2 Guided Material Authoring

MatFormer conditions generation on a partial graph so users can explore multiple completions and progressively refine a material graph. The examples use successive completion rounds and randomly selected nodes at each level.

4.2 Guided Material Authoring

  • Users can condition MatFormer on existing nodes and edges, then explore multiple completed graphs and iteratively refine their choices.Source: §4.2 · p. 10
  • The illustrated workflow uses two completion rounds: users choose a completion and a subset of its nodes before generating further alternatives.Nodes are colored by round; gray nodes are not passed to the next level.Source: §4.2 · p. 10
  • Randomly selecting nodes at each level can support both novice users and advanced users exploring material spaces.Source: §4.2 · p. 10

5 CONCLUSION AND FUTURE WORK

MatFormer generates material graphs and supports progressive authoring through suggested completions. The conclusion identifies training-data diversity as a constraint and outlines future directions for image-conditioned generation, semantic controls, recurring subgraphs, and hierarchical graphs.

5 CONCLUSION AND FUTURE WORK

  • MatFormer is a transformer-based autoregressive model that generates operation nodes, their parameters, and edges describing information flow.Source: §5 · p. 10
  • The authors describe MatFormer as, to their knowledge, the first generative model for procedural material graphs.Source: §5 · p. 10
  • The current method cannot condition generation on a target material image; doing so would require an efficient way to backpropagate from images to graphs.The proposed direction could support image-conditioned graph generation and training from larger image datasets.Source: §5 · p. 10
  • One proposed image-to-graph direction predicts intermediate subgraphs, uses autocomplete and possibly beam search, then refines graph parameters.Preserving image mesostructure may require a new material similarity metric.Source: §5 · pp. 10–11
  • Graphs often span 50+ nodes, making them complex to edit without knowing parameter effects; semantic sliders and value ranges are a proposed future direction.Source: §5 · p. 11
  • Discovering recurring subgraphs could yield reusable metanodes for more abstracted material-graph generation.Source: §5 · p. 11
  • Hierarchical graphs could guide users by separating coarse structures, local properties, and spatial effects, but suitable training data remains an immediate challenge.Source: §5 · p. 11

A SEMANTIC VALIDITY CHECKS

Semantic validity checks constrain generated parameter values and graph edges to satisfy the stated operator and connection requirements.

A SEMANTIC VALIDITY CHECKS

B CONDITIONAL TRANSFORMER GENERATOR

The conditional transformer block is modified to incorporate a condition vector alongside its input features.

B CONDITIONAL TRANSFORMER GENERATOR

  • The conditional transformer block takes condition vector c as an additional feature alongside input x, with the original block described using an MLP and layer normalization.Source: B CONDITIONAL TRANSFORMER GENERATOR · p. 12

C GRAPH STATISTICS

The section summarizes material graphs with statistics at both the whole-graph and operator-type levels.

C GRAPH STATISTICS

  • Whole-graph statistics measure node count, disconnected components, and the longest possible path.These capture graph size, connectivity, and path length.Source: C GRAPH STATISTICS · p. 12
  • Operator-level statistics count nodes by type, measure each node’s distance to the closest output, and count connected input slots by operator type.They also measure graph distance between all pairs of operator types.Source: C GRAPH STATISTICS · p. 12

D ARCHITECTURE AND TRAINING DETAILS

The models use a compact GPT-2-based transformer setup with specified sequence limits, and are trained separately for nodes, parameters, and edges.

D ARCHITECTURE AND TRAINING DETAILS

  • All transformer models use GPT-2 with 2 attention layers, 4 heads per layer, and 64-dimensional hidden features.Maximum sequence lengths are 400 nodes for the node generator and node encoder, 512 scalar parameter values, 700 edges (1400 tokens), and 800 slots; the node-type vocabulary contains 1207 types.Source: D ARCHITECTURE AND TRAINING DETAILS · p. 12
  • The node, parameter, and edge generators are trained with Adam at a 1e−4 learning rate and stop at minimum validation loss after 30, 11, and 10 epochs, respectively.Batch sizes are 64 for node and parameter generators and 16 for the edge generator; training takes roughly 2–3 days on one V100 GPU per model.Source: D ARCHITECTURE AND TRAINING DETAILS · p. 12

Related papers

Loading 2207.01044v2…