PaperlayerView exact revision on arXiv

Source-linked AI summary

GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds

Zekun Hao, Arun Mallya, Serge Belongie, Ming-Yu Liu

arXiv:2104.07659v1cs.CV

TL;DR

GANcraft addresses the lack of paired real images for semantic 3D block worlds and the view inconsistency of independent 2D translations. It uses pseudo-ground-truth images to train a neural renderer, and the authors report that it outperforms strong baselines.

  • Problem

    Neural rendering methods generally require real scene images, while independent 2D translations do not generate view-consistent outputs for block worlds.

    Source: §1 · p. 2

  • Method

    GANcraft trains a neural renderer with pseudo-ground-truth images and adversarial losses to produce view-consistent renderings from semantic block worlds.

    Source: §1 · p. 2

  • Results

    GANcraft outperformed strong baselines in converting block worlds to realistic-looking worlds.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The framework is presented as applicable to block-world representations beyond Minecraft, including voxels.

    Source: §1 · p. 2

    Pseudo-ground-truth images can contain artifacts or unrealistic sharp geometry, so reconstruction and adversarial losses must be balanced.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

We present GANcraft, an unsupervised neural rendering framework for generating photorealistic images of large 3D block worlds such as those created in Minecraft. Our method takes a semantic block world as input, where each block is assigned a semantic label such as dirt, grass, or water. We represent the world as a continuous volumetric function and train our model to render view-consistent photorealistic images for a user-controlled camera. In the absence of paired ground truth real images for the block world, we devise a training technique based on pseudo-ground truth and adversarial training. This stands in contrast to prior work on neural rendering for view synthesis, which requires ground truth images to estimate scene geometry and view-dependent appearance. In addition to camera trajectory, GANcraft allows user control over both scene semantics and output style. Experimental results with comparison to strong baselines show the effectiveness of GANcraft on this novel task of photorealistic 3D block world synthesis. The project website is available at https://nvlabs.github.io/GANcraft/ .

1. Introduction

GANcraft turns semantically labeled block worlds into view-consistent photorealistic renderings, addressing the lack of paired real images and the inconsistency of independent 2D translations.

1. Introduction

  • Block worlds offer an intuitive alternative to realistic 3D scene creation, which requires substantial expertise, software, assets, and development time.The paper points to entertainment, education, and rapid prototyping for artists as potential applications.Source: §1 · pp. 1–2
  • Paired block-world and real-image training data is costly or unavailable, while independent 2D translations do not produce consistent views.Existing neural-rendering methods also require real scene images and camera parameters or cannot handle the Minecraft–real domain gap.Source: §1 · p. 2, Table 1 · p. 2
  • GANcraft trains without paired ground truth by using pseudo-ground-truth photorealistic images for sampled camera views.The framework is presented as applicable to other block-world representations, including voxels.Source: §1 · p. 2
  • GANcraft introduces world-to-world translation: producing view-consistent photorealistic renderings from user-created 3D semantic block worlds.The task is described as a 3D extension of image-to-image translation.Source: §1 · pp. 1–2

2. Related Work

Related work spans image-to-image translation, pseudo-ground-truth training, and neural rendering; GANcraft combines ideas from these areas for large, complex block worlds.

2D image-to-image translation

  • 2D image-to-image translation maps images between domains, using paired examples in supervised settings and separate image sets in unsupervised settings.Unsupervised methods commonly use shared-latent-space assumptions or cycle-consistency losses.Source: §2 · p. 2

Pseudo-ground-truth training

  • Pseudo-ground-truth training uses predictions from a source-domain model as labels for target-domain data; GANcraft instead generates photorealistic targets with pretrained SPADE.The resulting pseudo pairs support reconstruction and perceptual losses in the world-to-world framework.Source: §2 · p. 3

3D neural rendering

  • Neural rendering combines differentiable 3D projection with learned 3D and 2D features, while many adversarial approaches have been limited to smaller or simpler scenes.NeRF-family methods learn scene geometry and appearance from posed real images, unlike GANcraft’s coarse voxel input without corresponding real images.Source: §2 · p. 3

GANcraft’s relation to prior methods

  • GANcraft refines supplied voxel geometry rather than learning scene structure from scratch and uses style conditioning, differentiable projection, and adversarial training.Its large, complex scenes and high-resolution outputs make adversarial loss alone insufficient for good results.Source: §2 · p. 3

3. Neural Rendering of Minecraft Worlds

GANcraft uses pseudo-ground-truth views to train without real images of user-created block worlds, then renders those worlds with a sparse voxel-based neural renderer.

3. Neural Rendering of Minecraft Worlds

  • The method targets landscape scenes larger than those typical of prior neural-rendering work, while learning details finer than a block without supervision.The input voxels and labels already specify coarse geometry and semantic arrangement.Source: §3 · p. 3

3.1. Generating pseudo-ground truth training data

  • Each iteration samples a camera pose and focal length, projects voxel labels, and uses SPADE with a random style code to create same-layout pseudo-ground-truth images for reconstruction and perceptual losses alongside adversarial loss.SPADE's generalizability and photorealistic generation help reduce the domain gap and label-distribution mismatch.Source: §3.1 · p. 4
  • Blockiness and some camera-pose/style-code combinations can yield artifacts in pseudo-ground truths, so reconstruction and adversarial losses must be balanced carefully.The method is designed to tolerate noise, varying styles, and inconsistencies in those generated images.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2. Sparse voxel-based volumetric neural renderer

  • A shared MLP predicts features and density from voxel-corner features, Fourier features, voxel labels, and style code; density does not depend on style.Learnable corner features are trilinearly interpolated, with shared vertices supporting smooth transitions across voxel boundaries.Source: §3.2 · pp. 4–5
  • Stratified sampling selects valid voxel-bounded ray points, while ray truncation and opacity regularization encourage opacity to saturate before the maximum distance.A modified Bresenham method samples valid points with complexity O(N), where N is the voxel grid's longest dimension.Source: §3.2 · pp. 5–6
  • Volumetric rendering blends per-sample features into pixel features, then a style-conditioned CNN produces RGB images; the two-stage design also reduces rendering computation and memory.Because the image-space CNN is not intrinsically view-consistent, its receptive field is limited to 9×9 pixels for local manipulations.Source: §3.2 · pp. 5–6

4. Experiments

Across five Minecraft worlds, GANcraft is evaluated against image-translation and neural-rendering baselines using image metrics, user preferences, and qualitative video comparisons. It achieves image-quality metrics close to SPADE and users prefer its videos for realism and temporal consistency.

Experimental setup

  • The evaluation uses five 512×512×256-block worlds with varied water, sand, forest, and snow compositions, plus a held-out set of 5,000 real images.FID and KID are computed on generated images from sampled camera views against held-out real landscapes; the dataset contains about 1 million landscape images, and 5,000 are set aside for testing.Source: §4 · pp. 6–7

Main results

  • Across all five block worlds, GANcraft achieves FID and KID values close to SPADE, beating MUNIT and NSVF-W.SPADE sets the lower bound on FID and KID but is not view-consistent.Source: Table 2 · p. 7, §4 · p. 7
  • GANcraft is more realistic and view-consistent than the baselines; NSVF-W produces dull, unrealistic outputs with artifacts.MUNIT and SPADE flicker, while wc-vid2vid is more view-consistent but fails for large motions; NSVF-W lacks an image-space CNN and uses an L2 loss, whereas adversarial loss is key to vivid results. GANcraft can also produce higher-resolution outputs by sampling more rays.Source: §4 · p. 7
  • Users chose GANcraft videos as more temporally consistent and realistic than videos from the compared methods.The human-preference comparison covered all five block worlds.Source: Table 3 · p. 7, §4 · p. 7

Ablations

  • Ablations show that removing pseudo-ground truth yields unrealistic outputs, omitting the CNN reduces fine detail, and removing adversarial losses makes images duller or blurrier.These ablations were trained on one Minecraft world due to computational constraints.Source: §4 · p. 7

5. Discussion

GANcraft converts block worlds into realistic-looking worlds, and its hybrid renderer outperformed strong baselines. The discussion identifies output limitations and directions for future improvement.

5. Discussion

  • GANcraft’s hybrid renderer, trained with real landscape images, pseudo-ground truths, and adversarial losses, outperformed strong baselines.The output-video comparison reports greater realism and view consistency than the baselines.Source: §5 · p. 7, Figure 4 · p. 8
  • Coarse input geometry can limit the realism of translated geometry, motivating smoother geometry as a future improvement.The paper also identifies using non-voxel inputs, such as meshes and point clouds, as a future direction.Source: §5 · pp. 7–8
  • Removing pseudo-ground truths, the CNN, or GAN losses degraded outputs in distinct ways compared with the full model.Without pseudo-ground truths outputs were unrealistic; without a CNN they lacked detail and had artifacts; omitting real-image GAN loss dulled colors; removing all GAN loss made outputs dull and blurry.Source: Figure 5 · p. 8
  • GANcraft is currently trained per world; enabling feed-forward generation on novel worlds remains future work.Source: §5 · p. 8

Supplementary Material

The supplementary material provides additional method information, results, and visual demonstrations through the project website and a video.

Supplementary Material

B. Method details

The supplementary method details describe numerical ray rendering and an efficient procedure for sampling only within occupied voxels.

B.1. Numerical volumetric rendering

  • The volumetric-rendering integral is approximated by discrete samples along each camera ray using quadrature.Source: §B.1 · p. 12

B.2. Point sampling algorithm

  • The point-sampling procedure traverses the voxel grid to find valid voxel segments, then samples only within those segments.Source: §B.2 · p. 12
  • Voxel traversal uses a 3D version of Bresenham’s line algorithm, stepping to the adjacent voxel across the face where the ray exits.Source: §B.2 · p. 12

B.3. Network Architecture

The supplementary details describe GANcraft’s style-conditioned rendering networks, semantic label translation, and voxel preprocessing.

B.3. Network Architecture

  • The six-network architecture includes a style-conditioned radiance-field MLP, sky-dome MLP, image-space CNN, style encoder and network, and label-conditional discriminator.The image-space CNN uses small kernels to encourage view consistency; the FPSE-based discriminator is described as more robust to label-distribution mismatch than a patch discriminator.Source: §B.3 · p. 12
  • GANcraft’s FPSE-based label-conditional discriminator handles label-distribution mismatch more robustly than a patch discriminator, which can cause training collapse.The discriminator uses translated semantic masks as conditional input.Source: §B.3 · p. 12, §B.4 · p. 14

B.4. Label Translation

  • A 12-class intermediate label set bridges Minecraft’s 680 material labels and COCO-Stuff’s 182 higher-level labels; random mappings to COCO-Stuff labels support pseudo-ground-truth generation.The same mapping scheme is used within a segmentation map, and the style encoder can explain away the mapping randomness.Source: §B.4 · pp. 14–15
  • The method works across five worlds with very different label distributions, including worlds dominated by sand or water.The cited examples contain over 50% sand and over 90% water, respectively.Source: Figure 12 · p. 15
  • Removing pseudo-ground truths can yield low FID and KID yet still produce unrealistic images with incorrect label-to-texture correspondence.Source: Figure 13 · p. 15

B.5. Voxel Preprocessing

  • Voxel preprocessing removes interior voxels and retains a four-voxel-thick shell, reducing typical occupancy from 28% to 3%.The passage states that this preprocessing also applies to voxel grids beyond Minecraft.Source: §B.5 · p. 15

C.1. Minecraft block worlds.

Across five Minecraft worlds with markedly different label distributions, GANcraft works despite the domain gap, though some style–world combinations produce unrealistic outputs. The appendix also specifies training and evaluation settings and the NSVF-W baseline changes.

C.1. Minecraft block worlds.

  • GANcraft works across five worlds despite label distributions that differ from real images, including worlds that are over 50% sand or over 90% water.Certain combinations of styles and worlds nevertheless produce unrealistic outputs, possibly because the styles are outliers.Source: §C.1 · p. 15, Figure 15 · p. 16

C.2. GANcraft settings

  • Camera-pose filtering requires sufficient depth and segmentation-label entropy so the mask can provide geometry hints for pseudo-ground-truth generation.The specified thresholds are mean depth 2 and label entropy 0.75.Source: §C.2 · p. 16
  • Training uses 24 samples per ray, while evaluation uses 32 and takes about 10 seconds per 1024×2048 frame on an NVIDIA Titan V.Training images are 256×256; camera poses are rejected when mean depth is below 2 or segmentation-mask label entropy is below 0.75.Source: §C.2 · p. 16

C.3. Baseline settings

  • The NSVF-W baseline uses only L2 and KL losses, reduces the KL weight to 0.01, and removes the image-space CNN renderer.Its per-sample MLP directly produces sigmoid-clipped RGB radiance instead of a feature.Source: §C.3 · p. 16

D. Additional results

Table 4 compares ablated versions of the full method on one block world using automated image-quality metrics, for which lower values are better.

D. Additional results

  • Table 4 compares ablated versions of the full method on a single block world using automated image-quality metrics, with lower values indicating better scores.Source: Table 4 · p. 16

D.1. Ablation study

Ablations generally worsen FID and KID, but removing pseudo-ground-truth images lowers both metrics while producing unrealistic outputs that do not map Minecraft labels meaningfully. The experiments cover one world, and the method can also produce blocky or unrealistic outputs under specified conditions.

D.1. Ablation study

  • The ablations were trained on one world because each model required four days on eight NVIDIA V100 GPUs.Source: §D.1 · p. 16
  • Ablated versions generally have higher FID and KID, but removing pseudo-ground-truth images lowers both metrics while failing to learn a meaningful Minecraft-to-real mapping.Visual inspection shows unrealistic outputs despite the lower scores.Source: §D.1 · p. 16
  • Coarse block geometry can yield jagged boundaries, while some style–world pairings, such as a forest with a red-sunset image, produce unrealistic or overly dark outputs.The style encoder sees pseudo-ground-truth images with the rendered Minecraft label distribution, not those combinations.Source: §D.1 · p. 16
Loading 2104.07659v1…