Minecraft blocks become generative tokens

Sakana AI's Dream-Cubed project transforms Minecraft blocks into tokens to generate playable and editable terrains in less than three minutes.

With Dream-Cubed, Sakana AI and New York University use Minecraft blocks as composable units, much like tokens for text. The project relies on over two million 32 × 32 × 32 block volumes, sourced from procedurally generated terrains and six maps designed by professional creators with their permission. The dataset represents tens of billions of blocks distributed across natural biomes, caves, villages, and human architectures.

Two 3D diffusion approaches were trained on the same 280-million-parameter architecture. The first works directly with block categories using MD4 masked diffusion. The second transforms each block type into a continuous representation before reconstructing the volume with a DDPM approach. Both produce terrains conditioned by biome, but the discrete formulation offers more direct control: blocks placed by the user remain fixed during generation and serve as constraints to complete the scenery.

Thus, just a few blocks are enough to suggest a spiral, a river, a volcano, or a slope. Dream-Cubed then builds the surrounding terrain, regenerates an existing area, or progressively extends the map through outpainting. The resulting volumes directly use the game's blocks: they remain editable and can be integrated into a playable Minecraft environment without any additional reconstruction steps.

The authors also conducted a study with 19 people familiar with Minecraft. According to their results, the generated terrains were preferred over real excerpts in 55.2% to 67.1% of the comparisons, notably because they sometimes represent the expected characteristics of a biome more clearly. The system remains experimental: the data comes from Minecraft 1.12.2, some blocks were grouped to reduce the vocabulary, and generating a single volume can take from 25 seconds to about 2 minutes and 30 seconds on an H100 GPU.