Building 3D Worlds

Imagine standing inside a movie set — not as a photograph, but as a space you can walk through, edit, and export into a game engine. That is the promise of 3D world generation: turning sparse visual evidence (a few frames, a panorama, or a handful of reference photos) into a coherent, navigable environment.

The field has exploded in the last two years. Research labs ask bold questions — Can a neural network replace a game engine? Can a model learn physics from video alone? — and industry teams ask practical ones — Can we get a mesh into Unreal Engine that a CG artist can actually use?

This post is a map of that landscape: what world models are, how the major approaches differ, what a modern generation pipeline looks like end to end, and where the hard problems still live.

What Is a World Model?

At its core, a world model is a system that understands (or simulates) how an environment behaves. That raises two natural questions:

Does it encode current state? Can the model represent what the world looks like right now — geometry, appearance, layout?
Can it predict what happens next? Given an action (move forward, turn left, open a door), does it produce a plausible next frame or next state?

Different systems answer these questions differently. Some prioritize interactive simulation (real-time, action-conditioned video). Others prioritize static reconstruction (accurate geometry you can edit offline). The choice of objective shapes everything downstream.

For a useful survey of the full research landscape, see World Models: A Survey.

Two Paradigms: Video Worlds vs. Static Meshes

Today’s world-generation systems broadly fall into two camps.

1. Video-Based World Models

These models generate frames on demand — you provide an action or camera motion, and the model hallucinates the next view. They power interactive experiences: game-like exploration, synthetic data, and research on embodied AI.

Representative work:

System	Core Question	Key Result
GameNGen (Google Research)	Can a neural network replace a game engine?	Yes — for a single, fixed world (e.g., DOOM)
Genie (DeepMind)	Can a model learn world dynamics from video alone?	Yes — but not real-time, and not always stable
Genie 2	Can this scale to diverse, open worlds?	Promising foundation model; architectural details in the DeepMind blog and SSRN analysis
Matrix Game 3.0, Sana World, WanX World	Can video models power game-world generation?	Fast inference, but quality trade-offs

Video world models are exciting for gaming and simulation. They excel when the output is video — when nobody needs to open the result in Blender.

But they carry structural limitations for production pipelines:

No persistent geometry. Each frame is generated fresh; there is no mesh to revisit, edit, or relight.
Hallucination and inconsistency. Revisiting the same location twice may yield different geometry.
Heavy rendering. High-quality diffusion video is slow and memory-hungry.
No engine export. Unity and Unreal expect meshes, textures, and materials — not MP4 files.

2. Static 3D Asset and Scene Generation

The second paradigm builds explicit 3D representations: point clouds, Gaussian splats, meshes, or NeRF-like fields. The output can be imported, edited, and assembled in a DCC tool or game engine.

Representative systems:

System	Approach	Notable Feature
HY-World 2.0 (Tencent Hunyuan)	Panorama → trajectory → video inpainting → 3DGS → mesh	Unreal Engine export support
Matrix-3D	Image/text to 3D world	Large-scale scene generation
WorldGen	Text/image conditioned world building	Flexible scene layout
Flash-Sculptor	Fast static scene sculpting	Speed-focused reconstruction

For computer graphics production — film sets, virtual production, game levels — this paradigm is usually the right one. A CG team needs:

Low-poly meshes with good appearance — not million-triangle raw reconstructions.
Engine-compatible assets — GLB, FBX, or direct Unreal/Unity import.
Editable geometry — the ability to move a wall, swap a prop, or fix a bad corner in 3D space.

Static mesh generation trades the magic of infinite interactive video for the reliability of something you can actually ship.

The HY-World 2.0 Pipeline: A Reference Architecture

Among static-scene systems, HY-World 2.0 is one of the most complete open pipelines. Understanding its stages clarifies how modern 3D world generation actually works — regardless of which tool you pick.

Input (panorama or multi-view images)
        ↓
  Trajectory Planning (VLM-guided camera paths)
        ↓
  Point-Cloud / Splat Rendering (conditioning video)
        ↓
  WorldStereo Diffusion (inpaint missing regions)
        ↓
  Frame Extraction + Camera Export
        ↓
  3D Gaussian Splatting Training
        ↓
  Mesh Extraction + Engine Export

Each stage solves a distinct sub-problem.

Stage 0: Scene Input

The simplest entry point is a single panoramic image plus a text prompt. The panorama encodes layout and appearance; the prompt guides semantic content. More advanced setups skip the panorama entirely and instead feed multi-view RGB-D images from a real location.

Panorama generation is often the bottleneck: building a coherent 360° view from scratch is harder than refining views you already have.

Stage 1: Trajectory Planning

A Vision-Language Model (VLM) — typically something like Qwen-VL — plans camera trajectories through the scene: forward paths, upward routes, and “wonder” exploration loops that cover unseen angles.

Good trajectories matter enormously. Jerky paths, large inter-frame motion, or trajectories that stare at voids will poison every downstream stage.

Stage 2: Conditioning Render

The planned cameras render the current best 3D representation — usually a point cloud splat. This produces two videos:

render.mp4 — RGB frames with gray holes where geometry is missing.
render_mask.mp4 — a binary mask (white = missing, black = valid geometry).

These are not the final output. They are conditioning signals for the diffusion model in the next stage.

Stage 3: WorldStereo Diffusion

WorldStereo is a stereo-aware video diffusion model that fills in the holes. It retrieves appearance information from a memory bank of previously seen views (typically four cardinal panoramas: front, right, back, left) and generates photorealistic frames along each trajectory.

Critically, WorldStereo treats the mask as soft ControlNet guidance, not hard inpainting. It prefers to generate in masked regions but does not force every missing pixel to be replaced. This architectural choice has deep consequences (covered in Limitations below).

Two model variants exist:

Variant	Denoising Steps	CFG	Trade-off
`worldstereo-memory-dmd`	4	1.0	Fast; lower quality
`worldstereo-memory`	Multi-step	5.0	Slower; significantly better hole filling

Stage 4: 3D Gaussian Splatting (3DGS)

Extracted frames — with known camera poses — train a 3D Gaussian Splat representation. Gaussian splatting represents a scene as millions of oriented 3D Gaussians, each with color, opacity, and covariance. It renders fast and captures fine detail.

Training typically runs for thousands of steps with losses on depth, normals, and sky regions. The result is a .ply point cloud of Gaussians that can be viewed interactively (e.g., via Viser) or converted further.

Stage 5: Mesh Extraction and Cleanup

The final production step converts splats or dense geometry into game-ready meshes:

Extract mesh from the trained Gaussian checkpoint.
Remesh for clean topology (QuadWild, QuadFlow, AutoRemesher).
Reduce polygon count while preserving silhouette and texture.
Transfer textures from the high-res source onto the reduced mesh.
Export as GLB/FBX for Unreal or Unity.

This stage is where art meets engineering. A mesh with 800,000 triangles that looks great in a viewer may be useless in a game engine that needs 30,000 well-flowed quads.

An Alternative Entry: Multi-View Images → Point Cloud

When a panorama is unavailable — for example, when reconstructing a real movie set from production stills — a different entry path works better.

Multi-view RGB images
        ↓
  Metric Depth Estimation (e.g., MoGe v2)
        ↓
  Global Point Cloud Assembly
        ↓
  Camera Trajectory Generation (PCD-anchored)
        ↓
  HY Scene Layout Preparation
        ↓
  (Same pipeline as above: render → WorldStereo → 3DGS → mesh)

Depth and Point Clouds

MoGe (and similar monocular depth models) produce metric depth maps from single RGB images. Given known or estimated camera intrinsics, each depth map unprojects into a partial point cloud. Multiple views — even non-overlapping ones — can be stitched into a global point cloud with hand-guided or optimization-based alignment.

The density and accuracy of this cloud determines everything downstream. Sparse clouds mean large holes in the conditioning render, which means the diffusion model must hallucinate more — and hallucination is where quality dies.

PCD-Anchored Trajectories

Without a panorama, a VLM cannot plan trajectories in the usual way. Instead, trajectories are generated relative to the point cloud: orbiting its center, sweeping along facades, or following paths that maximize coverage of known geometry while probing unknown regions.

Smoother trajectories with smaller inter-frame motion produce better WorldStereo results. Trajectories that leap across large unseen gaps force the model to invent entire scenes from scratch.

Object-Level Reconstruction

Scenes are made of objects. A complete pipeline also needs asset-level generation — chairs, props, characters — that can be placed inside the reconstructed environment.

Modern image-to-3D systems include:

System	Strength
Hunyuan3D-2	High-quality textured meshes from images
TRELLIS.2	Structured 3D generation (Microsoft)
TripoSG	Fast single-image to 3D
DeepMesh	Auto-regressive artist-quality mesh generation
MeshFlow	Efficient artistic mesh creation with good edge flow

The object pipeline faces the same mesh-quality challenge as scenes: raw generated meshes often have too many triangles, bad topology, or lost texture detail after reduction. The goal is a quad-dominant, low-poly mesh with embedded textures — the standard format for game assets.

Common remeshing tools and strategies:

MeshOptimizer — fast simplification, but can destroy edge flow.
QuadWild / QuadFlow — quad-dominant remeshing with controllable target face counts.
AutoRemesher — automated prep + remesh + GLB export in one pass.
Texture transfer — bake high-res appearance onto the reduced mesh so visual quality survives polygon cuts.

The ablation data is instructive: reducing to 10,000 triangles often fails outright; 20,000–30,000 is a practical sweet spot for many props; beyond 80,000, diminishing returns set in while engine performance suffers.

Where the Pipeline Breaks: WorldStereo in Detail

Understanding why generated worlds look wrong is as valuable as knowing how to run the pipeline. Most quality issues trace back to a small set of root causes.

The Mask Is Soft, Not Hard

WorldStereo uses the hole mask as ControlNet guidance — it tells the model where generation is preferred but does not mandate pixel replacement. Some holes stay gray even after diffusion. This is architectural, not a bug.

Implication: Do not evaluate hole filling from render.mp4. Always compare three files together:

File	What It Shows
`render.mp4`	Raw PCD splat (gray holes expected)
`render_mask.mp4`	White = should be generated; black = keep
`*_result.mp4`	Final diffusion output (evaluate here)

Sparse Point Clouds → Large Holes

When 50–68% of a frame is missing (white in the mask), the diffusion model must invent most of the image. Quality collapses.

Fixes:

Increase splat radius and points-per-pixel in the conditioning renderer.
Build denser global point clouds from more input views.
Reduce camera motion between frames.

Target: less than 30% hole coverage per frame.

Weak Memory Bank

WorldStereo retrieves appearance from a panorama memory bank. With only four cardinal views, unseen angles (especially diagonal or elevated) lack reference material.

Fixes:

Run the full scene sequentially so later trajectories benefit from earlier generations.
Use two-pass generation: first pass builds a rich memory bank; second pass regenerates only poor trajectories.
Expand the panorama bank with additional RGB-D views.
Increase --max_reference when calling the model directly.

Wonder Trajectories Start From Renders, Not Photos

The view0 trajectory begins from a real RGB image — strong conditioning. wonder_* trajectories start from sparse PCD renders — weak conditioning.

Fix: Replace wonder start frames with the nearest real RGB photo, resized to the model’s expected resolution (e.g., 832 × 480).

DMD vs. Full Model

The distilled DMD variant (4 steps, CFG = 1.0) prioritizes speed over quality. For any serious reconstruction, the full worldstereo-memory model is worth the extra compute.

From Movie Sets to Virtual Environments

One of the most compelling applications is movie set reconstruction: given reference frames from a film shoot, build a 3D environment that production teams can revisit, extend, or reuse.

The workflow:

Collect reference images from the set (production stills, behind-the-scenes photos, or extracted video frames).
Remove dynamic elements (actors, crew) via segmentation so static geometry is isolated.
Estimate depth and assemble a global point cloud.
Generate trajectories that cover the set from multiple angles.
Run WorldStereo to inpaint and expand coverage.
Train 3DGS and extract meshes.
Clean and reduce meshes for engine import.

The hard parts are not the tools — they exist and are open source. The hard parts are:

Corner and occluded regions that no camera ever saw.
Multi-view inconsistency when images come from different takes or lighting setups.
Depth drift that misaligns the global point cloud.
2D inpainting that ignores 3D structure — filling a hole in one view that creates a contradiction in another.

These are active research problems, not solved engineering tasks.

Open-Source Implementations Worth Knowing

The research papers are dense; the open-source ecosystem makes them runnable.

World Models (Interactive / Video)

Repository	Focus	Compute
GenieRedux	Multi-env world model + dataset	Multi-GPU
open-genie	Clean Genie v1 reimplementation	Single GPU
genie-bottle	Educational CoinRun demo	Single GPU
gameNgen-repro	DOOM + diffusion reproduction	TPU/GPU
tinyworlds	Minimal autoregressive baseline	Minimal

Static 3D Worlds

Repository	Focus
HY-World-2.0	Full panorama-to-mesh pipeline with UE export
Matrix-3D	Large-scale 3D world generation
WorldGen	Flexible scene generation
SyntheticEnvGeneration	Adapted HY-World pipeline for movie-set reconstruction

Key Papers and Resources

Resource	What You Get
Genie 1 (arXiv 2402.15391)	Original Genie paper (ICML 2024 Best Paper)
GenieRedux (arXiv 2409.06445)	Open reimplementation with runnable training details
GameNGen (arXiv 2408.14837)	Neural game engine on DOOM
World Models Survey (arXiv 2411.14499)	Comprehensive landscape overview
Genie mechanics gist	Readable walkthrough of Genie internals

Infrastructure Reality

Running these pipelines is not plug-and-play. A typical HY-World 2.0 setup involves:

GPU memory management — serving a VLM (e.g., Qwen3-VL-8B via vLLM) on one GPU while running diffusion on others.
CUDA library paths — PyTorch3D, gsplat, and NCCL each have specific version requirements; LD_LIBRARY_PATH tuning is routine.
Dependency pinning — NumPy versions, Transformers API changes (e.g., flattened CLIPVisionModel in v5.x), and vLLM wheel indices must align.
Multi-GPU orchestration — torchrun with FSDP for diffusion; single-GPU fallbacks require forward-pass patches to avoid NCCL errors.

This is normal for research-grade 3D pipelines in 2026. Expect a day of environment debugging before the first successful run.

Open Problems and Future Directions

The field is moving fast, but several problems remain unsolved.

Multi-Reference Scene Generation

Most pipelines assume a single panorama or a small set of cardinal views. Real movie sets provide dozens of non-overlapping photos from different takes, angles, and lighting conditions. Fusing these into one coherent 3D world — without seams, lighting jumps, or geometric contradictions — is largely unsolved.

3D-Aware Inpainting

Current inpainting operates in 2D image space. Filling a corner in one view may create geometry that is inconsistent when viewed from another angle. What is needed is inpainting that respects the underlying 3D structure — editing the point cloud or mesh directly, not just the pixels.

Controllable Editing

Once a world is generated, how do you edit it on the fly? Move a wall, change a material, add a prop? Today this requires exporting to a DCC tool and manual work. The dream is agentic AI that understands the 3D scene and accepts natural-language edits.

Avatar and Character Integration

Static environments are only half the problem. Production pipelines also need human avatars that match the visual style of the source material — another hard problem sitting adjacent to world generation.

Mesh Quality at Scale

Object and scene meshes that survive polygon reduction and retain visual fidelity and have clean quad topology for animation and lighting — this trifecta remains elusive. New methods like DeepMesh and MeshFlow are promising, but production-quality auto-remeshing is not yet a solved button-click.

Training Data

Two dataset gaps block further progress:

Panorama datasets for training better 360° generators.
Camera-controlled video datasets for training better view-synthesis models.

Without better training data, inference-time hacks (larger splats, richer memory banks, two-pass generation) can only go so far.

Choosing an Approach

Goal	Recommended Paradigm	Example Tools
Interactive game-like exploration	Video world model	Genie 2, Matrix Game, GameNGen
Production mesh for Unreal/Unity	Static 3D generation	HY-World 2.0, Matrix-3D, WorldGen
Single prop from a photo	Image-to-3D	Hunyuan3D-2, TRELLIS.2, TripoSG
Movie set reconstruction	Multi-view → PCD → pipeline	MoGe + HY-World adapter + 3DGS
Research on world dynamics	World model training	GenieRedux, open-genie, tinyworlds

The through-line: match the output format to the downstream consumer. If the CG team needs meshes, start with mesh-generation systems. If the goal is a playable demo, video world models are the right tool. Trying to extract production geometry from a pure video world model is fighting the architecture.

Closing Thoughts

Building 3D worlds sits at the intersection of computer vision, generative AI, and computer graphics — three fields that have historically optimized for different things. Vision cares about reconstruction accuracy. Generative AI cares about perceptual quality. Graphics cares about editable, efficient assets.

The best pipelines today are hybrids: they use generative models where reconstruction is impossible (inpainting holes, synthesizing unseen angles) and explicit 3D representations where precision matters (camera poses, point clouds, meshes). Neither pure video generation nor pure classical reconstruction wins alone.

The gap between a cool research demo and a set a CG team can ship is still wide — but it is narrowing. Trajectory planning is improving. Diffusion inpainting is getting better conditioned. Mesh extraction and remeshing tools are maturing. And the open-source ecosystem means anyone with a GPU can run the full stack and push on the remaining problems.

The question is no longer “Can we generate a 3D world?” It is “Can we generate one that is consistent, editable, and good enough to build on?” That is the problem worth working on.