DreamFusion

Poole 2023 · Paper

One-line summary — DreamFusion uses a pretrained 2D text-to-image diffusion model (Imagen) as a frozen prior to optimize a NeRF from a text prompt alone, via the Score Distillation Sampling (SDS) loss, enabling text-to-3D generation without any 3D training data.

Problem

Text-to-image synthesis was transformed by diffusion models trained on billions of image-text pairs, but adapting the recipe to 3D would require two things that did not exist: large-scale labeled 3D data and efficient architectures for denoising 3D data directly. NeRF had shown a 3D scene can be a network optimized purely through image-space losses on its renderings. DreamFusion asks whether a frozen 2D diffusion model can be that image-space loss — sampling in NeRF parameter space rather than pixel space, so all 3D knowledge is distilled out of a purely 2D prior.

Method & architecture

Diffusion prior. A text-conditioned diffusion model is trained with the weighted denoising objective $$\mathcal{L}_{\text{Diff}}(\phi, \mathbf{x}) = \mathbb{E}_{t \sim \mathcal{U}(0,1),\, \epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})}\big[ w(t)\, \| \epsilon_\phi(\alpha_t \mathbf{x} + \sigma_t \epsilon; t) - \epsilon \|_2^2 \big],$$ and DreamFusion uses its classifier-free-guided noise prediction $\hat{\epsilon}_\phi(\mathbf{z}_t; y, t) = (1+\omega)\,\epsilon_\phi(\mathbf{z}_t; y, t) - \omega\,\epsilon_\phi(\mathbf{z}_t; t)$ with an unusually large guidance weight $\omega = 100$.

Score Distillation Sampling. A differentiable image parameterization $\mathbf{x} = g(\theta)$ (here: a NeRF renderer) is optimized so its renders look like samples. Differentiating $\mathcal{L}_{\text{Diff}}$ through the U-Net is expensive and poorly conditioned; omitting the U-Net Jacobian gives the SDS gradient $$\nabla_\theta \mathcal{L}_{\text{SDS}}(\phi, \mathbf{x} = g(\theta)) \triangleq \mathbb{E}_{t,\epsilon}\Big[ w(t)\big(\hat{\epsilon}_\phi(\mathbf{z}_t; y, t) - \epsilon\big) \tfrac{\partial \mathbf{x}}{\partial \theta} \Big],$$ which the paper proves is the gradient of a weighted probability-density-distillation loss, $\nabla_\theta\, \mathbb{E}_t\big[ (\sigma_t / \alpha_t)\, w(t)\, \text{KL}\big(q(\mathbf{z}_t \mid g(\theta); y, t)\ \|\ p_\phi(\mathbf{z}_t; y, t)\big)\big]$. No backpropagation through the diffusion model is needed — it acts as a frozen critic whose noise residual points toward higher-density images for prompt $y$.

NeRF with shading. The 3D canvas is a mip-NeRF 360 variant whose MLP outputs density and albedo, $(\tau, \boldsymbol{\rho}) = \text{MLP}(\boldsymbol{\mu}; \theta)$, composited with standard volume-rendering weights $w_i = \alpha_i \prod_{j<i}(1-\alpha_j)$, $\alpha_i = 1 - \exp(-\tau_i\|\boldsymbol{\mu}_i - \boldsymbol{\mu}_{i+1}\|)$. Surface normals come from the density gradient, $\mathbf{n} = -\nabla_{\boldsymbol{\mu}}\tau / \|\nabla_{\boldsymbol{\mu}}\tau\|$, and each point is Lambertian-shaded by a randomly placed point light $\boldsymbol{\ell}$: $$\mathbf{c} = \boldsymbol{\rho} \circ \big(\boldsymbol{\ell}_\rho \circ \max(0,\ \mathbf{n} \cdot (\boldsymbol{\ell} - \boldsymbol{\mu}) / \|\boldsymbol{\ell} - \boldsymbol{\mu}\|) + \boldsymbol{\ell}_a\big).$$ Randomly swapping the albedo for white gives “textureless” renders, preventing degenerate flat-billboard solutions where scene content is painted onto flat geometry.

Per-prompt optimization loop. Each iteration: (1) sample a random camera (elevation $-10°$ to $90°$, full azimuth, distance 1–1.5) and light; (2) render the shaded NeRF at 64×64, choosing among illuminated, albedo-only, and textureless renders; (3) append view-dependent text (“front/side/back/overhead view”) to the prompt and compute the SDS gradient with the frozen 64×64 Imagen base model ($w(t)=\sigma_t^2$, $t \sim \mathcal{U}(0.02, 0.98)$); (4) update NeRF weights with Distributed Shampoo. 15,000 iterations take ~1.5 hours on a 4-chip TPUv4; opacity and orientation regularizers keep the density field clean.

Results

Evaluated with CLIP R-Precision (can CLIP retrieve the correct caption from a rendering?) on the 153 object-centric COCO prompts of Dream Fields, over both color and textureless geometry (“Geo”) renders:

DreamFusion approaches ground-truth-image consistency on color renders while being the only method with strong geometry scores (CLIP-trained baselines collapse to ~1 Geo, revealing texture painted on flat shapes). Ablations show viewpoint augmentation, view-dependent prompts, lighting, and textureless renders each improve geometry, with full renders improving by +12.5%. The optimized models can be viewed from any angle, relit, and composited into 3D environments. Known limitations from the paper: SDS is mode-seeking, yielding oversaturated/oversmoothed results and little diversity across seeds.

Why it matters for SLAM

DreamFusion connected two previously separate threads — neural rendering (NeRF) and generative diffusion models — and its SDS loss was immediately reused in a large family of follow-up text-to-3D works (Magic3D, ProlificDreamer, and others). For SLAM, it established the idea that a generative 2D prior can hallucinate plausible 3D structure: the same mechanism can, in principle, complete texture and geometry in the unobserved regions of a SLAM map, a recurring theme in generative map completion and Spatial AI research.

Related

• NeRF
• World Labs / Marble
• Sora / DiT
• Spatial AI
• World model