LERF

Kerr 2023 · Paper

One-line summary — Language Embedded Radiance Fields ground CLIP features inside a NeRF via volume rendering, enabling pixel-aligned, zero-shot 3D queries from open-ended natural-language prompts.

Problem

Humans refer to 3D locations through language covering “a vast range of properties: visual appearance, semantics, abstract associations, or actionable affordances” — “the yellow mug”, “something to write with”. CLIP can score such prompts against images, but it is “inherently a global image embedding and not conducive to pixel-aligned feature extraction”: one embedding per crop, no 3D structure. Prior open-vocabulary 3D work leaned on region proposals, masks, or fine-tuned detectors (LSeg, OWL-ViT), restricting queries to categories those detectors were trained on. LERF asks how to ground raw CLIP embeddings volumetrically in a NeRF so arbitrary language resolves to 3D locations.

Method & architecture

• A field over volumes, not points. Querying CLIP at a single 3D point is ambiguous, so LERF learns a field $F_{\mathrm{lang}}(\vec{x}, s) \in \mathbb{R}^{d}$ over volumes: position $\vec{x}$ plus a physical scale $s$ (side length of a cube centered at $\vec{x}$). The output is defined as the average CLIP embedding, across training views, of image crops containing that volume. The field is view-independent, so multiple views average into the same embedding.
• Volume rendering of language. Along a ray $\vec{r}(t) = \vec{o} + t\vec{d}$, the scale grows with distance, $s(t) = s_{\mathrm{img}} \cdot f_{xy} / t$ (a frustum), and embeddings are composited with the standard NeRF weights $w(t)$ from $T(t) = \int_t \exp(-\sigma(s)\,ds)$, $w(t) = \int_t T(t)\,\sigma(t)\,dt$:

$$\hat{\phi}_{\mathrm{lang}} = \int_t w(t)\, F_{\mathrm{lang}}\big(r(t), s(t)\big)\, dt, \qquad \phi_{\mathrm{lang}} = \hat{\phi}_{\mathrm{lang}} \big/ \lVert \hat{\phi}_{\mathrm{lang}} \rVert .$$

• Multi-scale CLIP supervision. A precomputed image pyramid of CLIP embeddings over crops (scales $s_{\min}=0.05$ to $s_{\max}=0.5$ in 7 steps, 50% overlap) supervises each rendered frustum; ground truth $\phi_{\mathrm{lang}}^{\mathrm{gt}}$ is trilinearly interpolated from the 4 nearest crops at the two adjacent scales. The loss maximizes cosine similarity:

$$L_{\mathrm{lang}} = -\lambda_{\mathrm{lang}}\, \phi_{\mathrm{lang}} \cdot \phi_{\mathrm{lang}}^{\mathrm{gt}}, \qquad \lambda_{\mathrm{lang}} = 0.01 .$$

• DINO regularization. A second head $F_{\mathrm{dino}}(\vec{x})$ predicts pixel-aligned DINO features (MSE loss, no scale input). DINO is never used at inference; sharing a backbone with the CLIP head regularizes the language field, fixing patchy, outlier-prone relevancy maps in sparsely viewed regions.
• Two separate hashgrids. Language optimization “should not influence the distribution of density”: one multi-resolution hashgrid (Instant-NGP style, 32 levels, table size $2^{21}$) feeds CLIP/DINO MLPs; a separate Nerfacto field handles color/density. $L_{\mathrm{lang}}$ and $L_{\mathrm{dino}}$ gradients never touch the NeRF outputs.
• Querying. A text prompt $\phi_{\mathrm{quer}}$ is scored against canonical phrases (“object”, “things”, “stuff”, “texture”) via a pairwise softmax relevancy score

$$\min_i \; \frac{\exp(\phi_{\mathrm{lang}} \cdot \phi_{\mathrm{quer}})}{\exp(\phi_{\mathrm{lang}} \cdot \phi_{\mathrm{canon}}^{i}) + \exp(\phi_{\mathrm{lang}} \cdot \phi_{\mathrm{quer}})},$$

and the scale $s$ is auto-selected by sweeping 0–2 m in 30 increments and keeping the highest-scoring one. Samples seen by fewer than 5 training views are discarded.

Results

• Setup: 13 in-the-wild handheld scenes (grocery store, kitchen, bookstore, teatime, figurines…) captured with Polycam on an iPhone at 994×738; OpenCLIP ViT-B/16 (LAION-2B); 30k steps ≈ 45 min on one A100 (~20 GB), usable results at 6k steps (8 min); querying is interactive/real-time in the Nerfstudio viewer.
• Localization (72 labeled objects across 5 scenes; success = highest-relevancy pixel inside the ground-truth box): LERF 80.3% overall vs. OWL-ViT 54.8% and LSeg distilled into 3D (DFF) 18.0%. Per scene: waldo_kitchen 81.5/42.6/13.0, bouquet 91.7/66.7/50.0, teatime 93.8/75.0/28.1, figurines 79.5/38.5/8.9 (LERF/OWL-ViT/LSeg); OWL-ViT wins only on ramen (92.5 vs 62.5).
• Existence determination (81 long-tail queries over 5 scenes): precision–recall curves show LSeg matches LERF only on in-distribution COCO labels and collapses on long-tail queries.
• Ablations: removing DINO degrades relevancy-map smoothness and boundaries; single-scale training (fixed 15% crop) fails on both large-context (“espresso machine”) and small (“creamer pods”) queries.
• Limitations: CLIP bag-of-words behavior (“not red” ≈ “red”), false positives on visually similar objects (zucchini vs other green vegetables), needs calibrated NeRF-quality multi-view capture.

Why it matters for SLAM

LERF established the paradigm of embedding vision-language features directly into a 3D scene representation — the semantic layer of Spatial AI: maps you can talk to. It directly inspired language-embedded Gaussian splatting (LEGS, LangSplat) and complements fusion-style approaches like ConceptFusion; its multi-scale CLIP + DINO recipe became the default for distilling 2D vision-language features into 3D fields. For robotics, “find something to write with” resolved to a 3D location is exactly what turns a SLAM map into an actionable world model.

Related

• LEGS
• ConceptFusion
• OpenScene
• NeRF
• CLIP