ZoeDepth

Bhat 2023 · Paper

One-line summary — Combines relative-depth pre-training (MiDaS/DPT style) with a lightweight Metric Bins Module to produce zero-shot metric depth, giving monocular SLAM a depth prior with absolute scale.

Problem

Monocular depth research had split into two camps: relative depth models (MiDaS, DPT) that generalize across domains but output depth only up to an unknown scale and shift, and metric depth models that predict absolute distances but overfit to the single camera/dataset they were trained on. SLAM needs both properties at once — meters, everywhere. Training a single metric model across datasets with very different depth scales (indoor ~10 m vs. outdoor ~80 m) usually deteriorates performance or diverges outright.

Method & architecture

Two-stage training. Stage 1 pre-trains a MiDaS-style encoder-decoder (DPT architecture with a BEiT384-L transformer encoder) for relative depth on the M12 mix of 12 datasets using the scale-shift-invariant multi-task loss. Stage 2 attaches one or more lightweight metric heads (each <1% of the backbone’s parameters) to the decoder and fine-tunes end-to-end on metric datasets (NYU Depth v2 and/or KITTI).

Metric bins module. The head hooks into the bottleneck plus four decoder levels (at 1/32, 1/16, 1/8, 1/4, 1/2 resolution). Following the adaptive-bins idea, depth at pixel $i$ is a probability-weighted combination of per-pixel bin centers $c_i(k)$:

$$d(i) = \sum_{k=1}^{N_{total}} p_i(k)\, c_i(k)$$

Unlike LocalBins (which starts with few seed bins and splits them at each decoder layer), ZoeDepth predicts all $N_{total}=64$ bin centers at the bottleneck and adjusts them at each decoder level with attractor layers: an MLP predicts $n_a$ attractor points $\{a_k\}$ per pixel, and each bin center moves by $c_i' = c_i + \Delta c_i$ with the inverse attractor

$$\Delta c_i = \sum_{k=1}^{n_a} \frac{a_k - c_i}{1 + \alpha |a_k - c_i|^{\gamma}}$$

where $\alpha, \gamma$ set attractor strength and $\{n_a^l\} = \{16, 8, 4, 1\}$ across decoder layers. Attracting is contracting (refinement without the sum-of-widths constraints splitting imposes).

Log-binomial probabilities. Because bins are ordered, the probability over bins is not a plain softmax but a binomial distribution with predicted mode $q$ and temperature $t$:

$$p(k; N, q) = \binom{N}{k} q^k (1-q)^{N-k}$$

computed in log space with Stirling’s approximation and normalized by $\mathrm{softmax}(\log(p_k)/t)$, preserving unimodality. Pixel supervision uses the scale-invariant log loss.

Domain heads + router. Separate metric heads act as indoor/outdoor experts over a shared backbone; a router MLP on bottleneck features picks the head per image. Three variants — labeled, trained, and auto router — allow deployment without any scene-type labels at inference.

Results

On the NYU Depth v2 benchmark: ZoeD-X-N (no relative pre-training) reaches REL 0.082, beating the prior SOTA NeWCRFs (REL 0.095) by 13.7% — validating the metric-bins design alone. ZoeD-M12-N (M12 pre-training, NYU fine-tuning) reaches REL 0.075, a total 21% improvement over NeWCRFs. Trained jointly on NYU + KITTI, the flagship two-head ZoeD-M12-NK holds REL 0.077 on NYU (only 2.6% off the NYU-only model), whereas NeWCRFs degrades ~15% (0.095 to 0.109) and AdaBins/PixelBins fail to converge in the joint setting; the single-head variant drops only 8% (0.081). In zero-shot transfer to eight unseen datasets, ZoeD-M12-NK improves the mean relative metric (mRI over $\delta_1$, REL, RMSE vs. NeWCRFs) by 5.3% (HyperSim) to 46.3% (DIODE Indoor) indoors and by 7.8% (Virtual KITTI 2) up to 976.4% (~11x, DIML Outdoor) outdoors, losing only on DDAD (−12.8%). Code and pre-trained models are public.

Why it matters for SLAM

Monocular SLAM is scale-ambiguous; relative-depth networks like MiDaS cannot fix that because their output is only defined up to an affine transform. ZoeDepth was the first practical model to deliver metric depth zero-shot across domains, so a single network can supply scale for monocular SLAM initialization, densification, or scale-drift correction. Its relative-then-metric training paradigm was adopted by successors such as Metric3D and Depth Anything, making ZoeDepth a key link between the relative-depth foundation models and metric-scale robotics use.

Related

• MiDaS — the multi-dataset relative-depth backbone and training recipe.
• DPT — the ViT-based dense prediction architecture ZoeDepth builds on.
• Metric3D — alternative canonical-camera route to zero-shot metric depth.
• Depth Anything — scaling up depth foundation models; uses the ZoeDepth framework for its metric fine-tuning.
• Depth from sensor — what metric depth networks aim to replace or complement.
• Scale ambiguity — the monocular SLAM problem a metric depth prior addresses.