Depth Anything

Yang 2024 · Paper

One-line summary — Depth Anything (CVPR 2024) is a foundation model for monocular depth estimation, built by scaling training to ~62M automatically annotated unlabeled images and achieving strong zero-shot generalization to any scene.

Problem

Supervised monocular depth models plateau because labeled depth data is scarce — roughly a million-plus images across public datasets, versus the effectively unlimited pool of unlabeled internet imagery. Depth Anything’s stated aim is deliberately unglamorous: “without pursuing novel technical modules”, build “a simple yet powerful foundation model dealing with any images under any circumstances” by making large-scale unlabeled data actually useful for depth.

Method & architecture

Base model. A DINOv2-initialized ViT encoder with a DPT decoder, trained in the MiDaS tradition: depth is converted to disparity $d = 1/t$ and normalized to $0 \sim 1$ per map, and multi-dataset training uses the affine-invariant loss

$$\mathcal{L}_l = \frac{1}{HW}\sum_{i=1}^{HW} \rho(d_i^*, d_i), \qquad \rho(d_i^*, d_i) = |\hat{d}_i^* - \hat{d}_i|$$

with per-image alignment $\hat{d}_i = \frac{d_i - t(d)}{s(d)}$, $t(d) = \mathrm{median}(d)$, $s(d) = \frac{1}{HW}\sum_i |d_i - t(d)|$. A teacher $T$ is trained this way on 1.5M labeled images from 6 datasets (BlendedMVS, DIML, HRWSI, IRS, MegaDepth, TartanAir); sky regions are segmented and set to zero disparity.

Data engine. The teacher pseudo-labels 62M unlabeled images from 8 public sources (SA-1B, ImageNet-21K, LSUN, BDD100K, Google Landmarks, Objects365, Open Images V7, Places365): $\hat{\mathcal{D}}^u = \{(u_i, T(u_i)) \mid u_i \in \mathcal{D}^u\}$. A student $S$ is re-initialized (not fine-tuned from $T$) and trained on labeled + pseudo-labeled data at a 1:2 ratio per batch.

Challenging the student. Naive self-training added nothing — teacher and student share architecture and make similar errors. Two strategies unlock the unlabeled data. (1) Strong perturbations on the student’s unlabeled inputs only: heavy color jittering and Gaussian blur, plus spatial CutMix, $u_{ab} = u_a \odot M + u_b \odot (1-M)$ with rectangular mask $M$ (50% probability); the unlabeled loss $\mathcal{L}_u$ is the area-weighted sum of affine-invariant losses against $T(u_a)$ and $T(u_b)$ inside $M$ and $1-M$ respectively, while the teacher always sees clean images. This compels the model “to actively seek extra visual knowledge and acquire robust representations”. (2) Semantic priors via feature alignment rather than an auxiliary segmentation task (which failed — discrete masks lose too much):

$$\mathcal{L}_{feat} = 1 - \frac{1}{HW}\sum_{i=1}^{HW} \cos(f_i, f_i')$$

aligning student features $f$ with a frozen DINOv2 encoder $f'$, but skipping pixels whose similarity already exceeds a tolerance margin $\alpha = 0.85$ — objects share semantics but not depth, so exact feature copying would hurt. The total loss averages $\mathcal{L}_l$, $\mathcal{L}_u$ and $\mathcal{L}_{feat}$.

Model family. ViT-S (24.8M), ViT-B (97.5M), ViT-L (335.3M) encoders trade accuracy against speed; metric variants fine-tune the encoder in the ZoeDepth framework on NYUv2 or KITTI.

Results

Zero-shot relative depth on six unseen datasets, versus the best MiDaS v3.1 model (ViT-L class): Depth Anything ViT-L wins everywhere — KITTI AbsRel 0.076 vs 0.127 ($\delta_1$ 0.947 vs 0.850), NYUv2 0.043 vs 0.048, Sintel 0.458 vs 0.587, DDAD 0.230 vs 0.251, ETH3D 0.127 vs 0.139, DIODE 0.066 vs 0.075 — and MiDaS wasn’t even fully zero-shot on KITTI/NYUv2 (it trains on their images; Depth Anything does not). The ViT-B model already beats MiDaS’s larger ViT-L, and even ViT-S (<1/10 the size) wins on Sintel, DDAD and ETH3D. Fine-tuned for in-domain metric depth (ZoeDepth framework), it lifts NYUv2 $\delta_1$ from 0.964 to 0.984 and AbsRel from 0.069 to 0.056 over the previous best (VPD), and KITTI $\delta_1$ from 0.978 to 0.982; the paper also reports stronger zero-shot metric transfer, strong semantic segmentation from the same encoder, and a better depth-conditioned ControlNet.

Why it matters for SLAM

Monocular SLAM has long wanted a depth prior that works everywhere — for scale, map density, initialization, and robustness in low-parallax motion. Depth Anything made “just call a depth foundation model” a realistic design choice: a single network that yields plausible dense depth on arbitrary indoor/outdoor imagery, with variants fast enough for real-time front-ends. It cemented the data-scaling paradigm for geometric perception — diverse unlabeled data beats more supervised labels — and (with V2) became the default depth backbone plugged into dense and neural SLAM pipelines.

Related

• MiDaS — the affine-invariant multi-dataset training recipe it inherits
• DPT — the dense-prediction decoder architecture
• Depth Anything V2 — successor trained on synthetic labels
• Metric3D — camera-aware route to metric depth
• Marigold — diffusion-based alternative for zero-shot depth
• ZoeDepth — the metric fine-tuning framework used for its metric variants