Depth Anything 3

Lin 2025 · Paper

One-line summary — A single plain transformer (vanilla DINOv2, no architectural specialization) trained on one minimal target — per-view depth maps plus per-pixel ray maps — that recovers consistent geometry and camera poses from any number of views, with or without known poses, beating VGGT by 35.7% in pose accuracy and 23.6% in geometric accuracy on average while also surpassing Depth Anything V2 at monocular depth.

Problem

Monocular depth, SfM, MVS, and SLAM all recover 3D structure from images, differing often by just the number of input views — yet the field builds a specialized model for each. Recent unified models (DUSt3R, VGGT) suffer key limitations: complex bespoke architectures, joint multi-task optimization from scratch, and an inability to fully leverage large-scale pretrained backbones. DA3 asks two minimal-modeling questions and answers both affirmatively: (1) is there a minimal set of prediction targets, or is joint modeling across numerous 3D tasks necessary? (2) can a single plain transformer suffice? A second obstacle is data: real-world depth (COLMAP, active sensors) is noisy and incomplete, while clean synthetic depth alone does not generalize camera pose estimation to the real world.

Method & architecture

Depth-ray representation. For $N_v$ input images the model predicts a depth map $\mathbf{D}_i$ and a ray map $\mathbf{M} \in \mathbb{R}^{H \times W \times 6}$ per view — each pixel stores a ray origin $\mathbf{t}$ and unnormalized direction $\mathbf{d} = \mathbf{R}\mathbf{K}^{-1}\mathbf{p}$ (magnitude preserves projection scale), sidestepping the orthogonality constraint of regressing $\mathbf{R}$ directly. A world point is then just an element-wise combination:

$$\mathbf{P} = \mathbf{t} + \mathbf{D}(u,v) \cdot \mathbf{d}$$

Camera parameters are recoverable from the ray map: the center is the mean of ray origins, and since $\mathbf{d}_{\text{cam}} = \mathbf{K}\mathbf{R}\,\mathbf{p}$ defines a homography $\mathbf{H} = \mathbf{K}\mathbf{R}$, solving $\mathbf{H}^{*} = \arg\min_{\lVert\mathbf{H}\rVert=1} \sum_{h,w} \lVert \mathbf{H}\mathbf{p}_{h,w} \times \mathbf{M}(h,w,3{:}) \rVert$ by DLT and RQ-decomposing yields $\mathbf{K}, \mathbf{R}$. Because that is costly at inference, a lightweight camera head predicts FOV $\mathbf{f} \in \mathbb{R}^2$, quaternion $\mathbf{q} \in \mathbb{R}^4$, translation $\mathbf{t} \in \mathbb{R}^3$ from one camera token per view (~0.1% of backbone compute). Ablations show depth+ray nearly doubles Auc3 over depth+camera and beats point-map or redundant multi-target setups.

• Single transformer backbone: a pretrained ViT (vanilla DINOv2) with input-adaptive cross-view self-attention — no architectural change, just token rearrangement. The first $L_s$ layers attend within each image; the remaining $L_g$ layers alternate cross-view and within-view attention ($L_s : L_g = 2{:}1$). With one image it naturally reduces to a monocular depth model at no extra cost.
• Camera condition injection: each view is prepended a camera token — an MLP encoding $\mathcal{E}_c(\mathbf{f}_i, \mathbf{q}_i, \mathbf{t}_i)$ when poses are known, a shared learnable token otherwise — so posed and unposed inputs are handled by one model (conditioning activated with probability 0.2 in training).
• Dual-DPT head: depth and ray branches share the reassembly modules and differ only in fusion layers, encouraging interaction between the two predictions; ablating it to two separate DPT heads consistently hurts.
• Teacher-student training: a monocular relative-depth teacher (DINOv2 + DPT, trained purely on synthetic data, predicting exponential depth rather than DA2’s disparity) generates dense pseudo-labels for real-world data, aligned to sparse/noisy metric depth by RANSAC least-squares scale-shift: $(\hat{s},\hat{t}) = \arg\min_{s>0,t} \sum_p m_p (s\tilde{\mathbf{D}}_p + t - \mathbf{D}_p)^2$. This preserves pose-depth coherence while adding DA2-level detail.
• Loss: normalized by the mean L2 norm of valid points, $\mathcal{L} = \mathcal{L}_D + \mathcal{L}_M + \mathcal{L}_P + \beta \mathcal{L}_C + \alpha \mathcal{L}_{\text{grad}}$ with $\alpha = \beta = 1$ — confidence-weighted L1 terms on depth, ray map, and the composed point map, plus camera and depth-gradient losses.

Trained on public academic datasets only: 128 H100 GPUs, 200k steps, base resolution 504x504, 2-18 views per sample, switching from ground-truth to teacher labels at 120k steps. The same recipe yields a monocular student, a metric-depth model, and a feed-forward 3DGS variant (extra GS-DPT head predicting pixel-aligned Gaussian parameters).

Results

• New visual geometry benchmark (HiRoom, ETH3D, DTU, 7Scenes, ScanNet++ — 89+ scenes): state of the art on 18 of 20 settings; on average +35.7% camera pose accuracy and +23.6% geometric accuracy over prior SOTA VGGT (abstract).
• Pose (Auc3): DA3-Giant (1.10B) 80.3 on HiRoom vs VGGT (1.19B) 49.1 and Pi3 67.0; 85.0 vs 62.6 on ScanNet++ (33% relative gain over the second best); at least 8% relative Auc3 improvement over all baselines.
• Reconstruction: +25.1% relative over VGGT and +21.5% over Pi3 on average; DA3-Large (0.30B backbone), 3x smaller than VGGT, still beats it in 5 of 10 settings.
• Monocular depth ($\delta_1$): DA3 scores 95.3 KITTI / 97.4 NYU / 75.5 SINTEL / 98.6 ETH3D / 95.4 DIODE, outranking DA2 (94.6 / 97.9 / 77.2 / 86.5 / 95.2); the monocular student reaches 97.1 / 98.0 / 82.3 / 98.8 / 96.5, over 10% better than DA2 on ETH3D. DA3-metric is SOTA on ETH3D metric depth ($\delta_1$ 0.917 vs UniDepthv2’s 0.863).
• Feed-forward NVS: fine-tuned DA3 hits 21.33 PSNR on DL3DV vs 20.96 for a VGGT backbone and 19.24 for specialized DepthSplat — geometry quality directly transfers to rendering quality.
• Efficiency: 37.6 FPS per image (DA3-Giant) vs VGGT’s 34.1 on an A100 at 504x336; handles 900-1000 images on one 80 GB A100 vs VGGT’s 400-500 (DA3-Large: 78.4 FPS, 1500-1600 images). A VGGT-style dual-transformer of comparable size drops to 79.8% of the single-backbone performance — the gap attributed to full pretraining vs two-thirds untrained blocks.

Why it matters for SLAM

DA3 merges the two deep-geometry threads this roadmap tracks. As the successor to Depth Anything V2, it is a stronger drop-in monocular depth prior for dense mono SLAM — and its metric variant addresses the scale problem directly. As a DUSt3R/VGGT-style any-view model, it emits exactly the front-end quantities a SLAM system needs — consistent per-view depth plus camera poses fusable into a clean point cloud in one forward pass — with better pose accuracy and roughly double the view capacity of VGGT, which matters for submap-based systems like VGGT-SLAM (the paper cites such SLAM applications as consumers of this model family). The camera-token conditioning is notable for SLAM specifically: a tracker’s pose estimates can be injected to improve geometry rather than being re-estimated from scratch. Honest caveats: DA3 is feed-forward reconstruction, not SLAM — there is no loop closure, relocalization, or global bundle adjustment, outputs are scale-normalized rather than metric (except the metric variant), and dynamic scenes are explicitly deferred to future work.

Related

• Depth Anything V2 — the monocular predecessor whose synthetic-teacher / pseudo-labeled-student recipe DA3 generalizes to any-view geometry
• DUSt3R — origin of feed-forward pointmap regression; DA3 argues depth+ray is the better minimal target
• VGGT — the prior any-view SOTA with a multi-stage architecture and redundant targets that DA3’s plain transformer surpasses