DUSt3R

Wang 2024 · Paper

One-line summary — Recast pairwise 3D reconstruction as direct regression of dense pointmaps from an image pair by a feed-forward Transformer, requiring no camera calibration, feature matching, or explicit geometric models.

Problem

Multi-view stereo in the wild requires first estimating camera intrinsics and extrinsics, which are tedious to obtain yet mandatory to triangulate corresponding pixels in 3D. The whole classical pipeline — calibrate, detect, match, estimate poses, triangulate — is a chain of fragile stages, each of which can fail. DUSt3R (“Dense and Unconstrained Stereo 3D Reconstruction”) takes the opposite stance: reconstruct arbitrary image collections with no prior information about calibration or viewpoint poses, by regressing 3D structure directly.

Method & architecture

Pointmap representation: a network F\mathcal{F} takes two RGB images I1,I2∈RW×H×3I^1, I^2 \in \mathbb{R}^{W \times H \times 3} and outputs two pointmaps X1,1,X2,1∈RW×H×3X^{1,1}, X^{2,1} \in \mathbb{R}^{W \times H \times 3} — per-pixel 3D coordinates — with confidence maps C1,1,C2,1C^{1,1}, C^{2,1}. Both pointmaps are expressed in camera 1’s frame, so a single forward pass jointly solves calibration, correspondence, and reconstruction. Feeding the same image twice yields monocular depth; the formulation unifies the monocular and binocular cases.

Architecture (CroCo-style, initialized from CroCo pretraining): a weight-sharing Siamese ViT-Large encoder, F1=Encoder(I1)F^1 = \text{Encoder}(I^1), F2=Encoder(I2)F^2 = \text{Encoder}(I^2), then two intertwined ViT-Base decoders in which every block performs self-attention, cross-attention to the other view’s tokens, and an MLP:

Gi1=DecoderBlocki1(Gi−11,Gi−12),Gi2=DecoderBlocki2(Gi−12,Gi−11),G_i^1 = \text{DecoderBlock}_i^1\big(G_{i-1}^1, G_{i-1}^2\big), \qquad G_i^2 = \text{DecoderBlock}_i^2\big(G_{i-1}^2, G_{i-1}^1\big),

with G0v:=FvG_0^v := F^v. A DPT regression head per branch maps all decoder tokens to a pointmap + confidence map. No geometric constraint is ever enforced — the network learns the priors from geometrically consistent training data.

Training objective: scale-normalized 3D regression for each valid pixel ii in view vv,

ℓregr(v,i)=∥1zXiv,1−1zˉXˉiv,1∥,\ell_{\text{regr}}(v,i) = \left\| \frac{1}{z} X_i^{v,1} - \frac{1}{\bar{z}} \bar{X}_i^{v,1} \right\|,

where z,zˉz, \bar{z} are the mean distances of all valid points to the origin, wrapped in a confidence-weighted loss that learns where predictions can be trusted:

Lconf=∑v∈{1,2}∑i∈DvCiv,1 ℓregr(v,i)−αlog⁡Civ,1,Civ,1=1+exp⁡Civ,1~>1.\mathcal{L}_{\text{conf}} = \sum_{v \in \{1,2\}} \sum_{i \in \mathcal{D}^v} C_i^{v,1} \, \ell_{\text{regr}}(v,i) - \alpha \log C_i^{v,1}, \qquad C_i^{v,1} = 1 + \exp \widetilde{C_i^{v,1}} > 1.

Everything falls out as by-products: pixel matches via reciprocal nearest neighbours in 3D pointmap space; the focal length by a Weiszfeld-style minimization of pixel-reprojection residuals over X1,1X^{1,1}; relative pose by Procrustes alignment of X1,1↔X1,2X^{1,1} \leftrightarrow X^{1,2} or PnP-RANSAC; absolute pose (visual localization) by scaling against a reference pointmap.

Global alignment for NN views: build a connectivity graph G(V,E)\mathcal{G}(\mathcal{V}, \mathcal{E}) over image pairs (network inference is roughly 40 ms per pair on an H100), then optimize world pointmaps χn\chi^n, one rigid pose PeP_e and scale σe\sigma_e per edge:

χ∗=arg⁡min⁡χ,P,σ∑e∈E∑v∈e∑i=1HWCiv,e∥χiv−σePeXiv,e∥,∏eσe=1.\chi^* = \arg\min_{\chi, P, \sigma} \sum_{e \in \mathcal{E}} \sum_{v \in e} \sum_{i=1}^{HW} C_i^{v,e} \left\| \chi_i^v - \sigma_e P_e X_i^{v,e} \right\|, \qquad \textstyle\prod_e \sigma_e = 1.

Unlike bundle adjustment this minimizes 3D projection error, not 2D reprojection error, and runs by plain gradient descent — a few hundred steps, mere seconds on a GPU. Substituting a pinhole parametrization for χn\chi^n recovers all poses PnP_n, intrinsics KnK_n, and depthmaps DnD^n.

Training data: 8.5M pairs from eight datasets (Habitat, MegaDepth, ARKitScenes, Static Scenes 3D, Blended MVS, ScanNet++, CO3D-v2, Waymo), trained first at 224 px then at 512 px with varied aspect ratios.

Results

Why it matters for SLAM

DUSt3R started the “3D foundation model” era of geometric vision: a single pre-trained network replacing the classical detect-match-triangulate pipeline, working even on uncalibrated images and as few as two views. It directly spawned MASt3R and MASt3R-SLAM, influenced feed-forward multi-view models like VGGT, and now anchors an entire branch of SLAM research where a learned pointmap regressor is the frontend and classical optimisation is the backend. Knowing its assumptions (offline pairing, global alignment cost, per-pair scale ambiguity) explains what the follow-up systems fix.