DSAC++

Brachmann 2018 · Paper

One-line summary — “Learning less is more”: strips the DSAC pipeline down to a single learnable component — an FCN regressing scene coordinates — scored by a parameter-free soft inlier count, and shows it can be trained from RGB images and ground-truth poses alone, discovering scene geometry without any 3D model.

Problem

DSAC had three shortcomings. Its scoring CNN overfits — it memorizes where reprojection errors occur in the image, a pattern that does not generalize to unseen views. Its initialization requires scene coordinate ground truth from RGB-D data or a 3D scene model, which may not exist (especially outdoors, where reconstructions demand tedious parameter search). And its end-to-end training was unstable because pose-refinement gradients were computed by finite differences, giving high gradient variance. DSAC++ asks whether learning a single component of the localization pipeline suffices — and whether it can be learned from camera poses alone.

Method & architecture

The DSAC-style pipeline is kept — scene coordinate regression, hypothesis sampling via PnP from 4-tuples, probabilistic selection, refinement — but with only one learnable component:

• Fully convolutional coordinate network (VGG-style, ~30M params): maps a 640x480 image to 80x60 scene coordinates $\mathbf{y}_i(\mathbf{w})$ with 41x41 px receptive fields — dense predictions reusing computation, unlike DSAC’s independent patches.
• Soft inlier count replaces the Score CNN: hypothesis consensus is a sigmoid-smoothed inlier count over reprojection errors $r_i(\mathbf{h},\mathbf{w})=\lVert C\mathbf{h}^{-1}\mathbf{y}_i(\mathbf{w})-\mathbf{p}_i\rVert$:

$$s(\mathbf{h})=\sum_i \mathrm{sig}\left(\tau-\beta\, r_i(\mathbf{h},\mathbf{w})\right)$$

A hypothesis $j$ is selected via the softmax $P(j;\mathbf{w},\alpha) \propto \exp(\alpha\, s(\mathbf{h}_j(\mathbf{w})))$, where $\alpha$ is auto-tuned by gradient descent on $|S(\alpha)-S^{*}|$, with $S(\alpha)$ the Shannon entropy of $P$ — keeping the distribution broad so end-to-end training stays stable instead of collapsing.

• Three-step training: (1) initialize coordinates on targets $\mathbf{y}^{*}$ — rendered from a 3D model if available, otherwise the constant-depth heuristic $\mathbf{y}_i^{*} \approx \mathbf{h}^{*}(\frac{d x_i}{f}, \frac{d y_i}{f}, d, 1)^T$ that merely disentangles camera views; (2) optimize reprojection error against ground-truth poses, $\tilde{\mathbf{w}}=\arg\min_{\mathbf{w}}\sum_i r_i(\mathbf{h}^{*},\mathbf{w})$ — single-view constraints from which the network recovers true depth, i.e. discovers scene geometry with no 3D model; (3) end-to-end, minimizing the expected pose loss $\mathbb{E}_{j\sim P}[\ell(\mathbf{R}(\mathbf{h}_j(\mathbf{w})),\mathbf{h}^{*})]$ as in DSAC.
• Analytic refinement gradients: refinement is Gauss-Newton on inlier reprojection errors, $\mathbf{R}^{t+1}=\mathbf{R}^{t}-(J_{\mathbf{r}}^{T}J_{\mathbf{r}})^{-1}J_{\mathbf{r}}^{T}\mathbf{r}(\mathbf{R}^{t},\mathbf{w})$, and its gradient is approximated by linearizing around the optimum $\mathbf{h}_\mathrm{O}$: $\frac{\partial}{\partial\mathbf{w}}\mathbf{R}(\mathbf{h})\approx-(J_{\mathbf{r}}^{T}J_{\mathbf{r}})^{-1}J_{\mathbf{r}}^{T}\frac{\partial}{\partial\mathbf{w}}\mathbf{r}(\mathbf{h}_{\mathrm{O}},\mathbf{w})$ — replacing DSAC’s unstable finite differences.

Results

• 7Scenes (% frames within 5 cm / 5 deg, complete set): 76.1% with a 3D model — +13.6% over DSAC trained on RGB-D; DSAC trained from a rendered model drops another 6.6%, so DSAC++ without any 3D model (60.4%) still beats model-trained DSAC by 4.5%.
• 12Scenes: 96.4% with a 16.7% margin over DSAC; 60.9% without a 3D model, comparable to a SIFT+PnP baseline.
• Median errors: e.g. 7Scenes Chess 0.02 m / 0.5 deg; Cambridge King’s College 0.18 m / 0.3 deg, Great Court 0.40 m / 0.2 deg — roughly 10x better than the PoseNet variants on many scenes and about 2x better than feature-based Active Search; without a 3D model it still surpasses most model-based competitors (e.g. King’s College 0.23 m / 0.4 deg). Like DSAC, it fails on the order-of-magnitude larger Cambridge Street scene.
• Ablation: swapping only DSAC’s scoring CNN for the soft inlier count lifts 7Scenes 55.9% to 58.9% (Heads +19%, Stairs +8%) and 12Scenes 79.7% to 89.6% — score regression generalizes poorly, coordinate regression well.

Why it matters for SLAM

DSAC++ removed the biggest practical barrier to scene coordinate regression — the need for 3D ground truth — making the approach compatible with the output of any SLAM/SfM system that provides poses. Its recipe (single FCN + classical geometric solving, soft inlier scoring, analytic Gauss-Newton gradients, reprojection-based self-supervision) became the template adopted by DSAC* and the ACE family, and it demonstrated early that an implicit neural scene map can be discovered from single-view reprojection constraints alone — an idea that reappears throughout neural mapping.

Related

• DSAC — the original differentiable-RANSAC formulation
• DSAC* — unified and further improved framework (TPAMI)
• ACE — accelerates SCR training from hours to minutes
• ACE Zero — removes even the pose supervision, learning maps from scratch
• PoseNet — the absolute pose regression approach SCR outperforms