Visual-SLAM why filter?

Strasdat 2012 · Paper

One-line summary — Rigorously showed that keyframe-based bundle adjustment gives more accuracy per unit of computing time than filtering, formalizing visual SLAM’s paradigm shift from filtering to optimization.

Problem

By 2012 the field had two working recipes for real-time monocular SLAM: MonoSLAM (2007) jointly estimated camera pose and landmarks with an EKF, while PTAM (2007) introduced keyframe-based bundle adjustment. Both ran in real time, but the community lacked a principled comparison. The two paradigms make opposite structural choices on the SLAM graph: filtering marginalises out all past poses, leaving a dense joint distribution over the current pose and all landmarks, while keyframe BA simply discards non-keyframe poses and their measurements, keeping the problem sparse. Strasdat, Montiel, and Davison asked: for the same computational budget, which choice buys more accuracy?

Method & architecture

Two carefully matched pipelines, both implemented and run in Monte Carlo simulation.

BA-SLAM: at each keyframe, initialise replacement points where features left the view, estimate the pose by motion-only BA, refine points by structure-only BA, then jointly optimise everything (g2o, Levenberg–Marquardt with Schur complement):

\chi^2(\mathbf{y}) = \sum_{\mathbf{z}_{i,j} \in Z_{0:i}} \big(\mathbf{z}_{i,j} - \hat{\mathbf{z}}(\mathbf{T}_i, \mathbf{x}_j)\big)^2, \qquad \chi^2(\mathbf{T}_i) = \sum_{\mathbf{z}_j \in Z_i} \big(\mathbf{z}_j - \hat{\mathbf{z}}(\mathbf{T}_i, \mathbf{x}_j)\big)^2,

over $\mathbf{y} = (\mathbf{T}_1, ..., \mathbf{T}_i, \mathcal{X})^\top$ with the first frame fixed as gauge.

Filter-SLAM: a state-of-the-art Gauss–Newton information filter, deliberately built to be the best case for filtering: points use anchored inverse-depth coordinates $\boldsymbol{\psi}_j := \mathrm{inv\_d}(A_{a(j)}\, \mathbf{x}_j)$ with $\mathrm{inv\_d}(\mathbf{a}) = \frac{1}{a_3}(a_1, a_2, 1)^\top$ , and each update minimises measurement residuals against the Gaussian map prior:

\chi^2(\Phi_i, \mathbf{T}_i) = (\Phi_i \boxminus \Phi_{i-1})^\top \Lambda_{\Phi_{i-1}} (\Phi_i \boxminus \Phi_{i-1}) + \sum_{\mathbf{z}_j \in Z_i} \mathbf{d}_j^\top \Lambda_z \mathbf{d}_j,

followed by the information-matrix update $\Lambda_i = \Lambda_{i-1} + D^\top \mathrm{diag}(\Sigma_z^{-1}, \ldots)\, D$ , where $D$ is the stacked reprojection Jacobian. The defining filter property: the linearisation point of absorbed measurements is fixed forever, whereas BA relinearises everything each iteration.

Metrics. Accuracy is the end-pose translation error, measured as RMSE and as entropy reduction in bits relative to the weakest configuration,

E = \frac{1}{2} \log_2 \frac{\det(\Sigma_{\langle M_{\min},\,15 \rangle})}{\det(\Sigma_{\langle M,N \rangle})},

for varying numbers of intermediate frames $M$ and points $N$ ; efficiency is entropy reduction per second of compute ( $E/c$ , bits per second). Asymptotics: BA costs $O(NM^2 + M^3)$ (Schur complement + reduced camera system), filtering costs $O(MN^3)$ — linear vs cubic in the number of points. Four camera motions are tested (sideways with full scene overlap; sideways with partial overlap; sideways + 30° rotation; sharp forward turn), each monocular and stereo, with $\sigma_z = \frac{1}{2}$ pixel measurement noise.

Results

Landmarks beat frames — the paper’s self-declared “single most important result”: increasing the number of points $N$ significantly reduces entropy in every configuration, while increasing the number of frames $M$ has only a minor accuracy effect (its real role is robustness, e.g. against monocular initialisation failures at $M=2$ ).
The well-parametrised filter matches BA accuracy in the three moderate settings — the difference is cost: since BA is linear in $N$ and filtering cubic, only BA can afford the many landmarks that accuracy demands.
In the sharp-forward-turn monocular setting, filter accuracy degrades below BA because measurement Jacobians are never relinearised — the classic Gaussian-filter consistency problem.
In combined accuracy/cost (bits per second), BA is more efficient in general; filtering is competitive only for low accuracy demands (small $M$ and small $N$ ).
Conclusion, verbatim in spirit: keyframe BA outperforms filtering because it gives the most accuracy per unit of computing time.

Why it matters for SLAM

This paper provided the theoretical justification for the shift that PTAM demonstrated empirically, cementing keyframe-based bundle adjustment as the standard visual SLAM backend. ORB-SLAM, LSD-SLAM, DSO, and essentially all subsequent systems are built on its conclusion, and its accuracy-versus-budget methodology became a standard way to argue SLAM design choices. Note the nuance that matters today: the argument is about vision-only SLAM with many landmarks — tightly-coupled VIO systems still use filters (e.g. MSCKF) where the trade-offs differ, and the authors themselves scoped the claim to local SLAM with loop closure left to appearance-based methods.

MonoSLAM — the filtering side of the comparison
PTAM — the keyframe-BA side of the comparison
ORB-SLAM — the archetypal system built on this conclusion
Filter-based vs Optimization-based — how the trade-off plays out in VIO
Schur complement / Sparsity — why BA scales so well
MAP inference as sparse nonlinear least squares — the estimation view underlying the whole argument

Visual-SLAM why filter?

Problem

Method & architecture

Results

Why it matters for SLAM

Related