MSCKF

Mourikis 2007 · Paper

One-line summary — The Multi-State Constraint Kalman Filter achieves efficient monocular VIO by keeping a sliding window of camera poses (not landmarks) in the EKF state and projecting feature measurements onto the left null space of the landmark Jacobian, so complexity stays linear in the number of features.

Problem

Classic EKF-SLAM puts every 3D landmark position in the state vector, so covariance updates scale quadratically with map size — prohibitive for vision-aided inertial navigation, where feature extractors routinely track hundreds of points per image. Pairwise alternatives (epipolar constraints, relative-pose measurements between image pairs) discard information and reuse the same pixels in statistically correlated constraints. The question MSCKF answered: can a feature track observed from multiple camera poses constrain the trajectory optimally, without the feature ever becoming a state variable?

Method & architecture

The filter follows a three-step loop (Algorithm 1 of the paper): propagate on every IMU sample, augment on every image, update when feature tracks complete.

• State. The evolving IMU state is $$\mathbf{X}_{\mathrm{IMU}} = \begin{bmatrix} {}^I_G\bar{q}^{\,T} & \mathbf{b}_g^T & {}^G\mathbf{v}_I^T & \mathbf{b}_a^T & {}^G\mathbf{p}_I^T \end{bmatrix}^T,$$ with unit quaternion ${}^I_G\bar{q}$ (global-to-IMU rotation), gyro/accel biases modeled as random walks, and velocity/position in the global frame; orientation error uses the minimal 3-DoF $\delta\boldsymbol{\theta}$ of the error quaternion $\delta\bar q$. The full state appends up to $N_{\max}$ past camera poses $({}^{C_i}_G\bar q,\, {}^G\mathbf{p}_{C_i})$.
• Propagation. IMU estimates are integrated with 5th-order Runge-Kutta; the covariance follows the Lyapunov equation $\dot{\mathbf{P}}_{II} = \mathbf{F}\mathbf{P}_{II} + \mathbf{P}_{II}\mathbf{F}^T + \mathbf{G}\mathbf{Q}_{\mathrm{IMU}}\mathbf{G}^T$ with numerically integrated state-transition matrix $\boldsymbol{\Phi}$.
• State augmentation. On each new image the camera pose ${}^{C}_G\hat{\bar q} = {}^{C}_I\bar q \otimes {}^{I}_G\hat{\bar q}$, ${}^G\hat{\mathbf{p}}_C = {}^G\hat{\mathbf{p}}_I + \mathbf{C}_{\hat q}^T\,{}^I\mathbf{p}_C$ is appended and the covariance expanded via its Jacobian.
• Structureless measurement model (the core contribution). When a feature $f_j$ tracked over $M_j$ poses is lost, its position ${}^G\hat{\mathbf{p}}_{f_j}$ is triangulated by Gauss-Newton least squares with inverse-depth parametrization. Stacking the linearized reprojection residuals of all its observations gives $$\mathbf{r}^{(j)} \simeq \mathbf{H}^{(j)}_{\mathbf{X}}\widetilde{\mathbf{X}} + \mathbf{H}^{(j)}_{f}\,{}^G\widetilde{\mathbf{p}}_{f_j} + \mathbf{n}^{(j)}.$$ Because the triangulation used the state estimate, $\mathbf{r}^{(j)}$ is correlated with $\widetilde{\mathbf{X}}$; projecting onto the left null space of $\mathbf{H}^{(j)}_f$ (basis $\mathbf{A}$) removes the feature error exactly: $$\mathbf{r}^{(j)}_o = \mathbf{A}^T(\mathbf{z}^{(j)} - \hat{\mathbf{z}}^{(j)}) \simeq \mathbf{A}^T\mathbf{H}^{(j)}_{\mathbf{X}}\widetilde{\mathbf{X}} + \mathbf{A}^T\mathbf{n}^{(j)},$$ a $(2M_j-3)$-dimensional constraint coupling all poses that saw the feature — optimal up to linearization, computed implicitly with Givens rotations in $O(M_j^2)$.
• Update. Residuals of all $L$ completed features are stacked, and a QR decomposition $\mathbf{H}_X = \begin{bmatrix}\mathbf{Q}_1 & \mathbf{Q}_2\end{bmatrix}\begin{bmatrix}\mathbf{T}_H \\ \mathbf{0}\end{bmatrix}$ compresses them to $\mathbf{r}_n = \mathbf{Q}_1^T\mathbf{r}_o = \mathbf{T}_H\widetilde{\mathbf{X}} + \mathbf{n}_n$ before the standard EKF update with gain $\mathbf{K} = \mathbf{P}\mathbf{T}_H^T\big(\mathbf{T}_H\mathbf{P}\mathbf{T}_H^T + \mathbf{R}_n\big)^{-1}$. Moving-object outliers are rejected with a Mahalanobis test. When the window is full, $N_{\max}/3$ evenly spaced poses are pruned (the oldest is kept — longer baselines carry more information).

Total cost: linear in features, at most cubic in the (bounded) number of windowed poses.

Results

Evaluated on a real-world urban drive in Minneapolis: a Pointgrey FireFly camera (640×480 @ 3 Hz) and an ISIS IMU (100 Hz) on a car, 1598 images over about 9 minutes, SIFT features, at most 30 camera poses in the state. Along the 3.2 km trajectory, 142,903 feature tracks were used for EKF updates, processed at 14 Hz on a single core of a 2 GHz Intel T7200 — faster than the 3 Hz sensor rate. The final position error was approximately 10 m, i.e. 0.31% of the traveled distance, with no loop closure and no motion priors; estimated 3σ accuracy was better than 1° in attitude and 0.35 m/s in velocity.

Why it matters for SLAM

MSCKF founded the filter-based branch of VIO and its structureless measurement model became standard well beyond filtering (e.g., smart factors in GTSAM/Kimera). It is the direct ancestor of S-MSCKF (stereo), ROVIO-era EKF designs, and OpenVINS, and the follow-up literature on its linearization behavior produced the observability/consistency analysis (First-Estimate Jacobians) that all modern filter-based VIO relies on. Its efficiency profile is why MSCKF-style estimators are widely associated with deployed AR/VR tracking stacks. When accuracy-per-CPU-cycle matters more than absolute accuracy, MSCKF is still the reference design.

Related

• OpenVINS — the modern open-source MSCKF with FEJ and online calibration.
• StereoMSCKF — the stereo extension (S-MSCKF).
• ROVIO — the other landmark filter-based VIO, using direct photometric updates.
• Filter-based vs Optimization-based — where MSCKF sits in the design space.
• Observability — the analysis tradition MSCKF spawned.
• Deployed VIO — where MSCKF-class efficiency matters most.