Filter-based vs Optimization-based

Within tightly-coupled VIO there are two estimation paradigms. Both solve the same probabilistic inference problem; they differ in how often they are willing to re-linearize it.

Filter-based (EKF)

Filter-based methods (MSCKF, ROVIO, OpenVINS) maintain a state vector $\mathbf{x}$ and covariance $\mathbf{P}$ updated recursively:

Propagate with IMU measurements: $\mathbf{x} \leftarrow f(\mathbf{x}, \tilde{\boldsymbol{\omega}}, \tilde{\mathbf{a}})$ , $\;\mathbf{P} \leftarrow \boldsymbol{\Phi}\mathbf{P}\boldsymbol{\Phi}^\top + \mathbf{Q}$ .
Update with visual measurements: with residual $\mathbf{r} = \mathbf{z} - h(\mathbf{x})$ and Jacobian $\mathbf{H}$ , apply the Kalman gain $\mathbf{K} = \mathbf{P}\mathbf{H}^\top(\mathbf{H}\mathbf{P}\mathbf{H}^\top + \mathbf{R})^{-1}$ and absorb the measurement into the Gaussian belief.

Each measurement is linearized once, at the moment it is processed. That gives constant-time, low-latency updates — but the linearization point is permanent: if the estimate later moves, the already-absorbed information cannot be re-linearized, and accumulated linearization error causes drift and (without care) inconsistency. MSCKF’s insight was to make the EKF window-shaped — keeping a sliding set of past camera poses in the state and applying feature constraints structurelessly — so even “filtering” VIO is secretly windowed.

Optimization-based (sliding-window BA)

Optimization-based methods (OKVIS, VINS-Mono, Basalt, ORB-SLAM3) keep a sliding window of recent keyframe states and minimize a joint nonlinear cost — reprojection residuals, preintegrated IMU residuals, and a marginalization prior — with Gauss-Newton or Levenberg-Marquardt:

$\min_{\mathcal{X}} \; \|\mathbf{r}_p\|^2 + \sum \|\mathbf{r}_{\text{IMU}}\|^2_{\Sigma^{-1}} + \sum \rho\left(\|\mathbf{r}_{\text{reproj}}\|^2_{R^{-1}}\right)$

Because every solver iteration re-linearizes all residuals in the window at the current estimate, linearization error is much smaller, and accuracy is correspondingly higher — at higher computational cost. Old states are compressed into the prior $\mathbf{r}_p$ by Schur-complement marginalization, which is where the paradigms meet (see below).

Comparison

	Filter (EKF)	Sliding-window optimization
Linearization	Once per measurement	Re-linearized every iteration
Cost	Low, constant-time	Higher, grows with window size
Accuracy	Good	Better (typically)
Latency	Very low (one update per measurement)	Higher (iterative solve per frame)
Consistency tools	First-Estimate Jacobians (FEJ), observability constraints	FEJ on the marginalization prior; nonlinear factor recovery (Basalt); delayed marginalization (DM-VIO)
Representative systems	MSCKF, ROVIO, OpenVINS	OKVIS, VINS-Mono, Basalt, ORB-SLAM3

The two families are closer than they appear

A sliding-window optimizer that marginalizes old states via the Schur complement is mathematically performing a filtering step on the marginalized part — the prior it creates is linearized once and frozen, exactly like an EKF update.
Conversely, an EKF is an optimizer that stops after a single Gauss-Newton iteration; the iterated EKF (used in ROVIO’s update step) re-linearizes within an update and moves further toward the optimization side.
Both families suffer the same disease at their frozen linearization points — spurious information gain along unobservable directions — and both use the same medicine (FEJ and its relatives). The consistency literature that grew out of MSCKF applies to the marginalization prior of every sliding-window system.

So the practical difference is not “recursive vs batch” but where each design spends its compute budget: filters spend almost nothing on re-linearization and everything on throughput; optimizers buy accuracy by re-linearizing a small, well-chosen window.

Choosing in practice

Hard real-time, embedded compute, high-rate control loops → filter (MSCKF/OpenVINS class). Constant, predictable per-frame cost; this efficiency profile is why filter-style estimators are associated with deployed AR/VR tracking.
Accuracy-critical mapping, offline or desktop-class compute → sliding-window optimization, or optimization plus a mapping back-end (Basalt, OKVIS2).
Study both failure modes: filter inconsistency shows up as overconfident covariance and yaw drift; optimizer latency shows up as dropped frames under load. The patch literature — FEJ, observability-constrained EKF, nonlinear factor recovery, delayed marginalization — exists precisely to treat these.

Common pitfalls

Comparing a filter and an optimizer at different compute budgets and concluding one paradigm is “better” — the classic “why filter?” analysis (Strasdat et al.) is a fixed-budget argument: re-linearizing a small keyframe window beats filtering a large state.
Treating the marginalization prior of a sliding-window system as exact — it is a one-shot linearization, and ignoring that (no FEJ handling) reintroduces filter-style inconsistency into an “optimization-based” system.
Assuming filters cannot be accurate: with FEJ, online calibration, and good front-ends, modern MSCKF implementations are competitive on standard benchmarks at a fraction of the cost.

Why it matters for SLAM

This is the VIO version of the classic “why filter?” question studied for visual SLAM (Strasdat et al.): given a fixed compute budget, re-linearizing a small window of keyframes beats filtering a large state. Knowing the failure mode of each side — filter inconsistency vs optimizer latency — tells you which system to pick for a given platform and which papers (FEJ, nonlinear factor recovery, delayed marginalization) exist to patch those failure modes.

Visual-SLAM why filter? — the foundational analysis of this trade-off.
MSCKF — the canonical filter-based VIO.
OKVIS — the canonical optimization-based VIO.
Basalt — nonlinear factor recovery to fix marginalization-prior linearization error.
Marginalization — the mechanism connecting both worlds.
Consistency — the estimator property at stake in this choice.