Tightly-coupled LiDAR-camera

A LiDAR-camera SLAM system is tightly coupled when point cloud registration residuals and visual residuals are optimized jointly in a single cost function (or a single filter update) over one shared state. It is loosely coupled when each modality runs its own odometry and only the resulting pose estimates are fused afterwards, e.g., in a pose graph or a simple filter.

The joint problem

In a tightly-coupled optimization backend, the state $\mathbf{X}$ (poses, velocities, IMU biases, sometimes landmarks) minimizes a sum of heterogeneous terms:

\mathbf{X}^* = \arg\min_{\mathbf{X}} \; \sum \|\mathbf{r}^{\text{IMU}}\|^2_{\Sigma_I} + \sum \|\mathbf{r}^{\text{visual}}\|^2_{\Sigma_V} + \sum \|\mathbf{r}^{\text{LiDAR}}\|^2_{\Sigma_L} + \sum \|\mathbf{r}^{\text{loop}}\|^2_{\Sigma_{lp}}

where LiDAR residuals are typically point-to-plane or point-to-line distances, and visual residuals are reprojection or photometric errors. LVI-SAM realizes this with a factor graph on GTSAM. The filter-based equivalent puts the same residuals into an iterated Kalman update: FAST-LIVO stacks the Jacobians of both modalities, $\mathbf{H} = [\mathbf{H}^{\text{LiDAR}}; \mathbf{H}^{\text{Visual}}]$ , into one update, while FAST-LIVO2 applies them sequentially within one iteration — LiDAR first, then vision on the intermediate state — to handle the very different dimensionalities of a LiDAR scan and an image.

By contrast, a loosely-coupled design would run LiDAR odometry and visual odometry independently and fuse only their 6-DoF pose outputs. That compression is exactly what tight coupling avoids.

Tight vs loose at a glance

	Tightly coupled	Loosely coupled
What is fused	raw residuals (points, pixels)	6-DoF pose estimates (+ covariance)
Information retention	full — cross-correlations with biases kept	lossy — each modality pre-marginalized
Cross-modal aiding	automatic through the shared state	only via hand-designed interfaces
Sensitivity to calibration errors	high — errors enter every residual	lower — absorbed per subsystem
Failure isolation	poor by default — outliers corrupt the joint state	good — a failing odometry can be ignored
Implementation effort	high	low

Why bother with the extra complexity?

Information is not thrown away. Loose coupling compresses each modality into a 6-DoF pose (plus covariance) before fusion; correlations between the raw measurements and the shared state (especially IMU biases) are lost.
Cross-modal aiding happens automatically. LiDAR depth can constrain visual features directly (as in LVI-SAM’s depth-enhanced features); the camera constrains the directions in which LiDAR geometry is degenerate.
Graceful degradation. When one modality’s residuals become uninformative, the joint problem is still well-conditioned as long as the other modality observes the weak directions.

Common pitfalls

Calibration debt. Extrinsics between LiDAR, camera, and IMU, plus time offsets, enter every residual; tightly-coupled systems convert calibration error directly into estimation bias, so calibration quality bounds achievable accuracy.
Outlier leverage. A bad measurement in one modality pulls the whole state, not just its own subsystem — robust weighting and explicit degradation detection are mandatory, not optional.
Weighing apples against oranges. The relative noise covariances $\Sigma_L$ vs $\Sigma_V$ decide which sensor “wins” disagreements; residuals in meters and residuals in intensity units have no natural common scale, and mis-tuned covariances silently bias the estimate.
Iterate or suffer linearization error. Both LiDAR and photometric residuals are nonlinear in the pose; single-shot updates degrade under fast motion, which is why the strong systems use iterated filters or re-linearizing smoothers.

Why it matters for SLAM

The tight-vs-loose distinction, familiar from VIO, is the single most important architectural axis in LiDAR-camera fusion: it largely determines a system’s accuracy ceiling and its failure behavior. Every major LVI system (LVI-SAM, R3LIVE, FAST-LIVO2) advertises tight coupling as a headline feature, and reading their cost functions is the fastest way to understand what each system actually fuses.

Tightly-coupled vs Loosely-coupled — the same concept in visual-inertial fusion
LVI-SAM — tightly-coupled factor-graph realization
FAST-LIVO — tightly-coupled filter realization
LiDAR-Visual-Inertial (LVI) — the broader triple-fusion picture
Factor graph — the machinery behind joint optimization
Degradation handling — the safeguards tight coupling requires