Camera models beyond pinhole

The pinhole model with radial-tangential distortion works well for narrow and moderate fields of view, but many SLAM platforms use lenses and sensors that break its assumptions. Wide field-of-view lenses see more of the scene — which means more parallax, more trackable features, and better robustness to fast rotation — at the cost of requiring different projection models.

Projection functions at a glance

All central camera models can be compared by how the image radius $r$ grows with the incidence angle $\theta$ (the angle between the incoming ray and the optical axis):

Model	$r(\theta)$	Behaviour at $\theta \to 90°$
Perspective (pinhole)	$f\tan\theta$	diverges — cannot represent FoV ≥ 180°
Equidistant fisheye	$f\theta$	finite
Equisolid fisheye	$2f\sin(\theta/2)$	finite
Stereographic	$2f\tan(\theta/2)$	finite

The pinhole projection blows up as rays approach 90° from the axis, which is the fundamental reason fisheye lenses need their own models rather than ever-higher-order distortion polynomials.

Fisheye: Kannala-Brandt

Fisheye lenses reach fields of view of 180° or more, where the pinhole perspective projection ( $r = f\tan\theta$ ) diverges. The Kannala-Brandt model instead expresses the image radius directly as a polynomial in the incidence angle:

$r(\theta) = k_1\theta + k_2\theta^3 + k_3\theta^5 + k_4\theta^7$

This generic model fits equidistant, equisolid, and other fisheye projections, and is the fisheye model implemented in OpenCV (cv::fisheye) and used by systems such as ORB-SLAM3 for wide-angle cameras. Note the structure: it is a replacement for the projection function itself, with the odd-power series playing the role that the distortion polynomial plays for pinhole cameras.

Double-sphere and omnidirectional models

The double-sphere model projects a 3D point through two unit spheres followed by a pinhole projection. It fits fisheye lenses with accuracy comparable to Kannala-Brandt while having a closed-form, computationally cheap unprojection — a practical advantage in real-time VIO, since unprojection (pixel to ray) runs for every feature on every frame; polynomial models like Kannala-Brandt need iterative root-finding for the same operation. The model originates from the group behind the Basalt VIO system.

Omnidirectional models (e.g., the unified camera model, which projects through a single sphere with a mirror/sphere offset parameter $\xi$ , and Scaramuzza’s polynomial model) cover catadioptric cameras — cameras with mirrors — and very wide fisheyes. They are supported by calibration tools such as Kalibr and OpenCV’s omnidir module.

Rolling-shutter awareness

Most low-cost CMOS cameras use a rolling shutter: image rows are exposed sequentially rather than simultaneously. When the camera or scene moves fast, each row is captured from a slightly different pose, producing skew and wobble. A geometric model that assumes one pose per frame is then wrong: a point observed on row $v$ was actually captured at time

$t(v) = t_0 + \frac{v}{H}\,t_{\text{readout}}$

where $t_0$ is the start of frame readout, $H$ the image height, and $t_{\text{readout}}$ the total readout time. High-speed SLAM either uses global-shutter hardware or explicitly models per-row capture time, interpolating the camera pose across the readout. At minimum, know which shutter your camera has before trusting the geometry.

Common pitfalls

Forcing a fisheye lens into the radial-tangential pinhole model: it fits well near the image centre and fails badly toward the border — reprojection error statistics will look deceptively good if the calibration views avoided the edges.
Undistorting fisheye images to a pinhole view throws away the wide FoV (cropping) or stretches the periphery enormously; modern systems keep the native model and adapt the solvers instead.
Mixing conventions between tools: the same lens calibrated in Kalibr, OpenCV, and a SLAM system’s config may use different parameter orders and model names; verify by reprojecting points, not by eyeballing numbers.

Why it matters for SLAM

Feeding fisheye images into a pinhole+distortion model, or ignoring rolling shutter on a fast platform, silently corrupts every measurement in the pipeline. Choosing the right camera model — and calibrating it with tools that support that model — is a prerequisite for accurate tracking on drones, AR headsets, and automotive surround-view rigs, which almost always use wide-angle or fisheye optics.