Stereo rectification

Stereo rectification warps the two images of a calibrated stereo pair so that they appear to come from two ideal, parallel cameras: image planes coplanar, optical axes parallel, and rows aligned. After rectification, the epipolar line of any pixel in the left image is simply the same row in the right image.

Why this works: for any pair of camera poses, epipolar geometry constrains the match of a left-image point to a 1D curve (a line for pinhole cameras) in the right image. Rectification is a pair of homographies $H_L, H_R$ applied to the images that map both epipoles to infinity along the horizontal axis, making all epipolar lines horizontal and vertically aligned.

How the rectifying transforms are computed

Given calibrated intrinsics $K_L, K_R$ , lens distortion, and the extrinsic transform $(R, t)$ between the cameras, the classic (Bouguet) construction used by OpenCV’s stereoRectify proceeds as:

Split the relative rotation. Rotate each camera by “half” of $R$ so the two optical axes become parallel while disturbing each image as little as possible.
Align with the baseline. Apply a common rotation so the new $x$ -axes point along the baseline vector $t$ . Both epipoles now sit at infinity, so epipolar lines are horizontal.
Choose common virtual intrinsics. A single new camera matrix $K_{\text{new}}$ (same $f$ , same principal point row) is chosen for both views, giving the rectified pair one focal length and one baseline for $Z = fB/d$ .
Bake remap tables. The per-camera warp $H = K_{\text{new}} R_{\text{rect}} K_{\text{old}}^{-1}$ , combined with the lens-undistortion model, is precomputed into lookup tables, so rectifying each frame costs a single image remap.

R1, R2, P1, P2, Q, roi1, roi2 = cv2.stereoRectify(K1, D1, K2, D2, size, R, T)
mapLx, mapLy = cv2.initUndistortRectifyMap(K1, D1, R1, P1, size, cv2.CV_32FC1)
mapRx, mapRy = cv2.initUndistortRectifyMap(K2, D2, R2, P2, size, cv2.CV_32FC1)
rectL = cv2.remap(imgL, mapLx, mapLy, cv2.INTER_LINEAR)
rectR = cv2.remap(imgR, mapRx, mapRy, cv2.INTER_LINEAR)

The returned $Q$ matrix reprojects a rectified pixel-plus-disparity $(u, v, d)$ directly to 3D — the disparity-to-depth conversion in matrix form.

What it buys you

1D correspondence search. Matching reduces from a 2D search to scanning along a single row, which is what makes dense disparity algorithms (block matching, Semi-Global Matching) and fast sparse stereo matching tractable in real time.
Simple disparity geometry. In the rectified frame, correspondence is purely horizontal, so $d = u_L - u_R$ and $Z = fB/d$ hold exactly, with a single common focal length $f$ and baseline $B$ for the virtual camera pair.
Outlier rejection for free. A candidate match lying off its scanline (beyond a small tolerance) violates epipolar geometry and can be discarded immediately.

Checking rectification quality

A quick sanity check: match features between the rectified images and inspect the vertical disparity $v_L - v_R$ . On a well-rectified pair it should be a small fraction of a pixel; a systematic offset or a gradient across the image means the extrinsic calibration has drifted. Rising vertical disparity over a robot’s lifetime is the standard symptom of a rig that has been bumped or is flexing thermally.

Practical caveats

Rectification is only as good as calibration. Errors in $(R, t)$ show up as vertical misalignment that silently degrades matching — the matcher searches the wrong row and either fails or returns biased disparities.
Wide-FoV and fisheye lenses suffer. Warping to a pinhole rectified pair loses significant image area at the periphery (or requires special rectification models); heavy crop factors waste exactly the wide field of view the lens was chosen for.
The baseline must stay rigid. Rectification assumes a fixed $(R, t)$ ; rigs that flex mechanically or thermally need periodic recalibration or online extrinsic refinement.
Shutters must be synchronized. If the two cameras expose at different instants, a moving platform effectively changes the baseline per frame — no static rectification can fix that.

Why it matters for SLAM

Nearly every stereo SLAM front-end — ORB-SLAM2’s per-keypoint stereo matching, S-PTAM, dense SGM-based mapping in Kimera — assumes rectified input, because scanline search is the only way to get per-frame depth at real-time rates. Understanding rectification also tells you where stereo pipelines break: bad calibration, unsynchronized shutters, or fisheye optics all violate the rectified model before any SLAM code runs.