The geometry of image formation

Computer vision begins with a lossy map. A point in three-dimensional space becomes a point on a two-dimensional sensor, and depth disappears. The familiar projection matrix is compact enough to hide where that loss occurs and which assumptions make the map useful.

Four coordinate systems, one observation

Let a world point be $P_w = (X_w, Y_w, Z_w, 1)^T$. A rigid transform places it in camera coordinates:

Perspective division then maps $P_c = (X_c, Y_c, Z_c)^T$ to the normalized image plane:

This division is the essential operation. Points on the same ray from the camera center share $(x_n, y_n)$, so a single image cannot recover their absolute depth without more information.

Pixels are not normalized coordinates

The intrinsic matrix converts normalized coordinates into sensor coordinates:

The focal lengths $f_x$ and $f_y$ are measured in pixels. The principal point $(c_x, c_y)$ describes where the optical axis meets the sensor. The skew $s$ is usually set to zero for modern sensors, but removing it is an assumption rather than an algebraic fact.

An implementation that preserves the stages

Compact matrix multiplication is convenient in production. During development, keeping the stages visible makes frame mistakes easier to find.

import numpy as np

def project(point_world, rotation, translation, intrinsics):
    point_camera = rotation @ point_world + translation
    if point_camera[2] <= 0:
        raise ValueError("point lies on or behind the camera plane")

    normalized = point_camera[:2] / point_camera[2]
    homogeneous = np.array([normalized[0], normalized[1], 1.0])
    pixel = intrinsics @ homogeneous
    return pixel[:2]

The explicit depth check matters. Algebra alone will happily project a point behind the camera into a plausible pixel coordinate.

What the matrix leaves out

Real cameras depart from this model. Radial and tangential lens distortion alter the mapping, rolling shutters expose rows at different times, and calibration parameters drift with focus and temperature. The pinhole camera is still the correct starting point because it separates projective geometry from those corrections.

The next step is not a larger matrix. It is a clear residual model: measure how observed pixels depart from the ideal projection, then decide which physical effects are worth estimating.

Further reading

• Richard Hartley and Andrew Zisserman, Multiple View Geometry in Computer Vision.
• Richard Szeliski, Computer Vision: Algorithms and Applications.
• The related notebook entry records a coordinate-normalization error that initially looked like poor calibration.