How Face Detection Works: Haar Cascades vs Deep Learning

Face detection answers one question: is there a face in this image, and where is it? The classic method, Haar cascades, is fast and light but only reliable for front-facing faces in good light. Deep learning detectors are more accurate in difficult conditions and cost more to run.

It is worth separating detection from face recognition. Detection finds a face. Recognition decides whose face it is. Almost every recognition system runs a detector first, so this is the step to understand before anything else.

How do Haar cascades work?

The Haar cascade method was introduced by Viola and Jones in 2001 and is still in use because it runs quickly on ordinary hardware.

It looks for simple patterns of light and dark. The eye region of a face is usually darker than the cheeks below it, and the bridge of the nose is usually brighter than the eyes on either side. A Haar feature is a small rectangle that measures one of these contrasts.

A single feature is a weak signal, so the method combines thousands of them and arranges them in a cascade. The first stages are cheap checks that throw out regions that are obviously not faces. Only the regions that pass go on to the more expensive stages. Most of an image is rejected almost immediately, which is where the speed comes from.

import cv2

face_cascade = cv2.CascadeClassifier(
    cv2.data.haarcascades + "haarcascade_frontalface_default.xml"
)

image = cv2.imread("photo.jpg")
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

faces = face_cascade.detectMultiScale(
    gray, scaleFactor=1.1, minNeighbors=5, minSize=(30, 30)
)

for (x, y, w, h) in faces:
    cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)

cv2.imwrite("detected.jpg", image)

The weakness is robustness. Haar cascades struggle with faces that are turned to the side, partly covered or poorly lit, and they tend to report faces where there are none.

How do deep learning detectors work?

Modern detectors learn what a face looks like from large sets of labelled images instead of relying on contrasts chosen by hand. A convolutional network learns its own features, which lets it handle angle, lighting and partial cover far better.

MTCNN is a well known example. It uses three small networks in sequence. The first proposes candidate regions, the second filters them, and the third refines the boxes and locates facial landmarks. Other detectors, such as RetinaFace and the face models built on YOLO, do the job in a single pass.

from mtcnn import MTCNN
import cv2

detector = MTCNN()
image = cv2.cvtColor(cv2.imread("photo.jpg"), cv2.COLOR_BGR2RGB)

results = detector.detect_faces(image)
for face in results:
    x, y, w, h = face["box"]
    confidence = face["confidence"]
    print(f"Face at ({x}, {y}), size {w}x{h}, confidence {confidence:.2f}")

Each result includes a confidence score, so you can decide how sure the detector must be before you accept a face.

Why do facial landmarks matter?

Many detectors also return landmarks: the positions of the eyes, the nose and the corners of the mouth. These points let you rotate and scale a face to a standard position, which is called alignment. Recognition models are much more accurate on aligned faces, so landmarks are the bridge between finding a face and identifying it.

Which method should you use?

Use a Haar cascade when you need something fast on modest hardware, the faces will be looking at the camera, and an occasional miss is acceptable. A simple attendance kiosk or a hobby project fits that description.

Use a deep learning detector when accuracy matters, when faces appear at angles or in poor light, or when you are feeding a recognition system afterwards. On small devices, look for a lightweight model built for that purpose instead of a large one.

Whichever you pick, test it on images from the place it will actually run. A detector that performs well on clean photos can fail under the lighting of a real corridor.

A note on responsibility

Faces are personal data. If you build something that detects or recognises people, tell them, store as little as you can, and check the law where you operate.

Face detection is a narrow problem, but it is a good one to learn on because both generations of the technique are easy to run and compare. If the ideas here were new, the companion article on how computer vision works covers the foundations.