How Computer Vision Works: A Practical Introduction

Computer vision is the part of AI that lets a program understand images and video. It works by turning pixels into numbers, learning patterns in those numbers, and using the patterns to answer a question such as what is in the picture or where it is.

It sits behind the camera app that finds faces in your photos and the sensor that flags a defective part on a factory line. This article explains the core ideas in the order I found most useful when I was learning them.

What does computer vision actually do?

It takes raw pixel data and produces structured information that a program can act on. That information usually falls into four kinds of task.

Classification answers what is in the image, such as a cat, a car or a faulty part. Detection answers where things are and how many there are, usually by drawing a box around each one. Segmentation goes further and decides which exact pixels belong to which object. Tracking follows an object from frame to frame in a video.

All four are built on the same foundation: represent the image as numbers, then learn what the numbers mean.

How does an image become something a model can learn from?

An image is a grid of numbers, one set for each pixel across the red, green and blue channels. Early computer vision relied on features designed by hand, such as edges, corners, gradients and colour histograms. Techniques like SIFT and HOG extracted those features, and classical models such as support vector machines learned to classify from them.

That approach worked in controlled conditions. It did not cope well with the variety of the real world, where lighting, angle and background change constantly and a hand-designed feature breaks easily.

What changed with deep learning?

Convolutional neural networks changed the field by learning the features directly from data. A CNN slides small filters across an image. The early layers learn to respond to edges, the middle layers to shapes and textures, and the deeper layers to whole parts of objects.

import torch
import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
        )
        self.classifier = nn.Linear(64 * 8 * 8, num_classes)

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        return self.classifier(x)

This example is deliberately small, but it shows the pattern behind most modern vision models. You stack convolutions to build up a representation of the image, then use that representation for the task you care about.

Which architectures are worth knowing?

Three names come up again and again. ResNet introduced residual connections, which made it possible to train much deeper networks without their accuracy degrading. YOLO, short for You Only Look Once, is an object detector that works in a single pass and is fast enough for real-time use, which makes it common in robotics and connected devices. Vision Transformers apply the transformer design from language models to patches of an image, and they now match or beat CNNs on many benchmarks.

Where does it show up in real products?

Phone cameras use it to focus on faces and adjust exposure. Retail systems use it to track stock on shelves. Manufacturing lines use it to catch defects before a product ships.

A growing share of this runs on small devices instead of in a data centre. Running a model on limited hardware keeps the response fast and keeps the images private, but it means the model has to be small and efficient without losing too much accuracy. A lot of the interesting engineering in the field is in that trade-off.

How should a beginner start?

Start by understanding how an image is represented and what a convolution does. Then build a simple classifier on a small dataset such as MNIST or CIFAR-10 using PyTorch or TensorFlow. After that, take a pretrained model such as ResNet or MobileNet and fine-tune it on your own images, which is far more practical than training from scratch. Once classification feels comfortable, move on to a detection framework like YOLO.

Understanding how images become features, and how features become decisions, makes every tool built on top of them easier to reason about. The next article in this pair looks at one of those tools in detail: how face detection works.