Pixels, Perception and Possibilities.

Deconstructing Pixels

Humans do not experience colour exactly as it exists in the physical world. What we call colour is our brain’s interpretation of light reflected from surfaces. I recently came across a fascinating example involving tigers: although humans perceive their fur as orange, many prey animals, such as deer, have a different visual system and may perceive that same colour as closer to green. Against the surrounding vegetation, this helps the tiger remain camouflaged and makes it easier to approach its prey unnoticed.

That fact left me with several uncomfortable but fascinating questions. Colours, pixels, light rays and reflections — what if much of what we see is not reality itself, but only our interpretation of it? In a way, vision becomes an example of relativity: the same object can appear differently depending on the observer and the system through which it is perceived.

This naturally led me to wonder: what do computers and machines see?

A computer does not see a person, a tiger, a tree or even a colour. It sees numbers. An image is represented as a grid of pixels, and each pixel contains numerical values describing properties such as brightness and colour intensity. Computer vision models process these values using mathematical operations, including filters and kernels, gradually learning patterns that allow them to identify shapes, textures, objects and relationships.

Perhaps this is one way to understand the relative nature of colour and vision. Humans interpret reflected light through biological systems, while machines interpret numerical representations through mathematical systems. Neither experiences the world in a completely direct or neutral way.

These ideas, questions and phenomena are what drew me toward computer vision and inspired me to understand it more deeply. This essay is my attempt to break computer vision down to its smallest building blocks so that anyone reading it can understand what a machine “sees,” how visual information is represented and interpreted, and why pixels are far more interesting than they initially appear.

Ultimately, this article is also a way for me to explain computer vision in the language that makes the most sense to my own mind. As Andrej Karpathy has suggested when discussing how to understand difficult concepts, one of the best tests of understanding is attempting to explain an idea clearly in your own words.

Pixel as Numbers

As I mentioned earlier, computers do not see objects, colours or shapes in the way humans do. They receive numerical representations of visual information. In a standard digital colour image, each pixel is typically described using three colour channels: red, green and blue. In an 8-bit image, the intensity of each channel is represented by a value ranging from 0 to 255. A pixel with values close to (255, 0, 0), for example, would appear strongly red, while (255, 255, 255) would appear white.

These values are organized into structures called tensors. A tensor can be understood as a multidimensional array of numbers. A scalar is a single number, a vector is a one-dimensional sequence of numbers, and a matrix is a two-dimensional grid of numbers. A tensor extends this idea into three or more dimensions. A colour image, for instance, is commonly represented as a three-dimensional tensor containing its height, width and colour channels. A collection of multiple images adds another dimension, producing a four-dimensional tensor called a batch.

To understand how computer vision systems represent and process images, some knowledge of arithmetic, linear algebra, matrices and probability is especially useful. Calculus becomes more important when studying how neural networks learn, because training involves calculating gradients and adjusting model parameters. However, the basic representation of an image can still be understood without advanced mathematics.

Consider a photograph of a cat. A human immediately recognizes fur, ears, eyes, whiskers and the overall shape of the animal. A computer initially receives only a structured grid of numerical pixel values. These RGB values form the image tensor that is passed into a computer vision model. The model then performs mathematical operations on the tensor to detect useful patterns and gradually construct a meaningful representation of the image.

This raises an important question: how can a computer make sense of a collection of numbers? How does it move from pixel intensities to recognizing that an image contains a cat?

The computer does not understand a cat in the human sense. Instead, it learns statistical patterns that frequently appear in images labelled as cats. During training, the model discovers combinations of edges, textures, curves, shapes and larger visual structures that help distinguish cats from other objects.

This brings us to one of the foundational ideas in modern computer vision: the convolutional neural network, commonly called a CNN. CNNs were designed to process grid-like data such as images. They use small filters, also known as kernels, that move across an image and respond to particular visual patterns. Early layers may detect simple features such as edges, lines and colour transitions. Deeper layers combine these features into more complex representations, such as eyes, ears, faces and eventually entire objects.

CNNs played a central role in the development of modern computer vision and remain widely used today. However, they are no longer the only dominant approach. Architectures such as Vision Transformers also process images effectively, often by dividing them into smaller patches and learning relationships between different regions. Regardless of the architecture, the underlying idea remains similar: the machine begins with numbers and gradually transforms them into representations that are useful for prediction.

CNNs, Vision Transformers and Zero-Shot Learning

Before discussing specific computer vision architectures, it helps to understand the basic idea behind neural networks.

A neural network is a mathematical system that learns patterns from data. It is made up of connected units, often called neurons, arranged in layers. Each connection has a numerical value called a weight, and many neurons also include a bias. During training, the network adjusts these weights and biases so that its predictions become more accurate.

Neural networks are often compared to the human brain because both involve interconnected units. However, this comparison should not be taken too literally. Artificial neural networks do not think or understand in the way biological brains do. They perform mathematical operations and learn statistical relationships within data.

A simple neural network usually contains three broad parts: an input layer, one or more hidden layers, and an output layer. The input layer receives the data. The hidden layers transform that data into increasingly useful representations. The output layer produces the final prediction.

The word “deep” in deep learning refers to the use of many hidden layers. These layers allow the model to learn increasingly complex patterns. In computer vision, early layers may respond to basic features such as edges, brightness changes and colour transitions. Later layers may combine those features into textures, curves, shapes, object parts and eventually complete objects.

For text, the process is different. A language model does not normally learn individual letters and sentence structures in the same way that a vision model learns edges and shapes. Instead, it learns numerical representations of tokens and the relationships between them. Across many layers, it learns patterns involving meaning, grammar, context and sequence.

Convolutional Neural Networks

A convolutional neural network, or CNN, is a type of deep neural network designed especially for grid-like data such as images.

As discussed earlier, an image can be represented as a tensor containing numerical pixel values. A CNN processes this tensor using small matrices called filters or kernels. These kernels move across the image and perform mathematical operations on small regions at a time.

Each kernel can learn to respond strongly to a particular visual pattern. One kernel may respond to vertical edges, another to horizontal edges, and another to changes in texture or colour. These patterns are not normally programmed by hand. The model learns them during training.

As information passes through multiple convolutional layers, the representations become more complex. The first layers may detect simple edges. Middle layers may detect textures, corners and curves. Deeper layers may identify features such as eyes, ears, wheels or faces. The final layers combine these features to make a prediction.

For example, when a CNN processes an image of a cat, it does not begin with the concept of a cat. It begins with pixel values. Through its layers, it may gradually detect fur-like textures, pointed ears, eyes, whiskers and the shape of the animal. The output layer then uses the learned representation to assign a probability to different classes, such as cat, dog or bird.

How a CNN Learns

The process of teaching the network is called training.

During training, an image is passed through the network in a process called the forward pass. The model produces a prediction. That prediction is then compared with the correct answer using a loss function.

The loss function measures how wrong the prediction is. A larger loss means the prediction is further from the correct answer. A smaller loss means the model is performing better.

The model then uses a process called backpropagation to determine how each weight contributed to the error. Calculus is used to calculate gradients, which describe how the loss would change if the weights changed slightly.

An optimization algorithm, such as gradient descent or Adam, uses these gradients to update the weights. The aim is to reduce the loss over time.

This process is repeated across many examples:

1. The model receives an image.
2. It produces a prediction.
3. The loss function measures the error.
4. Backpropagation calculates the gradients.
5. The optimizer updates the weights.
6. The process repeats.

The model is not being directly “punished” or “rewarded” in a human sense. Instead, the numerical parameters are adjusted in directions that are expected to reduce future errors.

Hyperparameters control how this training process is performed. Examples include the learning rate, batch size, number of training epochs, kernel size and number of layers. These values are chosen before or during training and are not learned in exactly the same way as the model’s weights.

Vision Transformers

Vision Transformers, commonly called ViTs, are another major architecture used in computer vision.

Unlike CNNs, Vision Transformers do not primarily process an image by sliding kernels across nearby pixels. Instead, they divide the image into smaller regions called patches.

For example, an image may be divided into square patches measuring 16 by 16 pixels. Each patch is flattened and converted into a numerical representation called an embedding. These patch embeddings are then processed as a sequence, in a way that is conceptually similar to how language Transformers process sequences of words or tokens.

The model also adds positional information so that it knows where each patch came from. Without positional information, the model would know which patches exist but not their original arrangement in the image.

The central mechanism in a Vision Transformer is self-attention. Self-attention allows every image patch to examine and interact with other patches. This helps the model learn relationships between distant parts of an image.

For example, when recognizing a person, the model may connect information from the face, arms, clothing and legs, even though those regions are separated across the image.

This is one important difference between CNNs and Vision Transformers. CNNs naturally focus on local regions first and gradually build larger representations. Vision Transformers can model relationships between distant image regions much earlier in the network.

A simplified Vision Transformer pipeline looks like this:

1. Divide the image into patches.
2. Convert each patch into an embedding.
3. Add positional information.
4. Pass the patch sequence through Transformer layers.
5. Use self-attention to learn relationships between patches.
6. Produce a final representation.
7. Use that representation for classification, detection or another task.

Vision Transformers often perform extremely well when trained on large datasets. However, they may require more data and computational resources than traditional CNNs. Modern computer vision systems also frequently combine ideas from both CNNs and Transformers rather than treating them as completely separate approaches.

Image Classification and Object Detection

It is important to distinguish image classification from object detection.

Image classification predicts what is present in an entire image. For example, a model may determine that an image contains a cat.

Object detection goes further. It identifies what objects are present and where they are located. The model usually produces a class label, a confidence score and a bounding box for each detected object.

Models such as YOLO and RT-DETR are primarily object-detection architectures.

YOLO, which stands for “You Only Look Once,” predicts object locations and classes in a single detection pipeline. It is widely used because it can perform object detection quickly, making it suitable for real-time systems.

RT-DETR is a real-time detection model based on Transformer concepts. It belongs to the broader DETR family, which treats object detection as a set-prediction problem. Rather than relying entirely on traditional detection components such as manually designed anchor boxes and post-processing pipelines, DETR-style models use Transformer-based mechanisms to predict a set of objects directly.

However, YOLO and RT-DETR are not automatically zero-shot models. They are object detectors. Whether they can perform zero-shot detection depends on how they were designed and trained.

A standard YOLO model trained only on a fixed list of classes can detect only those learned classes. For example, if it was trained on people, cars and bicycles, it would not automatically understand an entirely new category simply because the user typed its name.

Zero-Shot Learning

Zero-shot learning is not a specific model architecture. It is a capability or learning setting.

A zero-shot model can perform a task involving a class or concept it was not explicitly trained to predict in the usual labelled form.

For example, imagine a model trained to understand relationships between images and text. During inference, the model may be given an image and a list of text descriptions such as:

• a photograph of a tiger
• a photograph of a leopard
• a photograph of a wolf

Even if the model was not trained using a traditional classifier with those exact three output labels, it may compare the image representation with the text representations and select the best match.

CLIP is one well-known example of a model that supports zero-shot image classification. It was trained using large numbers of image-and-text pairs. It learned to place related images and descriptions close together in a shared representation space.

During zero-shot classification, an image is converted into an image embedding. Each possible text label is also converted into a text embedding. The model then compares them, usually using a similarity measurement. The text description most similar to the image becomes the predicted class.

A simplified zero-shot classification pipeline is:

1. Convert the image into an embedding.
2. Convert each text label into an embedding.
3. Compare the image embedding with the text embeddings.
4. Select the text label with the highest similarity.

The important idea is that the model is not restricted to a traditional fixed classification layer. Instead, language can be used to describe the classes at inference time.

Zero-Shot Object Detection

Zero-shot object detection combines object detection with language understanding.

Instead of detecting only a fixed set of categories learned during conventional training, a zero-shot detector may accept a text prompt such as:

• person
• red backpack
• construction helmet
• brown dog

The model then attempts to locate regions in the image that match those descriptions.

Models such as Grounding DINO and OWL-ViT are examples of architectures designed for open-vocabulary or zero-shot object detection. These models connect visual representations with text representations, allowing users to search for objects using natural-language descriptions.

Some newer YOLO-related systems may also support open-vocabulary detection when combined with vision-language training or text encoders. However, this capability should not be confused with standard YOLO object detection.

The distinctions are therefore important:

• CNN describes a type of neural-network architecture.
• Vision Transformer describes another type of visual architecture.
• YOLO and RT-DETR describe object-detection model families.
• Zero-shot learning describes the ability to work with unseen or dynamically provided classes.
• A model can use a CNN or Transformer architecture without being zero-shot.
• A detection model can be fast and accurate without supporting open-vocabulary prompts.
• Zero-shot systems usually depend on relationships learned between different modalities, especially images and language.

How These Ideas Connect

All of these systems begin with numerical representations.

A CNN processes local pixel patterns using learned kernels. A Vision Transformer divides an image into patches and learns relationships between them using self-attention. A traditional object detector learns to locate objects from a predefined set of classes. A zero-shot vision-language model connects images with textual descriptions so that it can reason about categories more flexibly.

Despite their differences, the broad process remains similar:

1. Convert the input into numbers.
2. Transform those numbers through multiple layers.
3. Learn useful representations.
4. Compare the output with a target during training.
5. Update the model’s parameters.
6. Use the trained model to make predictions on new inputs.

What changes from one architecture to another is how the numbers are organized, how the relationships are learned, and what type of output the model is designed to produce.

The Physics of Computer Vision

Across all these concepts, one pattern remains common: input, processing and output. The input may be an image, a video or a live camera stream. The processing stage attempts to interpret that visual information, while the output produces a prediction, detection, classification or action.

What fascinates me most, however, is everything that happens in the middle.

How does a computer move from simply seeing pixels to understanding the physics, dynamics and unpredictability of the real world? How does it understand not only what an object is, but what that object is likely to do next?

This is the part of computer vision that excites me the most, and I believe it may become one of the most important areas in the future of machine learning and possibly artificial general intelligence.

Current computer vision systems are already impressive. They can classify images, detect objects, segment scenes and track movement. Generative models such as GANs and diffusion models can create realistic images and videos. However, there is a difference between generating something that looks physically realistic and actually understanding the physical rules behind what is happening.

How do we teach a machine to predict and respond to real-world events with something close to the intuition humans naturally develop?

Consider something as simple as a coin falling to the floor. The coin may land exactly where it falls. It may bounce once or twice. It may spin, roll several metres away or change direction after hitting another object. Its behaviour will depend on its speed, angle, material, the type of floor, friction, nearby obstacles and several other conditions.

A truly intelligent visual system should not only detect the coin. It should understand that several outcomes are possible. It should predict those possible outcomes, adjust its expectations as the event unfolds and prepare to respond accordingly.

For example, if the machine is controlling a robot, the robot may need to decide whether to reach for the coin immediately, wait for it to stop moving or predict where it will land after its second or third bounce. That combination of seeing, predicting and reacting is far more interesting to me than simple object detection.

This question draws me deeper into computer vision because it moves the field from recognition toward understanding.

It involves understanding scenes, movement, relationships, affordances and consequences. A machine should not only know that an object is a cup. It should understand that the cup can be held, filled, poured from, dropped or broken. It should understand that a chair can be sat on, that a door can be opened and that a wet floor may be slippery.

This is where the concept of affordances becomes important. Affordances describe the possible actions associated with an object or environment. Recognizing an object tells the machine what it is. Understanding its affordances tells the machine what can be done with it and what may happen when it is used.

In my opinion, one possible way to teach machines this kind of understanding would be through carefully designed, sequence-by-sequence training.

Rather than training only on isolated images, the model could be trained on complete physical events. It could observe objects falling, rolling, bouncing, colliding, breaking, bending and reacting to external forces. Each sequence would show not only the beginning and end of an event, but the complete transition between them.

The first stage could involve large amounts of real-world data. The model could learn from the movement of different objects, changes in lighting, wind disturbances, surface textures, human interactions and other real-world conditions. The goal would be to expose it to as many variations of physical behaviour as possible.

From there, the model could be trained on generated simulations produced by physics engines, world models or video-generation systems. These simulations could create tens of thousands of controlled scenarios that would be difficult, expensive or dangerous to reproduce physically.

For example, the same coin could be dropped from different heights, onto different surfaces, at different angles and with different initial forces. The model could learn how small changes in the starting conditions may produce very different outcomes.

I imagine this training process happening in stages.

First, we would define the capabilities we want the model to learn at that point in its development. We would then collect high-quality data around those capabilities and use it for pretraining. The model would learn basic visual patterns, object behaviour, motion and scene relationships.

After that, it would be evaluated under carefully designed and stress-tested real-world conditions. The goal would be to identify where its understanding fails, especially when it encounters unfamiliar objects, unusual movement or conditions outside its training data.

The model could then be improved through additional training, including reinforcement learning where appropriate. By receiving feedback from its predictions and actions, it could gradually learn which behaviours produce successful outcomes and which ones do not.

Once those abilities became reliable, the system could be post-trained on more advanced and clearly defined capabilities. The process could continue step by step, moving from basic visual recognition to motion prediction, physical reasoning, planning and real-world interaction.

I do not believe this problem can be solved by one dataset, one architecture or one training method. It would likely require a combination of real-world video, simulation, self-supervised learning, reinforcement learning, memory, planning and continuous interaction with the environment.

It may also require what researchers often call a world model: an internal representation of how the environment works and how it may change over time.

A world model would allow the machine to go beyond reacting to the current frame. It could use previous observations to estimate what is happening, predict several possible futures and determine which action would be most appropriate.

This is the direction of computer vision that fascinates me most.

The field has already made significant progress in detecting, classifying, tracking and generating visual information. However, I believe its true potential lies in teaching machines to understand what they see, predict what may happen next and act intelligently within the physical world.

Beyond raw detection, classification and generation, computer vision could become one of the technologies that brings real automation into everyday life. Machines would no longer simply identify the world around them. They would understand how it behaves, anticipate how it may change and respond meaningfully to it.

Beyond the Pixels

This has been my ramble about computer vision and the deep interest I have developed in the field. I am currently looking to join teams and research labs working seriously on computer vision or recommendation systems — another area that makes my brain unusually happy.

I have applied to NVIDIA, Waabi, Shopify and a few other companies and labs here in Canada, and I hope I eventually get the opportunity to contribute my own ideas, learn from the people already pushing these fields forward and perhaps share my two cents once I am onboard.

Until then, I am continuing to build AVI, my personal assistant project. AVI brings together different modalities and ideas from several projects I have worked on, including vision, recognition, language and real-world interaction. It is still evolving, but the process has been genuinely exciting, and I cannot wait to demonstrate a working version operating inside my home.

For now, I will keep building and dropping my unsolicited human-animal color theories.

Keep yourselves curious.

Ciao.