But what is a neural network? | Deep learning chapter 1

Summary

An introductory lecture explaining the fundamental architecture and mathematics of neural networks through the example of handwritten digit recognition, designed for beginners with no prior machine learning experience.

Key Points

1. A neural network is fundamentally just a function with thousands of parameters (weights and biases) that transforms input numbers into output numbers through layers of weighted sums and activation functions
2. Neural networks ideally learn hierarchical patterns: early layers detect simple features like edges, middle layers recognize components like loops or lines, and final layers identify complete objects
3. Each neuron holds an 'activation' value between 0 and 1, computed by taking a weighted sum of previous layer activations, adding a bias, and applying a sigmoid function to squish the result into the 0-1 range
4. The MNIST digit recognition network has 784 input neurons (one per pixel), two hidden layers of 16 neurons each, 10 output neurons (one per digit), and approximately 13,000 total weights and biases to be learned
5. Understanding what weights and biases represent—what patterns they detect and when neurons activate—is essential for debugging networks and discovering unexpected solutions
6. Much of machine learning reduces to linear algebra: the transition between layers is compactly expressed as matrix-vector multiplication followed by element-wise activation functions
7. Learning means finding the right settings for all weights and biases so the network solves the problem, though networks may work for different reasons than initially expected

The Digit Recognition Problem and Network Architecture0:06

• Handwritten digit images in the MNIST dataset are rendered at extremely low resolution of 28x28 pixels.0:06
• Different handwritten 3s have very different pixel values from one image to the next, yet humans recognize them as the same digit effortlessly while the brain's light-sensitive cells firing are completely different for each image.0:22
• Writing a program that takes a 28x28 pixel grid as input and outputs a single number between 0 and 10 identifying the digit transforms from 'comically trivial' (for humans) to 'dauntingly difficult' (for traditional programming).0:54
• A neuron in a neural network is simply a thing that holds a number, specifically a number between 0 and 1.2:52
• The input layer of a digit recognition network contains 784 neurons (28x28 pixels), with each neuron holding a grayscale value from 0 (black pixels) to 1 (white pixels).3:08
• The number inside a neuron is called its 'activation', and you can visualize each neuron as being lit up when its activation is a high number.3:25
• The output layer has 10 neurons, each representing one digit (0-9), where the activation (between 0 and 1) represents how much the system thinks the given image corresponds with that digit.3:49
• Hidden layers are the layers between input and output that handle the process of recognizing digits. This example network uses two hidden layers, each with 16 neurons.4:03
• The choice of two hidden layers with 16 neurons each is somewhat arbitrary—the two layers were chosen for motivational purposes and 16 was just a nice number to fit on screen. In practice, there is a lot of room for experimentation with network structure.4:17

How Information Flows Through Layers4:33

• In neural networks, activations in one layer determine the activations of the next layer, loosely analogous to how groups of biological neurons firing cause certain others to fire.4:33
• When feeding an image into a trained network, you light up all 784 neurons of the input layer according to the brightness of each pixel, which causes cascading patterns through subsequent layers until the brightest neuron in the output layer represents the network's choice for what digit the image represents.5:03
• The ideal hope for middle layers is that each neuron in the second-to-last layer corresponds with one subcomponent of digits (like a loop at the top for 9 or 8), where any generally loopy pattern towards the top would set off a specific neuron with activation close to 1.6:07
• A 9 has a loop up top and a line on the right; an 8 has a loop up top paired with another loop down low; a 4 breaks down into three specific lines.5:54
• Recognizing subcomponents like loops can break down further: the second layer might recognize various little edges, where a long line (in digits 1, 4, or 7) is just a long edge or a pattern of several smaller edges.6:53
• The hoped-for hierarchical pattern detection: an image lights up neurons for 8-10 specific little edges, which light up neurons for the upper loop and long vertical line, which then light up the neuron for a 9.7:23
• Beyond image recognition, many intelligent tasks break down into layers of abstraction—parsing speech involves taking raw audio and picking out distinct sounds, which combine to make syllables, which combine to form words, which combine to make phrases and more abstract thoughts.8:08

Weights, Biases, and Activation Functions9:08

• To determine how one layer's activations influence the next, assign a weight to each connection between a neuron and neurons from the previous layer, then compute the weighted sum of all activations from the first layer according to these weights.9:08
• Weights can be visualized as a grid where green pixels indicate positive weights and red pixels indicate negative weights, with pixel brightness representing the weight's value.9:27
• To detect an edge in a specific region: set weights to zero for almost all pixels except positive weights in the target region, and add negative weights for surrounding pixels so the sum is largest when middle pixels are bright but surrounding pixels are darker.9:42
• The sigmoid function (also called logistic curve) squishes the real number line into the range between 0 and 1, where very negative inputs end up close to 0, positive inputs end up close to 1, and it steadily increases around input 0.10:32
• A bias is an additional number (like negative 10) added to the weighted sum before applying the sigmoid function, controlling how high the weighted sum needs to be before the neuron starts getting meaningfully active.11:11
• The weights tell you what pixel pattern a neuron is picking up on, and the bias tells you how high the weighted sum needs to be before the neuron starts getting meaningfully active.11:23
• Every neuron in the second layer connects to all 784 pixel neurons from the first layer, with each of those 784 connections having its own weight, plus each neuron has its own bias.11:38
• With a hidden layer of 16 neurons, there are 784 × 16 weights plus 16 biases just for connections from the first layer to the second; in total, this network has almost exactly 13,000 total weights and biases that can be tweaked to make the network behave in different ways.11:59

Mathematical Representation and What Learning Means12:31

• Learning in neural networks refers to getting the computer to find a valid setting for all the weights and biases so that it will solve the problem at hand.12:31
• Understanding what weights and biases actually mean provides a starting place for experimenting with how to change the network structure to improve performance when it doesn't work as anticipated.13:04
• When a network does work but not for expected reasons, examining the weights and biases is a good way to challenge assumptions and expose the full space of possible solutions.13:14
• Neural network connections can be represented compactly by organizing activations from one layer into a column vector and weights as a matrix, where each row corresponds to connections between one layer and a particular neuron in the next layer.13:40
• Taking the weighted sum of activations in the first layer according to the weights corresponds to one term in the matrix-vector product.13:58
• "So much of machine learning just comes down to having a good grasp of linear algebra."14:14
• Biases are represented by organizing them into a vector and adding the entire vector to the matrix-vector product, then applying the sigmoid function to each component of the resulting vector.14:29
• Writing the weight matrix and vectors as symbols allows the full transition of activations from one layer to the next to be expressed in an extremely tight expression, making code simpler and faster since libraries optimize matrix multiplication.14:55
• Each neuron is more accurately thought of as a function that takes in the outputs of all neurons in the previous layer and outputs a number between 0 and 1.15:27
• The entire neural network is just a function that takes in 784 numbers as input and spits out 10 numbers as output, involving 13,000 parameters in the form of weights and biases.15:39
• The network function involves iterating many matrix-vector products and the sigmoid squishification function, but it's just a function nonetheless.15:55

Insights

• The gap between human perception and computational difficulty reveals why neural networks are necessary: what seems 'comically trivial' to humans (recognizing a 3 regardless of handwriting style) becomes 'dauntingly difficult' when you must explicitly program rules, because the pixel patterns vary completely between instances
• The hierarchical abstraction pattern in neural networks mirrors how many intelligent tasks naturally decompose—from raw sensory input to low-level features to mid-level components to high-level concepts—suggesting this architecture captures something fundamental about information processing
• Examining weights and biases when a network works is as important as when it fails: networks may solve problems through unexpected mechanisms, and understanding the actual learned solution space challenges assumptions and reveals possibilities you hadn't considered
• The compactness of matrix notation is not just mathematical elegance—it directly translates to computational efficiency, as optimized linear algebra libraries can execute these operations far faster than explicit loops, making the mathematical abstraction practically essential

Action Items

• Visualize the 784 input neurons of a digit recognition network by mapping a 28×28 pixel image to activation values between 0 and 1 to understand how images become numerical inputs
• Experiment with different network structures (varying the number of hidden layers and neurons per layer) to develop intuition for how architecture choices affect performance
• Practice representing neural network computations using matrix-vector notation to build the linear algebra foundation essential for machine learning
• When working with trained networks, examine the learned weights and biases to verify whether the network is detecting the patterns you expected or solving the problem in surprising ways
• Study how the sigmoid function transforms weighted sums into activations by plotting it and testing different bias values to understand when neurons become 'meaningfully active'

Resources

• MNIST dataset — standard benchmark dataset of 28×28 pixel handwritten digit images used for training and testing digit recognition systems