How Neural Networks Learn: Walking Downhill in Fog

Introduction

Everyone says AI "learns", but what actually changes inside the computer? Some numbers, called weights, get nudged a tiny bit, thousands of times, until the predictions get better. Today you'll see exactly how, trace a prediction by hand, and build a small neural network from scratch with nothing but NumPy.

By the end of this lesson you'll be able to:

Describe a neuron as a weighted sum of inputs followed by a squash
Trace one prediction through a tiny network by hand
Explain loss as a single number that says how wrong the model is
Explain gradient descent as repeated small steps downhill, and the effect of the learning rate
Read a loss curve
Implement forward pass, backward pass and training loop for a 2-layer network in NumPy
Compare training accuracy with test accuracy

Working principle

A neuron is a vote with weights

A neuron takes several numbers, multiplies each by a weight (how much it matters), adds them, adds a bias (how easily it says yes), and squashes the result.

z = w1*x1 + w2*x2 + w3*x3 + b        (weighted sum)
p = 1 / (1 + e^-z)                    (sigmoid: squashes z to a number between 0 and 1)

Think of a panel of three judges for "is this a ball?". Each judge looks at one clue (brightness, roundness, greenness). The weights say how much you trust each judge. The sigmoid turns the total into a confidence between 0 and 1.

A network is neurons in layers

Layer	Job	In our lab
Input	The raw numbers	64 pixel values of an 8 by 8 patch
Hidden	Combines inputs into useful patterns	8 neurons with ReLU (keep positive values, turn negatives into 0)
Output	The answer	1 neuron: probability of "ball"

"Learning" means finding good values for all the weights (here 64 × 8 + 8 + 8 × 1 + 1 = 529 numbers).

Loss: one number for "how wrong"

For yes/no questions we use binary cross-entropy:

loss = -( y*log(p) + (1-y)*log(1-p) )

If the truth is "ball" (y = 1) and the model says 0.99, the loss is tiny. If it says 0.01, the loss is huge. It punishes confident mistakes hardest.

Gradient descent: the step

For every weight we ask: "if I nudge this weight up a little, does the loss go up or down, and by how much?" That answer is the gradient. Then:

weight = weight - learning_rate * gradient

Computing all gradients, from the output back to the input, is called backpropagation. You don't need to do the calculus by hand, but you should see that it's just the chain rule applied layer by layer.

The learning rate

Learning rate	What happens
Too small	Slow progress. Training takes forever.
About right	Loss falls smoothly.
Too big	Jumps around, or the loss explodes.

Learning rate0.45

The curve is the loss; the ball is the weights. Each step goes downhill by slope × learning rate. Now: good steps: reaches the valley. Loss = 3.240.

Train versus test

A model can score 100% on the data it trained on and fail on new data. That's why lesson 13 kept a test set. The gap between training accuracy and test accuracy is a signal. We see the first example of it today and study it properly in lesson 20.

What you'll need

Your cv_course environment with Jupyter and Matplotlib

Build it

Use a fresh notebook or script. Each cell runs on its own.

Cell 1: one neuron, by hand

import numpy as np
 
 
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
 
 
# One neuron deciding "ball or not?" from 3 numbers about a patch
x = np.array([0.9, 0.8, 0.1])        # brightness, roundness, greenness of the patch
w = np.array([1.5, 2.0, -3.0])       # how much each clue matters (the weights)
b = -1.0                             # the neuron's bias (how easily it says yes)
 
z = np.dot(w, x) + b                 # weighted sum
p = sigmoid(z)                       # squash to a probability between 0 and 1
print("weighted sum z =", round(z, 3))
print("probability of ball =", round(p, 3))
 
y = 1                                # the true answer for this patch
loss = -(y * np.log(p) + (1 - y) * np.log(1 - p))
print("loss =", round(loss, 3))

Check it on paper: 1.5 × 0.9 + 2.0 × 0.8 + (−3.0) × 0.1 − 1.0 = 1.35 + 1.6 − 0.3 − 1.0 = 1.65.

Questions: change the greenness to 0.9. What happens to z, p and the loss? Why does the third weight being negative make sense for a ball on grass?

Cell 2: walking downhill

import numpy as np
import matplotlib
matplotlib.use("Agg")              # remove this line if you want a plot window in a notebook
import matplotlib.pyplot as plt
 
 
# loss(w) = (w - 3)^2 + 1. The valley is at w = 3, but we only feel the slope under our feet.
def loss(w): return (w - 3) ** 2 + 1
def slope(w): return 2 * (w - 3)
 
 
for lr in (0.1, 0.9, 1.1):
    w = -2.0
    path = [w]
    for step in range(15):
        w = w - lr * slope(w)
        path.append(w)
    print(f"learning rate {lr}: final w = {w:.3f}, loss = {loss(w):.3f}")
    plt.plot(path, label=f"lr {lr}")
 
plt.axhline(3, color="grey", linestyle="--")
plt.xlabel("step")
plt.ylabel("w")
plt.legend()
plt.savefig("walk.png", dpi=100)

You should see: 0.1 creeps slowly towards 3, 0.9 swings back and forth across 3 but settles, 1.1 gets further away each step and explodes.

Question: which of the three learning rates would you pick? What would you try next?

Cell 3: a 2-layer network from scratch

The data is synthetic: 8 by 8 grey patches of noisy "grass" with, half the time, a bright round blob (the "ball") somewhere in it. Change NOISE and BALL_BOOST to make the problem harder.

import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
 
rng = np.random.default_rng(0)
NOISE = 0.08            # how noisy the "grass" is
BALL_BOOST = 0.5        # how much brighter the ball is than the grass
 
 
def make_patch(has_ball):
    """An 8x8 grey patch of grass-like noise, with a bright round blob if has_ball."""
    patch = rng.normal(0.3, NOISE, (8, 8))
    if has_ball:
        cy, cx = rng.integers(2, 6, 2)
        yy, xx = np.mgrid[0:8, 0:8]
        patch[(yy - cy) ** 2 + (xx - cx) ** 2 <= 2.5] += BALL_BOOST
    return np.clip(patch, 0, 1)
 
 
def make_data(n):
    y = rng.integers(0, 2, n)
    X = np.array([make_patch(label).ravel() for label in y])      # each patch becomes 64 numbers
    return X, y.reshape(-1, 1)
 
 
X_train, y_train = make_data(400)
X_test, y_test = make_data(200)
 
 
def sigmoid(z): return 1 / (1 + np.exp(-z))
 
 
# ---- a 2-layer network: 64 inputs -> 8 hidden neurons -> 1 output ----
W1 = rng.normal(0, 0.3, (64, 8)); b1 = np.zeros((1, 8))
W2 = rng.normal(0, 0.3, (8, 1));  b2 = np.zeros((1, 1))
lr = 0.5
history = []
 
for epoch in range(300):
    # forward pass: data flows through the network
    h = np.maximum(0, X_train @ W1 + b1)            # hidden layer with ReLU
    p = sigmoid(h @ W2 + b2)                        # output probability
    loss = -np.mean(y_train * np.log(p + 1e-9) + (1 - y_train) * np.log(1 - p + 1e-9))
 
    # backward pass: how much did each weight contribute to the error?
    dz2 = (p - y_train) / len(X_train)
    dW2 = h.T @ dz2
    db2 = dz2.sum(axis=0, keepdims=True)
    dh = dz2 @ W2.T * (h > 0)
    dW1 = X_train.T @ dh
    db1 = dh.sum(axis=0, keepdims=True)
 
    # step downhill
    W1 -= lr * dW1
    b1 -= lr * db1
    W2 -= lr * dW2
    b2 -= lr * db2
 
    history.append(loss)
    if epoch % 50 == 0:
        print(f"epoch {epoch:3d}  loss {loss:.4f}")
 
 
def predict(X):
    return sigmoid(np.maximum(0, X @ W1 + b1) @ W2 + b2)
 
 
train_acc = np.mean((predict(X_train) > 0.5) == y_train)
test_acc = np.mean((predict(X_test) > 0.5) == y_test)
print(f"train accuracy {train_acc:.2%}   test accuracy {test_acc:.2%}")
 
plt.plot(history)
plt.xlabel("epoch")
plt.ylabel("loss")
plt.savefig("loss.png", dpi=100)
 
fig, axes = plt.subplots(2, 6, figsize=(9, 3.2))
for ax, x, y in zip(axes.ravel(), X_test[:12], y_test[:12]):
    ax.imshow(x.reshape(8, 8), cmap="gray", vmin=0, vmax=1)
    ax.set_title(f"true {int(y[0])}  pred {predict(x[None])[0, 0]:.2f}", fontsize=7)
    ax.axis("off")
plt.savefig("preds.png", dpi=100)

With the default settings the loss falls from about 0.75 to under 0.02 and both accuracies reach about 100%. The task is easy.

Make it harder

Set NOISE = 0.15 and BALL_BOOST = 0.25, then run the cell again. A typical result is about 90% training accuracy and 79% test accuracy. The network has learned the training patches better than it has learned "ball". That gap is overfitting, and we'll study it in lesson 20.

Questions: where does the loss curve flatten? What happens with lr = 5? With lr = 0.001? What if the hidden layer has 2 neurons instead of 8?

If it goes wrong

Error	Cause	Fix
`ValueError: shapes not aligned`	A weight matrix has the wrong shape	Print `.shape` of each array. Inputs are (n, 64), `W1` is (64, 8), `W2` is (8, 1).
Loss is `nan`	Learning rate too high, or `log(0)`	Lower `lr`. Keep the `1e-9` inside the `log`.
Loss falls then rises	Learning rate too high	Halve it
Accuracy stuck near 50%	Learning rate too small, or too few epochs	Raise `lr`, or run more epochs
Different results every run	Random numbers	The seed `default_rng(0)` fixes this. Change it to see the spread.
You wonder "what is `h > 0`?"	The ReLU derivative	It is 1 where the neuron was active and 0 where it wasn't

Try it

Add a second hidden layer and compare
Plot training and test accuracy at each epoch on one chart
Try tanh instead of ReLU

Check what you learnt

What are the weights in a neural network?
What does the loss measure?
What is gradient descent?
What happens if the learning rate is too big?
Why check test accuracy, not just training accuracy?

Conclusion

A neuron multiplies its inputs by weights, adds them up and squashes the result. A network is layers of neurons. Training measures how wrong it is (the loss), works out which way each weight should move (the gradient), and takes a small step (the learning rate), over and over: walking downhill in fog.

What next?

Next: CNNs: convolutional layers explained. You'll meet the kind of network built for pictures: the CNN.