Neural Networks: From Learning Loop to NumPy Shapes¶

Why this matters¶

The previous lesson explained the learning loop:

make prediction -> measure loss -> compute gradients -> update weights

The implementation lesson turns that loop into NumPy code. The hard part is often not the idea of learning. The hard part is reading shapes like:

[batch_size, 784] @ [784, 50] = [batch_size, 50]

This bridge explains the shape mechanics before the full lesson uses MNIST, one-hot labels, weight matrices, transposes, and forward() / backward() methods.

You do not need advanced linear algebra here. The goal is to read the code without getting lost.

Mental model¶

Keep the cafe-order example from the first neural-network bridge.

Each order has three input features:

items ordered
drink size
includes food

Suppose the network predicts two possible classes:

small order
large order

Now add one hidden layer with two hidden units:

3 input features -> 2 hidden units -> 2 output classes

That tiny network has the same shape logic as the MNIST network in the full lesson:

784 pixel inputs -> 50 hidden units -> 10 digit classes

Only the numbers are larger.

Core ideas¶

A batch is a group of examples processed together.
A feature vector is one row of input numbers for one example.
A batch input matrix has one row per example and one column per feature.
A fully connected layer uses a weight for every input-to-output connection.
A bias adds one extra adjustable value per output unit.
Matrix multiplication lets the network process a whole batch at once.
.T means transpose: flip rows and columns so shapes line up.
A forward pass moves from inputs to hidden activations to outputs.
A backward pass computes gradients with the same shapes as the parameters they update.
The full MNIST lesson uses bigger numbers, but the same shape rules.

Walkthrough¶

One example becomes one row¶

Start with one cafe order:

order	items ordered	drink size	includes food
A	1	0	0

For teaching, encode drink size and food as simple numbers:

small drink = 0
large drink = 1
no food = 0
food = 1

So one order becomes one feature vector:

[1, 0, 0]

Shape:

[3]

That means:

3 input features

A batch becomes a table of rows¶

Neural-network code usually processes more than one example at a time.

Three orders can be stored as one input matrix:

X =
[
  [1, 0, 0],
  [3, 1, 1],
  [2, 0, 1],
]

Shape:

[3, 3]

Read that as:

[batch_size, input_features]

For this tiny batch:

batch_size = 3
input_features = 3

A hidden layer has one weight row per hidden unit¶

The hidden layer has two hidden units.

Each hidden unit receives all three input features:

hidden unit 1 sees: items, drink, food
hidden unit 2 sees: items, drink, food

So the hidden weight matrix can be stored as:

weight_hidden =
[
  [w11, w12, w13],
  [w21, w22, w23],
]

Shape:

[2, 3]

Read that as:

[hidden_units, input_features]

There are also two hidden bias values:

bias_hidden = [b1, b2]

Shape:

[2]

One bias per hidden unit.

Why the transpose appears¶

The forward pass needs this shape result:

[batch_size, input_features] @ [input_features, hidden_units] = [batch_size, hidden_units]

For the cafe example:

[3, 3] @ [3, 2] = [3, 2]

But the weights were stored as:

weight_hidden shape = [2, 3]

That is:

[hidden_units, input_features]

So the code uses .T to flip the weight matrix:

weight_hidden.T shape = [3, 2]

Then the multiplication lines up:

hidden_scores = X @ weight_hidden.T + bias_hidden

Shape check:

X:               [3, 3]
weight_hidden.T: [3, 2]
bias_hidden:        [2]
hidden_scores:  [3, 2]

The output has one row per order and one column per hidden unit.

Bias broadcasts across the batch¶

The bias has shape:

[2]

The hidden scores have shape:

[3, 2]

When NumPy adds the bias, it adds the same two bias values to every row:

row 1 hidden scores + [b1, b2]
row 2 hidden scores + [b1, b2]
row 3 hidden scores + [b1, b2]

This is called broadcasting. For this lesson, read it as:

reuse the same bias vector for each example in the batch

Activation changes values, not the shape¶

After the weighted sum, the network applies an activation function.

Plain version:

hidden_scores -> activation function -> hidden_activations

Shape:

[3, 2] -> [3, 2]

The activation changes the numbers. It does not change how many examples or hidden units there are.

The output layer repeats the same pattern¶

The output layer predicts two classes:

small order
large order

It receives two hidden values per example and produces two output values per example.

The output weights can be stored as:

weight_out shape = [2, 2]

Read that as:

[output_classes, hidden_units]

The forward pass uses the transpose again:

outputs = hidden_activations @ weight_out.T + bias_out

Shape check:

hidden_activations: [3, 2]
weight_out.T:       [2, 2]
bias_out:              [2]
outputs:            [3, 2]

The final output has:

one row per order
one column per class

One-hot labels match the output columns¶

If the classes are:

0 = small order
1 = large order

Then raw labels might look like:

[0, 1, 1]

For a two-output network, the targets can be written as one-hot rows:

[
  [1, 0],
  [0, 1],
  [0, 1],
]

Shape:

[3, 2]

That matches the output shape:

outputs: [3, 2]
targets: [3, 2]

This makes it possible to compare the model's two output values with the two target values for each example.

Backward pass shapes match the parameters¶

The bridge does not need the backpropagation formulas.

The shape rule is enough for now:

each gradient has the same shape as the parameter it updates

For the tiny cafe network:

weight_hidden: [2, 3]
grad_weight_hidden: [2, 3]

bias_hidden: [2]
grad_bias_hidden: [2]

weight_out: [2, 2]
grad_weight_out: [2, 2]

bias_out: [2]
grad_bias_out: [2]

The training loop can then update each parameter:

weight_hidden -= learning_rate * grad_weight_hidden
bias_hidden -= learning_rate * grad_bias_hidden
weight_out -= learning_rate * grad_weight_out
bias_out -= learning_rate * grad_bias_out

The subtraction works because each parameter and its gradient have matching shapes.

How this maps to MNIST¶

The full lesson uses handwritten digits instead of cafe orders.

The shape idea is the same:

Part	Cafe bridge	MNIST lesson
input features	3 order facts	784 pixel values
hidden units	2	50
output classes	2 order classes	10 digit classes
input batch	`[batch_size, 3]`	`[batch_size, 784]`
hidden weights	`[2, 3]`	`[50, 784]`
hidden activations	`[batch_size, 2]`	`[batch_size, 50]`
output weights	`[2, 2]`	`[10, 50]`
outputs	`[batch_size, 2]`	`[batch_size, 10]`

So when the full lesson shows:

z_h = np.dot(x, self.weight_h.T) + self.bias_h

translate it as:

take the batch of input rows
multiply by hidden weights
add one hidden bias per hidden unit
produce one hidden row per example

The numbers are bigger, but the shape story is unchanged.

Term Decoder¶

Use this table when reading the implementation lesson.

Full lesson term	Read it as
batch	group of examples processed together
feature vector	one row of input numbers
matrix	table of numbers
shape	row/column layout of an array
weight matrix	trainable connection table between layers
bias vector	trainable extra values added after the weighted sum
transpose	flipped rows and columns so multiplication lines up
activation	value-changing step after a weighted sum
one-hot target	class label written as a row with one `1`
gradient	update signal with the same shape as its parameter

Common traps¶

Do not read [3, 2] as one mysterious number

It means two facts: 3 rows and 2 columns. In a network batch, rows usually mean examples and columns usually mean features, hidden units, or classes.

Do not forget the batch dimension

The model is not only processing one example. Most code is written to process a batch of examples at the same time.

Do not treat .T as decoration

The transpose flips the weight matrix so the columns and rows needed for multiplication line up.

Do not expect activations to change the shape

Activation functions usually change the values inside an array, not the array's layout.

Do not derive backpropagation before reading the code

First, know that gradients match the parameters they update. The full lesson shows where those gradients come from.

Check yourself¶

If X has shape [8, 3], what does 8 mean?

The batch contains 8 examples.

If a hidden layer has 2 units and receives 3 input features, what shape can its weight matrix have?

[2, 3], meaning one row per hidden unit and one column per input feature.

Why does the code use weight_hidden.T?

Because the stored weight matrix is [hidden_units, input_features], but multiplication from the input batch needs [input_features, hidden_units].

What shape comes from [batch_size, 3] @ [3, 2]?

[batch_size, 2].

Why does a one-hot target for two classes have two columns?

Each column corresponds to one class.

What shape should a gradient have?

The same shape as the parameter it updates.

Now Read the Full Lesson¶

The full lesson uses the same mechanics with MNIST:

[batch_size, 784] -> [batch_size, 50] -> [batch_size, 10]

When you see NumPy code, translate it back to:

batch rows -> weighted sums -> activations -> class scores

Next: Implementing a Multi-Layer NN

Source anchors¶

Supports: study-guide/docs/lessons/09b-implementing-a-multi-layer-nn.md
Source file: notebooks/Module2/09b-Implementing a Multi-Layer NN.ipynb
Key source concepts prepared here: batches, feature vectors, weight matrix shapes, bias vectors, transposes, one-hot targets, forward pass shapes, gradient shapes, MNIST shape mapping