Attention Mechanisms: Why the Math Looks Like This¶
Why this matters¶
The first bridge lesson gave the plain idea:
The full lesson then starts using vector words:
- dot product
- same direction
- softmax
- sum to 1
- weighted sum
- scaled dot-product attention
This page explains why those pieces are there.
You do not need calculus for this. The goal is to make the math-shaped parts feel motivated instead of random.
Mental model¶
Attention needs a controlled way to answer two questions:
1. Which tokens match what I am looking for?
2. After I know that, how much should I borrow from each token?
The math exists because the model is working with vectors, not English words.
So the transformer needs numeric versions of these ideas:
| Human idea | Attention mechanism |
|---|---|
| Does this token match what I need? | dot product score |
| Which matches matter most? | softmax weights |
| How do I combine the useful information? | weighted sum of values |
The full lesson's attention formula is just a compact version of that table.
Core ideas¶
- A dot product is a simple way to score whether two vectors point in a similar direction.
- "Same direction" means the number patterns line up.
- Raw scores are not yet good borrowing amounts.
- Softmax turns raw scores into attention weights.
- Attention weights sum to 1 so they behave like one full unit of attention split across tokens.
- Queries and keys decide the weights.
- Values provide the information that gets mixed.
- Scaling keeps dot product scores from becoming too large before softmax.
Walkthrough¶
Start with the question attention is answering¶
Imagine the model is updating the token it in this sentence:
The model wants to know:
A human might say:
The model needs to do that with numbers.
Vectors are number patterns¶
A vector is just a list of numbers.
For teaching, use tiny two-number vectors:
Do not treat these as real model numbers. They are small fake numbers chosen to make the idea visible.
Read the query for it as:
Read each key as:
What "same direction" means¶
In vector language, "same direction" means the number patterns line up.
These point in the same direction:
The second vector is basically a bigger version of the first. Both numbers increased in the same pattern.
These point in the opposite direction:
The signs are flipped.
A tiny arrow picture:
When the full lesson says two vectors point in a similar direction, you can read it as:
Why use a dot product?¶
A dot product gives a single score for how much two vectors line up.
For two-number vectors:
Use the query for it:
Compare it with the keys:
key for "cat" = [3, 1]
score = 2*3 + 1*1 = 7
key for "mat" = [0, 4]
score = 2*0 + 1*4 = 4
key for "because" = [1, 1]
score = 2*1 + 1*1 = 3
So the raw scores are:
The model has not borrowed anything yet. It has only measured match strength.
Higher dot product means:
Raw scores are not good borrowing amounts¶
The scores tell us that cat is the best match, but they are not yet clean attention weights.
Raw scores can be:
- too large
- negative
- hard to compare across different rows
- not adding up to a meaningful total
For borrowing, we want something like percentages:
That is why attention normalizes the scores.
Why should the weights sum to 1?¶
Weights that sum to 1 behave like a recipe.
For example:
context for "it" =
70% information from "cat"
20% information from "mat"
10% information from "because"
Those weights add to 100%.
That means the model is splitting one full unit of attention across tokens.
In decimal form:
So when the full lesson says attention weights sum to 1, it means:
Why not just divide by the total?¶
It is tempting to do this:
Sometimes that works in simple examples. But attention scores can be negative.
Example:
If you divide by the total:
Now one "weight" is negative.
That is awkward for the beginner mental model:
Softmax avoids that problem.
What softmax does¶
Softmax is a score-to-weight converter.
It does three useful things:
1. makes every weight positive
2. makes all weights add up to 1
3. makes bigger scores stand out more
For our raw scores:
Softmax turns them into weights like this:
The exact percentages are less important than the job softmax performs:
Softmax is not just normal division. It exaggerates stronger matches, while still making the result positive and sum to 1.
Values are what actually get mixed¶
Queries and keys decide the attention weights.
Values provide the information that gets mixed.
For the same example:
query from "it" dot key from "cat" -> score for cat
query from "it" dot key from "mat" -> score for mat
query from "it" dot key from "because" -> score for because
scores through softmax -> attention weights
attention weights mix the values
Plain version:
queries and keys answer: how much should I look there?
values answer: what information do I get from there?
So if the weights are:
Then the new representation for it mostly borrows from the value of cat.
The weighted sum¶
A weighted sum is just the recipe being applied.
Imagine the value vectors are:
And the attention weights are:
The context vector is:
You do not need to calculate the final numbers here.
The point is:
the new "it" vector is mostly cat information,
with a small amount of mat and because information mixed in
That is attention's "borrowing" step.
Why scale the scores?¶
The full lesson also divides the scores by the square root of the key/query size.
This is called scaling.
The beginner intuition:
A two-number vector has two chances to add to the score.
A sixty-four-number vector has sixty-four chances to add to the score.
Large scores can make softmax too confident too quickly:
Scaling cools the scores down before softmax.
So when you see:
read it as:
How this maps to the full lesson code¶
After the plain example, the code shape is much less mysterious:
Read it as:
queries @ keys.T -> compare queries with keys
softmax(scores) -> turn raw scores into weights
weights @ values -> mix value information
Scaled dot-product attention adds one cooling step:
scores = queries @ keys.T
scores = scores / key_size**0.5
weights = torch.softmax(scores, dim=-1)
context = weights @ values
Read the whole flow as:
Term Decoder¶
| Term | Friendly meaning |
|---|---|
| vector | list of numbers representing a token |
| direction | the pattern of numbers in a vector |
| same direction | similar number pattern |
| dot product | score for how much two vectors line up |
| attention score | raw match strength before normalization |
| softmax | converter from scores to positive weights that sum to 1 |
| attention weight | how much to borrow from a token |
| weighted sum | recipe-style mixture using weights |
| query | what the current token is looking for |
| key | what another token can be matched by |
| value | information another token can contribute |
| scaling | cooling down scores before softmax |
Common traps¶
Dot product is not magic
In this lesson, read dot product as a similarity score. It gives a bigger number when the query and key number patterns line up more strongly.
Summing to 1 is not decoration
Weights sum to 1 so the model has a controlled mixture. It is splitting one full unit of attention across tokens.
Softmax is not simple division
Simple division can behave badly with negative scores. Softmax makes weights positive, makes them sum to 1, and emphasizes stronger matches.
Values do not decide the weights
Queries and keys decide the scores and weights. Values are the information mixed after the weights are known.
Same direction does not require a literal picture
The arrow picture is only a teaching aid. For model vectors, same direction mostly means the number patterns are similar.
Check yourself¶
If two vectors point in a similar direction, should their dot product be high or low?
High. A higher dot product means the vectors line up more strongly.
Why should attention weights add up to 1?
So they behave like a controlled recipe: one full unit of attention split across tokens.
Which pair decides the score: query-key, query-value, or key-value?
Query-key. The query from the current token is compared with keys from tokens it might attend to.
What does softmax do to raw attention scores?
It turns them into positive weights that sum to 1, while giving stronger matches more influence.
What gets mixed to create the context vector?
The value vectors, using the attention weights.
Why are scores scaled before softmax?
To keep dot product scores from becoming too large when vectors have many dimensions.
Now Read the Full Lesson¶
The full attention lesson uses the same flow with real tensors:
query-key dot products -> scores
scale scores -> keep them controlled
softmax -> attention weights
weights times values -> context vectors
When you see the compact formula or PyTorch code, translate it back to:
Next: Attention Mechanisms
Source anchors¶
- Supports the dot-product attention, softmax, scaling, query/key/value, and weighted-sum sections in
13-attention-mechanisms.md. - Uses simplified teaching numbers rather than notebook-faithful tensor values.