Skip to content

Classification Finetuning: Why a GPT Can Classify

Why this matters

The previous lesson showed a GPT-style model learning to predict the next token:

tokens -> hidden states -> vocabulary scores

The next lesson uses that same kind of model for spam classification:

message -> GPT-style model -> not spam or spam

That can feel like a sudden switch. If GPT was built to predict words, why can it now predict a class label?

This bridge explains the mechanical change before the full spam lesson adds datasets, padding, balancing, training loops, and evaluation code.

Mental model

A GPT model has two broad parts:

reader body -> output head

The reader body turns tokens into hidden states. Each hidden state is a learned numeric summary of what the model knows at that token position.

The output head turns those hidden states into scores.

For pretraining, the output head answers:

Which vocabulary token should come next?

For classification finetuning, the output head answers:

Which class label fits this whole input?

So the bridge from generation to classification is mostly a change in what the final scores mean.

pretrained GPT:
hidden state -> 50,257 vocabulary scores

classifier GPT:
hidden state -> 2 class scores

Core ideas

  • The transformer body can still read the message and build hidden states.
  • The original output head turns hidden states into vocabulary-sized token scores.
  • A classification head turns hidden states into class scores instead.
  • Prompting asks the old vocabulary head to generate answer tokens.
  • Classification finetuning trains a new head to score labels directly.
  • A classifier produces one target label for the whole message, not one target token per position.
  • In a causal GPT model, the last token position can attend to all earlier tokens.
  • The last token position is therefore used as the message-level summary.
  • Cross-entropy can train class scores just as it trained vocabulary scores, but the target now means a class label.

Walkthrough

Start from next-token prediction

During pretraining, the model receives token IDs and returns vocabulary logits.

For a batch of two short sequences:

input shape:  [2, 4]
output shape: [2, 4, 50257]

Read that as:

2 examples
4 token positions per example
50,257 vocabulary scores at each position

Each row of 50,257 scores answers:

What token should come next after this position?

That is perfect for generation, because generation chooses the next token and appends it to the text.

Prompting still uses the vocabulary head

Suppose we ask a pretrained model:

Is this message spam? Answer yes or no.

The model still uses its original vocabulary head. It is not directly choosing between class 0 and class 1.

It is generating answer tokens:

"yes"
"no"
" maybe"
" this"
...

That can work sometimes, but it is indirect.

Plain version:

prompting = ask the language model to write an answer
classification = train the model to score the labels directly

The spam lesson shows why the direct classifier is useful.

Replace vocabulary scores with class scores

For binary classification, the model only needs two output scores:

0 -> not spam
1 -> spam

So the large vocabulary output head is replaced with a smaller classification head.

before:
hidden state -> 50,257 vocabulary logits

after:
hidden state -> 2 class logits

The transformer body is still useful. It has learned how to turn text into hidden states. The new head changes how those hidden states are interpreted.

The model still returns one row per token position

Replacing the head does not remove the token dimension.

For one four-token message:

before replacement:
[1, 4, 50257]

after replacement:
[1, 4, 2]

There are now two class scores at each token position.

That means every position has a possible answer like:

position 1 -> [not_spam_score, spam_score]
position 2 -> [not_spam_score, spam_score]
position 3 -> [not_spam_score, spam_score]
position 4 -> [not_spam_score, spam_score]

But the training label is not one label per token.

The message has one label:

"free prize now" -> spam

So the classifier needs one pair of class scores for the whole message.

Why use the last token position?

GPT uses causal attention.

That means each token position can only attend to itself and earlier positions:

position 1 can see: position 1
position 2 can see: positions 1-2
position 3 can see: positions 1-3
position 4 can see: positions 1-4

The last position has the widest view.

For a message like:

free prize now

the final token position can use information from the whole message before making its class scores.

That is why the spam classifier uses the last position:

logits = model(input_batch)
class_logits = logits[:, -1, :]

Shape translation:

logits:       [batch, tokens, 2]
class_logits: [batch, 2]
targets:      [batch]

The -1 means:

take the final token position from each example

The final : means:

keep both class scores

One message, one label

For classification, the target batch might look like:

targets: [0, 1, 1, 0]

Read that as:

example 1 -> not spam
example 2 -> spam
example 3 -> spam
example 4 -> not spam

The model output used for loss has matching rows:

class_logits:
[
  [score_for_not_spam, score_for_spam],
  [score_for_not_spam, score_for_spam],
  [score_for_not_spam, score_for_spam],
  [score_for_not_spam, score_for_spam],
]

Cross-entropy compares each row of scores with the correct class label.

Cross-entropy did not disappear

Pretraining used cross-entropy like this:

vocabulary scores -> correct next-token ID

Classification finetuning uses cross-entropy like this:

class scores -> correct class ID

The loss function is familiar. The meaning of the scores and targets has changed.

For spam classification:

target 0 means not spam
target 1 means spam

The model is not trying to generate the word spam. It is trying to make the correct class score larger than the incorrect one.

Term Decoder

Term Friendly meaning
vocabulary logits scores for possible next tokens
class logits scores for possible class labels
output head final layer that turns hidden states into scores
classification head output head whose columns represent class labels
target label correct class ID for one example
last token position final sequence position, used because it can see the whole earlier message
binary classification choosing between two labels

Common traps

Do not treat class labels as generated words

The classifier is not generating the word spam. It is scoring class 0 and class 1, then choosing the larger score.

Do not use every token position as a separate message prediction

The spam task has one label per message. The model returns scores at every token position, but the lesson uses the last position for the whole-message prediction.

Do not forget what the output head changed

The transformer body can stay mostly the same, but the output head changes the meaning and size of the final scores.

Do not confuse accuracy with the training loss

Accuracy counts correct labels. Cross-entropy provides the differentiable loss that trains the class scores.

Check yourself

What does the original GPT output head produce?

Vocabulary logits: scores for possible next tokens.

What does the classification head produce?

Class logits: scores for the allowed class labels.

Why is prompting different from classification finetuning?

Prompting uses the vocabulary head to generate answer tokens. Classification finetuning trains a head that directly scores labels.

Why does the classifier use the last token position?

In a causal GPT model, the last position can attend to the earlier positions in the message, so it has the best single-token view of the whole input.

For a batch of 8 messages and 2 classes, what shape should class_logits have after selecting the last token?

[8, 2]: one row per message and one score per class.

What does cross-entropy compare during classification finetuning?

It compares the class logits for each message with the correct class label.

Now Read the Full Lesson

The spam classification lesson adds the concrete workflow:

SMS data -> tokenized messages -> padded batches -> GPT classifier -> spam/not spam accuracy

When you see:

model(input_batch)[:, -1, :]

translate it as:

run the GPT classifier
take the final token position
keep the two class scores

Next: Spam Classification Finetuning

Source anchors

  • Supports: study-guide/docs/lessons/17-spam-classification-finetuning.md
  • Source file: notebooks/Module2/17-Spam Classification Finetuning.ipynb
  • Key source concepts prepared here: vocabulary logits, class logits, output-head replacement, prompting versus direct classification, last-token selection, one label per message, cross-entropy over class scores