Dimensionality Reduction: LDA in Code¶
Why this matters¶
The PCA code bridge showed this pattern:
LDA adds one important ingredient:
This page uses one tiny code example to make that difference visible.
The goal is not to build an ideal classifier. The goal is to see that LDA needs labels and creates a lower-dimensional view designed around class separation.
Mental model¶
PCA asks:
LDA asks:
So LDA needs both:
Without y, LDA does not know which groups it should separate.
Core ideas¶
- LDA is supervised dimensionality reduction.
- Supervised means the method uses labels.
- The feature table is still called
X. - The labels are usually called
y. lda.fit_transform(X_scaled, y)uses both features and labels.- With two classes, LDA can create at most one useful component.
- The reduced values are new class-separating coordinates, not original feature columns.
One tiny LDA run¶
This invented apartment dataset has six rows, three numeric feature columns, and one label per row.
The label says whether the apartment is budget or luxury.
import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.preprocessing import StandardScaler
X = np.array([
[35, 1, 210],
[40, 1, 230],
[45, 2, 260],
[85, 3, 520],
[95, 4, 610],
[105, 4, 690],
])
y = np.array([
"budget",
"budget",
"budget",
"luxury",
"luxury",
"luxury",
])
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
lda = LDA(n_components=1)
X_lda = lda.fit_transform(X_scaled, y)
if X_lda[y == "budget"].mean() > X_lda[y == "luxury"].mean():
X_lda = -X_lda
class_means = {}
for label in np.unique(y):
class_means[label] = round(float(X_lda[y == label].mean()), 2)
print("Original shape:", X.shape)
print("Label shape:", y.shape)
print("Reduced shape:", X_lda.shape)
print("\nFirst three apartments after LDA:")
print(np.round(X_lda[:3], 2))
print("\nClass means in LDA space:")
print(class_means)
print("\nExplained variance ratio:")
print(np.round(lda.explained_variance_ratio_, 2))
Output:
Original shape: (6, 3)
Label shape: (6,)
Reduced shape: (6, 1)
First three apartments after LDA:
[[-6.02]
[-4.10]
[-4.12]]
Class means in LDA space:
{'budget': -4.75, 'luxury': 4.75}
Explained variance ratio:
[1.]
How to read the output¶
LDA uses labels¶
The feature table has this shape:
That means:
The label array has this shape:
That means:
This is the first practical difference from PCA. PCA fits with only X. LDA fits with X and y.
The reduced data has one new column¶
After LDA:
That means:
With two classes, LDA can create at most one useful discriminant component.
Plain version:
The LDA values are class-separating coordinates¶
The first three apartments after LDA are:
Those first three rows are all budget apartments.
The class means in LDA space are:
Read this as:
budget apartments are centered on the negative side
luxury apartments are centered on the positive side
The exact sign is not the important part. The important part is that LDA found a one-dimensional view where the labeled groups are far apart.
Explained variance has a different feel here¶
The explained variance ratio is:
Because there are only two classes, LDA has only one useful class-separating direction in this example.
So all of the discriminant information LDA can represent here lives in that one component.
This is different from saying the original dataset has no other details. It only says that, for separating two classes with LDA, one discriminant axis is enough.
When the LDA signal is weaker¶
The output above is a strong output signal for separation because the class centers are far apart:
Not every LDA result looks like that.
Suppose an LDA result printed:
The class centers are close together.
That is a weaker output signal.
It suggests LDA did not find a strong class-separating direction in this reduced view.
Use this careful wording:
LDA is eligible because labels are available, but this output gives weak evidence
that LDA separated the classes well. I would compare classifier performance with
and without LDA before relying on it.
How this connects to the full lesson¶
The full lesson uses the Wine dataset with three classes.
That matters because LDA can produce at most:
So:
The same reading habit applies:
X tells LDA the feature values
y tells LDA the known classes
the output columns are new class-separating coordinates
For deciding whether LDA is appropriate before running code, see Dimensionality Reduction: Why PCA and LDA Work Differently.
Common traps¶
LDA is not PCA with a different name
PCA ignores labels and preserves spread. LDA uses labels and tries to separate classes.
LDA needs labels
lda.fit_transform(X_scaled, y) needs y because class labels define the groups LDA should separate.
The LDA numbers are not original columns
The reduced values are positions on a learned class-separating axis. They are not size_m2, rooms, or price_k.
One component here does not mean one original feature
The single LDA component is a new direction built from the original features.
Eligible does not mean useful
LDA may be logically allowed when labels exist, but you still need output or metrics to decide whether it separated the classes well.
Check yourself¶
What extra input does LDA use that PCA ignores?
LDA uses the class labels, usually called y.
What shape did the data have before LDA?
X had shape (6, 3): six rows and three original feature columns.
What shape did the data have after LDA?
X_lda had shape (6, 1): six rows and one new LDA component column.
Why can this two-class example have only one useful LDA component?
LDA can produce at most number of classes - 1 useful components.
What did the class means show?
The budget and luxury apartments landed on different sides of the learned LDA axis.
If the LDA class means are close together, what should you say?
LDA is eligible if labels exist, but the output gives weaker evidence that it separated the classes well.
Next¶
Next, read Dimensionality Reduction.
The full lesson uses these same ideas with the Wine dataset, where LDA can create two class-separating components because there are three wine classes.
Source anchors¶
- Supports:
study-guide/docs/lessons/05-dimensionality-reduction.md - Source file:
notebooks/Module2/05-Dimensionality Reduction.ipynb - Key source concepts prepared here: LDA, labels, supervised dimensionality reduction, class-separating components, maximum LDA components