Dimensionality Reduction: Why PCA and LDA Work Differently

Why this matters

The first bridge lesson explained the basic goal:

many columns -> fewer useful summary columns

The full lesson then uses words like:

• axes
• variance
• projection
• covariance
• eigenvectors
• eigenvalues
• explained variance
• class separation

This page explains those mechanics without asking you to hand-compute the linear algebra.

Mental model

Think of your data as a cloud of points.

Each feature is an axis.

If your dataset has two features, a row becomes a point on a 2D chart:

x-axis: apartment size
y-axis: price

If your dataset has three features, a row becomes a point in 3D.

If your dataset has 13 features, like the Wine dataset in the full lesson, each row is still a point, but it lives in a space that is hard to draw.

Dimensionality reduction asks:

Can we make a lower-dimensional view that keeps the useful structure?

PCA and LDA answer that question differently.

PCA: keep the biggest overall spread
LDA: separate known classes

Core ideas

• Features can be imagined as axes on a chart.
• A row becomes a point in the feature space.
• Reducing dimensions is like making a useful shadow of a higher-dimensional object.
• PCA chooses directions that preserve spread.
• Variance means spread.
• Covariance means features move together.
• Eigenvectors are important directions PCA discovers.
• Eigenvalues say how much spread those directions capture.
• Explained variance ratio says how much of the original spread a component keeps.
• LDA uses labels and chooses directions that separate classes.

Walkthrough

Features as axes

Start with two apartment features:

size
price

You can draw them as chart axes:

price
  ^
  |
  |
  +------------> size

Each row becomes one point.

Add a third feature:

distance_to_station

Now each row needs three axes.

Add many more features, and the same idea continues, even though you can no longer draw it easily.

Projection means making a shadow

Projection is the idea of showing high-dimensional data in fewer dimensions.

A simple analogy:

a 3D object casts a 2D shadow

Different light angles create different shadows.

Some shadows preserve the shape better than others.

For data:

many-feature data -> high-dimensional object
fewer components  -> lower-dimensional shadow

A shadow is useful, but it is not the full object.

That is why dimensionality reduction can help while still losing information.

PCA chooses a shadow with the most spread

PCA means Principal Component Analysis.

PCA looks for directions where the data is most spread out.

Imagine a flat cloud of points:

      .       .
   .     .       .
.      .     .
   .       .

If the cloud stretches mostly from bottom-left to top-right, PCA chooses that stretched direction first.

That first direction is called the first principal component.

Plain version:

principal component = a new axis chosen by PCA

The first principal component keeps the largest spread.

The second principal component keeps the next largest spread that is not just a repeat of the first.

Variance means spread

Variance is a measure of spread.

Low variance:

points are packed close together

High variance:

points are spread far apart

PCA cares about variance because spread often carries structure.

But this has a trap:

high variance can be useful signal,
but it can also be noise or scale effects

PCA does not know your target labels. It only sees the feature values.

Why standardization matters

PCA looks for large numeric spread.

That means feature scale matters.

Example:

bedrooms: 1 to 6
size_m2: 20 to 300
price_euros: 100000 to 2000000

price_euros has huge numbers compared with bedrooms.

Without standardization, PCA may treat price as extremely important mostly because its numeric scale is large.

Standardization means:

put features on comparable scales before PCA judges spread

In the full lesson, scikit-learn's StandardScaler does this.

Covariance means features move together

Covariance asks:

when one feature goes up, what tends to happen to another feature?

Examples:

size and bedrooms:
often move up together

distance_to_station and price:
may move in opposite directions

door_color and price:
probably no consistent movement together

Plain meanings:

Covariance type	Friendly meaning
positive	features tend to increase together
negative	one tends to increase while the other decreases
near zero	no strong linear movement together

A covariance matrix is just a table of these pairwise movement relationships.

PCA uses that table to find strong directions of shared movement.

Eigenvectors and eigenvalues are role names here

The full lesson mentions eigenvectors and eigenvalues.

You do not need the deep linear algebra meaning to read the lesson.

For PCA, use these role names:

eigenvector = important direction PCA discovered
eigenvalue  = how much spread is captured in that direction

Mapping:

PCA idea	Linear algebra word
principal component direction	eigenvector
importance of that direction	eigenvalue

So if an eigenvalue is large, that direction captures a lot of spread.

If an eigenvalue is small, that direction captures less spread.

Explained variance ratio is the sanity check

Explained variance ratio answers:

how much of the original spread did this component keep?

Example:

component 1 keeps 45%
component 2 keeps 25%

Together:

45% + 25% = 70%

That means a 2D PCA view keeps about 70% of the spread.

It also means 30% is not shown in that 2D view.

This is why a pretty PCA plot can still be incomplete.

LDA uses labels and changes the goal

LDA means Linear Discriminant Analysis.

Like PCA, it creates a lower-dimensional view.

Unlike PCA, it uses class labels.

Suppose apartments are labeled:

budget
midrange
luxury

PCA asks:

Which direction keeps the biggest overall spread?

LDA asks:

Which direction separates budget, midrange, and luxury best?

That is a different goal.

LDA wants far-apart groups with tight interiors

LDA likes a direction where:

class centers are far apart
points inside each class are close together

Plain version:

good LDA view =
groups are separated from each other
and each group is not too scattered

This is why LDA can choose a direction that PCA would not choose.

PCA may prefer a direction with lots of total spread.

LDA may prefer a direction with less total spread if it separates the labels better.

PCA vs LDA in one picture

Use this mental picture:

PCA:
Find the viewing angle where the whole cloud looks most spread out.

LDA:
Find the viewing angle where labeled groups are easiest to tell apart.

That is the main difference.

How this maps to scikit-learn

The full lesson uses small scikit-learn calls.

For PCA:

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_std)
print(pca.explained_variance_ratio_)

Read that as:

learn two spread-preserving summary columns from the standardized features

For LDA:

lda = LDA(n_components=2)
X_lda = lda.fit_transform(X_std, y)

Read that as:

learn two class-separating summary columns using features and labels

The key difference:

PCA fits with X
LDA fits with X and y

y contains the labels.

How to decide whether PCA or LDA would help

Mock exams often ask:

Would PCA or LDA help in this situation?

Do not answer as if you can know the final result before checking output or metrics.

Use two steps:

eligibility = can this method logically be used here?
usefulness  = did it actually improve the result after checking evidence?

Step 1: check eligibility

PCA is eligible when you have numeric features.

It does not need labels.

So PCA can be considered for:

compression
visualization
noise reduction
preprocessing before another model

LDA is eligible when you have numeric features and known class labels.

It needs labels because it tries to separate known classes.

So LDA can be considered for:

labeled classification data
class-separating visualization
supervised dimensionality reduction before a classifier

For ordinary unsupervised clustering, LDA is usually not appropriate because the true labels are not available before clustering.

Step 2: check usefulness

Eligibility does not guarantee usefulness.

PCA may be eligible, but a 2D PCA view may lose too much spread.

LDA may be eligible, but the classes may still overlap in the LDA space.

So a careful answer says:

This method may help because ...
To know if it actually helps, I would check ...

Examples of evidence:

Method	What to check
PCA	explained variance ratio, 2D plot quality, model performance with and without PCA
LDA	class separation in the reduced space, classifier performance with and without LDA

Decision table

Situation	Best answer	Why
Many numeric features, no labels, want compression or visualization	PCA may help	PCA preserves high-variance directions without labels
Labeled classes, want lower-dimensional class separation	LDA may help	LDA uses labels to separate classes
Already 2D clustering data	Neither is necessary for visualization	The data can already be plotted directly
Ordinary clustering with no true labels	PCA may help; LDA is not appropriate	PCA does not need labels, but LDA does
Need original feature names for interpretation	Feature selection may be better	PCA and LDA create mixed component features
Curved or nonlinear structure	PCA/LDA may be weak	Both are linear transformations
Features have different scales	Standardize first	Scale can dominate PCA and distance-based methods

Exam answer template

Use this shape:

This is a [clustering/classification/visualization/preprocessing] setting.
PCA [may help / is not necessary] because ...
LDA [may help / is not appropriate] because ...
To know whether it actually helps, I would check ...

Example:

In this clustering task, PCA may help if the original data has many numeric features,
because it can reduce the data for visualization or noise reduction without labels.
But if the data is already 2D, PCA is not necessary for visualization. LDA is not
appropriate for ordinary clustering because LDA needs class labels, and clustering
does not have labels before the model runs. To know whether PCA actually improves
the result, I would compare the cluster plot or clustering metrics with and without PCA.

Term Decoder

Term	Friendly meaning
axis	one direction on a chart
point cloud	all rows imagined as points in feature space
projection	lower-dimensional shadow of the data
component	new summary axis
principal component	PCA component that preserves spread
variance	spread
covariance	how two features move together
covariance matrix	table of pairwise feature movement
eigenvector	important PCA direction
eigenvalue	importance of that direction
explained variance ratio	percent of spread kept by a component
standardization	putting features on comparable scales
class separation	how well labeled groups are pulled apart
eligibility	whether a method can logically be used in a setting
usefulness	whether a method actually improves the result after evidence is checked

Common traps

Projection is useful, not perfect

A lower-dimensional view is like a shadow. It can show structure, but it does not contain every detail from the original data.

Variance does not always mean useful signal

PCA keeps spread. It does not know whether that spread helps the prediction task.

Eigenvectors are not the point of the course reading

For this lesson, treat eigenvectors as PCA directions and eigenvalues as their importance scores.

Standardization is not cosmetic

PCA is sensitive to numeric scale. Large-scale features can dominate if features are not standardized first.

PCA and LDA optimize different things

PCA preserves overall spread. LDA separates known classes. Neither is automatically better for every task.

Check yourself

What does projection mean in this lesson?

Making a lower-dimensional view of the data, like a shadow of a higher-dimensional object.

What does PCA try to preserve?

The biggest overall spread in the data.

What does variance mean in plain language?

Spread.

What does covariance ask?

Whether two features tend to move together, move in opposite directions, or have no strong linear movement together.

In PCA, what is an eigenvector?

An important direction PCA discovered.

In PCA, what is an eigenvalue?

A number saying how much spread is captured by that direction.

Why check explained variance ratio?

It tells how much of the original spread the selected PCA components kept.

What does LDA use that PCA ignores?

Class labels.

What is the difference between eligibility and usefulness?

Eligibility asks whether a method can logically be used. Usefulness asks whether it actually improves the result after checking evidence.

Why is LDA usually not appropriate for ordinary clustering?

Because clustering is unsupervised and does not have true class labels before the model runs.

Now Read the Full Lesson

The full lesson uses the same ideas with the Wine dataset:

standardize features
find PCA directions
check explained variance
compare with LDA directions
use reduced features for plotting or modeling

When you see the full lesson's technical terms, translate them back to:

projection              -> useful lower-dimensional shadow
variance                -> spread
covariance matrix       -> table of features moving together
eigenvector             -> PCA direction
eigenvalue              -> direction importance
explained variance      -> how much spread was kept
LDA                     -> label-aware separation view

Next: Dimensionality Reduction: PCA in Code

Source anchors

• Supports: study-guide/docs/lessons/05-dimensionality-reduction.md
• Source file: notebooks/Module2/05-Dimensionality Reduction.ipynb
• Key source concepts prepared here: PCA, LDA, projection, variance, covariance, eigenvectors, eigenvalues, explained variance, standardization