Dimensionality Reduction: First Intuition¶

Why this matters¶

Many machine-learning datasets are tables with lots of columns.

That can create three practical problems:

hard to visualize
slower to train models
easier for models to get distracted by noise

Dimensionality reduction is the idea of turning many columns into fewer useful summary columns.

This page gives the basic idea before the full lesson introduces PCA, LDA, covariance, eigenvectors, and explained variance.

Mental model¶

Imagine an apartment listing table:

apartment	size	bedrooms	bathrooms	price	distance to station	neighborhood score
A	35	1	1	210000	1.2	7
B	80	3	2	520000	0.4	9
C	55	2	1	330000	2.5	6

There are many columns, but some columns are related.

For example:

size, bedrooms, bathrooms -> home_size
distance_to_station, neighborhood_score -> location_quality

The reduced version might be:

apartment	home_size	location_quality
A	small	medium
B	large	high
C	medium	lower

That is the plain idea:

many detailed columns -> fewer useful summary columns

Core ideas¶

A dataset is often a table.
Rows are examples.
Columns are features.
A dimension usually means one direction a data point can vary along.
In a table, an original dimension often corresponds to one feature column.
Dimensionality reduction creates fewer dimensions than the original data had.
The new dimensions are often summary columns, not original columns.
Reducing dimensions can make data easier to visualize, store, and model.
Reduction loses some detail, so the goal is to lose less important detail first.

Walkthrough¶

Start with columns, not math¶

Suppose each apartment has six numeric columns:

size
bedrooms
bathrooms
price
distance_to_station
neighborhood_score

Each row is one apartment.

Each column gives one way the apartment can vary.

In machine-learning language, each column is a feature.

What "dimension" means here¶

The word dimension can sound abstract, but start with this:

one numeric feature column = one original dimension

If a dataset has six numeric feature columns, each row needs six numbers to describe it.

So you can say:

this dataset has six original dimensions

Dimensionality reduction tries to describe each row with fewer numbers.

For example:

six original dimensions -> two new dimensions

The new dimensions might not be original columns. They might be combined summaries.

Why fewer columns can help¶

With one feature, you can draw a line.

With two features, you can draw a normal 2D chart.

With three features, you can imagine a 3D chart.

With 20, 100, or 1000 features, you cannot directly see the shape of the data.

Dimensionality reduction can create a smaller view:

many columns -> two or three summary columns -> plot on a chart

That chart is not perfect, but it can reveal useful patterns.

Reducing dimensions is not free¶

Compression always gives up something.

If you turn six apartment columns into two summary columns, some details disappear.

For example:

home_size may summarize size, bedrooms, and bathrooms

That is useful, but it no longer keeps each detail separately.

The goal is not:

keep everything perfectly

The goal is:

keep the important structure while using fewer numbers

Feature selection keeps original columns¶

Feature selection means choosing some original columns and dropping others.

Example:

keep:
size
price
distance_to_station

drop:
door_color
listing_id

The remaining columns are still original columns.

Feature selection answers:

Which existing columns should I keep?

Feature extraction creates new columns¶

Feature extraction means creating new columns from combinations of old columns.

Example:

home_size = blend of size, bedrooms, bathrooms
location_quality = blend of distance_to_station, neighborhood_score

The new columns are not original columns.

They are summary features.

Feature extraction answers:

What new summary columns can represent the old columns well?

PCA and LDA are feature extraction methods¶

The full lesson focuses on PCA and LDA.

Both create new features.

They do not simply pick existing columns.

Plain version:

PCA creates new summary columns that keep the biggest overall patterns.
LDA creates new summary columns that separate known groups.

You do not need to understand the mechanics yet.

For now, remember:

PCA and LDA reduce columns by creating new summary columns.

Why labels matter¶

Sometimes each row has a known class label.

For apartment listings, maybe the label is:

budget
midrange
luxury

Some reduction methods ignore labels.

Other reduction methods use labels.

This is the difference that will matter later:

PCA ignores labels.
LDA uses labels.

So PCA might find the biggest overall pattern, even if that pattern does not separate budget, midrange, and luxury very well.

LDA looks specifically for a view that separates the labeled groups.

Term Decoder¶

Term	Friendly meaning
row	one example in a dataset
feature	one input column
dimension	one numeric direction a data point can vary along
original dimension	an original feature column
new dimension	a summary feature created from old features
dimensionality reduction	replacing many dimensions with fewer useful ones
feature selection	keeping some original columns
feature extraction	creating new columns from combinations of old columns
PCA	method that keeps the biggest overall patterns
LDA	method that uses labels to separate groups

Common traps¶

Dimension does not have to mean sci-fi space

In this lesson, dimension usually means a numeric feature direction. In a table, that often starts as a column.

Reduction does not preserve everything

A reduced view is a summary. It can be useful while still losing some detail.

PCA and LDA do not just delete columns

They are feature extraction methods. They create new summary features from old features.

A simpler view is not automatically better

Fewer dimensions can help, but too much reduction can hide important information.

Labels change the goal

If labels are used, the method can look for class separation. If labels are ignored, it only sees the feature patterns.

Check yourself¶

In a table, what is an easy way to think about an original dimension?

As one numeric feature column.

What is the difference between feature selection and feature extraction?

Feature selection keeps some original columns. Feature extraction creates new summary columns from old columns.

Why might dimensionality reduction help visualization?

It can turn many columns into two or three summary columns that can be plotted.

Why is reduction not free?

Because some detail is lost when many columns are summarized with fewer columns.

What is the biggest plain-language difference between PCA and LDA?

PCA ignores labels and keeps big overall patterns. LDA uses labels and tries to separate groups.

Next¶

Next, read Dimensionality Reduction: Why PCA and LDA Work Differently.

That bridge explains the mechanics: axes, point clouds, spread, projection, covariance, eigenvectors, eigenvalues, and explained variance.

After that, read Dimensionality Reduction: PCA in Code, Dimensionality Reduction: LDA in Code, then Dimensionality Reduction.

Source anchors¶

Supports: study-guide/docs/lessons/05-dimensionality-reduction.md
Source file: notebooks/Module2/05-Dimensionality Reduction.ipynb
Key source concepts prepared here: dimensions, features, feature selection, feature extraction, PCA, LDA, labels