Principal Component Analysis for IRIS DATA
Meet the Iris data Find the principal components See Iris after reduction From PCA to classification
Imagine three Iris flowers on a table. One has a short, narrow sepal and a small petal. Another has medium petals. A third has long, wide petals. The classic Iris dataset turns these visible differences into four numbers for every flower: sepal length, sepal width, petal length, and petal width. Sepal length simply means the length of one of the green, leaf-like structures that protect the flower before it opens; if it measures 5.1 centimeters, that 5.1 is the sepal-length feature. Now put those flowers into a coordinate system. For a simple picture, choose sepal length, petal length, and petal width as the three axes. Each flower becomes one point in three-dimensional space. Points from the same species tend to gather together, but the clouds overlap. You can rotate this cloud and look from different directions. This raises the real problem. You began with four measurements, not three. So even our three-dimensional picture has already thrown one feature away. And even inside these three dimensions, some directions may show much more useful variation than others. The key point is that the coordinates we started with were chosen by biology, not because they are the best coordinates for understanding the dataset. Look at petal length and petal width together. You can often see one species sitting apart quite clearly, while the other two are closer. That tells you something important: the measurements are not equally useful, and some combinations of measurements carry more structure than any single feature alone. Suppose you were allowed to rotate the entire cloud without changing any distances between flowers. Could you choose a new set of axes so that the cloud stretches mostly along the first axis, less along the second, and very little along the third? That is exactly the geometric question behind principal component analysis. And notice what PCA is not doing. It is not looking at the species names and learning a boundary between them. It is looking only at the measurements and asking: in which directions does this dataset vary the most? So before we classify anything, we first have to discover better coordinates. But how do we find them? Start with all four measurements for all 150 flowers. Because the features have different average values, first center each feature by subtracting its mean. After centering, zero means average flower for that measurement. The whole cloud is now balanced around the origin, which makes directions of variation easier to compare. Imagine shining a flashlight on this four-dimensional cloud. We cannot literally see four dimensions, but we can ask what the shadow looks like along any chosen direction. Some directions produce a very long shadow because the flowers spread out strongly there. Other directions produce a short shadow because the flowers barely differ along them. Now choose the direction that gives the longest possible shadow. Call that direction the first principal component. The key point is that this direction captures the maximum variance, meaning the greatest spread of the centered data. It is a new axis, usually a mixture of all four original measurements rather than one original feature. Then impose one condition: the next direction must be perpendicular to the first. Among all such directions, choose the one with the greatest remaining spread. That becomes the second principal component. Repeat once more for a third component. PCA is therefore building a new coordinate system, one direction at a time, ordered by how much variance each direction captures. Mathematically, these directions come from the covariance matrix of the centered features. Its eigenvectors give the principal directions, and the corresponding eigenvalues tell us how much variance each direction captures. You do not need to memorize that yet. The picture comes first: eigenvectors point along the important axes of the cloud, and eigenvalues tell us how much structure lives along each axis. Now imagine keeping only the first two or three principal components and dropping the rest. You have compressed every flower from four original numbers into fewer numbers while trying to preserve as much variation as possible. This is dimensionality reduction. Pause and finish this thought: if the first component captures seventy percent of the variance and the second captures twenty percent, what percentage do the first two preserve? The answer is ninety percent. So two new coordinates can preserve ninety percent of the original spread even though you started with four measurements. But does preserving that spread also preserve the flower classes? Take each centered flower and project it onto the first principal component. That gives one new number for the flower. Project it onto the second component and you get a second number. Add the third component and each flower now has three PCA coordinates. We have transformed four original measurements into three new coordinates. Draw those three PCA coordinates as a three-dimensional scatter plot. Rotate the plot slowly. The important thing is that the axes are no longer sepal length, sepal width, petal length, and petal width. They are directions discovered from the data itself. A flower's position tells you how strongly it expresses each of these new patterns of variation. Now color the points by their known Iris species, but only for inspection. This color is not used to calculate PCA. As you rotate the cloud, one species becomes visibly separated from the others, while two species remain closer together. The reduced representation has kept enough of the original structure to make the biological groups visible. Here is the subtle part. PCA maximizes variance, not class separation. A direction can have enormous spread and still be poor for distinguishing two classes. So we should never say that PCA guarantees the best classifier. Instead, say that PCA creates a lower-dimensional coordinate system that can make the dataset easier to visualize, compress, and sometimes classify. Suppose you keep only the first two components and draw a flat scatter plot. You may already see three regions, with one species clearly apart and the other two partially overlapping. Add the third component and some points that were stacked on top of each other in two dimensions move apart in depth. This is why a three-dimensional visualization can be useful even when the actual model may use fewer dimensions. So pause here and ask yourself: what did we really accomplish? We started with four measurements describing flowers. We found directions of maximum variance without looking at the labels. Then we projected the flowers into those directions and inspected the resulting geometry. The classification question comes only afterward: does this new space make the species easier for a classifier to separate? Suppose the first two principal components already explain most of the variance. You can train a classifier using only those two numbers instead of all four original features. A two-dimensional feature space is easier to visualize, cheaper to store, and sometimes less noisy. But before declaring victory, test it rather than trusting the picture. Compare a classifier trained on all four original features with one trained on the reduced PCA features. Use the same training and test split for both. Measure accuracy or another appropriate metric. If the reduced model performs similarly, you have compressed the representation with little loss. If performance drops sharply, the discarded directions contained information useful for classification even though they explained less overall variance. This distinction is the central lesson. Variance is a property of the input data. Classification quality depends on the labels. PCA is unsupervised because it does not know the species. Methods designed specifically to use class labels, such as linear discriminant analysis, answer a different question: which directions separate the classes? The Iris dataset makes this easy to see because the four biological measurements are correlated. Petal length and petal width, for example, tend to move together. PCA can combine correlated features into a smaller set of axes that preserve much of their joint variation. Instead of carrying four partly redundant numbers, you can carry a few coordinates that summarize the main patterns. Now the visual picture becomes intuitive. The original axes are like the walls of a room chosen before you arrive. PCA rotates the room so that the longest direction of the object cloud lines up with the first axis. The analogy breaks here: PCA is not physically moving the flowers, and the new axes are computed from covariance and eigenvectors, not from a human eyeballing the cloud. So here is the final question. Can you look at a three-dimensional PCA plot and immediately claim the species are perfectly classifiable? No. You can say the structure is visually promising. Then you verify it with a classifier and held-out data. The key point is that dimensionality reduction is not simply throwing away features; it is choosing a smaller coordinate system that preserves what matters most about the data.