Eigen Values , Vectors & Applications
From three variables to a matrix Finding the special directions From vectors to pixels and filters Embeddings, search, and transformers
Imagine a tiny robot describing its position using three numbers: how far east it is, how far north, and how high it is. One position might be two, one, and zero. We can draw that as an arrow from the origin to the point with those coordinates. The three numbers together are one three-dimensional vector. Now suppose the robot's world applies the same rule every time it moves a vector: its east component becomes twice the old east component, its north component stays the same, and its height becomes half the old height. We have a repeatable transformation, not a new guess for each arrow. A matrix is a compact way to store that rule. Put two, zero, zero in the first row; zero, one, zero in the second; and zero, zero, one half in the third. Multiplying this matrix by a vector applies all three rules together. The output is another vector, so the transformation can be drawn as the original arrow and its transformed arrow. Try the vector two, one, zero. After the transformation it becomes four, one, zero. It grew more eastward, but its direction changed. The matrix did not merely stretch every arrow uniformly. It stretched some coordinate directions more than others, so most arrows turn as well as change length. But look at an arrow pointing purely east: one, zero, zero. It becomes two, zero, zero. It points in precisely the same direction; only its length doubles. A purely north-pointing arrow stays the same size. A purely vertical arrow keeps its direction but is cut to half its length. This gives us a useful question. Instead of asking what happens to every possible arrow, can we find the special arrows whose directions survive the transformation unchanged? Mathematicians call these special directions eigenvectors, and the scale factor attached to each one is its eigenvalue. The name sounds abstract, but the picture is simple: same line, possibly a different length. The key point is that an eigenvector is not just any vector, and an eigenvalue is not a second vector. The eigenvector is a nonzero direction that a matrix sends back along the same line; the eigenvalue tells you the signed amount of scaling. A negative value reverses the arrow, zero collapses it, and a positive value preserves its orientation. For this first matrix, the east direction has eigenvalue two, the north direction has eigenvalue one, and the vertical direction has eigenvalue one half. Pause and predict: which of these directions is stretched most, and which is compressed? The east direction doubles, while the vertical direction halves. Next, how do we discover these directions when the matrix is not already so convenient? We have seen a special arrow visually: after multiplying by a matrix, it stayed on its original line. To find one systematically, we need to say that in an equation. If a matrix called A transforms a vector called v, and the result is the same vector multiplied by a number called lambda, we write A times v equals lambda times v. Rearrange that statement by bringing both sides together. A times v minus lambda times v equals zero. Because both terms contain v, we can write the matrix A minus lambda times the identity matrix, multiplied by v, equals zero. The identity matrix is the matrix that leaves every vector exactly as it is. The zero on the right means we are looking for a vector that the adjusted matrix sends to zero. The all-zero vector would always work, but it tells us nothing about a direction, so we need a nonzero answer. This is why we cannot stop at the trivial solution. A determinant is a number calculated from a square matrix that tells us, among other things, whether the matrix squashes space so much that it cannot be reversed. For a two-by-two matrix, you multiply the top-left and bottom-right entries, then subtract the product of the other diagonal. For larger matrices the calculation is more involved, but the same idea helps test invertibility. If the determinant is zero, the matrix is singular: it has no ordinary inverse, because at least one direction has been collapsed. That is exactly when our adjusted matrix can have a nonzero vector sent to zero. So we set the determinant of A minus lambda times the identity matrix to zero and solve for lambda; the answers are the eigenvalues. Use a real three-dimensional example where each output mixes the inputs. Let the first output be two times x plus y, the second be x plus two times y, and the third be three times z. That rule becomes a matrix with rows two one zero; one two zero; and zero zero three. Now you have a matrix whose first two coordinates interact, while the third is separate. First inspect the horizontal plane. Try the vector one, one, zero. The first output is three, the second output is three, and the third is zero. So the transformed vector is three, three, zero: exactly three times the original. That makes one, one, zero an eigenvector with eigenvalue three. Now try one, negative one, zero. The first output is one, the second is negative one, and the third is zero. It comes back unchanged, so its eigenvalue is one. The vertical vector zero, zero, one becomes zero, zero, three, so it has eigenvalue three. Pause: what makes each of these a valid eigenvector? In each case, the output is just a scalar multiple of the input. An image may look like color and texture to you, but a computer stores it as numbers. A tiny grayscale image with three rows and three columns can be written as nine brightness values. If we line those values up, we get a vector in a nine-dimensional space. A color image has several numbers per pixel, so its vector is longer still. A matrix can transform that image vector. In principle, a matrix with one row and one column for every pixel value can mix nearby brightness values, blur an image, sharpen edges, or apply other linear operations. The output vector can then be reshaped into rows and columns to display a new image. But an important correction: most familiar image filters are not described by finding the eigenvectors of the particular image. A blur or edge filter is often implemented by a small kernel sliding over local neighborhoods. That is a convenient structured linear operation, and it can be represented as a large matrix, but eigen-analysis is not required each time you apply it. So where do eigenvectors enter imaging? Imagine repeatedly applying the same linear operation to a pattern. Some patterns may retain their shape and only become brighter, dimmer, or reversed. These are eigen-patterns of that operator. Studying them can reveal which spatial patterns an operation preserves or suppresses, and can help analyze stability and frequency behavior. Consider a simple pair of pixels. A blur might replace each pixel partly with its neighbor. The pattern where both pixels are equally bright may be preserved by a suitably normalized blur, while a pattern where one is bright and the other dark may shrink toward equality. These are illustrative patterns that can correspond to distinct eigenvalues for a chosen blur matrix. Now compare that with sharpening. Sharpening tends to emphasize differences between neighboring pixels rather than smooth them away. The constant pattern and the difference pattern can therefore respond very differently. Eigenvalues tell us the gain for a pattern that the operation preserves in shape, while eigenvectors tell us which patterns those are. There is one more imaging connection: principal component analysis, often used for compression, can use eigenvectors of a covariance matrix to identify directions of greatest variation in image data. Large eigenvalues correspond to directions that explain more variation. Keeping the strongest directions can reduce storage while preserving much of the structure. Pause and distinguish two claims. A filter can be applied as a linear operation without explicitly computing eigenvectors. Eigen-analysis is useful when you want to understand the operation's special patterns or the structure of the data. The question to carry forward is: how do these same vectors help computers compare meanings? A text embedding turns a piece of text into a list of numbers. A sentence might become a vector with hundreds or thousands of coordinates. Those coordinates are not usually labeled as plain English features; together, they encode patterns learned during training. Texts used in similar contexts often land near each other under a chosen similarity measure. Vector search takes a query embedding and compares it with stored embeddings. A common method is cosine similarity, which asks how closely two vectors point in the same direction. In that setting, direction matters more than raw magnitude. This is a geometric idea about vectors, but it does not require eigenvectors to work. Why are the ideas related at all? Both rely on linear algebra. Vectors let us represent things as lists of numbers, and matrices give us repeatable rules for changing those lists. Geometry then helps us compare how the resulting vectors point and relate to one another. Imagine a cloud of thousands of points, each representing one image, sentence, or document. Each point has many coordinates, so you cannot draw all the axes on paper. Eigenvectors can help us find important directions through this cloud, and the associated eigenvalues tell us how much the data varies along each direction. One common method is principal component analysis. It studies how the coordinates of the data vary together, builds a covariance matrix, and finds special directions called eigenvectors. The covariance matrix summarizes how pairs of measurements change together. The eigenvalue for each direction tells us how much variation lies along it. In practice, principal component analysis often keeps the directions with the largest eigenvalues. Those directions capture more of the differences across the data, so representing each example using only a few of them can reduce dimensions while preserving much of the important variation. A transformer also uses vectors and matrices extensively. First, learned matrices turn each token's representation into three different vectors called a query, a key, and a value. These names describe their roles in attention, not ordinary English meanings stored in separate boxes. The model compares queries with keys to calculate attention scores. A high score means one token is allowed to draw more information from another token in that step. The scores become weights, and those weights determine how the value vectors are combined into updated token representations. That attention calculation is not the same as finding eigenvectors of one matrix. Still, researchers can study eigenvalues and eigenvectors of attention-related or covariance matrices to understand dominant directions, stability, and patterns in learned representations. Eigen-analysis is one possible lens for studying a transformer, not the basic recipe that makes it work. Return to the three-variable example. The matrix transformed an input vector, and eigenvectors showed directions that kept their line while being scaled. In an embedding system, the input is a representation of text; learned matrices transform it, and similarity scores compare it with other representations. The common foundation is vector geometry and matrix operations, not a guarantee that every embedding is an eigenvector. Here is a final small test. If two sentence embeddings point in nearly the same direction, cosine similarity may rank them as similar even if their lengths differ. Does that automatically mean either vector is an eigenvector? No. To be an eigenvector, it must be transformed by a particular square matrix into a scalar multiple of itself. The key point is that shared linear algebra connects these topics, while their specific jobs remain different.