Episode 12a — Residual Connections
Episode 12a — Residual Connections
Before we continue, let's quickly remember where we are. Every transformer layer performs two important jobs. First, Attention gathers information from the rest of the sentence. Then, the Feed Forward Network refines that information. Simple enough. Now I want you to pause and think. If we keep repeating this process over and over again... Attention... Feed Forward... Attention... Feed Forward... what problem could eventually appear? Think about it for a moment. Here's a hint. Imagine you're asked to rewrite an essay. The first rewrite improves it. The second rewrite improves it even more. Then a third. Then a fourth. After twenty rewrites... are you absolutely sure that every useful idea from the original essay is still there? One more hint. Suppose someone gives you directions to reach a destination. Every person in the chain slightly modifies the instructions before passing them to the next person. After fifty people... would you trust that the original message is still completely intact? If you thought, "Important information could slowly disappear," you're absolutely right. The same challenge appears inside deep neural networks. Every transformer layer changes the vector it receives. If this continues through dozens of layers, some useful information from earlier layers may gradually become weaker or even disappear. So how do we solve this? Instead of throwing away the original vector... the transformer keeps it. It takes the transformed vector produced by Attention and the Feed Forward Network... and adds it back to the original input vector. In other words, the layer says, "I'll keep everything you already knew... and add only what I've learned. " This simple idea is called a Residual Connection. Every layer contributes new information without discarding what was already valuable. Residual Connections also make very deep transformers much easier to train because information and learning signals can flow smoothly through many layers. Let's summarize. Attention gathers information. Feed Forward refines information. Residual Connections preserve information. Now let's think one step ahead. We're repeatedly adding new information to these vectors. Do you think the numbers inside those vectors will always remain well behaved? Or could they gradually become too large, too small, or unstable? That's exactly the next challenge. And that's why transformers use Layer Normalization. Before moving to the next episode, take the quiz. A quiz doesn't simply check your understanding. It forces your brain to retrieve, organize, and connect what you've just learned. That's when information begins to become understanding.