Attention in transformers, step-by-step | Deep Learning Chapter 6

Because it doesn't work like how you think at all. You're still thinking it works like Chain of Thought. It doesn't. And the difference is key! It works by introducing probabilistic noise, and exploring N paths fully (each with noise) in parallel (all compressed). It's reasoning at a much, much smaller (probabilistic) level than running everything through the expensive large model (deterministic) and sometimes catching that it said, "I think 1.12 is greater than 1.9 because 12 is bigger than 9, final answer". The easiest way to think about it is: if you understand how hyper words work, it's as if it's searching for different versions of the hyper words that probilisticslly would lead to better outcomes IF it fed them to the LLM before it even does. That's not actually how it works exactly. But I think it is close enough to be helpful to understand where the gain is, a rough idea of what's happening (searching paths), and how it can potentially have huge orders of magnitude improvements (doing so without paying the full price of exploring the paths through the expensive and huge model). And also why it is so much harder to determine what it's "thinking". If you aren't familiar with hyper words, this is an amazing series: ▶ this video The general idea is that a token is a multidimensional vector to represent a word -> think like "man" is a [noun, singular, English, pronoun, masculine, contemporary, ...]. Each time is sees a new word, it mutates this word to mean some new token (ofte

in “Claude Opus 4.8” · View on HN

This is good, but what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. 0. [link] 1. ▶ this video

in “The Illustrated Transformer (2018)” · View on HN

If you want the actual maths instead of handwaving, I recommend: [link] For something gentler, 3Blue1Brown: ▶ this video (this is part of a series)

in “Can I Buy Your KV Cache?” · View on HN

> when an LLM tells us that the sun rises in the east, it's not because it truly understands astronomy or the solar system. Rather, it has seen the phrase "sun rises in the east" repeated so many times in its training data that this becomes the highest probability answer. It's less about understanding and more about pattern recognition. This is not too different from most human person's experience with basic education ( [link] ). Though it is obvious that humans can do more than that too. And while "voting" that may be a close-enough metaphor for the multi-layer perceptron layers, it is not what is happening in the attention layers. I suggest checking out 3blue1brown's videos on gradient descent [link] and "How might LLMs store facts" [link] versus how attention works ▶ this video

in “LLMs Aren't Thinking, They're Just Counting Votes” · View on HN

There are many others that are better. 1/ The Annotated Transformer Attention is All You Need [link] 2/ Transformers from Scratch [link] 3/ Andrej Karpathy has really good series of intros: [link] Let's build GPT: from scratch, in code, spelled out. [link] GPT with Andrej Karpathy: Part 1 [link] 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning [link] Attention in transformers, visually explained | Chapter 6, Deep Learning ▶ this video Full 3Blue1Brown Neural Networks playlist [link]

in “The Engineer’s Guide to Deep Learning: Understanding the Transformer Model” · View on HN

About a year ago, I watched this 3Blue1Brown LLM tutorial ( ▶ this video ) on how a model’s self-attention mechanism is used to predict the next token in a sequence, and I was surprised by how little we know about what actually happens when processing the sentence “A fluffy blue creature roamed the verdant forest.” A year later, the field of mechanistic interpretability has seen significant advancements, and we're now able to "decompose" models into interpretable circuits that help explain how LLMs produce predictions. Using the second iteration of an LLM "debugger" I've been working on, I compare the hypothetical representations used in the tutorial to the actual representations I see when extracting a circuit that describes the processing of this specific sentence. If you're into model interpretability, please take a look! [link]

in “3Blue1Brown Follow-Up: From Hypothetical Examples to LLM Circuit Visualization” · View on HN