Let's build GPT: from scratch, in code, spelled out. on HackerTube: recommended by 14 Hacker News commenters across 8 separate months, 2023–2025.
tutorial · intermediate · evergreen
Andrej Karpathy builds a GPT from scratch in code, following Attention Is All You Need and GPT-2/3, in about two hours using PyTorch. He connects it to ChatGPT and assumes familiarity with his earlier makemore videos.
It is the go-to recommendation for learning how transformers and LLMs actually work, praised for its clarity and Karpathy's teaching.
“the best video I have seen on the topic”
20 Hacker News comments link this video. Each one links back to its thread.
+1, Andrey is an amazing educator! I'd also recommend his ▶ this video and checking out his github repos. MinGPT, for example, implements a small gpt model that's compatible with HF API, whereas more modern nanoGPT shows how to use newer features such as flash attention. The quality of every video, every blog post is just so high.
in “Implementing a ChatGPT-like LLM from scratch, step by step” · View on HN
Andrej Karpathy, the director of AI at tesla, published an AMAZING video about how the current models are built which I think everyone in the tech space should watch. It is intuitive, easy to follow and quite frankly the best video I have seen on the topic. Here is the link: ▶ this video If you conclude, after watching this, that this is all intelligence is - namely cascading optimization and data structure problems chained in a row, then you and a fair number of people here wont ever find common ground. That is not to say that you're wrong, it just seems as if there is a missing component here that we haven't yet discovered.
in “Google DeepMind CEO Says Some Form of AGI Possible in a Few Years” · View on HN
Very impressive. Recently I watched this really amazing lecture on building GPT from scratch from Karpathy, I was blown away: ▶ this video
in “GPT in 60 Lines of NumPy” · View on HN
Great compilation, would be great to see Vision Transformer (ViT) included. Andrej Karpathy's GPT video is a must have companion for this ▶ this video I was going nuts trying to grok Key, Query,Position and Value until Andrej broke it down for me.
in “The Transformer Family” · View on HN
Attention is all you need and GPT-2 were well known at that point. Many might doubt whether this approach leads to "general" intelligence – depends on the definition. BTW, Karpathy has a nice video tutorial about building an LLM: ▶ this video
in “A ChatGPT clone, in 3000 bytes of C, backed by GPT-2 (2023)” · View on HN
Yes, it's doable. Your model won't be as large and as performant as a real large scale model, but you could train something. You could watch this for a start: ▶ this video
in “Ask HN: Is it feasible to train my own LLM?” · View on HN
I initially thought this was his older video, but I see this video was published just an hour ago (from the time of this comment). I was thinking of this other video he had published early this year: "Let's build GPT: from scratch, in code, spelled out" ▶ this video Karpathy is generally well-reputed as a good tutor, especially for complex topics in AI / ML.
in “Andrej Karpathy: Intro to Large Language Models [Video]” · View on HN
I love all the series about writing your own X in 100 lines of code. It gives you the understanding of technology and removes a lot of unnecessary details. The great examples of this are 'A from-scratch tour of Bitcoin in Python' [link] and 'Let's build GPT: from scratch, in code, spelled out' ▶ this video from Andrej Karpathy I wonder if anybody tried to collect all such projects together and built his own 'Internet in just 100 lines of code'
in “Introducing "Implement DNS in a Weekend"” · View on HN
Those Computerphile videos[0] by Rob Miles helped me understand transformers. He specifically references the "Attention is all you need" paper. And for a deeper dive, Andrej Kharpaty has this hands-on video[1] where he builds a transformer from scratch. You can check-out his other videos on NLP as well they are all excellent. [0] [link] , [link] [1] ▶ this video
in “Ask HN: Can someone ELI5 Transformers and the “Attention is all we need” paper” · View on HN
Here are a few resources that might help get you started: 1. A thread explaining the internal working of transformers: [link] 2. Paper by DeepMind which provides pseudo-code for important algorithms for Transformer models: [link] 3. Another thread specifically on large language models: [link] Once again these are not courses per se, but do provide intuitive explanations for how transformers work. There is also the nanoGPT series of videos by Karpathy on youtube. First video here: ▶ this video
in “Ask HN: How to Learn LLM Fundamentals” · View on HN
I agree that Karpathy's YouTube video is an excellent resource for understanding Transformers from scratch. It provides a hands-on experience that can be particularly helpful for those who want to implement the models themselves. Here's the link to the video titled "Let's build GPT: from scratch, in code, spelled out": ▶ this video Additionally, for more comprehensive resources on Transformers, you may find these resources useful: * The Illustrated Transformer by Jay Alammar: [link] * MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention: [link] * Karpathy's course, Deep Learning and Generative Models (Lecture 6 covers Transformers): [link] ... These resources cover different aspects of Transformers and can help you grasp the underlying concepts and mechanisms better.
in “What Are Transformer Models and How Do They Work?” · View on HN
Andrej Karpathy's 2 hour video and code is really good to understand the details of Transformers: "Let's build GPT: from scratch, in code, spelled out." ▶ this video
in “Transformers from Scratch” · View on HN
I think you are taking a very major assumption that LLMs are deterministic infact they are exactly opposite. They are probabilistic systems. - They do not transpile Rough English to Deterministic English. Infact they do not do any transpiling at all. - LLMs learn the probabilities of words in the dataset for all contexts from that dataset (context is a ordered set of words). This is called training. - Once training is done, LLMs can generate text given a prompt and the probabilities that it has learned. The analogy of LLMs being auto-complete on steroids is very apt. - Whether the text generated by LLMs is factual or not is purely coincidental. I would highly recommend watching and working though the code (if possible) of NanoGPT by Andrej Karpathy. ▶ this video A lot of things which seem like magic about LLMs will get demystified. Now one can argue that LLMs are showing human like reasoning/intelligence/sentience as an emergent behavior. This is hard to argue against because all these terms are extremely hard to define. IMO, the only emergent behavior that LLMs are showing is the output they generate looks like it might have been generated by a human which should not be surprising given that LLMs like ChatGPT has trained on a large amount of human written text available on internet.
in “Flow-Based Programming, a way for AI and humans to develop together” · View on HN
I recommend this video on the subject. "Let's build GPT: from scratch, in code, spelled out." ▶ this video
in “Deep Learning Tuning Playbook” · View on HN
The one you linked to is based on Karpathy's tutorial: ▶ this video , except changing to train on tinystories in stead of Shakespeare. smolGPT also looks inspired by nanoGPT, also from Karpathy: [link]
in “SmolGPT: A minimal PyTorch implementation for training a small LLM from scratch” · View on HN
Excuse my ignorance, is this different from Andrej Karpathy ▶ this video Anyway I will watch it tonight before bed. Thank you for sharing.
in “Building LLMs from the Ground Up: A 3-Hour Coding Workshop” · View on HN
That's... my point. You're agreeing with me. Siri and Alexa are bad . Very, very bad. They pretend to understand spoken English, but they don't , because they're just a huge set of hard-coded rules written out one-at-a-time by enormous numbers of very expensive developers. This is the 1990s approach to AI: Fuzzy logic, heuristics, solvers, dynamic programming, etc... That approach has been thoroughly blown out of the water by Transformers, which does all of that and much more with a thousand lines of code that can be banged out in a couple of hours by one guy while talking in a YouTube video: ▶ this video Transformers will revolutionise this entire space, and more that don't even exist yet as a market. Take for example humanoid robots: Boston Dynamics has had the hardware working well enough for a decade, but not the software . You can't walk up to one of their robots, point at something, and tell the robot to complete a task. It can't understand what it is seeing, and can't understand English instructions. Programming that in with traditional AI methods would take man-millenia of effort, and might never work well enough . If we could speed up GPT-4o (the one with vision) to just 10x or 50x its current speed, with some light fine-tuning it could control a humanoid robot right now with a level of understanding comparable to C3P0 from Star Wars!
in “AI's $600B Question” · View on HN
There was this comment a couple of weeks ago ( [link] ). Perhaps also episode 7 of Karpathy’s “Neural Networks: From Zero to Hero” course ( ▶ this video )
in “Attention is Turing complete [pdf]” · View on HN
2-hour video posted a month or two ago in a comment here: "Let's build GPT: from scratch, in code, spelled out." ▶ this video (I haven't gotten around to watching it yet)
in “A Completely Non-Technical Explanation of Deep Learning” · View on HN
I remember Andrej mentioning in [1] that is a challenging part, what materials/lectures/papers have you found or what ideas on what to do about that so far for that? [1] ▶ this video
in “Training NanoGPT on My Journal” · View on HN