[1hr Talk] Intro to Large Language Models on HackerTube: recommended by 9 Hacker News commenters across 6 separate months, 2023–2026.
talk · intro · dated
Andrej Karpathy gives a one-hour general-audience introduction to large language models, covering what they are, how they are trained, comparisons to operating systems, and security challenges such as jailbreaks.
HN recommends it as the dense, approachable starting point for understanding LLMs and often cites it for specific claims about scaling and training.
“Start with his short intro to LLMs, only 1 hour long but is dense with valuable stuff”
20 Hacker News comments link this video. Each one links back to its thread.
All of Andrej Karpathy's stuff on YouTube is absolutely top tier. He's a rare example of a brilliant contributor in his field who is an equally brilliant educator. Start with his short intro to LLMs, only 1 hour long but is dense with valuable stuff: ▶ this video
in “Why has LLM progress seemingly stalled around the GPT-4 level? Or has it?” · View on HN
Short answer is No. I highly recommended watching Andrej Karpathy's Intro to LLMs talk, particularly the section on System 1 vs System 2 thinking. Long story short, what you are describing, using more processing to prepare a better response, is something that is an area of interest, but is not currently part of ChatGPT (or any other LLM that I am aware of). See: ▶ this video
in “Ask HN: Do LLMs get "better" with more processing power and or time per request?” · View on HN
Andrej Karpathy's "Zero to Hero" series on YouTube is the ultimate guide to building LLMs. Extremely information-dense but as complete as it gets: [link] Also, an amazing high-level overview of LLMs, including extensive discussion about attack vectors, that he published a couple days ago: ▶ this video
in “Generative AI for Beginners” · View on HN
The weights are just numbers. I don't what technical background you have in other areas of computing, but I think that this is a good, short introduction that doesn't assume too much: [link] To quote part of it, Training a model can be thought of as tuning the dials on a really big machine. The way that a language model behaves is entirely determined by these many different continuous values, usually called parameters or weights. Longer and slightly more technical, "Intro to Large Language Models" by Andrej Karpathy: ▶ this video
in “The Unbearable Cheapness of Open Weight Models” · View on HN
Look into the training process - ▶ this video The goal of the initial pretraining phase is to make it good at predicting the next word. The rest of the training process is aimed at making it (1) helpful and (2) as correct as possible. I think some people oversimplify things by calling LLMs "next token predictors" and they leave out the tuning towards helpfulness and correctness.
in “Ask HN: Why has ChatGPT a bias towards correctness?” · View on HN
In addition to the other great suggestions, point him to Karpathy's YouTube channel[1]. Karpathy has an approachable communication style. Here's his "1 hour intro to LLMs" video: ▶ this video 1. [link]
in “Ask HN: 16 yo Nephew, in E. Africa, wants to train an LLM with on disk Wikipedia” · View on HN
I suspect these videos would be higher quality than most offerings from universities or business schools (given that it's a fast moving field, and most of the true experts are practitioners, and so if they're teaching, they're not teaching actual classes, at most they're doing youtube videos or interviews): - [link] (Jeremy Howard Hacker's Guide to LLMs) - ▶ this video (Karpathy Intro to LLMs)
in “Ask HN: AI Training for Executive Level” · View on HN
Hey, imho best overall technical intro to LLMs (I guess that´s your main interest as you mentioned qlora + llama) is by Simon Willis [1]. Additionally or if you prefer videos, the recent 1h "busy persons intro" by Andrei Karpathy is great + dense as well [2]. [1] [link] [2] ▶ this video EDIT: Maybe I misunderstood as you asked about papers, not general intros. I don´t think that reading papers is the best way to "catch up" as the pace is rapid and knowledge very decentralized. I can confirm what Andrej recently wrote on X [3]: "Unknown to many people, a growing amount of alpha is now outside of Arxiv, sources include but are not limited to: - [link] - HN - that niche Discord server - anime profile picture anons on X - reddit" [3] [link]
in “Ask HN: AI/ML papers to catch up with current state of AI?” · View on HN
My view is to focus on doing stuff. That's what I did. Pick up some task you want the model to do, try finetuning llama, playing with APIs from OpenAI, etc. Googling and asking GPT along the way. Foundational model training got so expensive that unless you can get hired by "owns nuclear power plant of GPUs" you are not going to get any "research" done. And as the area got white-hot those companies have more available talent than hardware nowadays. So just getting into the practitioner area is the best way to get productive with those models. And you improve as a practitioner by practicing, not by reading papers. If you're at the computer, your time is best spent writing code and interacting with those models in my opinion. If you cannot (e.g. commute) I listen to some stuff (e.g. ▶ this video - Anything from Karpathy on youtube, or [link] channel).
in “Ask HN: AI/ML papers to catch up with current state of AI?” · View on HN
* "[1hr Talk] Intro to Large Language Models" by Andrej Karpathy: ▶ this video from [link] * Andrej Karpathy: [link]
in “The Intuition Behind How Large Language Models Work, Part I” · View on HN
I watched Karpathy's Intro to Large Language Models [0] not so long ago and must say that I'm a bit confused by this presentation, and it's a bit unclear to me what it adds. 1,5 years ago he saw all the tool uses in agent systems as the future of LLMs, which seemed reasonable to me. There was (and maybe still is) potential for a lot of business cases to be explored, but every system is defined by its boundaries nonetheless. We still don't know all the challenges we face at that boundaries, whether these could be modelled into a virtual space, handled by software, and therefor also potentially AI and businesses. Now it all just seems to be analogies and what role LLMs could play in our modern landscape. We should treat LLMs as encapsulated systems of their own ...but sometimes an LLM becomes the operating system, sometimes it's the CPU, sometimes it's the mainframe from the 60s with time-sharing, a big fab complex, or even outright electricity itself? He's showing an iOS app, which seems to be, sorry for the dismissive tone, an example for a better looking counter. This demo app was in a presentable state for a demo after a day, and it took him a week to implement Googles OAuth2 stuff. Is that somehow exciting? What was that? The only way I could interpret this is that it just shows a big divide we're currently in. LLMs are a final API product for some, but an unoptimized generative software-model with sophisticated-but-opaque algorithms for others. Both are utterly in need fo
in “Andrej Karpathy: Software in the era of AI [video]” · View on HN
It might be a preview or something because I have YT premium and doesn't show up that anywhere. Can you share a video that works for that? Like this one. ▶ this video
in “TexTube: Chat with any YouTube video transcript in ChatGPT fast” · View on HN
Actually the improvement graphs are still scaling exponentially with training/compute being the bottleneck. So there isn't yet any evidence of diminishing returns. source: ▶ this video
in “OpenDevin: An Open Platform for AI Software Developers as Generalist Agents” · View on HN
I watched the latest Karpathy video, where he shows that you can create a prompt suffix that looks like jibberish, using an iterative approach. It looks like this technique will be impossible to defend against, because it discovers for an arbitrary string that causes the jailbreak. Looks like this paper is building on the token level attack from that work. The base64 encoding technique was amusing as well ▶ this video
in “Tree of Attacks: Jailbreaking Black-Box LLMs Automatically” · View on HN
Seems like they have made progress in combining reinforcement learning and LLMs. Andrej Karpathy mentions it in his new talk (~38 minutes in) [1], and Ilya Sutskever talks about it in a lecture at MIT (~29 minutes in) [2]. Finding a proper reward function to train LLMs in a reinforcement learning setup, and using it to train a model to solve math problems in a similar fashion to how AlphaGo used self-play to learn Go, would indeed be a tremendous breakthrough. [1] ▶ this video [2] [link]
in “Sam Altman's ouster was precipitated by letter to board about AI breakthrough” · View on HN
Current architectures have a lot of juice left across several axes, using extraordinarily accurate scaling laws. We're nowhere close to a wall. The industry has been focusing hard on 'System Two' approaches to augment our largely 'System One'-style models, where optimal decision policies make Q-learning a natural approach. The recent unlock here might be related to using the neural nets' general intelligence/flexibility to better broadly estimate their own states/rewards/actions (like humans). [EDIT: To be clear, the Q-learning stuff is my own speculation, whereas the 'System Two' stuff is well known.] Serendipitously, Karpathy broadly discussed these two issues yesterday ! Toward the lecture's end: ▶ this video
in “About That OpenAI "Breakthrough"” · View on HN
Karpathy in his recent video [1] agrees, but at this point scaling is a very reliable way to better accuracy. [1]: ▶ this video
in “Training for one trillion parameter model backed by Intel and US govt has begun” · View on HN
Seems like he actually disagrees here: If you train a bigger model on more text, we have a lot of confidence that the next-word prediction task will improve. So algorithmic progress is not necessary, it's a very nice bonus, but we can sort of get more powerful models for free, because we can just get a bigger computer, which we can say with some confidence we're going to get, and just train a bigger model for longer, and we are very confident we are going to get a better result. ▶ this video (23:43)
in “Training for one trillion parameter model backed by Intel and US govt has begun” · View on HN
Are you sure? In Andrej Karpathy's intro to LLMs[1], he says pre-training uses about 10TB of text from the web. It's hard to believe that a comparable amount of text had been created, even across all languages, in the entire history of humanity up to the 17th century. [1] ▶ this video
in “MonadGPT – What would have happened if ChatGPT was invented in the 17th century?” · View on HN
Well, I was discussing Large Language Models (LLMs) as a technology, and esp, via this video [0] with a friend, just now. He made a striking remark. LLMs compress the information they ingest in a variably-lossy way, and when you query them, they rebuild a representation from whole or part of this compressed data in a statistical manner. As a result they're a lossy storage medium. Not unlike some music formats which store audio data in a lossy way. You know, this data can be restored with mathematical and statistical wizardry. You don't get everything you put in, but 97%-99% out of it, and companies and RIAA went bonkers for years, because even if it's not an exact reproduction, it was a reproduction enough, and this is a copyright violation. If an LLM can reproduce what I have written, or coded with 97%-99% accuracy without any license information, and I licensed this thing with less than permissive licenses, and sue the maker of that LLM, what will happen? - Will it be a copyright infringement? - Will it be fair use? It's the first if you look at it fairly, but it'll be probably be the latter, because money, fame and other corporate points will be at stake, otherwise. [0]: ▶ this video Edit: I forgot to add the video. :)
in “Let's Not Flip Sides on IP Maximalism Because of AI” · View on HN