Deep Dive into LLMs like ChatGPT

This is a good resource, however for about 99.99% of people, you are most likely to just use a foundation model like ChatGPT, Claude, Gemini etc. so this knowledge/training will get you neither here or there. I would suggest you look into another Karpathy's video -- Deep Dive into LLMs like ChatGPT. ▶ this video

in “Neural Networks: Zero to Hero” · View on HN

Modal's GPU glossary is a good overview about how GPUs work [0]. Karpathy's LLM overview is a good high level overview on LLMs [1]. 3b1b's video (and subsequent videos) on transformers was excellent at helping me understand the math at a high level [2]. This matrix multiplication optimization worklog helped me understand writing better CUDA (not for beginner intro though) [3]. During this process I also asked ChatGPT a lot of questions. I'm definitely open to suggestions about "how to learn" with all the new tools we have. I felt this has not been straightforward to figure out. [0] [link] [1] ▶ this video [2] [link] [3] [link]

in “Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken” · View on HN

> The models still _generate_ response token-by-token, but they pick tokens _not_ based on tokens that maximize probabilities at each token. This is also not how base training works. In base training the loss is chosen given a context, which can be gigantic. It's never about just the previous token, it's about a whole response in context. The context could be an entire poem, a play, a worked solution to a programming problem, etc, etc. So you would expect to see the same type of (apparent) higher-level planning from base trained models and indeed you do and can easily verify this by downloading a base model from HF or similar and prompting it to complete a poem. The key differences between base and agentic models are 1) the latter behave like agents, and 2) the latter hallucinate less. But that isn't about planning (you still need planning to hallucinate something). It's more to do with post-base training specifically being about providing positive rewards for things which aren't hallucinations. Changing the way the reward function is computed during RL doesn't produce planning, it simply inclines to model to produce responses that are more like the RL targets. Karpathy has a good intro video on this. ▶ this video In general the nitpicking seems weird. Yes, on a mechanical level, using a model is still about "given this context, what is the next token". No, that doesn't mean that they don't plan, or have higher-level views of the overal structure of their response, or whatever.

in “Tracing the thoughts of a large language model” · View on HN

There is a very good intro on how LLM work by Kharpaty. It's ▶ this video and it's 3.30 hours, so way below the 100 hours mark :)

in “Ask HN: If you had 100 hours to learn something new, what would you learn?” · View on HN

I agreed when I read your comment, but that turned out to be almost directly from the video ▶ this video From around the 2:20 mark he says: “[…] actually ends up being only about 44 TB of disk space. You can get a USB stick for like a TB very easily, or I think this could fit on a single hard drive almost today” So it’s just slightly altered from what was said in the original video. And the LLM rewritten version of it also says “roughly” where he said “almost”, and I guess 44 TB is pretty roughly or pretty almost 32 TB. Although I’d still personally probably put it as “can fit on a pair of decently sized hard drives today” (for example across two 24 TB drives). Regardless, it’s close enough to what was said in the source video that it’s not something the LLM just made up out of nowhere.

in “Show HN: How LLMs Work – Interactive visual guide based on Karpathy's lecture” · View on HN

That's post-training No, it is not. Read the paper. They are discussing an emergent property of the context itself: "For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model." I'm talking about tasks like multiplying two 4-digit numbers (let's say 8-digit, just to be safe, for reasoning models), which 5th or 6th graders in the US are expected to be able to do with no problem - without using a calculator. So am I. See, for example, Karpathy's discussion of native computation: ▶ this video (starts at 1:46:56). The first few tokens in the context actually serve as some sort of substrate for general computation. I don't pretend to understand that, and it may still be something of an open research topic, but it's one more unexpected emergent property of transformers. You'd be crazy to trust that property at this stage -- at the time Karpathy was making the video, he needed to explicitly tell the model to "Use code" if he didn't want it to just make up solutions to more complex problems -- but you'd also be crazy to trust answers from a 5th-grader who just learned long division last week. Feeling quite comfortable being classed in the same camp with the greatest living physicist, honestly. Not a great time for you to rest on your intellectual laurels. Same goes for Penrose.

in “Using secondary school maths to demystify AI” · View on HN

Anyone else read the book that the author mentions, Build a Large Language Model (from Scratch) [0]? After watching Karpathy's video [1] I've been looking for a good source to do a deeper dive. [0] [link] [1] ▶ this video

in “The maths you need to start understanding LLMs” · View on HN

That's actually pretty much what Andrej Karpathy mentions as a mitigation for hallucinations here: ▶ this video

in “Persona vectors: Monitoring and controlling character traits in language models” · View on HN

Can also just fetch the subs already in YouTube rather than retranscribing. eg: yt-dlp --write-auto-subs --skip-download " ▶ this video "

in “Yt-transcriber – Give a YouTube URL and get a transcription” · View on HN

Why would you expect them to be able to given how they work? [1] Someone where I work was trying to get an LLM to evaluate responses to an internal multiple-choice quiz (A, B or C), putting people into different buckets based on a combination of the total number of correct responses and having answered specific questions correctly. They spent a week "prompt engineering" it back and forth, with subtle changes to their instructions on how the scoring should work, with no appreciable effect on accuracy or consistency. That's another scenario where I felt someone was asking for something with no mechanical sympathy for how it was supposed to happen. Maybe a "thinking" model (why do "AI" companies always abuse terms like this? (rhetorical)) would have been able to get enough stuff into the context for it to be able to get closer to a better outcome, but I took their prompt asked it to write code instead, and got it translated into some overly-commented but simple-enough code which would do the job perfectly every time, including a comment that the instructions they'd provided had a gap where people answering with a certain combination of answers wouldn't fall into any bucket. [1] ▶ this video

in “AI LLMs can't count lines in a file” · View on HN

I watched an Andrej Karpathy video recently. He said that hallucination was because in the training data there were no examples where the answer is, "I don't know". Maybe I'm misinterpreting what he was saying though. ▶ this video

in “Ask HN: Any insider takes on Yann LeCun's push against current architectures?” · View on HN

Okay, I feel like there might've been a breakthrough here. After watching Karpathy's video [0], he mentioned how hard it is for LLMs to have humor and be funny but it seems like Claude 3.7 really nailed it this time? Like, most of these posts are legit funny. [0] - ▶ this video

in “Claude 3.7 Sonnet and Claude Code” · View on HN