The spelled-out intro to language modeling: building makemore on HackerTube: recommended by 2 Hacker News commenters across 2 separate months, 2024.
tutorial · intro · evergreen
Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.
HN recommends it as a long but worthwhile way to demystify how language models work technically.
“It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs.”
4 Hacker News comments link this video. Each one links back to its thread.
You've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - [link] - Highly recommend this one ▶ this video - [link] - [link] 1: ex: [link]
in “Show HN: Next-token prediction in JavaScript” · View on HN
I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's. It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. ▶ this video
in “Markov Chains are the Original Language Models” · View on HN
For reference, this project is from Eureka Labs, which is the new education startup from Andrej Karpathy. * [link] It appears to be a generalized version of the bi-gram project, which he links to here: * ▶ this video It's cool to see him putting in even more effort on the edu work. It's a real gift he has, and was visible all the way back in his grad student projects.
in “The n-gram Language Model” · View on HN
This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it. I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise. [0] ▶ this video
in “GPT in 500 Lines of SQL” · View on HN