Build a Large Language Model (From Scratch)

Description

How to implement LLM attention mechanisms and GPT-style transformers. In Build a Large Language Model (from Scratch) bestselling author Sebastian Raschka guides you step by step through creating your own LLM. Each stage is explained with clear text, diagrams, and examples. You’ll go from the initial design and creation, to pretraining on a general corpus, and on to fine-tuning for specific tasks. Build a Large Language Model (from Scratch) teaches you how to: • Plan and code all the parts of an LLM • Prepare a dataset suitable for LLM training • Fine-tune LLMs for text classification and with your own data • Use human feedback to ensure your LLM follows instructions • Load pretrained weights into an LLM Build a Large Language Model (from Scratch) takes you inside the AI black box to tinker with the internal systems that power generative AI. As you work through each key stage of LLM creation, you’ll develop an in-depth understanding of how LLMs work, their limitations, and their customization methods. Your LLM can be developed on an ordinary laptop, and used as your own personal assistant. Purchase of the print book includes a free eBook in PDF and ePub formats from Manning Publications. About the technology Physicist Richard P. Feynman reportedly said, “I don’t understand anything I can’t build.” Based on this same powerful principle, bestselling author Sebastian Raschka guides you step by step as you build a GPT-style LLM that you can run on your laptop. This is an engaging book that covers each stage of the process, from planning and coding to training and fine-tuning. About the book Build a Large Language Model (From Scratch) is a practical and eminently-satisfying hands-on journey into the foundations of generative AI. Without relying on any existing LLM libraries, you’ll code a base model, evolve it into a text classifier, and ultimately create a chatbot that can follow your conversational instructions. And you’ll really understand it because you built it yourself! What's inside • Plan and code an LLM comparable to GPT-2 • Load pretrained weights • Construct a complete training pipeline • Fine-tune your LLM for text classification • Develop LLMs that follow human instructions About the reader Readers need intermediate Python skills and some knowledge of machine learning. The LLM you create will run on any modern laptop and can optionally utilize GPUs. About the author Sebastian Raschka is a Staff Research Engineer at Lightning AI, where he works on LLM research and develops open-source software. The technical editor on this book was David Caswell . Table of Contents 1 Understanding large language models 2 Working with text data 3 Coding attention mechanisms 4 Implementing a GPT model from scratch to generate text 5 Pretraining on unlabeled data 6 Fine-tuning for classification 7 Fine-tuning to follow instructions A Introduction to PyTorch B References and further reading C Exercise solutions D Adding bells and whistles to the training loop E Parameter-efficient fine-tuning with LoRA Review: Amazing book. Exceeded my expectations! - The book is amazing. Much better than I expected. I was minimally familiar with neural networking techniques (finished 6-months course on Coursera, and by now have forgotten most of it). So, I had a vague idea about forward and backward propagation, remembered such terms as dropout, normalization etc. without actually remembering their meaning. From the Andrew Ng course I remembered the term "transformer" (since he had a few good introductory explanations of it), but by now I completely forgot how it works. My knowledge of Python was very limited (and mostly forgotten). I knew nothing about PyTorch. When I saw the references to the book on Facebook, I decided that it might be helpful for me to recall these concepts, and especially interesting was to learn the concept of transformers and self-attention which I knew belong to the foundation of modern LLMs. The book exceeded my expectations. It is written in an excellent methodical style. Introduces concepts one by one, helps experimenting with them in the real code. It provided an excellent introduction to PyTorch (in Appendix A, which the author recommended to consume before reading the rest of the book). The introduction is short, not overwhelming the reader with millions potential concepts of the huge ecosystem of Python and PyTorch, and still sufficient for productive consuming the entire book that uses both. All the concepts are defined in easy-to-consume steps, leading eventually to a complete overall understanding of GPT model. I am not naive to think that I can develop LLMs by myself now, but I definitely got more than expected. And enjoyed the material a lot. I did not use the code from GitHub (by the book's reference). Instead, I meticulously re-entered all the examples from the book's text into several Jupyter Notebooks in VSCode. This way I moved a bit slower but understood material better. Even found a few minor (typo-level) issues in the code. I am working on an ordinary Surface Book (no GPU), and all examples work instantaneously so far (obviously, it will change when I come to training). I am now in the position after chapter 4: Built the untrained GPT model and cannot wait when I will start training and using it. Highly recommend the book to everyone who wants to make their hands "dirty" with the AI. Review: The Book That Makes LLMs Feel Buildable - Clear Explanations Without Hiding the Hard Parts What I like most is that it doesn’t treat LLMs like magic. It breaks the process down step by step—from tokenization and embeddings to attention, training, and generation—so I could see how all the pieces connect. Hands-On Learning That Actually Sticks The examples make a huge difference. Instead of only reading theory, I was able to follow along and understand how each part works in code. That made concepts like transformers and self-attention feel much less intimidating. Great for Going Beyond Surface-Level AI This isn’t just a “what is AI?” book. It helped me understand the mechanics behind modern language models and gave me a much stronger foundation for experimenting on my own. Challenging, But Worth It Some sections take focus, but that’s part of why I liked it. It pushes you to really understand the material.