Opleiding: Building Large Language Models
The course Building Large Language Models from SpiralTrain teaches you how to design, train, fine-tune, and deploy transformer-based LLMs using PyTorch and modern tooling.LLM Intro
The course starts by explaining what LLMs are, where they’re used, and the lifecycle of building vs. using them. We introduce the Transformer/GPT architecture, how models learn from large datasets, and when to use classic QA versus RAG.
Working with Text Data
You’ll move from raw text to model-ready tensors: tokenization (e.g., BPE), token→ID mapping, special/context tokens, and sliding-window sampling. We cover embeddings and positional encodings while handling unknown words and basic sentence structure.
Attentions Mechanism
This module demystifies self-attention for long-sequence modeling: queries, keys, values, and causal masking to hide future tokens. We add positional encoding, multi-head attention, and stacked layers to capture dependencies across different parts of the input.
Pytorch Deep Learning
This module explains PyTorch fundamentals—tensors, core operations, and training loops—with the tooling to measure model quality. We cover feature scaling/normalization (including categorical features), activation and loss functions, and backpropagation.
Neural Networks
Next the course proceeds with building MLPs and CNNs in PyTorch while choosing appropriate activations and losses and implementing backprop. We touch NLP-specific preprocessing and walk through end-to-end binary and multi-class classification.
GPT from scratch
Next you will implement a minimal GPT with layer normalization, residual connections, and attention + feed-forward (GELU) blocks.
Pretraining
Then pretrain the LLM on unlabeled text with next-token prediction, tracking training vs. validation losses. You will explore decoding strategies (e.g., temperature, top-k), control randomness for reproducibility, and save/load PyTorch weights.
Tuning for Classification
Then the course covers preparing datasets and dataloaders, initializing from pretrained weights, and add a classification head with softmax. Train and evaluate with loss/accuracy, culminating in an LLM-based spam-classification example.
Fine-Tuning
Finally you will practice supervised instruction tuning: format datasets, batch efficiently, and fine-tune a pretrained LLM. Also evaluate outputs, export responses/checkpoints, and apply parameter-efficient methods such as LoRA.
Audience Course Building Large Language Models
The course Building Large Language Models is intended for engineers who want to design transformer-based LLMs.
Prerequisites Course Building Large Language Models
Participants should be comfortable with Python. Prior exposure to PyTorch or a similar Deep Learning framework is helpful.
Realization Training Building Large Language Models
The training blends concise theory with guided, hands-on labs. Through code-alongs you’ll build a mini-GPT, prepare datasets, run pretraining and fine-tuning and deploy models.
Building Large Language Models Certificate
After completion, participants receive a certificate of participation for the course Building Large Language Models.
Modules
Module 1 : LLM Intro
- What is an LLM?
- Applications of LLMs
- Stages of Building LLMs
- Stages of Using LLMs
- Transformer Architecture
- Utilizing Large Datasets
- GPT Architecture Internals
- Learn Language Patterns
- Retrieval Augmented Generation
- Question and Answer Systems
- QA versus RAG
- Building an LLM
Module 2 : Working with Text Data
- Word Embeddings
- Decoders and Encoders
- Decoder Only Transformer
- Tokenizing text
- Convert Tokens into IDs
- Special Context Tokens
- Understand Sentence Structure
- Byte Pair Encoding
- Unknown Words
- Sampling with Sliding Window
- Creating Token Embeddings
- Encoding Word Positions
Module 3 : Attentions Mechanism
- Modeling Long Sequences
- Capturing Data Dependencies
- Attention Mechanisms
- Attending Different Input Parts
- Using Self-Attention
- Trainable Weights
- Hiding Future Words
- Positional Encoding
- Causal Attention
- Masking Weights with Dropout
- Multihead Attention
- Stacking Attentions Layers
Module 4 : Pytorch Deep Learning
- Deep Learning Intro
- Overview of PyTorch
- PyTorch Tensors
- Tensor Operations
- Model Evaluation Metrics
- Feature Scaling
- Feature Normalization
- Categorical Features
- Activation Functions
- Loss Functions
- Backpropagation
Module 5 : Neural Networks
- Neural Networks Intro
- Building NN with PyTorch
- Multiple Layers of Arrays
- Convolutional Neural Networks
- Activation Functions
- Loss Functions
- Backpropagation
- Natural Language Processing
- Stopword Removal
- Binary Classification
- Multi-class Classification
Module 6 : GPT from scratch
- Coding an LLM Architecture
- Layer Normalization
- Normalizing Activations
- Feed Forward Network
- GELU Activations
- Adding Shortcut Connections
- Connecting Attention
- Weight Tying
- Linear Layers in Transformer Block
- Coding the GPT Model
- Generating Text
Module 7 : Pretraining
- Pretraining on Unlabeled Data
- Calculating Text Generation Loss
- Training Losses
- Validation Set Losses
- Training an LLM
- Decoding Strategies
- Control Randomness
- Temperature Scaling
- Saving Model Weights in PyTorch
- Loading Pretrained Weights
Module 8 : Tuning for Classification
- Categories of Fine-Tuning
- Preparing the Dataset
- Creating Data Loaders
- Top-k Sampling
- Soft-Max Function
- Initializing with Pretrained Weights
- Adding Classification Head
- Classification Loss and Accuracy
- Fine-tuning on Supervised Data
- Using LLM as Spam Classifier
Module 9 : Fine-Tuning
- Instruction Fine-tuning
- Supervised Instruction
- Preparing a Dataset
- Organizing Training Batches
- Creating Data Loaders
- Loading a pretrained LLM
- Fine-tuning the LLM
- Extracting and Saving Responses
- Evaluating Fine-tuned LLM
- Fine Tuning with LoRA
