For Day 5/30 I focused on the first step of building a language model — embedding tokens and encoding their positions. 🔡 Token Embeddings: I used PyTorch’s nn.Embedding to convert token IDs (from a BPE tokenizer) into 256-dimensional vectors. This captures what each token is, but not where it is in the sequence. 📍 Positional Embeddings: Since transformers are position-agnostic, I created a separate embedding layer for positions (shape: [context_length, 256]) and added them to the token embeddings to encode order. 🧠 Fun fact: Unlike the original Transformer model, OpenAI’s GPT uses absolute positional embeddings that are optimized during training — not predefined. This helps GPT better understand long-range dependencies and varying sequence lengths. 💡 After this step, I now have input_embeddings of shape [batch_size, sequence_length, embedding_dim] — ready to be fed into the attention mechanism! Onward to building the self-attention module next! #deeplearning #generativeai #3...