Hello Folks! The work week has begun in full force. I managed to spend some cycles on my challenge nevertheless. It’s day 19 of building a LLM from scratch ✨Today I coded evaluation of generative language models during pretraining. So how do we start? ✨We first initialize a GPT-style transformer and observe that untrained text generation produces incoherent outputs. ✨To quantify model quality, we introduce cross-entropy loss, computed over next-token prediction targets created by shifting input sequences by one position. The model outputs logits across the full vocabulary, which are converted into probabilities via softmax. ✨Cross-entropy measures how well the predicted distribution aligns with ground-truth tokens, while perplexity (exp of loss) provides an interpretable uncertainty metric. ✨Finally, we compute training and validation losses to monitor learning progress. #deeplearning #generativeai #womenwhocode #largelanguagemodels #womenintech