# @awomanindatascience on Instagram

- **Type:** Video
- **Original URL:** https://www.instagram.com/p/DUCz11nk6jz
- **Gondola URL:** https://gondola.cc/posts/61230115-awomanindatascience-instagram
- **Thumbnail:** https://img.gondola.cc/tr:w-,h-,fo-auto/postThumbnails/2ce225591f.jpg
- **Posted:** 2026-01-28T06:18:41.000+00:00
- **Account Owner:** Chinar Arora (@awomanindatascience) — https://gondola.cc/awomanindatascience

## Caption

Hello Folks! The work week has begun in full force.
I managed to spend some cycles on my challenge nevertheless.

It’s day 19 of building a LLM from scratch 
✨Today I coded evaluation of generative language models during pretraining. So how do we start?
✨We first initialize a GPT-style transformer and observe that untrained text generation produces incoherent outputs. 
✨To quantify model quality, we introduce cross-entropy loss, computed over next-token prediction targets created by shifting input sequences by one position. The model outputs logits across the full vocabulary, which are converted into probabilities via softmax. 
✨Cross-entropy measures how well the predicted distribution aligns with ground-truth tokens, while perplexity (exp of loss) provides an interpretable uncertainty metric. 
✨Finally, we compute training and validation losses to monitor learning progress.

#deeplearning #generativeai #womenwhocode #largelanguagemodels #womenintech

## Stats

- **Views:** 0
- **Likes:** 218
- **Shares:** 0
- **Comments:** 3

## Tags

generativeai, womenintech, largelanguagemodels, deeplearning, womenwhocode

---
Copyright (c) Gondola