# @awomanindatascience on Instagram

- **Type:** Video
- **Original URL:** https://www.instagram.com/p/DIILjtupvu2
- **Gondola URL:** https://gondola.cc/posts/61230121-awomanindatascience-instagram
- **Thumbnail:** https://img.gondola.cc/tr:w-,h-,fo-auto/postThumbnails/7ed24741c4.jpg
- **Posted:** 2025-04-07T02:04:13.000+00:00
- **Account Owner:** Chinar Arora (@awomanindatascience) — https://gondola.cc/awomanindatascience

## Caption

For Day 5/30 I focused on the first step of building a language model — embedding tokens and encoding their positions.

🔡 Token Embeddings: I used PyTorch’s nn.Embedding to convert token IDs (from a BPE tokenizer) into 256-dimensional vectors. This captures what each token is, but not where it is in the sequence.

📍 Positional Embeddings: Since transformers are position-agnostic, I created a separate embedding layer for positions (shape: [context_length, 256]) and added them to the token embeddings to encode order.

🧠 Fun fact: Unlike the original Transformer model, OpenAI’s GPT uses absolute positional embeddings that are optimized during training — not predefined. This helps GPT better understand long-range dependencies and varying sequence lengths.

💡 After this step, I now have input_embeddings of shape [batch_size, sequence_length, embedding_dim] — ready to be fed into the attention mechanism!

Onward to building the self-attention module next! 

#deeplearning #generativeai #30dayschallenge #womenwhocode #datascience #womenintech #largelanguagemodels

## Stats

- **Views:** 0
- **Likes:** 37
- **Shares:** 0
- **Comments:** 0

## Tags

womenwhocode, generativeai, datascience, womenintech, largelanguagemodels, deeplearning, 30dayschallenge

---
Copyright (c) Gondola