# @awomanindatascience on Instagram

- **Type:** Video
- **Original URL:** https://www.instagram.com/p/DTKHltnE7Si
- **Gondola URL:** https://gondola.cc/posts/61230177-awomanindatascience-instagram
- **Thumbnail:** https://img.gondola.cc/tr:w-,h-,fo-auto/postThumbnails/d43f9c4f96.jpg
- **Posted:** 2026-01-06T05:52:04.000+00:00
- **Account Owner:** Chinar Arora (@awomanindatascience) — https://gondola.cc/awomanindatascience

## Caption

It’s Day 11 of Building a LLM from scratch. 
Today I learn about how to implement the self attention mechanism used in the original transformer architecture, the GPT models and most popular LLMs.  This is also called scaled dot product attention. 

Self-attention with trainable weights is how transformers learn context instead of following fixed rules. Each token embedding is projected into Query, Key, and Value using learned matrices. Queries ask what am I looking for?, keys say what do I contain?, and values pass the actual information. The model computes similarity between queries and keys, scales and normalizes them with softmax, and uses the result to weight the values. In language models, a causal mask blocks future tokens. Because Q, K, and V are learned, the model discovers grammar, meaning, and long-range relationships on its own.

With the pace I’m going I’ll most likely extend this challenge beyond 30 days so stay tuned 😉

#deeplearning #generativeai #womenwhocode #largelanguagemodels #womenintech

## Stats

- **Views:** 35,239
- **Likes:** 2,818
- **Shares:** 0
- **Comments:** 55

## Tags

generativeai, womenintech, largelanguagemodels, deeplearning, womenwhocode

---
Copyright (c) Gondola