# @awomanindatascience on Instagram

- **Type:** Video
- **Original URL:** https://www.instagram.com/p/DIQGS9GRLvW
- **Gondola URL:** https://gondola.cc/posts/61230119-awomanindatascience-instagram
- **Thumbnail:** https://img.gondola.cc/tr:w-,h-,fo-auto/postThumbnails/edb673d6a1.jpg
- **Posted:** 2025-04-10T03:49:01.000+00:00
- **Account Owner:** Chinar Arora (@awomanindatascience) — https://gondola.cc/awomanindatascience

## Caption

🧠 Day 7/30 Building a Large Language Model from Scratch and today I learnt about Building Self-Attention from Scratch!
Today I learnt how to create a simple self-attention mechanism — no trainable weights, just pure math magic ✨

Here’s the breakdown:

🔹 Goal: Compute a context vector for each token that blends in info from all other tokens in the input.
🔹 Step 1: Use the dot product between the query (e.g., token 2) and all tokens to get attention scores.
🔹 Step 2: Normalize these scores using softmax to get attention weights (how much each token contributes).
🔹 Step 3: Multiply each input vector by its attention weight and sum them up to get the context vector.

🧪 Example input:
“Your journey starts with one step.”
Each word is a 3D embedding.
Context vector for “journey” ≈ [0.4419, 0.6515, 0.5683]

It’s amazing how attention lets models focus on relevant parts of a sentence — like how we do when understanding language!

📚 Next up: generalizing attention across all tokens and adding trainable weights!
#deeplearning #generativeai #30dayschallenge #womenwhocode #datascience #largelanguagemodels

## Stats

- **Views:** 976
- **Likes:** 31
- **Shares:** 0
- **Comments:** 0

## Tags

largelanguagemodels, deeplearning, 30dayschallenge, womenwhocode, generativeai, datascience

---
Copyright (c) Gondola