# @awomanindatascience on Instagram

- **Type:** Video
- **Original URL:** https://www.instagram.com/p/DTPRwUtk27Q
- **Gondola URL:** https://gondola.cc/posts/61230176-awomanindatascience-instagram
- **Thumbnail:** https://img.gondola.cc/tr:w-,h-,fo-auto/postThumbnails/c88e193f09.jpg
- **Posted:** 2026-01-08T06:01:21.000+00:00
- **Account Owner:** Chinar Arora (@awomanindatascience) — https://gondola.cc/awomanindatascience

## Caption

Hi Folks! For day 12 of Building a LLM from scratch I understand what is causal self- attention. 

Causal self-attention ensures an LLM never looks into the future. By masking attention weights above the diagonal, each token can attend only to itself and past tokens, making next-word prediction truly autoregressive. Instead of masking after softmax (which breaks probability normalization), we efficiently mask attention scores with -∞ before softmax. 

We also apply dropout on attention weights during training to reduce overfitting. 

Finally, this logic comes together in a compact, batch-aware CausalAttention module used in GPT-style models. 
I understood how to code a ClassAttention module which will be used in the next section of multi head attention.
See you soon! ✌️

#deeplearning #generativeai #largelanguagemodels #womenintech

## Stats

- **Views:** 1,934
- **Likes:** 166
- **Shares:** 0
- **Comments:** 5

## Tags

generativeai, womenintech, largelanguagemodels, deeplearning

---
Copyright (c) Gondola