🌴Today’s day 3/30 days of Building a Large Language Model from Scratch. I focused on preparing the training dataset today. This involves splitting text into individual word and sub word tokens which can then be encoded into vector representations for the LLM. 🌴Deep neural network models can’t process raw text directly. Therefore we need a way to represent words as continuous values vectors. The concept of converting data into a vector format is often referred to as embedding. 🌴The steps require for preparing the embedding include splitting text into words, converting words into tokens and turning tokens into embedding vectors. 🌴The text I tokenize for this LLM training is “The Verdict”, a short story by Edith Wharton which has been released into the public domain. Tomorrow I’ll convert them into embedding. #deeplearning #generativeai #30dayschallenge #womenwhocode #datascience #womenintech