(AI #12) LLM - Implementing a GPT model from scratch to generate text

Lets quickly recap we I have discussed in previous blog about Attention Mechanism. Regarding Attention Mechanism, we have seen what is self-attention, drawbacks of legacy systems, Self-attention without trainable weights, Self-attention with trainable weights, Causal attention, Masked multi-headed attention.

Inside Self-Attention, we have discussed about :

• Attention Scores
• Attention Weights
• Context Vector

Importance of context vector is, it is going to maintain the semantics of all the tokens(not just about a single token).

Self-attention with trainable weights :

• Inputs
• Weight, Query, Value matrix (Wq, Wk, Wv) and cal Q, K, V
• Query = Input @ Wq
• Key = Input @ Wk
• Value = Input @ Wv
• Above Query, Key, Value matrix are initially random values and they will be adjusted during training
• Calculate Attention score
• Query matrix @ Transpose of Key matrix
• Attention Weights
• Scaling = Square root of dimensions of Key matrix
• Normalization using softmax
• We are transforming data from Attention Scores to Attention Weights because we want to make interpretability easy. 
• Context vector 
• Attention Weights @ Value vector

Causal Attention :

• Inputs
• Weight, Query, Value matrix (Wq, Wk, Wv)
• Query = Input @ Wq
• Key = Input @ Wk
• Value = Input @ Wv
• Attention Scores
• Query matrix @ Key Transpose
• Attention Weights
• Scaling
• Masking (Future Values i.e. -infinity)
• Softmax
• Optional dropout
• Context Vector
• Attention Weights @ Value matrix

Masked Multi-Headed Attention :

• Same as Causal Attention but this have multi heads
• This is to increase the parallelism 
• For GPT-2 small model, it has 12 layers & 12 heads per layer (Total heads = 144) 

LLM Architecture

Above image represents each and every single step involved in implementing a GPT model from scratch. For GPT-2 model, we have 12 layers and in each layer there will be 12 heads. So, as shown in the above images Transformer Block will be repeated 12 times for each layer. By the end this blog, we will know in detail about all the blocks shown in the above image. 

Out input sentence : "Every effort moves you" ==> this will break into multiple tokens (token could be a character, word, sub-word), internally GPT-2 is using a algorithm called BPE. Tokens won't carry any meaning, we have to convert tokens into Token Embeddings which will carry the meaning of tokens as well. Then we need to identify the position of token, which are called as Positional Embeddings, then we will combine both Token Embedding & Positional Embedding and create Input Embeddings. 

We have discussed, in Neural Network Dropout prevents Overfitting & Under-fitting problem. To prevent we will use Dropout mechanism. 

In the above image, blue colour block is called Transformer Block. It has some important block called as Masked Masked Attention mechanism. Transformer blocks are nothing but layers and we have 12 layers in GPT-2 model. In each block, we have 12 heads which are nothing but Masked Multi-headed attention blocks. 

So, total number of Masked Multi-headed attention blocks are 144 in GPT-2 model. 

Also, note that a connection which is starting before LayerNorma-1 and joining '+' after Dropout is called Skip/Residual Connection.  

Feed Forward is a Neural Network which consists of input, hidden, output layers. In hidden layers, we generally are going to use activation functions called Relu etc.  In this hidden layer, we use GeLU activation function.  

Also see , another Skip/Residual Connection started after first Dropout after '+' and connected to next '+' at the end of transformer layers. 

This entire grey colour box which internally contain a Transformer block is called a GPT model. We will program this entire architecture in this blog. That's the goal for this blog. 

GPT-2 small model (124 million parameters) : 

We need below configuration for implementing GPT model.

• Vocabulary size, vocab_size = 50,257
• In above image, during tokenization we need this vocabulary
• Token Embedding 
• embedding size, emb_dim = 768
• each word/token will be represented in a 768 dimensional space
• Context Length
• context_length = 1024
• means at a time, we will consider 1024 tokens for progressing
• Heads 
• n_heads = 12
• Layers
• n_layers = 12
• Dropout 
• drop_rate = 0.1

Thank you for reading this blog !

Arun Mathe