Lets quickly recap we I have discussed in previous blog about Attention Mechanism. Regarding Attention Mechanism, we have seen what is self-attention, drawbacks of legacy systems, Self-attention without trainable weights, Self-attention with trainable weights, Causal attention, Masked multi-headed attention. Inside Self-Attention, we have discussed about : Attention Scores Attention Weights Context Vector Importance of context vector is, it is going to maintain the semantics of all the tokens(not just about a single token). Self-attention with trainable weights : Inputs Weight, Query, Value matrix (Wq, Wk, Wv) and cal Q, K, V Query = Input @ Wq Key = Input @ Wk Value = Input @ Wv Above Query, Key, Value matrix are initially random values and they will be adjusted during training Calculate Attention score Query matrix @ Transpose of Key matrix Attention Weights Scaling = Square root of dimensions of Key matrix Normalization using softmax We are transforming data from Attention Scor...