DataSphere

We have seen about RNN and LSTM in my previous blog, which are the foundation concepts for LLM. In today's blog, we will start looking at the actual LLM concepts like Transformers, LLM, Agents etc. LLM Vs Agents LLM(Large Language Model)  is a Neural Network, trained to predict the next word and provide meaningful context/output. Agents is a system that use LLM + tools + memory + decision logic to achieve some goals. An Agent is not just a model - its a system architecture. LLM LLM takes input from user and it will produce the output, but every run it may produce a different output. This is called Non-Deterministic output. Once conversation is over, it won't remember anything. This is called State-less . That's the reason, we can't directly use LLM and build an application, as it may produce different outputs each time. It is not designed specifically for this application and it behaves inconsistent. Agents  It will use LLM , T ools , M emory , O bservability & E v...

A Large Language Model is a deep Neural Network, trained with self-supervised learning on large amounts of data. It is an AI system, trained on massive amounts of data to understand and generate human-like language. Examples : GPT-4 ChatGPT LLaMa Gemini  A LLM is a model that : Reads text Learns patterns in language Predicts the next word Generates meaningful response At it's core, it is a next word prediction machine trained on millions/billions of word s. Most LLM's are built using the transformer architecture introduced in a white paper, named  "Attention is all you need!"  You can download the paper from :  https://arxiv.org/pdf/1706.03762   Note : Modern LLM's are using the "Transformer" architecture (recommended in the above white paper) as a foundation, but traditional old models had used RNN, LSTM as base. Let us understand the concepts of RNN, LSTM, pros and cons of them, then it will be easy for us to understand why we are using Transformers i...