DataSphere

This blog will explain a clear picture on what will happen inside a Neural Network(NN).  But before going through NN, we need to have some knowledge on some of the basic concepts in Calculus(Maths) & architecture of a Neural Network.  Note :   I recommend you to read the following blog(link mentioned below) and then start reading this blog. Previous blog link :  https://arunsdatasphere.blogspot.com/2026/01/deep-learning-and-neural-networks.html   At-least try to  understand the basic layers of NN, weights, biases, activation function, loss function etc. Lets start with Derivatives. Derivatives :                      Derivatives are originally a core concept of calculus (maths) . They answer one question which is  “How fast is something changing?”  Why derivatives appear in Machine Learning ? Machine Learning uses Math as its foundation. In ML, derivatives help answer : If I sligh...

I was curious to learn Artificial Intelligence and thinking what is the best place to start learning, and then realized that Deep Learning and Neural Networks is the heart of AI. Hence started diving into AI from this point. Starting from today, I will write continuous blogs on AI, especially Gen AI & Agentic AI. Incase if you are interested on above topics then please watch out this space. What is Artificial Intelligence, Machine Learning & Deep Learning ? AI can be described as the effort to automate intellectual tasks normally performed by Humans. Is this really possible ? For example, when we see an image with our eyes, we will identify it within a fraction of milliseconds. Isn't it ? For a computer, is it possible to do the same within same time limit ? That's the power we are talking about. To be honest, things seems to be far advanced than we actually thing about AI.  BTW, starting from this blog, it is not just a technical journal, we talk about internals here. ...

 Please keep below points in mind before creating a table in HIVE. HIVE is not a database but it is a data warehouse which is designed for data analytics(OLAP) HIVE doesn't have any internal storage to store data, it will use HDFS to store table data and RDBMS(like MySQL) to store schema related information. Hence we need to have a RDBMS like MySQL. When we create a database, HIVE will create a folder in HDFS. Later when we create a HIVE table, it will create another folder inside database folder. After creating database, table, once we load/insert data into it then that data will be stored as a file in HDFS and schema information will be stored in MySQL. Delimiter is very important while loading the data, we will see some examples below. As this is analytics, we know what type of data is there(as data is historical) and we will create schema based on the data. Based on the data, we need to create a table. HOW TO CREATE DATABASE ----------------------------- CREATE DATABASE IF NOT ...

Pair RDD operations are the real time operations which we use in projects. We can solve all real time issues using pair RDD's.  In distributed environment, to handle complex problems, we can't go with just value based approach. We should also have a key associated with it. Remember in Map Reduce, internal calls will happen using record which is a Key, Value pair. In RDBMS, multiple columns will be available associate to one primary key. Record  <key, Value>  Values can be multiple but it will associate with a Key Example : scala> val namesrdd = sc.parallelize(List("raj", "venkat", "sunil", "kalyan", "anvith", "raju", "dev", "hari"), 2) namesrdd: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[38] at parallelize at <console>:23 scala> val prdd1 =namesrdd.map(x => (x, x) ) prdd1: org.apache.spark.rdd.RDD[(String, String)] = MapPartitionsRDD[39] at map at <console>:2...