Skip to main content

Spark : Spark context and RDD

Spark context is the entry point for any Spark operations. Suppose we are running Spark in single node, multi nodes, all required details are available as part of an object called Spark context.

Once we have a Spark context, we can create a RDD out of this and proceed with whatever the operations we want to perform like min(), max(), groupBy, filter etc, We can do two important things in Spark, one is Transformation and other one is Action. 

Learning spark is nothing but learning how to transform data and perform actions on them.


Below is the flow :

SPARK CONTEXT --> RDD -->(Transformations, Actions)


RDD(Resilient Distributed Dataset) : Resilient means failover(recover quickly from difficult conditions), Distributed means data is distributed(not in single server/machine), Dataset is a collection of data

  • Transformation 
  • Action


Simple use case to understand difference between Transformation & Action :

Example 1 : f(x) = x + 1
If we give multiple inputs to above function, we will get multiple outputs. It is called a Transformation.


Example 2 : f([x1, x2, x3, ....xn]) = min(f(x)) OR max(f(x)) OR sum(f(x))
Even if we give multiple inputs for above function, we will get only one output. It is called a Action.



Functionalities of RDD :
  • Immutable (can't be modified)  
    • Incase if we have to transform an RDD, then convert it into another RDD
      • Example :RDD1 --> RDD2
    • This is called Transformation
    • Spark is meant of Analytics (NOT TRANSACTIONS - OLAP(not OLTP)) 
    • If RDD's are not immutable then distributed parallel processing will produce inconsistent results. 
  • Cacheable 
    • It will persist information for future operations
    • It can be at either Disk level or Memory level 
    • Spark is ~100x faster because of this Cache technique
    • It will reuse existing results for further computations
      • Example : Google results will retain to show for next users who use same search string
    • You can Cache entire data in Hard Disk
    • You can Cache entire data in Memory
    • You can Cache data in both Hard Disk + Memory
      • It will try to keep as much data in memory, rest of the data into Hard Disk
  • Lazy evaluation
    • RDD follow bottom to top approach
    • It won't execute everything, before execution it will come up with plan on what to execute
    • Main constraint is 
      • if information is static, then we can go for lazy evaluation
      • if information is dynamic, then we can't go for this approach
  • Distributed, Partitioning, replication
  • Type Infer

Let's learn more information on further blogs. Have a great day!





Arun Mathe

Gmail ID : arunkumar.mathe@gmail.com










Labels:

Comments

Post a Comment

Popular posts from this blog

(AI #1) Deep Learning and Neural Networks

I was curious to learn Artificial Intelligence and thinking what is the best place to start learning, and then realized that Deep Learning and Neural Networks is the heart of AI. Hence started diving into AI from this point. Starting from today, I will write continuous blogs on AI, especially Gen AI & Agentic AI. Incase if you are interested on above topics then please watch out this space. What is Artificial Intelligence, Machine Learning & Deep Learning ? AI can be described as the effort to automate intellectual tasks normally performed by Humans. Is this really possible ? For example, when we see an image with our eyes, we will identify it within a fraction of milliseconds. Isn't it ? For a computer, is it possible to do the same within same time limit ? That's the power we are talking about. To be honest, things seems to be far advanced than we actually thing about AI.  BTW, starting from this blog, it is not just a technical journal, we talk about internals here. ...
Post a Comment
Read more

(AI #3) Deep Learning Foundations - Activation & Loss Functions, Gradient Descent algorithms & Optimization techniques

It is extremely important to have a deep knowledge while designing a machine learning model, otherwise we will end up creating ML models which are of no use. We have to have a clear understanding on certain techniques to confidently build a ML model, train it using "training data", finalize the model and to deploy it in production. So far, from blog #1, #2, we have seen about the fundamentals of Deep Learning and Neural Network, architecture of a Neural Network, internal layers and components etc.  Providing the links of Blogs #1 , #2 below for quick reference. Deep Learning & Neural Networks : https://arunsdatasphere.blogspot.com/2026/01/deep-learning-and-neural-networks.html Building a real world neural network: A practical usecase explained : https://arunsdatasphere.blogspot.com/2026/01/building-real-world-neural-network.html Now let's dive through below concepts/criteria to help gaining confidence on building your ML model: Activation Functions (Forward Propaga...
Post a Comment
Read more

(AI #2) Building a Real-World Neural Network: A Practical Use Case Explained

This blog will explain a clear picture on what will happen inside a Neural Network(NN).  But before going through NN, we need to have some knowledge on some of the basic concepts in Calculus(Maths) & architecture of a Neural Network.  Note :   I recommend you to read the following blog(link mentioned below) and then start reading this blog. Previous blog link :  https://arunsdatasphere.blogspot.com/2026/01/deep-learning-and-neural-networks.html   At-least try to  understand the basic layers of NN, weights, biases, activation function, loss function etc. Lets start with Derivatives. Derivatives :                      Derivatives are originally a core concept of calculus (maths) . They answer one question which is  “How fast is something changing?”  Why derivatives appear in Machine Learning ? Machine Learning uses Math as its foundation. In ML, derivatives help answer : If I sligh...
Post a Comment
Read more