Skip to main content

Spark : What is Spark ?

Apache Spark is an open-source, distributed system for processing large amounts of data. It's used for analytics, machine learning, and other applications that require fast processing of large data sets.


History of Spark :

Around 2009, a project called Mesos started in Berkeley university. It is a resource management system, similar to yarn in Hadoop. 

In Hadoop, we have a data processing module called Map Reduce. It consists of processes called JobTracker and TaskTracker. People who started Mesos aware of drawbacks of Map Reduce. To test this project called Mesos, these people implemented Spark, but their primary goal is Mesos. 

Initial Spark program is just 100 lines of code, they observed that Spark is almost 10x faster than Hadoop. Then focus shifted from Mesos to Spark. Around 2013 they made this project as open source.

In 2014, around Aug, Spark becoming top level project in Apache. Instead of using Map Reduce, people started using Map Reduce(MR). It is almost 100x faster than MR.

Now, Spark is a mandatory Big Data technology for data processing.



What is the main programming language for Spark(Spark 1.x) ?

Main programming language of Spark is Scala. To support other programming languages, some wrappers are available. We also have good number of use cases where we have to use Python(using library Py4J). 

We can use any of the below programming languages :

  • Scala
  • Python
  • Java
  • R
  • SQL


Why scala ? 

Scala is implemented on top of Java. Advantage of Scala is, it is having all Java features, you can directly use Java code in Scala, it is a scalable language. Scala development started in 2002, its main goal is to fix all issues existing in Java. Also Hadoop is built using Java.


Points to remember:

  • We have to use either Java/Scala to implement new features in Spark.
  • Major programming language is Scala
  • We can use Python as well using Py4J library
  • We can run Java in Scala but vice versa is not possible
  • We can directly call Java programs inside a Scala program

Note : Above context is for Spark 1.x


From Spark2, unified engine for large scale data analytics approach came into picture :

  • Performance will be same even if we use any programming language
  • API's used in Scala, Java, Python are same (almost ~90% are same)
  • Advantage of this approach is, lets say we learnt Spark in Python, then switching to Scala needs just the basics of Scala


Note : Spark3 fixed all the minor issues from Spark2 as well.


Below code snippet confirm that code will be almost similar in all programming languages like Python, Scala & Java. So, just remove the Myth that you need in depth programming knowledge to learn Spark. We need to be strong in Spark, programming basics are good enough.


Python code : Created a data-frame to read data from logs.json

df = spark.read.json("logs.json")

df.where("age > 21").select("name.first").show()


Scala code : Created a variable to read data from logs.json

val df= spark.read.json("logs.json")

df.where("age > 21").select("name.first").show()


Java code : Created dataset data-frame to read data from logs.json

Dataset df = spark.read.json("logs.json")

df.where("age > 21").select("name.first").show()



More information about Spark :

  • Heart of Spark is Spark core and we have below 4 main libraries in it 
    • Spark SQL
    • Spark Streaming
    • Spark MLLib
    • Spark GraphX
  • Above 4 libraries are built  on the top of Spark context & RDD
  • Spark Context & RDD are the primary concepts of Spark
  • Spark Context 
    • Entry point for any operations(to filter, groupBy, min, max etc,)
    • Using Spark context we will create an RDD
    • On top of RDD, we can run above operations


Conclusion :

Basically spark is all about processing large amount of data(Big Data). Going forward, I will be discussing more about Spark, like how to process large amounts of data, also how to do same in Cloud(AWS, Azure etc,). Lets see more about Spark in coming blogs. 

Have a great day!







Labels:

Comments

Post a Comment

Popular posts from this blog

(AI #1) Deep Learning and Neural Networks

I was curious to learn Artificial Intelligence and thinking what is the best place to start learning, and then realized that Deep Learning and Neural Networks is the heart of AI. Hence started diving into AI from this point. Starting from today, I will write continuous blogs on AI, especially Gen AI & Agentic AI. Incase if you are interested on above topics then please watch out this space. What is Artificial Intelligence, Machine Learning & Deep Learning ? AI can be described as the effort to automate intellectual tasks normally performed by Humans. Is this really possible ? For example, when we see an image with our eyes, we will identify it within a fraction of milliseconds. Isn't it ? For a computer, is it possible to do the same within same time limit ? That's the power we are talking about. To be honest, things seems to be far advanced than we actually thing about AI.  BTW, starting from this blog, it is not just a technical journal, we talk about internals here. ...
Post a Comment
Read more

(AI #3) Deep Learning Foundations - Activation & Loss Functions, Gradient Descent algorithms & Optimization techniques

It is extremely important to have a deep knowledge while designing a machine learning model, otherwise we will end up creating ML models which are of no use. We have to have a clear understanding on certain techniques to confidently build a ML model, train it using "training data", finalize the model and to deploy it in production. So far, from blog #1, #2, we have seen about the fundamentals of Deep Learning and Neural Network, architecture of a Neural Network, internal layers and components etc.  Providing the links of Blogs #1 , #2 below for quick reference. Deep Learning & Neural Networks : https://arunsdatasphere.blogspot.com/2026/01/deep-learning-and-neural-networks.html Building a real world neural network: A practical usecase explained : https://arunsdatasphere.blogspot.com/2026/01/building-real-world-neural-network.html Now let's dive through below concepts/criteria to help gaining confidence on building your ML model: Activation Functions (Forward Propaga...
Post a Comment
Read more

(AI #2) Building a Real-World Neural Network: A Practical Use Case Explained

This blog will explain a clear picture on what will happen inside a Neural Network(NN).  But before going through NN, we need to have some knowledge on some of the basic concepts in Calculus(Maths) & architecture of a Neural Network.  Note :   I recommend you to read the following blog(link mentioned below) and then start reading this blog. Previous blog link :  https://arunsdatasphere.blogspot.com/2026/01/deep-learning-and-neural-networks.html   At-least try to  understand the basic layers of NN, weights, biases, activation function, loss function etc. Lets start with Derivatives. Derivatives :                      Derivatives are originally a core concept of calculus (maths) . They answer one question which is  “How fast is something changing?”  Why derivatives appear in Machine Learning ? Machine Learning uses Math as its foundation. In ML, derivatives help answer : If I sligh...
Post a Comment
Read more