DataSphere

In this blog, we are going to see how Spark streaming will work, we will also see how to write Spark code to read streaming data, and store in some other place, let's say AWS RDS. Spark Architecture :  We have below layers in Spark architecture.  Data Storage (HDFS, HBase, Cassandra, Amazon S3) Resource Management (Hadoop Yarn, Apache Mesos, Kubernetes) Processing Engine (Spark Core) Libraries ( Spark streaming , Spark SQL, GraphX, MLlib) API's (Scala, Python, Java, R) Spark core :  Spark core is the heart of Spark architecture. By default, it will process only batch data(historical data), on the top of this we can use libraries to perform multiple activities. That means, if you need to run any SQL queries on the top of Spark, it won't support directly, hence we use Spark SQL. We have to understand that there is a limitation in Spark where we can't read live incoming traffic, at-least we need to let that incoming traffic wait for few seconds, and then only read, process...

This is one of the important concept where we will see how an end-to-end pipeline will work in AWS. We are going to see how to continuously monitor a common source like S3/Redshift from Lambda(using Boto3 code) and initiate a trigger to start some Glue job(spark code), and perform some action.  Let's assume that, AWS Lambda should initiate a trigger to another AWS service Glue as soon as some file got uploaded in AWS S3 bucket, Lambda should pass this file information as well to Glue, so that Glue job will perform some transformation and upload that transformed data into AWS RDS(MySQL). Understanding above flow chart : Let's assume one of your client is uploading some files(say .csv/.json) in some AWS storage location, for example S3 As soon as this file got uploaded in S3, we need to initiate a TRIGGER in AWS Lambda using Boto3 code Once this trigger is initiated, another AWS service called GLUE(ETL Tool)  will start a Pyspark job to receive this file from Lambda, perform so...

Below code is to delete a existing RDS and also to create a new RDS in AWS RDS using boto3 python package : import boto3 # Creating a client session for RDS using region name, aws_access_key_id & aws_secret_access_key client = boto3 . client ( 'rds' , region_name = "ap-south-1" , aws_secret_access_key = 'YOUR_AWS_SECRET_ACCESS_KEY' , aws_access_key_id = 'YOUR_AWS_ACCESS_KEY_ID' ) # Deletig an existing instance # DB instance ID is enough, make sure to skip final snapshot & delete any automated backup's response = client .delete_db_instance( DBInstanceIdentifier = 'newpoc' , SkipFinalSnapshot = True , DeleteAutomatedBackups = True ) # To cross check if any RDS is available response = client .describe_db_instances() print ( response ) # To create a new RDS in AWS # DBInstanceIdentifier is the name of RDS # Engine must be your expected RDBMS name # Provide user name and...

Boto3 is the Amazon Web Services software development kit for Python, which allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2. Boto3 is maintained and published by AWS. Please find latest documentation at : https://boto3.amazonaws.com/v1/documentation/api/latest/index.html Command to install it : pip install boto3 Local storage Vs Cloud storage: Local file system is block oriented, means storage is divided into block with size range 1-4kb Collections of multiple blocks is called a file in local storage Example : 10MB file will be occupying almost 2500 blocks(assuming 4kb each block) We know that we can install softwares in local system (indirectly in blocks) Local system blocks managed by Operating system But Cloud storage is a object oriented storage, means everything is object No size limit, it is used only to store data, we can't install software in cloud storage Cloud storage managed by users We need to install either Pyc...

What is Athena ? Amazon Athena is a serverless, interactive query service that allows users to analyze data stored in Amazon Simple Storage Service(S3). To make it more clear, it is a kind of workbench for working on HIVE queries in more visual way.  AWS took open source Apache HIVE and modified it and released their own product called Athena. Home page of Athena looks as below(login to AWS and type Athena under search) : Once you are in Athena home page, it will show a prompt to launch it on right hand side, click launch to open Athena editor(as shown in above screenshot). Also, before start working on Athena, we need to set up a query result location in Amazon S3. Let's understand why we do this, if you remember, HIVE doesn't have a storage, we need a relational database to store schema information and HDFS storage to store table data(records). HIVE is just a processing layer on the top of Hadoop.  Similarly, even in Athena, we need to set up a query result location in Amazo...

What is the use case ?   Let's see how to read data from either AWS S3/RDS, apply transformation and load it in HIVE using EMR service in Amazon Web Services. Pre-requisites : We need an AWS account Create a EMR system with Hadoop, Hive & Spark installed in it Create a S3 bucket and have some .csv files with same data Create a RDS system in AWS (preferable MySQL as it's free)  Sample EMR service will be as below : Now, click on Connect to the primary node using SSH option as shown in below screenshot. Now copy the SSH command as shown below.  Now from your local desktop/laptop, open command prompt and paste copied SSH command, press enter and then say yes and then enter. It will connect to EMR service as shown below. Use below command to cross check installed software, it will display the location of those softwares. To enter into HIVE shell, just type hive and press enter, it will take you to hive shell. If you want to go to Hadoop prompt from HIVE, just do Ctrl+C. ...