DataSphere

This is a end-to-end data flow pipeline created using Apache Nifi, Kafka-Spark structured streaming and Snowflake. Flow of data in this pipeline : Server(https://randomuser.me/api/) -----> Nifi (Using REST API) -----> Kafka(Kafka brokers) -----> Consumer(Kafka Structured streaming) ----->Snowflake(To store data) For this project, I have used above online data generation website(randomuser.me) to collect the data stream. This will act as Server. We have to configure Nifi to catch this data stream using InvokeHTTP processor We have to configure Nifi to send this data stream to Kafka Producer using another processor called PublishKafkaRecord_2_6 So that data stream will be continuously happening from Online website to Kafka Producer Using PyCharm/VSS, create code for Kafka Consumer to receive this data using Spark structured streaming And store it in snowflake (after doing required transformation if needed) What knowledge required to understand this pipeline ?  We need to ...

Apache Kafka : Apache Kafka is an open-source data streaming platform that stores, processes, and analyzes large amounts of real-time data. It's used to build real-time data pipelines and applications that can adapt to data streams.  Event streaming :  Event streaming is the digital equivalent of the human body's central nervous system. It is the technological foundation for the 'always-on' world where businesses are increasingly software-defined and automated, and where the user of software is more software. Technically speaking, event streaming is the practice of capturing data in real-time from event sources like databases, sensors, mobile devices, cloud services, and software applications in the form of streams of events; storing these event streams durably for later retrieval; manipulating, processing, and reacting to the event streams in real-time as well as retrospectively; and routing the event streams to different destination technologies as needed. Event strea...