This is a end-to-end data flow pipeline created using Apache Nifi, Kafka-Spark structured streaming and Snowflake. Flow of data in this pipeline : Server(https://randomuser.me/api/) -----> Nifi (Using REST API) -----> Kafka(Kafka brokers) -----> Consumer(Kafka Structured streaming) ----->Snowflake(To store data) For this project, I have used above online data generation website(randomuser.me) to collect the data stream. This will act as Server. We have to configure Nifi to catch this data stream using InvokeHTTP processor We have to configure Nifi to send this data stream to Kafka Producer using another processor called PublishKafkaRecord_2_6 So that data stream will be continuously happening from Online website to Kafka Producer Using PyCharm/VSS, create code for Kafka Consumer to receive this data using Spark structured streaming And store it in snowflake (after doing required transformation if needed) What knowledge required to understand this pipeline ? We need to ...