After working for more than 4 years on Data Intensive applications in a startup, consultancy and product-based companies. I think that the most important design and decision of any successful project is choosing the right Database and Data Storage at different layers of application. Whenever I am discussing my design…

One of the biggest pains of Data Engineers is small file problems. Let me tell you a short story and explain how one of the efficient tools solves this problem. A few days ago, while UPSERTING data to my Apache Hudi table, I was observing the pipeline result and noticed that my small files were being compacted and converted into larger files…

I love BigQuery and think It is one of the best products ever made by the Google Cloud Platform. As someone who works in data engineering, I have found BigQuery to be one of the best products offered by the Google Cloud Platform. …

In the previous article of this series i.e. Spark Streaming in layman’s terms we have understood the following things. Different streaming sources Stateful vs Stateless transformation For those who are reading this article without reading the previous article of this series, I recommend reading the last article or watching the…

In the first article about the spark streaming series, we have understood the following important concept. We have also written word count code to understand these concepts. For those who are reading this article without reading the first article of this series, I recommend reading the previous article or watching…

A few months back, I was given a codebase that used Spark Streaming and it was written in scala. We were supposed to make major changes and go for the new version of the project. I had worked on Spark Structure Streaming before but not in Spark Streaming. I found…

I recently tried onboarding disaster management for our streaming pipeline which involves Kafka, Spark Streaming and MongoDb in one of our use cases at Walmart Global Tech The last few weeks were a good learning curve for me and I really enjoyed all these awesome implementations of the streaming pipeline…

One of the most fundamental decisions to make in the Data Engineering world is to choose the proper file formats in different zones of the Big Data Pipeline. It helps the team to fetch the data faster and lower the cost of the project. …

When we are working with big data files like Parquet, ORC, Avro etc then you will mostly come across different compression codec like snappy, lzo, gzip, bzip2 etc. In this article, we will try to understand some of these compression codecs and discuss basic fundamental differences between them. Before starting…