Ankur Ranjan – Medium

Sep 12

File Formats in Big Data World — Part 1

One of the most fundamental decisions to make in the Data Engineering world is to choose the proper file formats in different zones of the Big Data Pipeline. It helps the team to fetch the data faster and lower the cost of the project. …

Data Engineering

7 min read

Mar 22

Demystify different compression codec in big data

When we are working with big data files like Parquet, ORC, Avro etc then you will mostly come across different compression codec like snappy, lzo, gzip, bzip2 etc. In this article, we will try to understand some of these compression codecs and discuss basic fundamental differences between them. Before starting…

Big

4 min read

Mar 21

LeetCode Curated SQL Solutions and Discussion — Week 1

SQL is a must when you are in the domain of Data. Let it be Data Engineering, Big Data, Data Analyst or BI Developer, everyone who is working with Data should have a good understanding of SQL. I feel that reading SQL theoretically is not gonna help that much. So…

Leetcode

6 min read

Published in Analytics Vidhya

·Apr 1, 2021

Apache Sqoop - One smart tool for Big Data World.

When we talk about the big data world then there are always three things involved and they are storage, processing & scalability. Here we always like to store a massive amount of data, process it efficiently in a given amount of time & above all design systems that are highly…

Sqoop

8 min read

Mar 25, 2021

Spark Interview Guide : Module 1: Spark Core

Nivedita Mondal

I think there is one mistake in the definition of Executors i.e.

I think there is one mistake in the definition of Executors i.e. `Executor in Spark are worker nodes`. Actually, Executors are not worker nodes instead they are the worker nodes processes. And I must agree this is really very well written.

1 min read

I think there is one mistake in the definition of Executors i.e. `Executor in Spark are worker nodes`. Actually, Executors are not worker nodes instead they are the worker nodes processes.

And I must agree this is really very well written.

Published in Analytics Vidhya

·Jul 28, 2020

Apache Airflow — Part 1

Every programmer loves automating their stuff. Learning and using any automation tool is fun for us. A few months ago, I came across a wonderful open source project called apache-airflow. I have tried to discover this open source project and use it in my existing codebase. This blog series is…

Airflow

5 min read

Published in Analytics Vidhya

·Jul 5, 2020

Part 2- A Beginners Guide to Time profiling in Python.

Hello folks, welcome back. If you are joining back from my last blog then pretty much context has been set about time profiling, If you have not yet gone through the last blog, I highly recommend you to do so, though it is not mandatory. Using %lprun for time profiling. …

Python

3 min read

Published in Analytics Vidhya

·Jun 27, 2020

A Beginners Guide to Time profiling in Python.

Writing a python code gives us great power to showcase our idea by easily programming them but like someone has rightly said “With great power comes great responsibility”. As python is a high level interpeter based language, there is always a need to check the time complexity and code profiling…

Python

3 min read