Nikhil is an Engineering Professional with a BTech degree in IT from Savitribai Phule University. He has a keen interest in problem-solving and designing proficient solutions for business use cases. His experience spans cloud data engineering, ETL development, and building complex data pipelines.
Experience
Software Engineer 2
Epsilon
Building framework to migrate Map Reduce jobs to Spark. Improved ORC file and Kafka compression thus reducing HDFS Disk footprint by 3 times using techniques like ( tuning linger and batch size, Sticky Partitioner, Sorting high cardinality columns etc). Build Kibana Dashboard for Spark Job metrics using ELK stack. Troubleshooting and debugging data pipeline on rotation basis to support 300B records/ day. Shell script for replacing expensive hive msck repair command.
Cloud Data Engineer
Hashmap Inc
WASTE MANAGEMENT (Oct 2020- June 2021): Responsible for integrating On prem Data Sources (Oracle, Netezza) and Cloud datasources (Google Analytics, S3, APIs) with Snowflake. Responsible for writing Matillion ETL jobs and supporting a range of Business consumption of patterns. Cloud Formation scripts to automate test and deployment of AWS resources. LAM RESEARCH (Dec 2018- Sept 2020): Build Spark batch application for parsing complex JSON files. Ingesting data into HBase and creation of HDF5 files using Java. Build Pyton library for loading, querying and performing analytical operations on time-series HBase data using SQL like syntax. Handling Kafka backpressure and message loss ( custom offset management). MURPHY OIL (June 2018-Nov 2018): Build accelerator to automate ETL process by using crealytics library for excel ingestion.
Education
Savitribai Phule University
BTech
IT
Licenses & Certifications
Snowflake SnowPro Core
Google Cloud Certified Professional Data Engineer
Applied Data Science 1: Scientific Computing and Python (with honors)
WorldQuant University