Adit Chauhan

Working on scheduling ETL Pipelines for Dimension Tables using Spark, Dbt and Airflow.

Implemented complex python logics for data analysis. Collaborated with other teams for infrastructure setup and worked on various AWS tech stack to meet the design requirements. Worked on exception handling and to send out relevant mails to the support team to debug. Developed ETL jobs in Airflow to build audit process for input tables and to launch EMR cluster and deploy spark applications. Implemented Backend process with AWS glue, S3, step-function, lambda, PostgreSQL and event bridge using python and spark scripts. Created triggers and functions to PostgreSQL tables. Involved in creating Hive tables using various partition techniques, and analyzed data using Hive queries. Data Ingestion and refresh from AWS RDS (PostgreSQL) to Hive external tables using psycopg2, boto3 and processing data through Spark Core and Spark SQL. Implemented and deployed change requests on production. Loaded and transformed large sets of data from RDS (PostgreSQL) to S3 and vice-versa using Spark. Performance Tuning and Optimization of Spark jobs. Fixing production issues and troubleshooting spark/python jobs as part of developer support.

Implemented scheduled Data movement from Sftp server to Snowflake. Integrated the flow to AWS Lambda and S3.

Implemented end to end real time workflow for Visualization team to build real time reports. Got an opportunity to work on web-hooks, API integration, Lambda and DynamoDB. Spent time on designing the workflow architecture.