Dipesh Chaudhary
@dipeshchaudhary
DATA ENGINEER at CLOUDGRITZ TECHNOLOGIES
Noida, India
Dipesh Chaudhary is an experienced Data Engineer skilled in building robust data pipelines and migrating complex data structures. He has expertise in AWS services, including Glue, S3, and RedShift, and utilizes PySpark for efficient data transformation. His background includes setting up CI/CD workflows and maintaining real-time data lakes using Databricks and Kafka.
Experience
DATA ENGINEER
CLOUDGRITZ TECHNOLOGIES
Developed ETL pipelines using AWS Glue & PySpark to efficiently migrate and transform over 200 SQLServer tables to AWS RedShift. Automated infrastructure provisioning using AWS CloudFormation, reducing manual setup time and ensuring consistency. Set up CI/CD workflows using GitHub Actions and Spinnaker, automating deployments and version control. Optimized RedShift table structures and performance, improving query speed by 40%. Maintained real-time data pipelines using Kafka, Databricks and PySpark to process high-volume customer data in the sports and betting domain, ensuring scalability and fault tolerance. Implemented the Medallion Architecture in Databricks, organizing data into Bronze, Silver, and Gold layers to streamline data ingestion, cleansing, and transformation. Automated CI/CD workflows with Jenkins to deploy PySpark and SQL scripts, reducing manual deployment errors and accelerating pipeline updates.
DATA ENGINEER INTERN
CLOUDGRITZ TECHNOLOGIES
Tested data pipelines and databricks jobs to ensure they function correctly and meet the requirements. Gained expertise in performance tuning and resource management in Spark, and learnt optimal techniques for the same. Created interactive dashboards using the Databricks SQL Dashboards to explore and visualize data using SQL queries.
Education
Amity School of Engineering & Technology
B.Tech - ECE