DGDivyanshu is a systematic and dedicated team player with an analytical mindset. He has approximately one year of experience in Big Data Engineering. His expertise includes building end-to-end data pipelines and utilizing technologies like Apache Spark, AWS, and Snowflake for real-time insights.
Experience
Big data Engineer
BluePi Consulting Pvt. Ltd.
Created end to end Big data pipeline for a leading Real Estate Company to provide real time insights. Setup AWS Database Migration service to migrate data from various sources (MySql, PostgeSQL, Google Analytics) to datalake (AWS S3). Developed spark applications in python (PySpark) to load huge number of files with different schema into Delta tables of databricks. Worked on data cleaning and pre-processing to perform feature engineering. Wrote spark script to transform and load data to snowflake staging and main tables. Complete ownership of enterprise reporting database build with dimensional modelling: - SCD-Type1 and SCD-Type2. Created live Dashboard on Metabase and Tableau. Created Data quality dashboards using Great Expectations.
Education
National Institute of Technology, Jaipur
Bachelor of Technology
Mechanical Engineering
Licenses & Certifications
Databricks Lakehouse Platform Essentials
Databricks
An Introductory Guide to SQL
Educative Course
Apache Spark Programming with Databricks
Databricks