Dipesh Gupta

Responsible to develop new ETL pipelines in ADF. Took ownership of all the pipelines created and existed in ADF. Responsible to write the business transformation logics in Databricks using PySpark/Scala. Responsible to refresh all the pipelines weekly or monthly based on the business need and debug the error faced, if any, while refreshing the pipelines. Understand the end-to-end pipeline to enhance the existing/new pipelines. Follow up with the zone teams to send the raw data for which we don’t have the access. Connect with the zone teams to understand and validate the data and mapping files. Eliminate all the manual process existed while refreshing the pipelines. Extract the raw data from On-Prem DB(using self hosted IR in ADF) and load it in the ADLS, Transform the data by reading it from the ADLS in Databricks and Load the transformed data into the Azure SQL DB. Later, Power BI is connected to this SQL DB for visualization. Create Linked Services, Datasets and Integration Runtime(IR) in ADF while creating pipelines in ADF. Send mail notifications using Web Activity in ADF which will trigger the Logic app to send an email after pipeline ran successfully or failed.

Designed a GUI using python tkinter module. The design was based on real time project of one client. Integrated python GUI with Database Server, created QRcode, used webbrowser module, downloaded the image stored in the database and many more things.

Solved many problem statements using our own approach by integrating various tools together like Git, GitHub, Jenkins, Docker, RHEL8, Machine Learning, Deep Learning and some Kubernetes. Completed every task within the given period of time(Deadline) and researched many things when I faced some difficulties. Completed Face Mask Detection project using MLOps. Configured Jenkins to send mail notifications after Job completed successfully. Created a dedicated Docker environment for training the Machine learning model inside it using base image as CentOS and all the machine learning libraries installed in it.