DEEPAK KUMAR

Developed an Easy, configurable Generic XML parser Notebook for Upstream Service in Azure DataBricks(PySpark), reducing manual effort to infer schema and flatten StructType data. Optimized the existing ETL framework for parallel & deadlock-free data load in snowflake, reducing load time by 70%. Developed CI Pipeline for Snowflake (DWH) in Azure-DevOps, including SQL Syntax validation, Meta Data validation, and Deploying to remote Control-m Server. Developed & Automated the Actuarial Report Generation by analyzing and translating business requirements into SQL for 8 different Data Sources. Developed a Python library to generate custom reconciliation control values by parsing 2300(CSV, EXCEL) files through pandas, decreasing overall parsing time to around 7 minutes through Python multiprocessing module.

Contributed to Python-based cloud Agnostic ETL(Batch) framework for snowflake(target) which loaded data from 12 source systems (S3, SQL-Server, Profisee) into modeled Data-Vault tables. Optimized the framework by limiting parallel connections, reducing the overall batch Load time by 50%, and developed features like checkpoints/restart-ability and automated notification process for data anomaly. Independently analyzed & modeled business requirements into a Data Warehouse using DataVault-2.0, creating Data-Marts for reporting. Involved in migrating the onPrem ETL framework into AWS (server-less stack: AWS s3, Batch, CloudWatch, Fargate), resulting in cost savings for the customer.