Palwinder Singh

Implemented metadata driven orchestration framework using Azure data factory, Azure SQL server, Azure function app and Azure Key vault to Batch ingest data from 35 different applications. Created Metadata driven Databricks template to ingest data and create trusted data tables using lakehouse architecture for 4000 tables. Implement external hive metastore to enable working with multiple Databricks workspaces. Implemented custom logging for Datalake and Datawarehouse using log analytics. Created metadata driven execution engine to generate Datalake Data Marts. Implemented Metadata driven view generator on Datalake tables. Implemented governance and security in Datalake and Datawarehouse.

Worked on Rule Engine to process 42 million records in 1.5 hours. Designed and implemented monitoring dashboard to track progress of multiple asynchronous dependent jobs. Designed and implemented data factory pipelines to implement eventual consistency between different database in microservices environment.

Created SQL Scripts to generate test data and speeding it up by 30%. Created and managed test plans and bug tracking process to manage Quality control phase. Improved software and data quality by testing relevant functional points and reporting to appropriate team. Worked on SQL store procedure performance improvement speeding up performance by 15%. Worked in Agile team to Develop and deliver modules to collect grades and generate results within 10 weeks. System collected 3.4 million grades in 2 days without any performance hiccups and helped client to move business in pandemic.