PRAGAT TIWARI

Developed the pyspark scripts for data loader as per defined configuration in delta tables. Developed the python scripts for automatic spark sql generation as per metadata and execute in multiple threads as per configurtion defined. Wrote unit test cases for the data shield. Velocity POC: Extracting data model from the informatica xml to generate data model for source files. Generate ddl scripts based on data model configuration for spark external table and views. Prepare Metadata for Data Loader based on configuration such as file path , partitions and bucketing. Created ADF pipeline parameter based. Written script to load metadata dynamically from central metadata sheet. Developed utility to merge multiple output files into one.

Worked as Python and Pyspark developer writing SAS written procedures into Databricks using Python,Pyspark. Analyse the requirement or the functionality of the current SAS code and designe and implement the best possible solution in Databricks. Performed unit testing for all the procedures. Also have experience on Databases like SQL, MySQL.