Sarath Maya

Designed and implemented a data pipeline using the Medallion architecture, organizing data into Bronze, Silver, and Gold layers for efficient processing and analytics. Ingested raw data from Azure Data Lake into the Bronze layer using Databricks, ensuring data traceability and availability for further processing. Cleaned and structured data in the Silver layer using PySpark, implementing Slowly Changing Dimensions (SCD) to maintain accurate historical records, and applied salting techniques to address data skew. Applied business logic to transform data into the Gold layer, optimizing Delta tables for high-performance analytics. Enhanced processing efficiency through strategic repartitioning, coalescing, and caching, improving the overall performance of Spark jobs. Integrated the pipeline with Azure DevOps and Git for continuous integration and deployment.

Designed and implemented data ingestion pipelines in ADF to transfer data from on-premises SFTP to Azure Data Lake Gen2. Created date hierarchy-based partitioning in ADLS Gen2 for efficient storage and retrieval. Developed SQL-based transformation logic using Azure Synapse to process and refine data. Organized and populated data in MS SQL tables using Stored Procedures to support multiple source systems. Built Dashboard on Power BI and Orchestrated the whole ETL process using Azure Data Factory.

Responsible for developing the Metadata UI web page application using Angular 6 or above, HTML5, CSS3, TypeScript, bootstrap etc. Created the Roles and Permissions for each screen using Swagger UI. Used Microsoft Visual Studio TFS as a source control and Octopus for deploying Application and Database. Designed applications to extract, transform and load data on the data warehousing systems. Extracted data from internal data warehouse system to SSRS.