Prithiviraj Manimaran

Devised ETL pipelines using Azure Databricks and Data Factory to ingest batch data by connecting to API from various platforms like SAP, Infor M3 to do incremental load automatically and perform transformations with highest efficiency. Contributed to data migration efforts from on premise to cloud infrastructure, with specific focus on building views for downstream reporting. Designed and implemented delta lake tables on top of ADLS Gen2 as data lake reducing data latency by 60% for business users while querying the table and contributed to Spark optimization to achieve best performance. Identified bottlenecks and optimized data processing pipelines which improved performance by 25%. Worked on CI/CD pipelines and deployed solutions in the production environment with Azure DevOps. Developed a tool using Python which creates meta scripts for PostgreSQL with single click for the tables to be ingested, saving 12 hours of manual work per week, and eliminating errors. Developed validation framework using PySpark which is an automated validation in higher environment. This resulted in saving 4 hours of manual validation and errors. Worked with semi structured data like JSON on Azure Cosmos DB and have handled tables with more than 10TB + data as input sources. Implemented monitoring system to ensure smooth data operations. Leveraged Snowflake and Azure synapse analytics for data warehousing.