Vaishnavi Parkhi

Designed and optimized large-scale Spark data pipelines, improving workflow execution time by 30% using broadcast joins, caching, and repartition strategies. Implemented Medallion (Bronze–Silver–Gold) Lakehouse architecture in Delta Lake on ADLS, improving scalability and analytical efficiency. Built and maintained batch + streaming pipelines using ADF and Databricks, reducing data turnaround time by 25%. Processed 20M+ records per batch, ensuring high reliability, data quality, and governance compliance. Led Azure DevOps CI/CD deployment pipelines for versioning, testing, and automated releases.

Developed PySpark workflows and ETL integrations in Azure Synapse Notebooks and Databricks. Converted complex business logic into Spark SQL, RDD, and DataFrame transformations, improving pipeline scalability. Built ingestion pipelines handling CSV, JSON, Parquet, Delta, ensuring efficient schema inference and validation. Monitored and resolved ETL failures using AutoSys and Airflow, improving uptime and reducing job failures. Developed SQL-based transformations and supported SSIS, SSRS workloads. Analyzed 400+ execution logs via Databricks and Unix to troubleshoot bugs and ensure smooth sprint cycles.