Prathamesh Patange

• Designed and developed scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks, ingesting and processing 500GB+ of banking transaction and operational data daily from multiple upstream systems into Azure Data Lake Storage Gen2. • Built scalable ETL/ELT pipelines using PySpark and Spark SQL, processing 50M+ records per batch to support large-scale banking data transformations via batch and incremental loads. • Designed and implemented Medallion Architecture on Delta Lake to organize Bronze, Silver, and Gold data layers for scalable analytics and reporting. • Developed incremental data ingestion pipelines using Change Data Capture (CDC) and watermarking techniques, reducing full data loads and improving pipeline performance. • Developed a metadata-driven ingestion framework in PySpark to automate and standardize ingestion of source datasets into the data lake. • Optimized Spark-based data processing and analytical queries through partitioning strategies, caching, and Delta file compaction to improve performance of large-scale analytical workloads. • Improved analytical query performance by 20% through Databricks query profiling, ZORDER indexing, and Delta Lake optimizations on large Delta tables. • Reduced Delta Lake storage costs using OPTIMIZE and VACUUM scheduling, partition tuning, and removal of redundant intermediate tables. • Delivered curated datasets for Power BI dashboards and reporting, enabling business stakeholders across operations and analytics teams to access reliable data insights. • Implemented data governance and access control using Unity Catalog and Azure Key Vault, with service principal authentication across ADF linked services and Databricks notebooks, supporting compliance requirements. • Collaborated with data analysts, product owners, and business stakeholders to deliver scalable data solutions within an Agile/Scrum development environment.