Ipsita Sarkar

Architected production-grade ETL/ELT pipelines using PySpark and Spark SQL, processing 100M+ records daily with 30-40% performance improvement through partition optimization, broadcast joins, and adaptive query execution; Designed multi-layer lakehouse architecture (Landing → Bronze → Silver → Gold) ensuring data governance, lineage, and quality across enterprise platform serving 500+ business users; Developed reusable PySpark transformation modules with complex joins, window functions, and aggregations, reducing code duplication by 40% and accelerating feature delivery by 2-3 weeks; Orchestrated end-to-end workflows using Azure Data Factory and GCP Cloud Composer, enabling reliable cross-platform data movement between ADLS, BigQuery, and Cloud Storage with 99.8% pipeline success rate; Built optimized SQL models with materialized views, partitioned tables, and clustered indexes, reducing dash-board query latency from 45s to 18s (60% improvement) and supporting real-time analytics; Implemented comprehensive automated data quality framework including schema validation, duplicate detec-tion, null checks, and threshold-based alerts, reducing production defects by 30% and improving data