Kshitija Ingle

Designed and implemented a large-scale data pipeline using AWS Databricks and PySpark to efficiently process and analyze massive datasets. Streamlined data ingestion from multiple sources into Amazon S3, enhancing retrieval speeds by 30% through partitioning and compression. Built distributed ETL workflows in PySpark, reducing processing time by 40% and improving data quality by 25% with Python-based validations. Automated data transitions with AWS Step Functions, minimizing downtime by 15% and ensuring seamless workflows. Monitored and optimized system performance using CloudWatch, proactively resolving bottlenecks. Utilized EC2 and Databricks clusters efficiently, achieving significant cost reductions. Stored processed data in MySQL, enabling effective reporting and data visualization with optimized SQL queries.