Deep jethwa

Built scalable ETL/ELT pipelines using Databricks and PySpark, enabling efficient ingestion and processing of large-scale datasets. Implemented Lakehouse architecture with Medallion Architecture (Bronze–Silver–Gold) using Delta Lake to deliver reliable, analytics-ready data layers. Developed batch and incremental ingestion pipelines on Azure Data Lake Storage (ADLS Gen2) using Auto Loader to process structured and semi-structured data (CSV, JSON, XML). Implemented data quality checks, validation rules, and data cleansing processes to ensure reliable and consistent datasets. Applied Delta Lake features such as MERGE, schema evolution, and time travel to support reliable data versioning and governance. Optimized Apache Spark workloads using partitioning, caching, and query tuning to improve pipeline performance and execution efficiency.

Built a production-grade ETL data pipeline in Linux-based environments using Python, MongoDB, Airflow, Docker to ingest and process large yard-management datasets. Performed data validation, cleansing, and normalization to ensure high-quality, consistent datasets for downstream analytics. Automated multi-interval data processing (hourly to yearly aggregations), improving workflow scalability and operational visibility. Developed automated data workflows that reduced manual reporting effort by ~80% and improved data delivery efficiency. Designed analytics-ready data models supporting business reporting, forecasting, and downstream analytics.