Shubham Mehar

Architected a scalable ETL pipeline for Amazon Paid Search data processing millions of records daily, reducing data latency by 40% and enabling real-time business analytics. Developed PySpark and Python transformation workflows to clean, enrich, and standardize large datasets, accelerating downstream consumption by 30%. Automated Silver-to-Gold layer schema conversion via Python scripts in Unity Catalog — handling table/view recreation and permission preservation — cutting manual governance effort by 50%. Refactored legacy SQL workflows to PySpark DataFrames in Databricks, achieving 35% faster query times and 20% lower cloud compute costs. Orchestrated pipeline automation using Apache Airflow, eliminating manual interventions and improving SLA adherence by 25%. Enforced data governance and compliance standards using Azure Databricks, Delta Lake, and Unity Catalog.

Designed a scalable Azure Data Lake architecture consolidating 1TB+ daily data from customer usage, recharge, and network logs across multiple on-premise sources. Built PySpark ETL pipelines ingesting and partitioning 1TB+ daily CDR data, enabling near-real-time analytics for 5M+ telecom subscribers. Engineered an automated data profiling and quality framework in Python/PySpark, improving data accuracy by 15% and reducing processing time by 25%. Boosted pipeline efficiency by 20% via Spark performance tuning — bucketing, salting, partitioning, and caching strategies.