Shivani Upadhyay
@Shivani
Data Engineer at Spark Brains Pvt Ltd
Chandigarh
Results-driven Data Engineer with 4 years of experience architecting scalable solutions on the Databricks Lakehouse platform. Expert in designing Medallion Architectures (Bronze/Silver/Gold) utilizing PySpark, Delta Lake, and Unity Catalog to ensure high data quality and governance. Proven track record of optimizing end-to-end ETL/ELT pipelines for enterprise clients like P&G, leveraging Azure Data Factory and Databricks Lakeflow to reduce analytical processing time by 70%. Focused on transforming complex raw data into actionable insights through high-performance SQL and automated data orchestration.
Experience
Data Engineer
Spark Brains Pvt Ltd
Travel Analytics Platform: Engineered a declarative data pipeline using Databricks Lakeflow to ingest high-volume S3 data into a Unity Catalog-managed Medallion Architecture. Implemented automated schema enforcement and data quality checks at the Silver layer. Dubai Municipality: Spearheaded the migration of complex legacy Oracle SQL stored procedures to Impala (Hue), optimizing query logic for large-scale municipal datasets. Developed end-to-end PySpark ETL workflows orchestrated via Oozie.
Data Engineer
Tranzita Systems (Client: Procter & Gamble)
HS&E Command Centre: Architected Databricks ETL pipelines to ingest and harmonize semi-structured data from SharePoint and enterprise catalogs. Truck Load Optimization: Engineered automated data pipelines to synchronize mission-critical logistics data between SQL Server and Delta Lake. On-Shelf Availability (OSA): Engineered a daily automated reporting engine to calculate inventory stock levels. Services as Measured by Customers (SAMBC): Developed a Python/Pandas-based automation tool for Root Cause Analysis (RCA) of order delays.
Education
Buddha Institute of Technology
Bachelor of Technology
Computer Science