Shoubhit Kumar

Built and operated PySpark-based ETL pipelines on Databricks handling incremental ingestion, late-arriving data, and historical backfills for enterprise reporting workloads. Implemented Delta Lake MERGE patterns and watermarking logic, reducing full reload dependency and improving pipeline efficiency by 40%. Designed Bronze, Silver, and Gold datasets with enforced schemas and data contracts, enabling consistent consumption across BI and analytics teams. Enforced data validation standards across pipelines covering schema conformity, null handling, and record uniqueness, improving downstream data reliability by up to 98%. Optimized large Delta tables using partitioning and Z-ordering, significantly improving SQL query performance and Power BI refresh latency. Automated ServiceNow to analytics ingestion workflows using Python and SQL, reducing manual SLA validation effort by 80%. Contributed to AI and GenAI enablement by preparing clean, point-in-time datasets for incident clustering, forecasting, and conversational assistants, supporting feature readiness and validation for ML workflows.