Pankaj Singh

Architected and owned large-scale batch and near real-time data pipelines processing 100M+ records weekly, reducing insight delivery time by 40% for operational and financial analytics. Designed and implemented a robust ETL framework for financial and operational datasets, consistently maintaining 95%+ data accuracy and near real-time data availability. Automated end-to-end data ingestion and validation workflows, reducing manual effort by 70% and improving pipeline throughput by 40%. Optimized PySpark and Databricks jobs through partitioning, caching, and query tuning, improving load efficiency by up to 90%. Implemented hybrid batch and near real-time processing patterns, enabling 30% faster operational decision-making across business teams.

Led migration of 50+ legacy DataStage jobs to Azure-based data pipelines, improving overall processing efficiency by 40% and reducing execution time by 30%. Independently delivered 9 of 15 business-critical KPIs, supporting improved reporting accuracy and data-driven decision-making across stakeholders. Optimized complex SQL queries on large datasets, reducing query execution time by 40% and improving report responsiveness. Re-engineered legacy DataStage workflows into PySpark, reducing processing time by 30% and cutting post-migration defects by 90%.