Adarsh Singh

Designed and developed scalable ETL pipelines using Databricks, PySpark, and Python to process over 500M+ records weekly. Integrated Apache Kafka for real-time streaming, reducing data latency by 30%. Designed and developed robust ETL pipelines from scratch, creating core components including data quality checks, workflow control, audit tracking, batch and job detail logging, and data reconciliation mechanisms. Spearheaded optimization of insurance coverage data analysis by identifying common attributes across products to deliver consistent reporting by building a centralized Snowflake multi-product coverage data mart, enabling faster decision-making across 5+ business units. Deployed and managed data pipelines on Azure Data Factory and Azure Databricks, improving data refresh frequency from daily to hourly and reducing manual intervention by 90%. Led enhancements and issue resolution, working directly with business stakeholders to drive improvements aligned with business goals.

Developed a Machine Learning model to accurately predict outstanding payment amounts from companies, improving financial forecasting capabilities. Designed and implemented a complex User Interface using ReactJS, Material UI, and Highcharts, enabling intuitive data visualization and user interaction. Managed and optimized a MySQL database to store outstanding payment data and user information, ensuring efficient data retrieval and integrity.