Pintu Singh

Optimized overall process performance through Spark performance tuning, improving job run times by 20% and efficiently managing a 12TB dataset containing approximately 10 billion records. Implemented Spark optimization techniques such as caching, multithreading, and broadcast joins, resulting in a 20% decrease in processing time for handling a daily load of around 2 million records. Successfully migrated legacy on-premise processes to the cloud using Spark, resulting in a 20% reduction in processing time. Developed multiple automated data pipelines to fetch data from various Kafka topics. Optimized queries and data processing, resulting in 30% faster data retrieval and analysis, supporting timely and accurate reporting.

Created ETL workflows to extract data from 20+ sources, transforming it into a standardized format and loading it into a data warehouse, improving data integration and accessibility by 25%. Implemented data quality checks in Apache Airflow DAGs, ensuring 99.6% accuracy in data transformations and load operations. Automated incident logging for ETL pipeline failures using machine learning, reducing manual intervention by 60% and improving response times. Analyzed data to solve a wide variety of business problems, creating data visualizations that drove strategic direction and improved decision-making processes. Collaborated with cross-functional teams to identify areas for data-driven improvement, implementing solutions that increased operational efficiency. Conducted exhaustive root cause analysis for data discrepancies, presenting actionable insights to key stakeholders, which enhanced data-driven decision-making. Demonstrated reliability and expertise through multiple on-call rotations, effectively resolving critical production issues to ensure uninterrupted system functionality.

Designed and developed an optimized Data Model for Drugs and Vaccine Distribution Management System (DVDMS) for Assam State.