Khushank Madaan

Enhanced business insights by implementing a data pipeline in Azure Synapse Analytics that automates the ETL of healthcare and clinical data from multiple sources such as SQL Server, Sharepoint, Azure SQL DB, Emails, ADLS, etc. Strengthened business intelligence by engineering Delta/Incremental Loading of data using a metadata-driven configuration, optimizing data processing efficiency. Enabled Data Warehousing to support Power BI reporting and Web Portal visualization for stakeholders. Optimized data pipeline performance, reducing runtime by 55% and costs by 40%, through efficient SQL queries utilizing CTEs, indexing, and other best practices. Leveraged Data Flows backed by Spark clusters for parallel processing, further enhancing efficiency and scalability. Improved operational efficiency by enhancing pipeline monitoring, debugging, and performance tracking through the integration of logging into Azure Data Lake. Optimized storage costs and ensured efficient log retention and auditability by implementing lifecycle management policies. Proactively managed operational costs and pipeline stability by implementing cost and failure alerting mechanisms with email notifications, enabling timely issue detection and resolution to prevent delays and ensure seamless performance. Streamlined data processing by formulating an AWS Lambda solution in Python to download and extract ZIP files from an SFTP server to an S3 bucket, process the data, and insert it into an SQL Server database, ensuring security and compliance through AWS Secrets Manager for credential management. Enhanced accuracy by 34% and improved reliability by developing an Azure Function in Python to analyze PDFs using the OpenAI API, update the SQL Server DB, and ensure security via Azure Key Vault, with robust error handling and logging.

Engineered a high-performance data pipeline in Azure Synapse, optimizing workflows to achieve a 60% reduction in runtime, significantly boosting operational efficiency and responsiveness, based on a reference ADF pipeline. Devised a robust and configurable Python parser to process X12 EDI institutional claims, ensuring accurate data extraction, enhancing processing efficiency, and adapting effectively to evolving requirements.