Anukool Tiwari

Modernized Architecture (ADF + Databricks) | Client: Humana. Developed and orchestrated ADF pipelines to ingest data into Azure Data Lake Storage (ADLS Gen2) following bronzesilvergold layering. Architected and engineered end-to-end curation pipelines using PySpark, Delta Lake, and Azure Databricks, achieving a 40% data quality gain. Utilized Azure Databricks notebooks to perform complex data transformations, validations, and aggregations. Processed and managed 10M+ records daily using incremental Change Data Capture (CDC) techniques. Led performance optimization initiatives in Databricks by tuning Spark configurations, joins, and transformations, achieving a 50% improvement in data processing speed. Reduced overall pipeline execution time by 30% and enhanced data accuracy and consistency by 40%. Leveraged ADLS Gen2 for staging and intermediate storage, resulting in a 25% reduction in storage costs. Optimized PySpark jobs in Azure Databricks using partitioning, caching, and efficient Spark execution strategies.

StreamSets-based Ingestion | Client: Humana. Designed and implemented ETL pipelines using StreamSets to ingest data from Genesys REST APIs, ensuring reliable batch ingestion. Stored raw ingested data in Google Cloud Storage (GCS) and maintained MongoDB audit collections for ingestion tracking and reconciliation.