Sayan Chakraborty

Architected an end-to-end real-time data platform on Azure Databricks using Medallion Architecture (Bronze/Silver/Gold) to support scalable analytics and AI/ML workloads. Built scalable batch and streaming pipelines using PySpark Structured Streaming and Delta Lake, enabling real-time visibility of logistics operations. Optimized truck load planning using data-driven models, improving vehicle fill rates by 18% and delivering over $1M in annual cost savings. Developed a material recommendation engine using PySpark, improving operational efficiency and inventory planning accuracy. Migrated legacy ingestion pipelines to Delta Lake, enabling ACID transactions, schema evolution, and significantly improving data reliability. Implemented event-driven ingestion pipelines for shipment data from TMS and FourKites, powering near real-time logistics analytics. Established CI/CD pipelines for data workflows using Git and automated deployments, improving release reliability and reducing manual intervention. Built a GenAI-powered NL-to-SQL platform using LangChain, enabling business users to query operational data using natural language.

Built Python-based data extraction pipelines and dashboards for business insights.

Automated batch workflows and improved job scheduling reliability for enterprise systems. Supported cloud migration initiatives improving system performance and stability. Collaborated with global teams to troubleshoot production issues and ensure SLA adherence.

Developed big data workflows using Hadoop, Spark, and Hive for large-scale analytics. Improved data quality by 30% through validation, cleansing, and deduplication processes.