Sidhant Patra

Architected and maintained complex Apache Airflow DAGs to orchestrate large-scale ELT workflows, implementing task parallelization and dynamic configuration to improve data throughput by 25% for critical pipelines. Led migration of legacy Talend ETL pipelines to GCP, redesigning ingestion and transformation logic using Cloud Composer, BigQuery, and Dataflow to build a scalable cloud-native data platform. Identified reusable patterns across pipelines and built Python automation scripts to programmatically generate and migrate 300+ DAGs, significantly reducing manual engineering effort and accelerating migration timelines. Developed and optimized Google Cloud Functions for event-driven data processing and API integrations, tuning performance to achieve sub-500ms latency for high-priority data feeds. Served as technical point of contact for the department, gathering requirements from product stakeholders and translating them into robust, production-grade data solutions with clear SLAs. Modified existing Apache Beam code in JAR files to incorporate new features, enhancing pipeline flexibility and enabling seamless integration of real-time analytics for 5+ production workflows.

Engineered scalable data pipelines using Python and Apache Spark on AWS S3 to process large-scale retail transaction data, designing dimensional models (fact and dimension tables) to power high-performance point-of-sale analytics. Designed and implemented a sales performance incentive engine processing 100GB of daily data, ensuring 100% accuracy of commission calculations through robust validation and reconciliation logic, replacing manual spreadsheet workflows. Optimized Spark jobs, partitioning strategies, and file layouts on S3 to reduce end-to-end processing time and compute costs, improving report availability and reducing delays for stakeholders.

Validated and analysed financial datasets in BigQuery and HiveSQL to meet business and regulatory requirements, implementing data quality checks that increased trust in executive-level reporting. Built interactive Power BI dashboards to visualise payment metrics, customer behaviour trends, and portfolio performance, enabling leadership to move from weekly to near-daily data-driven decisions. Collaborated in Agile sprints with data engineers, analysts, and product owners to refine requirements, prioritise backlog items, and deliver a scalable analytics environment ahead of schedule.