Priyank Chauhan

Design and Develop ETL pipelines to fetch the data from other 3rd party services that helps to grow the D2C business using Pull APIs, Apache Airflow with Python over GCP Composer. Build and Deploy streaming ETL pipelines for payments data from different platform using GCP technologies: Pub/Sub, Dataflow and Big Query. Construct and Implement scraping ETL pipelines for get competitors data from different Marketplaces using GCP technologies: Pub/Sub, Cloud Functions. Build and Deploy streaming ETL pipelines for the real time data analysis for customer acquisition platforms using Push APIs, Kafka, Spark and Spark Streaming Jobs running over Data proc clusters. Shape and Deliver validation, altering framework and working on code reviews, Git CI/CD integrations with Jenkins. With the business requirements tracking all the competitive brands via web-scrapping and app-scraping leading to the development of inhouse tool with analytics team to analyze the D2C spend structure. With Cloud Infra under supervision all the resources allocations, Managing the IAM rules, setting of firewall rules, Creation of VPC and HA-VPN tunneling. Cloud Cost Optimization is another role under supervision including the optimizations regarding BigQuery which includes Bigquery OnDemand slot inclusion module, Bigquery editions cost structure. Later Cost Optimization for the running VMs in the Infra with the precise allocation of services. Developed a Kubernetes K8 cluster for the hosting of nodes and pods to run the Ongoing ETL Pipelines with Jenkins CI/CD deployment structure leading to the data orchestration tool for the business. Worked upon deployment of pipelines with Apache Airflow on GCP Virtual Machine.