Venu Gopal

Build a configuration driven extension to airflow where developers can define their workflows in YAML format and consistently produce automatic DAG code which airflow can interpret. Conduct data validation on source and staging tables using DAG factory with a YAML configurational approach. Execute table transformations using Airflow and migrate data into other SQL environments. Integrated and processed pharmaceutical data from multiple sources including AWS, Azure, and on-premise database handling various file formats like JSON, CSV, and Parquet. Implemented Spark optimization techniques such as caching, multithreading, and broadcast joins, resulting in a 20% decrease in processing time for handling a daily load.

Created an API service using Python to generate dynamic DAGs in Apache Airflow. Designed and implemented advanced scheduling capabilities using Airflow for data pipeline orchestration, reducing manual intervention time by 80% and streamlining workflow efficiency. Developed a solution leveraging Amazon MSK (Managed Streaming for Apache Kafka) to push messages for real-time tracking of processing status (success or failure).

Worked within the marketplace team with the primary objective of enhancing seller experience. Integrated and transformed data from various digital platforms, prepared data, built ETL and ELT processes, and orchestrated and monitored pipelines utilizing AWS Glue. Optimized overall process performance through Spark performance tuning, improving job run times by 20% and efficiently managing a Terabyte(TB) dataset. Worked on data ingestion pipeline to ingest the flat file in the Data lake.

Worked extensively with large marketing datasets concerning user behavior, identifying trends, and conducting in-depth analysis to draw meaningful conclusions. Utilized standard metrics such as Click-Through Rate, Conversion Rate, ROI, Cost per Click, etc., to gauge the success of campaigns.