Anshul Malhotra

Implemented Data Pipelines to process Customer Feedbacks from over 8+ ingestion sources. Implemented Spark Batch Applications to process parquets, AVRO and JSON from S3 and persisted results to Hive after enrichment. Implemented and Scheduled Data Pipelines with Airflow. Implemented metrics capturing mechanism to codebases for pushing metrics to Datadog. Configured Monitors in Datadog with metrics captured to Alert team in case of any issues in data pipelines. Worked on North star vision of project to design system that can receive feedback in real-time. Documented Component level LLD for creating Scalable and Fault Tolerant systems. Implemented APIs using Spring Boot to receive events in real time and publish those events to Kafka and deployed application over Kubernetes.

Developed complex Big Data projects by gathering, parsing, managing, analysing, interpreting, and visualizing large datasets to extract valuable insights and convert those insights into actionable business decisions. Used Spark SQL over Hortonworks Hadoop YARN to perform analytics on data in H-Base. Implemented efficient Big Data solutions by integrating multiple Big Data tools. Transformed spark RDD jobs to spark SQL jobs. Transformed Cron schedules to NiFi based scheduling. Implemented REST API’s using Spring Boot. Designed and implemented scalable applications for automating the bring up process of virtual machines in private data centres. Directed software design and development across multifaceted team to meet product’s needs for functionality, timeline and performance. Reviewed requirements, specifications and technical design documents to provide timely and meaningful feedback for Sprint discussions. Adjusted design parameters to boost performance and incorporate new features.