Sulbha Goyal

Developed various batch and real-time pipelines to aggregate data from multiple sources. Built data migration utilities (replacing sqoop) and created Airflow DAGs for scheduling Spark ETL jobs. Built real-time pipelines aggregating 15+ streams using Spark Streaming and Kafka, integrating Aerospike for state management. Developed ETL jobs to compute business metrics (atv, gmv, gr, sp) and built dashboards using Glue and AWS Athena.

Developed data pipelines to aggregate and transform incoming sensor data using big data technologies. Owned maintenance of multiple pipelines and provided technical solutions for the Big Data layer. Developed Rest APIs for application integration.

Implemented frameworks for importing existing Hive tables and applying custom UDFS. Developed frameworks for complex data operations (joins, aggregations) using Spark or Hive, and for saving query results to HDFS. Implemented framework to import data from HDFS into user databases.

Developed data ingestion framework to import data from various JDBC/Non-JDBC sources (MySQL, Oracle, SFTP, FTP) into HDFS and Hive. Implemented data transformation in Scala (e.g., CSV to Parquet, JSON to Parquet) using Apache Spark. Worked on HIVE SQL transformations and handled data type compatibility issues.

Trained in object-oriented programming (Java), DBMS, and software engineering, covering various phases of SDLC including requirement elicitation and deployment.