Shivam Mistry

AWS ETL Pipeline Implementation for Holcim Sales Data Integration and Analysis: Ingested sales data from multiple sources (ERP, CRM, spreadsheets) into an S3 bucket. Transformed data using AWS Glue for analytics and reporting (cleaning, enrichment, aggregation). Loaded transformed data into AWS Redshift data warehouse for efficient querying and reporting. Validated data to ensure quality standards and business rules were met. Automated pipeline using AWS Lambda, and monitored using AWS CloudWatch. Also conducted S3 Data Analysis using PySpark and Databricks, analyzing data in S3, creating visualizations and dashboards using Databricks notebooks, and collaborating with business stakeholders to understand their data analysis needs.

Executed Customer Data Analysis using Sqoop, Hive, Spark and S3 bucket, involving data extraction from MySQL via Sqoop to HDFS, transformation using Hive, and analysis using Spark SQL. Also worked on setting up ETL process leveraging Hadoop data pipeline for TFS, utilizing Sqoop to transfer data to HDFS raw, defining schema, creating HIVE tables, and performing data processing such as aggregation, joins, and filtering.