Ashish Meshram

The motive of this project to build framework to import data from AWS S3 to Redshift data warehouse for analyzing marketing data for particular product and implement marketing strategies. The data pipeline architecture consists of AWS S3, AWS Glue, Redshift data warehouse. We had used Spark as execution engine to process data. Roles and Responsibilities: Perform ETL Transformation for Data Cleaning. Monitoring Glue Jobs and Troubleshooting if any Error. Implementing Optimization Technique for Better Performance. Implement Business Requirements using DSL Approach. Raise JIRA for Infrastructure, Platform issues. Engage with BA to understand the requirements clearly. Define all the possible Test Cases along with the Test Data. Comes up with missing scenarios and get them clarified with BA. Implement Data Validation, Quality Checks, Profiling. Perform UAT of Big Data implementation. Resolve JIRA tickets raised by Client.

The motive of this project to analyzed the fuel card transactions and provide better control over payment programs for business. For that we have built framework to import full load transactional data from AWS S3. The data pipeline architecture consists of AWS S3, AWS Glue, PySpark, Redshift. Roles and Responsibilities: Monitoring and Understand the Nature and Characteristics of Data and Check how data Flow. Engage with BA to Understand the Requirements Clearly. Study Incoming Schema and Based on Schema Creation Redshift Table. Involve in Creating queries for Data Analysis to Meet the Business Requirements. To Perform Transformations using Spark-SQL as Per Client Requirement. Creating Views for analysis purpose for client. Make the config entries for files in file as client per client. Perform Optimization Techniques in order to enhance performance of Spark Job. Monitoring Job Daily, Fixing Production Bugs, and Raising Ticket for Production activity. Managed to help Admin Team and Data Analysis Team with knowledge sharing. Monitoring Pre-Validation Process to Validate the Client Data.