AARZOO AGRAWAL

Project 2 - BI Migration: Migration from Birst BI system to AWS Quicksight using AWS S3, Redshift, Step function and Quicksight to build reports. Created Snaplogic pipelines to ingest data from multiple data sources like Redshift, MySQL, SFTP server, Email, SQLServer, Rest API in S3. Built a script using python which uses copy command to load data from S3 to redshift tables. Configured glue jobs to use fractional DPU resource using pythonshell instead of spark shell which greatly reduced the cost. Implemented RLS on Redshift tables which contains PII data to control access to specific users and roles. Created data views by joining tables in presentation layer which is used in Quicksight for dashboarding purpose. Project 1 - Payroll Integration: Designed and Constructed real-time data pipeline (writing glue jobs) to process semi-structured (json) data from workday, build a generic utility to export data through api for business use reducing cost by 60%. Ingesting data from sources like sharepoint, oracle, aws redshift into athena after performing schema validation and further writing scripts using Spark in Python to perform transformations in consumption layer. Writing SQL queries to fetch required data from Athena tables and build generic utility to calculate revenue, sum from data based on logics involving working days, absences which reduced entire manual workload saving 95% of time. Automated ETL processes across billions of rows of data, which reduced manual workload.

Learned about Azure platform, its services, types of storage accounts, load balancer, functions, data recovery, Azure AD and Basic Linux commands with hands on exercises. Created Azure Virtual Machines, applications, database, and functions on azure portal. Designed virtual networks to support workloads with the highest security and performance.