Sandeep Lakhera

Designed and Implemented data lake on AWS EMR using S3, Glue and DMS. Optimized DynamoDB queries by implementing appropriate GSI and choosing the best partition and sort key which improved speed by 67%. Managed data pipeline up-time of 99.8% while ingesting streaming and transactional data across different primary data sources using Spark, S3, and Python. Designed and implemented a real-time data pipeline to process semi-structured data by integrating 150 million raw records from 10+ data sources using PySpark. Developed end-to-end data pipelines and workflows in Infoworks, streamlining data integration processes, and automated them through Autosys which reduced manual work by 80%. Designed and implemented a Load script for compressing data using LZO compression, exporting it from AWS S3 to DynamoDB, and performing a decompression process for data retrieval.

Owned the Infoworks ETL operations for imports from diverse databases (Teradata, MDM, SQL Server, DB2,On prem, HDFS) and performing ETL operations, tokenization, and storing data in AWS S3 using Spark and Hive. Led and executed the migration from on-premises data processing infrastructure to Amazon EMR (Elastic MapReduce).