Simran Arora

Developed and maintained data pipelines for historical and incremental data load in batches using Python programming language. Designed and executed a versatile YAML framework for data warehouse that enables integration with multiple jobs for the purposes of parallel execution and reusability. Created Python ETL jobs to extract complex JSON structures from MongoDB collections and transform them into a tabular format for reporting purposes. Deployed a Python job to purge data from various databases including AWS S3, MongoDB, Oracle, and Postgres. Collaborated on refreshing master data and stabilizing digital platforms using PostgresSQL functions and SQL queries. Optimized AWS Glue code with PySpark to efficiently load large volumes of data from a Redshift database to Postgres, resulting in significant runtime and cost reductions. Transformed Talend ETL jobs into Python, optimizing the execution time of jobs. Automated all Python jobs developed on EC2 instance using crontab, thereby reducing manual effort and saving time. Provided assistance and implemented improvements to the Python code of an API developed with Flask. Actively involved in developing a data lake project and creating a pipeline using PySpark.