Shagun Bansal

Engineered scalable and fault-tolerant ETL pipelines using PySpark and Python. Streamlined API-based data ingestion pipelines, improving ingestion stability for high-volume data loads. Built a Snowflake Streamlit based chatbot using Analyst features for self service data exploration. Automated pipeline monitoring with email alerts and logs for failure scenarios. Orchestrated end-to-end Snowflake and Spark data pipelines using Apache Airflow. Enforced data security and governance by developing a Python based Row Level Security framework and implementing data masking controls. Addressed 25+ client-reported issues promptly and managed daily and weekly workflow troubleshooting. Worked closely with business stakeholders to translate business requirements into scalable data solutions.

Developed and optimized SQL driven ETL workflows, reducing data refresh cycles by 20%. Created documentation including BRDs, STTMs, and KPI definitions to support ETL processes, architecture understanding, and onboarding.

Built Snowflake-native ELT pipelines ingesting data from AWS S3 data lakes via Snowpipes and external stages. Implemented Star schema based fact and dimension tables, improving data accuracy by 25%. Developed Python scripts to process semi-structured data, including JSON and Parquet files, sourced from Amazon S3. Improved query performance in Snowflake using clustering and warehouse tuning, reducing execution time by 40%. Monitored and managed data pipelines in cloud environments, ensuring high availability and performance. Supported CI/CD style deployments, environment refreshes, and production releases while managing more than 150 JIRA and ServiceNow tickets. Conducted unit testing, system integration testing, and excel based reconciliation for production ready data.