Marivini Naveen

Migrated 100+ production PySpark ETL pipelines from EMR 6.8 to 7.2 in a live production environment. Redesigned Spark write logic to eliminate duplicate data generation post-migration. Optimized Spark configurations improving execution time by ~30%. Developed SQL-based reconciliation framework for multi-million record validation. Stabilized Airflow DAGs (daily/weekly/monthly) during production validation cycle. Reduced pipeline failures by 25% through upstream data validation checks. Supported CI/CD deployments and production monitoring in Agile delivery model. Client: NatWest Group

Designed and developed end-to-end Snowflake ETL workflows for remediation processes. Implemented multi-layer data warehouse architecture (RAW, CLEANSED, STAGING, HISTORIC). Built complex SQL transformations using CTEs, window functions, and incremental delta logic. Developed Python UDFs for transformation and validation. Integrated AWS S3 staging with Snowflake ingestion pipelines. Automated Airflow DAGs reducing manual effort by 40%. Implemented data quality controls ensuring >99% dataset accuracy. Ensured production stability and collaborated with stakeholders in Agile environment. Client: Royal Bank of Scotland