Anurag Pandey

Client - Atlassian. Developed a configuration-driven Python framework to automate Databricks job execution, dynamically processing SQL files per table for flexible (incremental/overwrite) data ingestion into Delta/Parquet. Enhanced performance and scalability through environment-aware storage management, automated freshness validation, Z-ordering, and vacuuming. Developed custom Airflow operators to streamline workflow automation while building and managing multiple Airflow Dags to orchestrate scheduled and backfill Databricks jobs. Developed a Python-based QA script in Databricks that connects via JDBC to compare table data between MySQL/HANA and Databricks using primary key joins or set-difference logic and generates Excel reports detailing per-column discrepancies, cutting manual verification efforts by 30%. Optimized Spark jobs by addressing data skew and recursive query inefficiencies: filtered nulls and applied key salting to prevent task hang-ups, and introduced check pointing to break lineage in recursive queries reducing execution time from 6 hours to 15 minutes and achieving up to 90% cost reduction. Developed a utility function that transmits real-time task status to a Streaming Endpoint, enhancing monitoring and operational insights. Leveraged REST APIs to integrate Capterra and Impact data into existing pipelines, automating ingestion and enhancing data accessibility. Enhanced CI/CD processes by integrating Bitbucket with AWS S3 and Databricks. Developed functionality to trigger single-task Databricks jobs directly from Bitbucket via dynamic payload generation based on configuration files, reducing staging deployment complexity. Received commendations from Head of Data Engineering at Atlassian for my exceptional contributions to the C360 CAR project, driving a successful beta launch, resolving 95% of critical issues, and accelerating the release timeline by 20%.

client - Clarivate. Migrated Snowflake stored procedures to PySpark SQL, reducing transformation costs and enabling the creation of aggregated consumption tables for end-user analytics. Developed ETL pipelines to ingest CSV data from FTP sources, cleanse data, and build reliable source-of-truth base tables. Implemented HIPAA-compliant data masking through hashing techniques to secure sensitive information. Leveraged Docker and Kubernetes for containerization and orchestration, streamlining deployment processes and enabling scalable application infrastructure.

client - VMWare. Migrated SQL Server/HANA stored procedures to PySpark, increasing query response efficiency by 30% by redesigning normalized tables as denormalized one's in Hadoop to minimize data shuffling. Crafted and refined complex PySpark scripts to replicate PostgreSQL procedures, including handling cursors and iterative logic for accurate transformation. Engineered an automated data migration framework in Python, enabling seamless transfer of source data to Hadoop (in ORC format via Sqoop) with minimal configuration. Optimized Sqoop and PySpark jobs by fine-tuning mappers and partitioning, while leveraging techniques such as salting, broadcasting, caching, and distribute by operations to enhance performance. Conducted impact analysis to identify bottlenecks and implement targeted performance enhancements, ensuring efficient benchmarking and system reliability.

Developed a comprehensive framework to migrate upstream data from HANA to Hadoop using Spark, JDBC connectors, and created Hive external tables on top that. Converted HANA stored procedures to PySpark, creating denormalized consumption tables that boosted data processing efficiency by 40% and streamlined end-user analytics. Replaced the legacy middleware with a Python-based solution to dynamically generate queries from UI filter selections. Developed a high-performance solution to precalculate 12,000 dynamic queries using Python multiprocessing and Spark, storing results in JSON for near-instantaneous UI responses and offloading live queries from HANA — reducing HANA load. Worked in an agile environment by continuously refining data flows, optimizing Sqoop and PySpark jobs, and conducting impact analyses to identify and resolve performance bottlenecks.

client - coop bank. Worked on two marketing projects at COOP Bank, helping to improve campaign performance. Provided production support and built over 100 Oracle SQL and PL/SQL programs to keep systems running smoothly.