Srikant Padhy

Designed and implemented real-time Spark Streaming pipelines processing 1 TB/day, leveraging structured streaming, dynamic resource allocation, and executor tuning, improving throughput by 30% and reducing batch processing time by 25%. Optimized Cassandra cluster performance by redesigning partitioning strategies, implementing replication tuning and indexing, reducing write latency by 40%, and improving real-time query performance by 35%. Collaborated with business teams to build a re-targeting campaign platform, driving a 45% increase in user engagement and a 20% rise in conversion rates. Pioneered a schema standardization framework to flatten nested JSON data using Spark SQL and custom parsers. Spearheaded a Delta Lake migration strategy, replacing legacy batch pipelines, leveraging Z-Order indexing and data compaction, improving data freshness by 90%, and reducing storage costs by 25%. Led a team of 4 engineers, conducting hands-on mentorship on Spark performance tuning and query optimization.

Designed and optimized distributed data pipelines handling 6 TB/day, reducing end-to-end latency by 40% using memory tuning, efficient shuffle operations, and partition pruning. Developed production-grade ETL workflows for a price prediction ML model, using Apache Airflow, PySpark, and feature engineering techniques, achieving 99.9% SLA compliance and improving forecast accuracy by 15%. Engineered high-throughput Spark ingestion jobs processing 14 K records/minute with exactly-once semantics. Collaborated on ML-driven customer retention models (Python, Scikit-Learn) that reduced churn by 20%. Automated 15+ manual workflows using Bash/Python, reducing operational overhead by 10 hours/week.

Built 15+ predictive models (XGBoost, Random Forest, L1-regularized Logistic Regression) in Python to solve resource allocation challenges in distributed systems, optimizing hyperparameters via 5-fold cross-validation. Accelerated model training by 35% using parallel programming techniques (MPI, OpenMP) to distribute workloads across multi-core architectures.