Mehul Patil

Mastered core big data technologies, including MapReduce, HDFS, Hive, and Apache Spark 3.0, with expertise in both low-level resilient distributed datasets (RDD) and high-level APIs. Designed and optimized data pipelines on fully distributed computing clusters by managing data storage solutions and file compressions.

Implemented data cleaning techniques and data validation tests like schema validation on input data, leading to enhanced data accuracy by ~35%. Redesigned the data pipeline architecture, resulting in a decrease of 40% in processing time and manual data handling tasks. Maximized the storage space by ~22% and data retrieval speed by 3 to 4 minutes using Parquet files. Optimized cluster utilization to 95% by strategically reallocating tasks and tuning partition sizes, core allocation, and memory distribution. Developed and executed hive queries to construct hive tables, extracting valuable analytical insights with 100% accuracy and reliability. Leveraged version control system git, automatic deployment tool Jenkins, unit test framework Pytest, and JIRA for productivity and short release cycles. Partnered with 3 cross-functional teams, including IT support, data science, and business clients, using an agile model to address technical challenges and define project scope. Integrated Spark & hive with TIBCO Statistica 13.0 for analysis and visualization using Livy and ODBC driver. Communicated complex data processes and results to management through detailed reports and presentations. Authored 10+ articles online regarding managing data nodes in TIBCO Statistica.