Srinivas More

Designed and delivered scalable, secure, and cost-efficient Azure-based data solutions for enterprise analytics. Developed scalable ETL/ELT pipelines using Azure Data Factory and Azure Databricks to automate end-to-end data workflows. Implemented Medallion Architecture (Bronze, Silver, Gold) to build structured and reusable data lake layers. Built Databricks pipelines and jobs to process incremental data loads and complex transformations. Developed Databricks notebooks for data cleansing, transformation, deduplication, and missing value handling in the Silver layer. Performed merge (upsert) operations in the Gold layer using Databricks and ADF to maintain data integrity. Implemented Slowly Changing Dimensions (SCD Type 1 and Type 2) for historical data tracking in analytical datasets. Collaborated with DevOps teams to enable CI/CD pipelines for automated deployment of data workflows and notebooks.

Developed and maintained ETL pipelines using Azure Data Factory to ingest structured and semi-structured data from SFTP sources into Azure Data Lake Storage Gen2 (ADLS Gen2). Implemented data transformation workflows using PySpark and Spark SQL in Azure Databricks. Built metadata-driven ADF pipelines capable of dynamically processing multiple data sources, improving pipeline reusability and reducing ADF artifacts by 50%. Integrated ADLS Gen2 with Azure Databricks using mount points for efficient large-scale data processing. Performed data validation, schema enforcement, and transformation logic to ensure data quality before downstream consumption. Assisted in optimizing data pipelines and improving job performance in Databricks environments.

Assisted in building and maintaining ETL pipelines for data extraction, transformation, and loading processes. Worked with Hadoop-based file formats such as Parquet for efficient storage and processing of large datasets. Supported basic data transformation tasks using PySpark and SQL in big data environments. Documented workflows using MkDocs to improve knowledge sharing and onboarding across teams. Assisted in troubleshooting and resolving issues in data processing pipelines.