Ashwini Dsilva

Design, develop, and maintain scalable and efficient data pipelines for extracting, transforming, and loading (ETL) data from various sources to destination systems. Proficiently manage the ingestion of diverse file types (e.g., fixed width, delimited files) into Azure Data Lake Storage (ADLS) using NiFi. Collaborate with cross-functional teams to understand data requirements and ensure the seamless flow of information across the organization. Create and optimize logical and physical data models, ensuring data structures align with business needs and best practices. Write and optimize code in languages such as Python, SQL, or Scala to implement ETL processes and data transformations. Develop and implement data quality checks and validation processes to ensure the accuracy and reliability of data. Develop test cases and scripts to validate data integrity and accuracy during migration. Write efficient Hive queries for data analysis, addressing diverse data requirements. Load .xl and .csv files into Hadoop, ensuring proper validation of resulting tables using tools like Hue. Rewrite or convert on-premises table’s scripts in databricks using the appropriate syntax. This might involve translating SQL DDL statements into databricks Data Frame API calls or spark SQL. Optimize the table's data processing for performance in the databricks environment. This may involve partitioning tables, optimizing join operations, and tuning cluster configurations. Implements the workflow design by developing and configuring data processing tasks. Ingesting the data from on-premises sources to azure databricks.