Md Farman

Migration from Dremio to Databricks & Data Validation: Designed and implemented workflows to extract and load data from ADLS into Delta tables in Databricks. Developed and automated Python/PySpark scripts for data validation and schema alignment between Dremio and Databricks. Created reusable SQL validation scripts for row counts, schema checks, and sample data comparisons to ensure accuracy. Designed and monitored data pipelines (ADF + Databricks) with built-in test cases for completeness, validity, and accuracy. Collaborated with cross-functional QA and DevOps teams to ensure smooth migration with minimal business disruption. Database Optimization and Automation: Automated transformation logic testing using PySpark and SQL, improving accuracy and reducing manual testing effort. Worked extensively on test automation for data quality: schema validation, deduplication checks, and completeness validation. Integrated validation jobs with CI/CD pipelines to improve reliability of production deployments. Implemented performance optimizations (cache, persist, broadcast joins) for both transformation and test workflows.