Aditya Firake

Design and developed Databricks workflow to ingest streaming calls data and convert it into audio transcribes for ML & AI use cases. Created Databricks SQL alerts and dashboard to monitor the pipeline. Created Airflow pipeline to periodically ingest data from the Bigquery(GC) and land it on AWS S3.

Design and developed Airflow DAG to fetch data from Bloomberg API. Configure usage right flags to provide grained access for downstream consumers. Investigate and fix data quality issues reported by downstream teams. API Integration for different API sources. Apply Data Contracts for Data Quality Assurance. Create Docker images for various integration projects.

Design, Develop, and maintain data pipelines built using Python and PySpark. Integration of code with monitoring and alert tools like Opsgenie, Grafana, Graphite, Datadog. Created Terraform script to provide s3 access to prod and stage roles and validate access using Boto3 script. Configure and Create EMR cluster using shell script, Gitlab CICD jobs.

Setup Snowflake database connection in Boomi. Migrate workday reports on Snowflake using Boomi integration. Migrate existing finance data (tables and Stored Procedures) from MS SQL to Snowflake. Snowflake query tuning, access management, and handled semi-structured files.

Created AWS lambda function to trigger AWS data pipelines. Gather all the GIS Open data for each county in US states. Process shape files and generate Avro files to further process them using Spark jobs. Calculate and Generate geolocation Housing data using TomTom street data.

Export incident data from BMC remedy users on HDFS and prepare health reports using Hive and Tableau. Handled responsibilities like writing RESTful web services and exposing JSON objects to NG-grid in Angular JS. Contribution to the design and development of the feedback service for the ADR Public website.