Senior Data Engineer

We are seeking aSenior Data Engineer to build an MLOps platform and enable automated model delivery, deployment, and feature engineering at scale. You will evolve the featureOps stack using Azure Databricks, Azure Data Factory, and Azure DevOps to run CI/CD into the Databricks Feature Store for Model Lab use cases.

Responsibilities

• Migrate current Spark feature engineering pipelines to Azure
• Partner with the team to define and build a forecasting model
• Assist with taking the developed machine learning model into production
• Help design and implement the CI/CD pipeline that feeds the Databricks Feature Store
• Collaborate closely with Data Scientists and engineers to match feature engineering to ML model needs
• Maintain reliability and scalability of data pipelines across the platform
• Diagnose and improve existing Spark-based workflows throughout the migration

Requirements

• Hands-on experience of 3+ years working with PySpark
• Solid background in Data Science core skills
• Proven track record in feature engineering and machine learning pipeline development
• Practical experience migrating data pipelines to cloud environments
• English proficiency at B2 (Upper-Intermediate) level or higher

Nice to have

• Experience with MLflow and Docker
• Working knowledge of Azure Kubernetes Services, Azure Data Factory, and Azure Databricks
• Familiarity with ML model deployment practices