Lead Data Engineer

We're looking for aLead Data Engineer to construct an MLOps platform that supports automated model delivery, deployment, and large-scale feature engineering. In this role, you'll advance the featureOps stack by leveraging Azure Databricks, Azure Data Factory, and Azure DevOps to implement CI/CD pipelines feeding into the Databricks Feature Store for Model Lab applications.

Responsibilities

• Transition existing Spark feature engineering pipelines over to Azure
• Work alongside the team to design and develop a forecasting model
• Support the deployment of the machine learning model into a production environment
• Contribute to designing and building the CI/CD pipeline that supplies the Databricks Feature Store
• Work in close coordination with Data Scientists and engineers to align feature engineering with ML model requirements
• Ensure data pipelines remain reliable and scalable across the entire platform
• Troubleshoot and enhance current Spark-based workflows during the migration process

Requirements

• A minimum of 5 years of practical PySpark experience
• Strong foundational knowledge in core data science principles
• Demonstrated success in building feature engineering and machine learning pipelines
• Hands-on background in moving data pipelines into cloud-based environments
• B2 (Upper-Intermediate) English proficiency or above

Nice to have

• Familiarity with MLflow and Docker
• Working knowledge of Azure Kubernetes Services, Azure Data Factory, and Azure Databricks
• Understanding of best practices for deploying ML models