Senior AI Engineer with Databricks

We are looking for aSenior AI Engineer with Databricks expertise to design, deploy and maintain scalable machine learning pipelines using the Databricks platform. In this role, you will deliver production-ready ML pipelines, automated training and retraining workflows, deployed models, monitoring dashboards and CI/CD pipelines for ML systems.

Responsibilities

• Design, implement and maintain end-to-end ML pipelines on Databricks
• Build workflows for data ingestion, preprocessing, feature engineering, training and inference
• Leverage PySpark, Spark ML and Databricks notebooks/jobs
• Manage model versioning, experiment tracking and reproducibility using MLflow
• Package and deploy models for batch and real-time inference
• Monitor model performance, drift and retraining cycles
• Develop scalable ETL/ELT pipelines using Databricks Delta Lake
• Optimize data storage and access patterns through partitioning, Z-ordering and caching
• Integrate with data sources such as Azure Data Lake, S3, APIs and databases
• Implement CI/CD pipelines for ML workflows using Azure DevOps, GitHub Actions and Databricks Repos and Jobs API
• Configure clusters, autoscaling and cost optimization while applying Infrastructure as Code with Terraform, ARM and Bicep
• Implement logging, alerting and observability to ensure high availability and fault tolerance of ML systems

Requirements

• 3+ years of experience in machine learning engineering or related roles
• Expertise in the Databricks platform including workspaces, jobs and clusters
• Proficiency in Apache Spark, PySpark and Python with pandas and scikit-learn
• Skills in MLflow for tracking, registry and deployment
• Competency in CI/CD pipelines, Docker containerization and REST APIs for model serving
• Familiarity with version control using Git
• Background in Azure including Azure Databricks, ADLS, ACR and AML
• Knowledge of data preprocessing, feature engineering and model training and evaluation
• Understanding of libraries such as XGBoost, LightGBM and CatBoost
• English proficiency at B2 level or higher

Nice to Have

• Familiarity with AWS including S3, EMR and SageMaker
• Skills in streaming pipelines with Spark Structured Streaming and Databricks Feature Store
• Knowledge of Kubernetes
• Competency in monitoring tools such as Prometheus and Grafana
• Experience with large-scale production systems