Lead Data Software Engineer (Databricks)

We are searching for aLead Data Software Engineer to join our team and oversee the development of our cutting-edge supply-chain data analytics platform.

This platform delivers a comprehensive view of suppliers, products, material categories, shipments, and compliance activities while employing artificial intelligence to generate insights and recommendations tailored to our clients’ needs.

If this role excites you – apply!

Responsibilities

• Design and develop data architectures and ETL pipelines utilizing Databricks and external orchestrators such as Airflow
• Collaborate with machine learning teams to incorporate AI-driven insights into the platform
• Develop system-level improvements and components for data engineering workflows
• Optimize the performance and scalability of data-intensive applications
• Adhere to software engineering standards, including containerization, unit testing, linting, and code style reviews
• Maintain and improve data workflow, Delta Lake, and Delta Live Tables within Databricks
• Provide technical mentorship and leadership to the data engineering team
• Engage with clients and stakeholders to understand their requirements and offer customized solutions
• Tackle challenges independently and ensure successful project delivery

Requirements

• Experience leading data engineering efforts on projects utilizing Databricks
• Solid understanding of data architectures and skills in data modeling
• Background in designing and building ETL pipelines in Databricks with external orchestrators such as Airflow
• Hands-on expertise in Databricks (Delta Lake, workflows, Delta Live Tables, deployment, and versioning)
• Proficiency in Python and cloud-native technologies
• Competency in Spark/PySpark
• Engineering experience with either AWS or Azure
• Knowledge of big data and techniques for optimizing data-intensive applications
• Strong communication skills with a proactive attitude and client-facing experience
• Ability to navigate ambiguity and complete projects independently
• Comfort working in a dynamic and transparent startup environment
• Fluent English communication skills at a C1 level

Nice to have

• Proficiency in configuring and managing CI/CD pipelines in Azure DevOps
• Familiarity with Data Observability and Data Quality Monitoring practices
• Capability to integrate data quality checks into data pipelines