Senior Data Engineer

We are seeking aSenior Data Engineer with proven expertise in Azure Data Platform technologies, Databricks, and PySpark to strengthen our growing data engineering organization. In this position, you will craft and deliver scalable, cloud-native data solutions that drive enterprise analytics, reporting, and business intelligence efforts. You will manage large-scale datasets, develop reusable ingestion frameworks, refine data pipelines, and play a key role in evolving modern data platforms built on Azure and Databricks. The right candidate pairs strong hands-on engineering capabilities with a background in designing robust, metadata-driven data solutions, along with a genuine drive to deliver high-quality, maintainable data products.

Responsibilities

• Design, build, and sustain scalable data ingestion, transformation, and delivery pipelines
• Deliver cloud-native data solutions using Azure and Databricks technologies
• Create and refine ETL/ELT processes with PySpark, Spark SQL, and Delta Lake
• Apply metadata-driven and configuration-based ingestion frameworks to boost scalability and maintainability
• Build and sustain batch and incremental data processing pipelines
• Set up data quality controls, validation frameworks, monitoring, logging, and error handling mechanisms
• Work closely with business stakeholders, architects, and engineering teams to capture requirements and deliver data solutions
• Address production issues and push forward continuous improvements in platform reliability and performance
• Track and fine-tune Databricks workloads for efficiency, scalability, and cost management
• Uphold engineering best practices including code reviews, automated testing, CI/CD, and infrastructure automation
• Contribute to data governance, security, and access management initiatives using Unity Catalog and Azure services
• Guide team members and shape engineering standards and best practices

Requirements

• 3+ years of experience in Data Engineering or Software Engineering
• Expertise in Azure Databricks, Apache Spark, and PySpark
• Proficiency in Spark SQL and a track record of building solutions on Azure cloud platforms
• Advanced Python programming skills paired with strong SQL development and optimization experience
• Background in Unity Catalog and enterprise data governance practices
• Familiarity with Git-based source control workflows
• Experience designing and maintaining CI/CD pipelines
• Knowledge of orchestration tools such as Databricks Workflows and Azure Data Factory (ADF)
• Strong analytical, troubleshooting, and problem-solving skills
• Excellent communication and stakeholder management abilities
• English proficiency at B2 level or higher

Nice to have

• Knowledge of Terraform and Infrastructure as Code (IaC)
• Experience with Databricks Asset Bundles
• Skills in building reusable data engineering frameworks and accelerators
• Expertise in advanced Spark performance tuning and optimization
• Familiarity with enterprise-scale lakehouse architectures
• Exposure to data security, compliance, and governance initiatives
• Experience working within Agile delivery teams