Senior Data Quality Engineer

We are seeking a skilledSenior Data Quality Engineerto join our team. The successful candidate will play a crucial role in ensuring the integrity, reliability, and quality of our data engineering pipelines and platforms.

Responsibilities

• Design and develop a robust automated testing framework and strategy for data engineering pipelines, with a focus on integration with Databricks, PySpark, Scala, Spark SQL, and other relevant technologies
• Implement automated tests to ensure data integrity, quality, and reliability across all data pipelines and platforms
• Integrate automated testing within Azure DevOps CI/CD pipelines to ensure seamless testing, build, and deployment processes
• Conduct and execute performance tests to assess the efficiency and scalability of data processing systems
• Work closely with data engineers and other stakeholders to understand requirements and ensure testing strategies are aligned with business objectives
• Establish monitoring for automated tests, providing regular reports on test coverage, defects, and data quality issues

Requirements

• Minimum of 3 years of experience in Data Quality Engineering or a related field
• Solid understanding of automated testing frameworks and data engineering principles
• Proficiency with Databricks, PySpark, Scala, Spark SQL, Terraform, and Azure Event Hubs. Solid understanding of automated testing frameworks and data engineering principles
• Experience in integrating testing frameworks with Azure DevOps and managing source control with Git
• Knowledge of pipeline configuration and deployments utilizing Terraform
• Familiarity with Jira
• Strong analytical abilities with a proven track record in troubleshooting and resolving complex data and testing challenges
• Fluency in English with a B2+ level

Nice to have

• Experience with Power BI, Azure Data Lake, Spark Streaming, and other data processing and visualization tools
• Prior experience in a data-intensive industry or with large-scale data environments
• Proficient in implementing solutions through Infrastructure as Code (IaC)