TC
Test Architect
TechDigital Corporation
Location not stated
Remote
11 months ago
- Azure Databricks
- Python
- PySpark
- ETL
- Azure Synapse
- Data Modeling
- ELT
- CI/CD
- Git
- Azure DevOps
- Azure
- AWS
- GCP
- Delta Lake
- MLflow
- Databricks
- Agile
- Scrum
11 months ago
Key Responsibilities:
• Manage Testing of end-to-end data pipelines using Azure Databricks, Python, and PySpark.
• Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders to understand data requirements.
• Ensure data quality, integrity, and governance across all migration and transformation processes.
• Mentor junior engineers and provide technical leadership within the team.
• Develop reusable frameworks and automation tools to streamline migration and data processing tasks.
• Stay updated with the latest trends and best practices in cloud data engineering and migration.
Required Skills & Qualifications:
• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• 12+ years of experience in data engineering with expertise in Python and PySpark.
• 3+ years of experience working with Azure Databricks and other Azure data services (e.g., Azure Data Lake, Azure Synapse).
• Proven experience in ETL and data migration projects, including strategy, execution, and validation.
• Strong understanding of distributed computing, data modeling, and ETL/ELT processes.
• Experience with CI/CD pipelines and version control tools (e.g., Git, Azure DevOps).
• Excellent problem-solving and communication skills.
• Ability to lead and manage multiple projects simultaneously.
Preferred Qualifications:
• Azure certifications (e.g., Azure Data Engineer Associate).
• Experience with other cloud platforms (AWS, GCP) is a plus.
• Familiarity with Delta Lake, MLflow, and other Databricks ecosystem tools.
• Experience in Agile/Scrum environments.
Test Architect · TechDigital Corporation