Lead Python Data Engineer

We seek a skilledLead Python Data Engineer to join our team and help us build and maintain our data pipelines.

As a Python Data Engineer, you will be responsible for designing, developing, and testing Python-based data pipelines and performing data cleansing, transformation and quality assurance.

Responsibilities

• Design, develop, test and maintain Python-based data pipelines
• Write high-quality and maintainable Python code to migrate data between different storage systems with modifications
• Create ETL jobs, airflow operators, and Spark jobs to enhance data processing capabilities
• Implement Data Intake processes for Data Sources (ETL/ELT, DWH Modelling and Optimization)
• Perform data cleansing, transformation and quality assurance to maintain data integrity
• Optimize data processing, storage and retrieval to improve performance
• Collaborate closely with cross-functional teams to ensure data accuracy and consistency

Requirements

• In-depth learning of Python programming with the ability to write maintainable and scalable code
• Strong understanding of distributed computation and multiprocessing
• Knowledge of data warehouse concepts and ETL processes
• Proficiency in working state machines, web back-end development, networking and protocols
• Experience with cloud computing platforms, such as AWS or GCP, and tools like Step Functions, Batch
• Knowledge of Spark, Hive, Databricks and other data processing frameworks
• Familiarity with data orchestration tools like Airflow
• Expertise in with SQL, NoSQL and cloud-based data storage solutions