Azure Data Engineer
- Azure
- AWS
- Databricks
- Apache Spark
- Apache Kafka
- Azure Databricks
- ETL
- ELT
- CI/CD
- PySpark
- AWS Cloud
- SQL
- Python
- Delta Lake
- IaC
- Snowflake
- Apache Hudi
- Airflow
Job Title: Azure Data Engineer (Azure to AWS Migration) – Spark, Kafka & Databricks
Location: Remote (USA)
Job Type: Contract
Work Arrangement: 100% Remote
Experience Required: 12+ Years
Domain: Telecom
Job Summary
We are seeking an experienced Azure Data Engineer with strong expertise in Apache Spark, Apache Kafka, Azure Databricks , and cloud migration projects , specifically migrating data platforms from Microsoft Azure to AWS . The ideal candidate will have extensive experience building scalable data pipelines, streaming solutions, and modern data lake architectures while driving cloud transformation initiatives.
This role requires hands-on experience with Azure services, Databricks, Spark optimization, Kafka event streaming, and AWS data services to support enterprise-scale migration efforts.
Key Responsibilities
- Design, develop, and maintain scalable batch and real-time data pipelines.
- Lead migration of enterprise data platforms from Microsoft Azure to Amazon Web Services (AWS) .
- Develop and optimize ETL/ELT pipelines using Azure Databricks and Apache Spark .
- Build and maintain real-time streaming applications using Apache Kafka .
- Migrate data lakes, data warehouses, and analytics workloads from Azure to AWS.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Work closely with solution architects to design cloud-native AWS data architectures.
- Implement CI/CD pipelines for data engineering deployments.
- Ensure data quality, governance, monitoring, and security throughout migration activities.
- Collaborate with business stakeholders, data scientists, and application teams to deliver reliable data solutions.
- Troubleshoot production issues and optimize distributed data processing environments.
Required Skills
Cloud Platforms
- Microsoft Azure
- Amazon Web Services (AWS)
- Azure to AWS Migration experience (Mandatory)
Mandatory Qualifications
- 8+ years of Data Engineering experience.
- 4+ years of hands-on experience with Azure Databricks .
- Strong expertise in Apache Spark and PySpark .
- Hands-on experience with Apache Kafka and streaming architectures.
- Proven experience in Azure to AWS cloud migration projects .
- Experience designing enterprise-scale data lake solutions.
- Strong SQL and Python programming skills.
- Experience with Delta Lake and modern Lakehouse architecture.
- Knowledge of CI/CD and Infrastructure as Code.
- Excellent communication and stakeholder management skills.
Preferred Qualifications
- AWS Certified Data Engineer – Associate or AWS Certified Solutions Architect.
- Microsoft Azure Data Engineer Associate (DP-203).
- Databricks Certified Data Engineer Associate/Professional.
- Experience with Snowflake, Iceberg, or Apache Hudi.
- Experience with Airflow or other orchestration tools.
Azure Data Engineer · Abode Techzone LLC