DNAHL_SQL, Databricks, PySpark, Delta lake, Azure Data Factory, Logic Apps, and Airflow
- AI
- Databricks
- PySpark
- Azure Data Factory
- Logic Apps
- Airflow
- ETL
- ELT
- Delta Lake
- Azure
- CI/CD
- GitHub Actions
- Machine Learning
- Apache Airflow
- Python
- SQL
- Event-Driven Architecture
- AI/ML
- MLflow
- Azure Databricks
- Position Overview
- We are seeking a Senior Data Engineer to drive cloud data modernization and build scalable, AI-ready data platforms. This role emphasizes expertise in Databricks, PySpark, Azure Data Factory, Logic Apps, and Airflow, with a strong focus on orchestration, pipeline reliability, and end-to-end data workflow management.
- Key Responsibilities
- Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow
- Develop and manage end-to-end orchestration frameworks integrating Airflow DAGs with ADF and Logic Apps
- Implement advanced workflow orchestration patterns (event-driven, micro-batch, hybrid scheduling)
- Ensure pipeline dependency management, execution reliability, and operational excellence
- Build high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture
- Implement data observability, monitoring, and alerting mechanisms across workflows
- Optimize pipelines and workflows for performance, scalability, and cost efficiency
- Integrate pipelines with Azure services such as ADLS Gen2, Blob Storage, and event triggers
- Implement CI/CD for pipelines and workflows using GitHub Actions or equivalent
- Ensure data quality, governance, and security compliance
- Collaborate with data science teams for ML/GenAI data readiness
- Mentor junior engineers and drive best practices for orchestration and pipeline design
- Required Qualifications
-10+ years of experience in data engineering
- Highly proficient inDatabricks (PySpark, Delta Lake), Azure Data Factory (ADF), AzureLogic Apps, and ApacheAirflow
- Strong programming skills inPython andadvanced SQL
- Experience in buildingorchestrated data platforms with multi-tool integration
- Strong understanding of ETL/ELT patterns and orchestration frameworks
- Experience in cloud-native data architectures (Azure preferred)
- Hands-on experience with monitoring, logging, and pipeline reliability
- Data Engineer (Databricks +Pyspark) JD + ADF/Logic Apps (Highly Proficient) + Airflow
- Preferred Qualifications
- Experience with event-driven architecture and API integrations
- Exposure to AI/ML data pipelines and MLflow
- Knowledge of Data Lakehouse architectures and governance frameworks
- Azure / Databricks certifications
|
Mandatory skills* |
SQL, Databricks, PySpark, Delta lake, Azure Data Factory, Logic Apps, and Airflow |
DNAHL_SQL, Databricks, PySpark, Delta lake, Azure Data Factory, Logic Apps, and Airflow · Diverse Lynx India