DL
Senior Data Engineer (Databricks & AWS Migration)
Diverse Lynx India
Location not stated
Senior
2 months ago
- ETL
- Databricks
- ELT
- PySpark
- SQL
- CloudWatch
- CI/CD
2 months ago
| • Analyze and understand existing ETL pipelines, data sources, and transformation logic across current platforms • Drive end-to-end migration of data pipelines to the new platform using Databricks • Design, build, and maintain scalable data pipelines for batch and near real-time processing in the new platform • Re-engineer legacy ETL/ELT workflows into optimized, scalable, and modular pipelines using Spark/Databricks • Integrate data from multiple heterogeneous sources and ensure seamless data flow across systems • Implement data transformation logic using PySpark/SQL on Databricks and ensure adherence to best practices • Validate migrated pipelines to ensure data accuracy, reconciliation, and consistency with source systems • Optimize pipelines for performance, scalability, and cost efficiency within cloud environments • Work closely with business stakeholders, analysts, and data teams to understand data requirements and use cases • Support reporting, analytics, and downstream applications by providing reliable and high-quality datasets • Monitor production pipelines, troubleshoot failures, and proactively resolve data-related issues • Ensure proper logging, alerting, and observability for all pipelines (using tools like CloudWatch / Databricks monitoring) • Follow data governance, security, and compliance standards during data handling and migration • Implement CI/CD pipelines for data workflows and maintain proper documentation of pipelines and processes |
Senior Data Engineer (Databricks & AWS Migration) · Diverse Lynx India