Azure Data Engineer

In this role, you will:

• Design, develop, and optimize data pipelines using Azure Databricks, PySpark.
• Implement and maintain ETL/ELT pipelines using Azure Data Factory (ADF) and Azure Synapse for orchestration.
• Develop and optimize complex SQL queries and Python-based data transformation logic.
• Work with version control systems (GitHub, Azure DevOps) to manage code and deployment processes.
• Implementing technical expertise and look for opportunities to reduce operational costs, eradicating unnecessary processes and exploring opportunities for service efficiency and rationalizing the service where possible.
• Working on large dataset of ELC and need to define/build data model for golden view.
• Automate deployment of data pipelines using CI/CD practices in Azure DevOps.
• Ensure data quality, security, and compliance with best practices.
• Monitor and troubleshoot performance issues in data pipelines.
• Ensure compliancy with Operational risk standards (E.g. Network, Firewall, OS, Logging, Monitoring, Availability, Resiliency).
• Collaborate with cross-functional teams to define data requirements and strategies.

To be successful in this role, you should meet the following requirements:

• 6–7+ years of experience in data engineering, working with Azure Databricks, PySpark, and SQL.
• Proficiency in Python for data processing and transformation.
• Expertise in machine learning for processing large data set.
• Strong expertise in Azure Data Factory (ADF) for building and managing ETL processes.
• Familiarity with GitHub and Azure DevOps for version control and CI/CD automation.
• Solid understanding of data modelling, warehousing, and performance optimization.
• Ability to work in an agile environment and manage multiple priorities effectively.
• Excellent problem-solving skills and attention to detail.
• Experience with Delta Lake and Lakehouse architecture.
• Hands-on experience with Terraform or Infrastructure as Code (IaC).
• Understanding of machine learning workflows in a data engineering context.
• In-depth knowledge of fundamental data warehousing concepts such as entity relationship modelling, dimensional modelling, star and snowflake schemas, Data marts.
• Experience in handling technical issues in Data models, Data engineering logic document and Power BI reports.