DL
Azure Data Engineer
Diverse Lynx India
Location not stated
2 months ago
- Microsoft Fabric
- Azure
- Databricks
- Unity Catalog
- Microsoft Purview
- ETL
- T-SQL
- Apache Spark
- Pandas
- Vault
- Azure Data Factory
- Azure Synapse
- Azure DevOps
- CI/CD
- IaC
- Agile
- Azure Active Directory
- Azure AD
- PowerShell
- Bash
- Bicep
- JSON
- Python
- SQL
2 months ago
JOB DESCRIPTION:
Experience in the following technologies is key to this role Knowledge of distributed data mesh deployment methodology.
- 6 years of experience as a platform and data engineer.
- Hands-on experience with Microsoft Fabric (Lakehouse, Warehouse, Data Pipelines, notebooks, and semantic models). Strong understanding of event streaming architectures using Azure Event Hubs.
- Hands-on experience with data governance frameworks and tooling, specifically Databricks Unity Catalog andor Microsoft Purview. Experience designing and maintaining real-time data processing solutions with Azure Stream Analytics.
- Experience in building robust and performant ETL processes. Excellent data analysis and exploration using T-SQL Demonstrated expertise in developing and optimizing large-scale data processing applications using Apache Spark. Proficient in writing complex Spark jobs to process and analyze vast datasets efficiently.
- Skilled in leveraging Pandas for data manipulation and analysis tasks.
- Capable of transforming trusted data into actionable insights by utilizing Pandas powerful data structures and functions.
- Knowledge and experience of data warehouse modelling methodologies (Data Vault 2.0, Kimball) Experience in Azure one or more of the following Microsoft Fabric, Azure Data Factory, Azure Event Hubs, Databricks, Azure Synapse Analytics, ADLS Gen2 and Azure Stream Analytics.
- Experience in Azure DevOps and managing source code repositories, building and releasing applications, and implementing continuous integration and delivery processes (CICD) and infrastructure automation, including automated deployment pipelines using infrastructure-as-code.
- Experienced in the Agile Methodology ways of work Understanding of web application security concepts, such as Azure Active Directory (Azure AD) and the ability to integrate, authentication and authorization mechanisms into applications using Azure AD, data validation, and protection against common vulnerabilities. Experience in Infrastructure as Code (IaC)
- scripting and automation via Azure CLI, PowerShell, Bash, Bicep and JSON ARM and Bicep templates. Understanding and experience in Azure bicep templates for automating deployment of pipelines. C, Python and SQL (stored procedures, functions) Effective communication and collaboration skills to work in cross-functional teams, participate in code reviews, and communicate technical concepts to non-technical stakeholders.
- Willingness to learn new technologies, keep up with industry trends, and adapt to evolving development practices and tools. Experience in Banking and Financial Services Experience in Open Banking technologies
Azure Data Engineer · Diverse Lynx India