DL
cloudops
Diverse Lynx India
Location not stated
2 months ago
- Azure Databricks
- Fabric
- Unity Catalog
- Microsoft Fabric
- FinOps
- CI/CD
- Azure
- Key Vault
- Databricks
- Delta Lake
- IaC
- Terraform
- Bicep
- ADO
- GitHub
- RBAC
- ACLS
- Network Security
- IAM
- Azure Monitor
- Incident Management
- Python
- PowerShell
- Bash
- SQL
- PySpark
- Azure DevOps
- GitHub Actions
- Zero Trust
- Devops
2 months ago
Job Description: Cloud Data Platform Engineer / SRE (Azure | Databricks | Fabric | Unity Catalog)
Role Overview
We are seeking highly skilledCloud Data Platform Engineers (4–5 resources) with deep hands-on expertise acrossAzure, Databricks, Microsoft Fabric, and Unity Catalog to strengthen our cloud data operations. These roles demandtechnical execution, not coordination—requiring strong engineering capabilities ininfrastructure automation, FinOps, CI/CD, platform governance, environment management, and Site Reliability Engineering (SRE) practices.
The ideal candidates will have proven experience managingenterprise-scale data platforms, ensuring secure, reliable, cost-efficient, and well-governed cloud environments.
Key Responsibilities
1. Cloud Platform Operations
- Manage and optimize Azure workloads—ADLS, VNets, Key Vault, ADF, Synapse, Fabric, Databricks.
- Configure and maintainDatabricks clusters, jobs, DLT pipelines, Delta Lake storage, and Unity Catalog policies.
- OperationalizeFabric Lakehouses, Pipelines, Warehouses, and Semantic Models for production workloads.
- Ensure robust platform governance across environments (DEV–QA–UAT–PROD).
- Build and maintainTerraform/Bicep templates for environment provisioning and configuration.
- Develop end-to-endCI/CD pipelines for Databricks, Fabric, and Azure components (ADO/GitHub).
- Automate deployment of notebooks, workflows, access policies, networking components, and Fabric artifacts.
- Enforce version control, release governance, and quality gates.
- ImplementFinOps dashboards, alerts, budgets, and spend governance practices.
- Perform Databricks and Fabriccost optimization—cluster sizing, autoscaling, idle management, job tuning.
- Conduct capacity planning for compute, storage, Fabric engines, and Databricks workloads.
- Develop cost-saving recommendations and automated consumption monitoring.
- Provision and manage Azure data environments with consistent policies and naming standards.
- ConfigureRBAC, ACLs, Unity Catalog grants, service principals, network security, Managed Identities.
- Implement governance standards fordata access, lineage, audit logging, compliance, and risk mitigation.
- Ensure secure connectivity using Private Endpoints, VNET integration, and enterprise IAM controls.
- Implement monitoring and alerting usingAzure Monitor, Log Analytics, Databricks Metrics, Fabric Admin APIs.
- Build runbooks, dashboards, and automated remediation workflows for platform reliability.
- Conduct performance tuning of data workloads, Fabric pipelines, Databricks jobs, and storage layers.
- Lead incident management, root‐cause analysis, and environment stabilization efforts.
Required Skills & Experience
- 6–12 years in cloud data engineering, SRE, or platform engineering roles.
- Strong hands-on expertise with:
- Azure Data Services (ADLS, ADF, Synapse, Key Vault, VNets)
- Azure Databricks (clusters, jobs, Delta Lake, DLT, Unity Catalog)
- Microsoft Fabric (Lakehouse, Pipelines, Warehouse, Dataflows)
- Unity Catalog governance (catalogs, schemas, access policies, lineage)
- Strong scripting and automation experience:Python, PowerShell, Bash, SQL, PySpark.
- Experience withTerraform/Bicep for IaC.
- Strong knowledge ofAzure DevOps or GitHub Actions CI/CD pipelines.
- Proven FinOps experience withcost governance and optimization across cloud workloads.
- Experience in SRE practices—SLIs, SLOs, operational readiness, automated recovery.
Preferred Qualifications
- Certifications inAzure Data Engineer, Azure DevOps Engineer, Databricks Data Engineer, FinOps Practitioner.
- Experience in highly regulated environments (BFSI, Healthcare, Retail).
- Understanding of zero-trust security models.
Suitable Job Titles
- Cloud Data Platform Engineer
- Databricks Platform Engineer
- Azure/Fabric DevOps Engineer
- Cloud DataOps / SRE Engineer
- Platform Reliability Engineer (Data Platforms)
cloudops · Diverse Lynx India