AC
Azure Data Engineer
APN Consulting Inc
- ๐ฎ๐ณ India
- On-site
- 22 hours ago
- ServiceNow
- AI/ML
- SQL
- PySpark
- Power BI
- DAX
- Fabric
- SSIS
- Databricks
- Git
- Azure DevOps
- GitHub
- CI/CD
- YAML
- GitHub Actions
- REST API
- Terraform
- Microsoft Fabric
- Bicep
- Key Vault
- SAST
- CodeQL
- SonarQube
- IaC
- Trivy
- pytest
- Great Expectations
- OIDC
- Azure
22 hours ago
Job Description
Job Title- Azure Data Engineer
Experience- 6-8 Year
Location- Bangalore (Hybrid)
- Build and maintain Lakehouses, Warehouses and SQL analytics endpoints; implement medallion layers with Delta tables, V-Order, OPTIMIZE/VACUUM and partitioning strategies.
- Develop ingestion and orchestration with Data Factory pipelines, Copy jobs, Dataflows Gen2 and Notebooks (PySpark, Spark SQL); configure OneLake shortcuts and Mirroring for source systems.
- Implement real-time scenarios with Eventstream, Eventhouse and KQL databases, Real-Time dashboards and Activator alerts.
- Build Power BI semantic models (Direct Lake, Import, composite), DAX measures, incremental refresh and RLS/OLS.
- Tune Spark environments (starter pools, custom pools, libraries, high-concurrency sessions) and SQL query performance; monitor capacity consumption with the Fabric Capacity Metrics app and Monitoring hub.
- Execute migrations from Synapse, ADF, SSIS and Databricks: inventory, conversion, reconciliation testing and cutover.
- Configure Fabric Git integration with Azure DevOps Repos or GitHub; enforce branching, pull-request policies and code-review standards for notebooks, pipelines and semantic models (TMDL/PBIP formats).
- Build CI/CD pipelines in Azure DevOps (YAML) or GitHub Actions using Fabric REST APIs, fabric-cicd, deployment pipelines and parameter files for environment promotion.
- Write and maintain Terraform (azurerm, azapi, Microsoft Fabric provider) or Bicep modules for capacities, workspaces, connections, networking, Key Vault and role assignments.
- Implement security gates in pipelines: SAST (CodeQL, SonarQube), secret scanning, IaC scanning (Checkov, Trivy), dependency scanning, unit tests for notebooks (pytest, Great Expectations / data-quality checks) and approval gates.
- Use OIDC / workload identity federation and managed identities; store secrets in Azure Key Vault; eliminate hard-coded credentials.
Azure Data Engineer ยท APN Consulting Inc