NC
Senior Data Engineer (Databricks, Lakehouse & AI-BI)
NR Consulting - India
Location not stated
Senior
2 months ago
- Databricks
- AI
- Azure Data Factory
- Delta Lake
- Unity Catalog
- ETL
- CI/CD
- PySpark
- SQL
- Git
2 months ago
Location: ABU DHABI
Exp: 6+ Years
Job Description:
Key Responsibilities
1. Pipeline Orchestration: Design and build robust data ingestion and orchestration pipelines, with Databricks (Lakeflow Jobs/Workflows) as the primary platform and Azure Data Factory supporting legacy/external needs — taking full ownership of pipeline reliability and the transition toward Databricks-native orchestration.
2. Lakehouse Data Engineering: Build and maintain scalable Delta Lake tables under Unity Catalog — schema design, access controls, lineage tracking, and data quality across bronze/silver/gold layers.
3. Lakeflow Pipeline Development: Build declarative ETL pipelines using Databricks Lakeflow, incorporating CDC, incremental processing, and data quality validations.
4. AI/BI & Genie Enablement: Design and maintain semantic layers, metrics, and certified datasets to support Genie spaces and AI/BI dashboards, enabling natural-language querying and self-service analytics.
5. SharePoint/External Source Integration: Configure and maintain Lakeflow Connect (or similar) integrations for ingesting SharePoint and other external data sources into the Lakehouse with minimal manual effort.
6. Governance & Best Practices: Apply Unity Catalog governance controls — access policies, data masking, audit logging — and uphold strong engineering practices (CI/CD, version control, testing).
Must-Have Skills
• Strong hands-on Databricks expertise — PySpark/SQL, Workflows/Jobs orchestration, Delta Lake, Unity Catalog
• Working knowledge of Azure Data Factory
• Experience with Lakeflow (Declarative Pipelines/Connect) including CDC and data quality validations
• Exposure to Databricks AI/BI or Genie (semantic layers, certified datasets)
• Familiarity with CI/CD and version control (Git, Databricks Repos/Asset Bundles)
• Demonstrated ability to own deliverables fully and resolve issues independently
• Strong communication skills for working with business/analytics stakeholders
Senior Data Engineer (Databricks, Lakehouse & AI-BI) · NR Consulting - India