High level requirement details:
Experience
- 5–10 years of experience in Data Engineering
Key Skills Required
- Strong hands-on experience withDatabricks:
- PySpark
- SQL
- Delta Lake
- Unity Catalog
- Databricks Workflows / Jobs orchestration
- Experience withAzure Data Factory (ADF)
- Experience buildingLakehouse architectures using Bronze, Silver & Gold layers
- Good knowledge ofDatabricks Lakeflow:
- Declarative Pipelines
- Lakeflow Connect
- CDC (Change Data Capture)
- Data quality validations
- Experience withAI/BI and Genie:
- Semantic layers
- Certified datasets
- Self-service analytics solutions
- Experience buildingAI Agents/Copilots using:
- LLMs
- RAG (Retrieval-Augmented Generation)
- Vector Search
- Prompt Engineering
- Good understanding ofUnity Catalog governance:
- Access controls
- Data masking
- Audit logging
- Lineage
- Experience withCI/CD, Git, Databricks Repos & Asset Bundles
Expected Responsibilities
- Design, develop, and support end-to-end data pipelines in Databricks
- Build and maintain Delta tables under Unity Catalog
- Develop Lakeflow pipelines with incremental processing and CDC
- Enable AI/BI use cases through semantic models and Genie spaces
- Integrate SharePoint and external data sources into the Lakehouse
- Ensure data quality, governance, and operational reliability
- Drive delivery independently and proactively resolve issues
Location: Offshore
Notice Period: Preferably Immediate to ≤ 30 days |
