DL
Lead Software Engineer-Databricks
Diverse Lynx India
🇮🇳 India
On-site
Staff / Principal
2 months ago
- Databricks
- PySpark
- Delta Lake
- SQL
- IAM
- Unity Catalog
- FinOps
- ETL
- CI/CD
- MLflow
- Agile
2 months ago
Skills Required: Digital: Databricks, Digital: PySpark
Relevant Experience Range in Required Skills: 8 to 10 Years
Job Description:
1. Design and implement scalable Lakehouse architecture using Databricks, Delta Lake, and Cloud Storage.
2. Develop robust ETLELT pipelines using PySpark, Spark SQL, and Databricks Notebooks.
3. Integrate Databricks with Cloud Services (AWSAzure) including storage, IAM, and event services.
4. Implement Unity Catalog, Data Lineage, Access Governance, and Auditing Standards.
5. Define and enforce data quality, observability, and FinOps monitoring.
Essential Skills:
1. Design and implement scalable Lakehouse architecture using Databricks, Delta Lake, and Cloud Storage.
2. Develop robust ETL pipelines using PySpark, Spark SQL, and Databricks Notebooks.
3. Integrate Databricks with Cloud Services (AWSAzure) including storage, IAM, and event services.
4. Implement Unity Catalog, Data Lineage, Access Governance, and Auditing Standards.
5. Define and enforce data quality, observability, and FinOps monitoring.
6. 7 to 10 years of IT experience with min 4 yrs exp in designing, building, and optimizing large-scale data processing solutions on the Databricks Platform
7. Skills: DBX, PySpark, Spark SQL, Delta Lake, SQL Warehouse, Cloud Platforms (AWSAzure), CICD, Data Governance.
8. Preferred DBX Certification, MLFlow Functional background.
9. Experience in Banking and Financial Services - particularly within Corporate Functions domain.
10. Methodology: Agile Behavioural Skills.
11. Ability to work in a cross-cultural team with multi geographical locations.
12. Ability to communicate (verbal and written) effectively with counterparts in SG Paris.
13. Functional background Experience in Banking and Financial Services - particularly within Corporate Functions domain Methodology Agile
Lead Software Engineer-Databricks · Diverse Lynx India