DL
Databricks Architect
Diverse Lynx India
Location not stated
2 weeks ago
- Databricks
- T-SQL
- PySpark
- SQL
- Delta Lake
- Shopify Liquid
- Unity Catalog
- Python
- Scala
- CI/CD
- Git
- MLflow
2 weeks ago
Role Overview
Serves as the core technical authority for designing, modernizing, and optimizing the Databricks Lakehouse architecture, code refactoring, and data execution pipelines.
Roles & Responsibilities
- Refactor legacy Synapse T-SQL stored procedures, scripts, and views into native PySpark, Databricks SQL, or Delta Live Tables (DLT)
- Transition legacy Synapse distribution schemas (HASH, ROUND_ROBIN, REPLICATE) to Delta Lake Liquid Clustering (CLUSTER BY) for optimal query performance and maintenance
- Design efficient batch and streaming ingestion patterns using Auto Loader and Delta Lake Lakehouse mechanisms
- Conduct performance tuning (OPTIMIZE, Z-ORDER, compute cluster sizing, Serverless warehouse setup)
Key Skills Required
- Designing end-to-end Target State Architectures Blueprint incorporating Databricks Lakehouse components (Unity Catalog, Delta Live Tables)
- Databricks: Databricks Runtime, Delta Lake, Delta Live Tables (DLT)
- Articulating design decisions, evaluating build vs. buy, and balancing performance with licensing costs
- Performance & Storage: Liquid Clustering, Partitioning Strategies, Auto Loader, DB SQL Serverless
- Languages: Python, PySpark, Advanced SQL, Scala (optional)
CI/CD & Tools: Databricks Asset Bundles (DABs), dbx, Git integration, MLflow
Databricks Architect · Diverse Lynx India