EI
Databricks Platform Engineering Lead
Echo IT Solutions, Inc
Location not stated
Staff / Principal
1 day ago
- Databricks
- ETL
- ELT
- Azure Databricks
- Unity Catalog
- Apache Spark
- PySpark
- Python
- SQL
- Delta Lake
- Azure Data Factory
- IaC
- Terraform
- CI/CD
- Secrets Management
- Azure
- Incident Management
- Devops
- Agile
- Risk Management
1 day ago
Warren, NJ 3 days hybrid
Mode of interview : 2 videos
Duration : 6+ months
Project Overview
We are seeking an experienced Databricks Platform Engineering Lead to support an enterprise-wide pricing application within a global reinsurance organization. The application leverages Databricks as its data and analytics platform and integrates with a SQL Server-based pricing application.
The resource will lead the engineering, governance, and ongoing enhancement of the Databricks platform supporting the pricing application. The role requires a strong combination of hands-on platform engineering, Databricks administration, data engineering, and technical delivery leadership.
The ideal candidate must be capable of managing the Databricks platform foundation while providing technical oversight to data-engineering teams building ETL/ELT pipelines, transformations, and pricing data products.
Technical and Delivery Expertise
• 10+ years of overall technology experience.
• 5+ years of hands-on experience with Databricks, cloud data platforms, or platform engineering.
• 3+ years of experience in a Platform Lead, Databricks Lead, Cloud Platform Engineer, Data Platform Lead, or similar role.
• Strong hands-on experience with:
o Azure Databricks
o Databricks account and workspace administration
o Unity Catalog
o Apache Spark and PySpark
o Python and SQL
o Delta Lake and Lakehouse architecture
o Azure Data Lake Storage
o Azure Data Factory or comparable integration tools
o Databricks Workflows and job orchestration
o Infrastructure as code using Terraform
o CI/CD and automated deployment practices
• Strong understanding of
o Databricks workspace provisioning and administration
o Development, test, and production environment management
o Compute provisioning and cluster configuration
o Cluster policies, pools, runtimes, and serverless compute
o Platform security, identity, and access management
o Service principals, managed identities, and secrets management
o Azure networking, private endpoints, and secure connectivity
o Unity Catalog architecture, permissions, lineage, and governance
o Platform monitoring, logging, auditability, and observability
o Databricks performance, capacity, and cost optimization
o Production support and platform-incident management
• Good understanding of data-engineering practices, including:
o Batch and streaming data ingestion
o ETL/ELT pipeline development
o Medallion architecture
o Data transformations and enrichment
o Data quality, validation, and reconciliation
o Schema management and data lineage
o Spark workload and query optimization
• Experience leading globally distributed platform, data-engineering, application, QA, DevOps, and production-support teams.
• Ability to participate in architecture reviews, solution-design discussions, code reviews, security assessments, and production-readiness evaluations.
• Strong understanding of Agile delivery, release management, risk management, dependency management, and stakeholder governance.
Databricks Platform Engineering Lead · Echo IT Solutions, Inc