Principal Data Engineer for Databricks

Principal Data Engineer for Databricks

Location: Warren, NJ. (2-3 days in-office)

Key Responsibilities

• Lead the architecture, design, and delivery of scalable enterprise data platforms using Azure Databricks, Delta Lake, and lakehouse patterns.
• Build and optimize reliable batch and real-time data pipelines for analytics, reporting, and AI workloads.
• Establish reusable engineering frameworks, coding standards, CI/CD practices, and performance and cost-optimization strategies.
• Implement data governance, security, lineage, and quality controls, including Unity Catalog capabilities.
• Provide technical leadership, mentor engineering teams, and collaborate with business, architecture, analytics, and data science stakeholders.

Key Skills

• Azure Databricks and Lakehouse: Databricks platform, Delta Lake, Unity Catalog, and enterprise lakehouse architecture.
• Data Engineering: Apache Spark, PySpark or Scala, advanced SQL, ETL/ELT design, and distributed processing.
• Azure Data Services: Azure Data Factory, ADLS Gen2, Synapse Analytics, Key Vault, Monitor, and Log Analytics.
• Architecture and Governance: Data modeling, security, data quality, lineage, scalability, and performance optimization.
• Engineering Leadership: Azure DevOps, CI/CD, Agile delivery, stakeholder management, mentoring, and technical roadmap development.