Databricks Platform Engineer

Job Title: Databricks Platform Engineer
Domain: Pharma
Location: East Coast Preferred (NJ Preferred)

Job Summary

We are seeking a highly skilled Databricks Platform Engineer to join an existing engineering team. This role focuses on designing, optimizing, and maintaining the Databricks platform and architecture to enable development teams to efficiently build and deploy data solutions.

The ideal candidate will act as a platform expert, ensuring scalability, reliability, governance, and performance of the Databricks ecosystem while supporting developers with best practices, tools, and frameworks.

Must-Have Skills

• 5+ years of experience in Data Engineering or Platform Engineering
• 3+ years of hands-on Databricks experience:
  • Architecture
  • Administration
  • Platform Engineering
• Apache Spark / Distributed Data Processing
• Cloud Platforms (Azure Preferred, AWS/GCP acceptable)
• Delta Lake & Data Lake Architecture
• CI/CD & DevOps Practices
• Terraform / Infrastructure as Code
• Governance & Security Frameworks

Key Responsibilities

Platform Engineering & Architecture

• Design, implement, and maintain Databricks platform architecture for enterprise-scale workloads
• Establish best practices for:
  • Workspace setup
  • Cluster management
  • Job orchestration
• Optimize platform performance, scalability, and cost efficiency
• Define and enforce standards for data engineering and analytics workloads

Developer Enablement

• Support development teams with guidance, documentation, and reusable frameworks
• Enable developers using:
  • CI/CD pipelines
  • Automation
  • Deployment standards
• Troubleshoot and resolve platform-related issues
• Provide technical mentorship and onboarding support

Governance, Security & Compliance

• Implement and manage:
  • Unity Catalog
  • RBAC
  • Access controls
  • Data governance
• Ensure compliance with enterprise and regulatory policies
• Monitor and audit:
  • Platform usage
  • Access patterns
  • Data lineage

Integration & Ecosystem Management

• Integrate Databricks with:
  • AWS / Azure / GCP
  • Data lakes
  • Enterprise systems
• Manage integrations with:
  • Power BI
  • APIs
  • Data ingestion frameworks
  • Orchestration tools
• Support batch and streaming ingestion frameworks using Delta Lake

Operational Excellence

• Monitor platform health, performance, and usage metrics
• Implement automation for:
  • Provisioning
  • Monitoring
  • Scaling
• Drive continuous improvements in reliability and developer experience

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience)
• 5+ years in Data Engineering or Platform Engineering
• 3+ years of hands-on Databricks experience
• Strong understanding of Apache Spark and distributed data processing
• Experience with Azure preferred (AWS/GCP also acceptable)
• Knowledge of:
  • Data Lake Architecture
  • Delta Lake
  • Modern Data Platforms
• Experience implementing:
  • Security frameworks
  • Governance models
  • Access controls
• Familiarity with:
  • CI/CD pipelines
  • DevOps practices
  • Terraform / Infrastructure as Code