Databricks Data Engineer – Sales Data Platform

​Role: Databricks Data Engineer – Sales Data Platform
Location: Harrisburg, PA – Remote
 

Role Summary

We are seeking an experiencedDatabricks Data Engineer to design and implement a scalable Sales Data Platform using Databricks on AWS. This is a hands-on, architecture-driven role focused on data ingestion, transformation, modeling, and optimization using modern lakehouse architecture patterns.

The ideal candidate will collaborate with data engineers, source system teams, cloud platform teams, and downstream consumers to deliver high-quality, governed, and performance-optimized sales datasets.

Key Responsibilities

1. Architecture and Design

• Define end-to-end lakehouse architecture on Databricks running on AWS for sales data domains.

• Design Medallion Architecture (Bronze, Silver, and Gold layers) aligned with enterprise data standards.

• Establish data modeling standards for sales facts, dimensions, hierarchies, and aggregations.

• Define scalable ingestion patterns for batch and incremental data loads.

• Drive performance, scalability, and cost optimization best practices.

2. Data Engineering and Implementation

• Develop and guide the implementation of PySpark-based data pipelines using Databricks.

• Implement Delta Lake capabilities, including ACID transactions, schema evolution, schema enforcement, time travel, and versioning.

• Design and optimize large-scale joins, aggregations, and window functions.

• Implement Change Data Capture (CDC) and incremental processing using watermarking and change detection.

• Build idempotent, restartable, and fault-tolerant data pipelines.

• Develop and maintain data transformations using dbt, PySpark, and Spark SQL.

3. AWS and Platform Integration

• Design and implement data solutions using AWS services, including:

  • Amazon S3 for data lake storage

  • AWS IAM for security and access control

  • AWS Glue and Glue Data Catalog

  • Amazon CloudWatch for monitoring and logging

• Optimize Databricks cluster configurations, including job and all-purpose clusters.

• Implement secure connectivity, secrets management, and access-control patterns.

4. Data Quality, Governance, and Reliability

• Define and implement data quality checks, validations, and reconciliation processes.

• Ensure data lineage, metadata capture, and traceability.

• Implement error handling, auditing, and monitoring frameworks.

• Support enterprise data governance, security, and access-control requirements.

• Maintain data accuracy, consistency, and reliability across pipelines.

5. DevOps and Operational Excellence

• Implement CI/CD pipelines for Databricks notebooks, jobs, and data engineering workflows.

• Establish code versioning, code review, and deployment standards.

• Design monitoring, alerting, and service-level agreement (SLA) tracking for data pipelines.

• Support production stabilization, troubleshooting, and performance tuning.

• Optimize pipeline execution, cluster utilization, and operational efficiency.

6. Collaboration and Technical Leadership

• Provide technical leadership and mentorship to Databricks Data Engineers.

• Collaborate with Sales, CRM, ERP, source system, cloud platform, and analytics teams.

• Work with downstream data consumers and application teams to understand data requirements.

• Translate business requirements into scalable technical designs.

• Conduct technical design reviews and promote reusable engineering standards.

Required Skills and Qualifications

Core Technical Skills

• 8+ years of experience in Data Engineering, Data Architecture, or related roles.

• 4+ years of hands-on Databricks experience.

• Strong expertise in PySpark and Spark SQL.

• Strong hands-on expertise in dbt.

• Deep understanding of Delta Lake.

• Strong knowledge of AWS services, including S3, IAM, AWS Glue, and CloudWatch.

Data Engineering Expertise

• Experience working with sales data domains, including orders, revenue, pricing, customers, and products.

• Strong understanding of fact and dimension modeling.

• Experience with Slowly Changing Dimensions (SCD Type 1 and Type 2).

• Experience designing large-scale data processing solutions.

• Experience handling high-volume, high-velocity datasets.

• Expertise in batch and incremental data processing patterns.

• Experience implementing Medallion Architecture and scalable lakehouse solutions.

Platform and Operational Skills

• Experience with Databricks job orchestration and scheduling.

• Strong skills in cluster sizing, performance tuning, and cost optimization.

• Experience implementing CI/CD for data platforms.

• Strong troubleshooting, debugging, and root cause analysis skills.

• Understanding of data quality, governance, monitoring, and reliability practices.

Preferred Qualifications

• Experience with enterprise OneData or Data Mesh initiatives.

• Exposure to real-time or near-real-time data ingestion.

• Experience integrating CRM and sales systems, such as Salesforce or SAP sales data.

• AWS or Databricks certifications.

• Experience delivering large-scale enterprise data platform transformation programs.