Data Engineer – Python with Databricks

Job Title: Data Engineer – Python with Databricks

Locations: New Jersey (NJ) & Boston, MA

Role Overview

We are looking for a Data Engineer skilled in Python and Databricks to join our team supporting critical data initiatives for State Street. You will design, build, and maintain scalable ETL pipelines and analytics solutions on the Databricks platform.

Key Responsibilities

Develop and maintain data pipelines and workflows in Databricks using Python

Ingest, cleanse, transform, and aggregate large datasets from disparate sources

Optimize Spark jobs for performance, scalability, and cost efficiency

Collaborate with data scientists, analysts, and business stakeholders to translate requirements into technical solutions

Implement data quality checks, monitoring, and alerting

Document data models, pipelines, and operational procedures

Participate in code reviews, agile ceremonies, and continuous improvement initiatives

Required Skills & Experience

Python: Strong programming skills for data processing and automation

Databricks & Apache Spark: Hands-on experience building and tuning Spark jobs

ETL & Data Modeling: Proven ability to design and implement robust ETL pipelines

SQL: Advanced query writing and performance tuning on large datasets

Cloud Platforms: Familiarity with Databricks deployments on AWS, Azure, or GCP

Version Control: Proficient with Git and CI/CD best practices

Agile Methodologies: Experience working in Scrum or Kanban environments