Sr. Data Engineer

Senior Data Engineer

Key Responsibilities

· Designs, develops, optimizes, and maintains data architecture and pipelines that adhere to Extraction, Transformation, Loading (ETL) principles and best practices using Azure Databricks (PySpark)Works to identify business problems that can be solved and product features that can be enhanced by data engineering and analytics

· Provides subject matter expertise as a product team member, for identification, estimation, and implementation of data solutions to business use cases

· Collaborates with data analysts, key data consumers and application developers to optimize models and algorithms for data quality, security, and governance

· Solicits, understands, and ensures compliance with data governance and data security requirements while creating and improving integrated and reusable data pipelines, to enable faster data access and improved time-to-solution for data and analytics initiatives

· Leads engineering communities of practice (CoP) to expand influence and quality across the digital organization

· Designs end to end DataOps and analytical workflows, including overall architecture, capabilities, platforms, tools, and governing processes

· Mentors' data team members to grow capabilities and mitigate product delivery risks

Skills and Experience

· Bachelors degree and 5 years' experience in a Data Engineering role, in a healthcare setting (required) or 7 years of related experience in lieu of degree. (required)

· Master's degree Computer Science, Statistics, Applied mathematics, or information systems and 6 years of related experience or 11 years of related experience and no degree. (preferred)

Minimum Years of Experience

· 5 years' experience in a data engineering or architecture role (required)
5 years' experience designing, building, deploying, and troubleshooting data extract and load jobs (required)

· 5+ years' experience in building cloud data pipelines using Azure Databricks (PySpark) and Azure data factory

· 3 years' experience building DB objects, authoring, and troubleshooting SQL queries, stored procedures and ETL processes (required)5 years' experience writing complex T-SQL queries, strong knowledge of DML, DDL, triggers, common table expressions (CTE), query tuning and performance optimization (required)

· 3 years' experience foundational data management practices, including:

o Implement Delta Lake architectures, including medallion data models (bronze/silver/gold layers)

o Demonstrated ability to design, build, manage and automate pipelines for data structures encompassing data transformation, data models, schemas, metadata, and workload management

o Build reusable frameworks for metadata‐driven ingestion, orchestration, and error handling.

· 3 years' experience working with DevOps and continuous integration capabilities for version control, build automation, testing, and release management (e.g., Visual Studio, Git, Jenkins, Azure DevOps) (required)

Other Knowledge, Skills and Abilities Required

· License/Certification - Azure Data Engineer Associate (preferred)

Additional Details

· 1 year experience working with C#, PowerShell, Azure functions and logic apps, Azure Key Vault, .Net core, Azure PaaS (preferred)

· 3 years' experience Exposure to hybrid deployments, cloud and on premise, with demonstrated ability to work across multiple deployment environments, operating systems (preferred)