AP
Senior Data Engineer
Atlantic Partners Corporation
- Location not stated
- Senior
- 21 hours ago
- System Design
- Snowflake
- dbt
- CI/CD
- Data Architecture
- Azure
- SQL
- Python
- Git
- Secrets Management
- PostgreSQL
- IaC
- Machine Learning
- AI
- Equity
21 hours ago
About the Role
The Senior Data Engineer will be the core engineer of the firm's data engineering team, building pipelines, reusable data models, and the infrastructure that enables clean, reliable, and fast data delivery for business and data science needs. This player-coach role will help establish the engineering practices that turn data from enterprise systems and external sources into trusted, production-ready datasets. You'll work closely with data scientists, analysts, business stakeholders, and engineering partners to deliver scalable solutions across multiple business domains. The role combines practical delivery with strong system design, ownership of pipeline reliability, and a focus on data quality, performance, security, and maintainability. This role would manage a growing team of junior data engineers who are responsible for developing and maintaining different pipelines across the firm. This position description outlines the basic tasks and requirements for the position noted. It is not a comprehensive listing of all job duties.
Key Responsibilities
• Design and implement automated ingestion and transformation pipelines that integrate data from APIs, enterprise applications, files, and third-party providers into Snowflake.
• Architect, test, and lead new engineering capabilities that address a firmwide data need, involving combinations of OLAP/OLTP and structured/unstructured data storage.
• Develop reusable data models and dbt transformations that standardize business logic, support consistent metrics, and provide trusted datasets for reporting, analytics, and data science.
• Support real-time and batch data needs across multiple business domains, selecting appropriate ingestion patterns, refresh schedules, and orchestration approaches.
• Maintain high data quality through automated testing, source-to-target reconciliation, validation rules, and clear processes for resolving data issues with source owners.
• Implement observability and monitoring for data freshness, completeness, pipeline failures, and performance; establish actionable alerts and troubleshoot production issues through resolution.
• Optimize Snowflake workloads and pipeline design to improve query performance, processing speed, scalability, and cost efficiency.
• Apply strong software engineering practices, including version control, code review, automated testing, CI/CD, and repeatable deployments across development and production environments.
• Collaborate with data scientists and project managers to translate business requirements into production-ready datasets with clear definitions, documented lineage, and agreed delivery expectations.
• Partner with Data Architecture, Platform Engineering, and Cybersecurity to implement secure access, appropriate handling of sensitive data, and governed data delivery.
• Own pipelines through deployment and ongoing support, including dependency management, retries, recovery, historical backfills, and changes to source schemas.
• Document data models, architecture decisions, dependencies, and operational procedures; contribute reusable engineering patterns and provide technical guidance to other team members.
Requirements
• 7–10+ years of experience in data engineering, including at least 3 years working with Snowflake, dbt, and unstructured storage (S3, Azure Blob, etc.) in production.
• 2+ years of experience managing distributed data engineering teams, mentoring junior talent, and providing actionable feedback to develop the capabilities of other data engineers.
• Advanced SQL and Python skills, with experience building maintainable transformation logic, ingestion services, and automation for production data workflows.
• Strong software engineering habits, including Git-based version control, code review, automated testing, CI/CD, and disciplined release management.
• Hands-on experience designing and operating Snowflake data pipelines and dbt projects, including reusable models, incremental processing, testing, documentation, and performance tuning.
• Experience implementing observability, monitoring, and pipeline reliability practices, including alerting, incident investigation, recovery, and data quality checks.
• Strong problem-solving and system design skills, with the ability to assess tradeoffs across scalability, latency, reliability, maintainability, and cost.
• Experience with workflow orchestration and both batch and real-time or near-real-time data integration, including APIs, incremental ingestion, and changing source schemas.
• Solid understanding of dimensional modeling, warehouse design, data lineage, and the development of datasets that support business analytics and data science.
• Ability to work closely with cross-functional stakeholders, clarify ambiguous requirements, and explain technical decisions and data limitations in plain language.
• Experience with cloud data infrastructure and enterprise data security, including access controls, secrets management, and appropriate handling of sensitive information.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical field, or equivalent practical experience.
Preferred Qualifications
• Experience in financial services, real estate, private equity, asset management, or another environment with complex data and governance requirements.
• Experience scaling a growing data platform or engineering team, including shared standards, reusable components, and operational processes.
• Familiarity with Microsoft Azure technical stack, OLTP databases (Postgres), infrastructure as code, and delivering data for machine learning/AI applications.
Senior Data Engineer · Atlantic Partners Corporation