Data Engineer

Job Description: Data Engineer – Datastore Migration

Role Overview

The Engineer will be part of the Datastore-Migration Factory team, responsible for end-to-end datastore migration from an on-premises Data Lake to an AWS-hosted Lakehouse. This is a high-visibility and crucial project for Client.

Key Responsibilities

1. Pipeline Migration

• Logic and Scheduling: Refactor and migrate extraction logic and job scheduling from legacy frameworks to the new Lakehouse environment.

• Data Transfer: Execute the physical migration of underlying datasets while ensuring data integrity.

• Stakeholder Engagement: Act as a technical liaison to internal clients, facilitating hand-off and sign-off discussions with data owners to ensure migrated assets meet business requirements.

2. Consumption Pattern Migration

• Code Conversion: Translate and optimize legacy SQL and Spark-based consumption patterns (raw and modeled) for compatibility with Snowflake and Iceberg.

• Usage Analysis: Understand usage patterns to deliver the required data products.

• Stakeholder Engagement: Act as a technical liaison to internal clients, facilitating hand-off and sign-off discussions with data owners to ensure migrated assets meet business requirements.

3. Data Reconciliation and Quality

• Apply a rigorous approach to data validation using reconciliation frameworks to ensure migrated data is functionally equivalent to data already used in production workflows.

• Work with the internal Data Management Platforms team.

• Demonstrate an aptitude for learning new workflows and language constructs as necessary.

Technical Skills

1. Basic Qualifications

• Education: Bachelor's or Master's degree in Computer Science, Applied Mathematics, Engineering, or a related quantitative field.

• Experience: Minimum of 3–5 years of hands-on coding experience in a collaborative, team-based environment.

• Troubleshooting: Ability to troubleshoot SQL and perform basic scripting.

• Programming Languages: Professional proficiency in Python or Java.

• Methodology: Deep familiarity with the full Software Development Life Cycle (SDLC), CI/CD best practices, and Kubernetes (K8s) deployment experience.

2. Core Data Engineering Competencies

Candidates must demonstrate a sophisticated understanding of the following modeling concepts to ensure data correctness during reconciliation:

• Temporal Data Modeling: Managing state changes over time, such as Slowly Changing Dimension (SCD) Type 2.

• Schema Management: Expertise in schema evolution, referencing Apache Iceberg, and schema enforcement strategies.

• Performance Optimization: Advanced knowledge of data partitioning and clustering.

• Architectural Theory: Understanding the balance between normalization and denormalization and the strategic use of natural versus surrogate keys.

3. Technical Stack Requirements

Candidates are not expected to be experts in every tool, but the collective team must cover the following technologies.

Extraction and Logic

• Kafka

• ANSI SQL

• FTP

• Apache Spark

Data Formats

• JSON

• Avro

• Parquet

Platforms

• Hadoop (HDFS/Hive)

• Snowflake

• Apache Iceberg

• Sybase IQ

Core Competencies

• Integrity and Ethics: Demonstrates strong integrity and consistently models good conduct and ethical decision-making.

• Teamwork and Collaboration: Acts as a trusted team player who collaborates effectively across multiple teams and functions.

• Communication: Communicates with clarity and confidence through concise written updates, structured verbal briefings, and proactive stakeholder management.

• Global Collaboration: Works effectively with global teams across time zones and cultures, builds alignment, and resolves issues constructively.

• Ownership and Delivery: Maintains a strong sense of ownership, drives work to closure, and meets commitments.

• Urgency and Quality: Brings high energy and urgency to achieve targets while maintaining quality and professionalism.

• Continuous Improvement: Shows intellectual curiosity, asks thoughtful questions, surfaces risks early, and seeks feedback to continuously improve.