Specialist - Data Engineering

Key Responsibilities

Design and implement complex data pipelines using Dataflow Apache Beam and Dataproc Spark

Lead Cloud Composer Airflow orchestration design optimizing DAG structure dependencies and retry strategies

Drive advanced Python and SQL implementations including reusable frameworks and utility components

Build and standardize custom Dataflow Beam templates and reusable pipeline patterns

Perform deep performance tuning for BigQuery query optimization partitioning strategies cost optimization

Troubleshoot complex pipeline failures and performance bottlenecks across processing and orchestration layers

Review code and design artifacts produced by engineers provide technical guidance and mentoring

Own technical quality for assigned modules including nonfunctional aspects performance reliability scalability

Act as the first point of escalation for technical issues before involving architects

Collaborate with architects to understand design intent and ensure correct implementation

Mandatory Technical Skills

Advanced GCP Knowledge

Strong understanding of IAM strategy service isolation and projectlevel design considerations

Programming Querying

Advanced proficiency in Python and complex GoogleSQL

Orchestration

Strong handson expertise with Cloud Composer Airflow including best practices and optimization

Data Processing

Deep handson experience with Dataflow and Apache Beam

Strong working knowledge of Dataproc Spark internals

Analytics Platform

Advanced experience with BigQuery including performance and cost optimization techniques

Good to Have

Exposure to streaming data pipelines and PubSubbased ingestion

Familiarity with CICD Gitbased version control and basic DevOps practices

Understanding of data quality checks logging and monitoring patterns

Exposure to AgileScrum delivery models

GCP certifications such as Associate Cloud Engineer or Professional Data Engineer nice to have