NC
Data Engineer – GCP Data Products Team
NR Consulting - India
🇮🇳 India
On-site
3 days ago
- GCP
- Flink
- ELT
- ETL
- NoSQL
- BigQuery
- Redshift
- Azure
- HBase
- MongoDB
- Apache Pulsar
- Pub/Sub
- Composer
- Airflow
- CI/CD
- Kubernetes
- Java
- Python
- Scala
- AWS
- SQL
- Devops
- IAM
- Snowflake
3 days ago
Location: PAN India
Exp: 10+ Yrs
Job Description:
Key Responsibilities
• Design and deliver end to end data pipelines on cloud platforms (GCP preferred).
• Build scalable data ingestion, transformation, and processing workflows using distributed technologies such as Spark, Flink, Storm, or similar.
• Develop robust ELT/ETL processing and migration pipelines, including support for legacy Datastage decommissioning and modernisation.
• Work with a variety of database technologies including relational, NoSQL, MPP and columnar stores (BigQuery, Redshift, Azure SQLDW, HBase, MongoDB).
• Implement streaming and messaging based pipelines using Kafka, Pulsar or Pub/Sub.
• Build optimised, scalable data models to support diverse consumption patterns, applying partitioning, sharding, bucketing and aggregation strategies.
• Apply performance tuning and optimisation across storage, compute and query layers.
• Ensure secure handling of data including authentication, authorisation, encryption in transit/at rest, and cloud native security controls.
• Implement monitoring, alerting and observability for large scale distributed data workloads.
• Use orchestration tools such as Cloud Composer, Airflow or equivalent to operationalise pipelines.
• Contribute to CI/CD, containerisation, Kubernetes-based deployments, and automated testing practices.
• Participate in data governance, metadata, catalogue and lineage processes as needed.
• Collaborate with engineers, architects and SMEs to deliver stable, high quality data products.
• Required Skills & Experience
• Strong programming skills in Java (preferred), Python, or Scala.
• Hands on experience with cloud data services (GCP preferred; Azure/AWS acceptable).
• Practical experience with distributed data processing frameworks such as Spark (Core/SQL/Streaming), Flink, or Storm.
• Experience across multiple database technologies—Relational, NoSQL, MPP, columnar.
• Strong knowledge of data ingestion, transformation and messaging systems: Kafka, Pulsar, Pub/Sub, etc.
• Understanding of designing scalable data models for varied access patterns.
• Experience with performance tuning, cost optimisation and scaling strategies.
• Experience delivering large scale big data solutions in batch and/or streaming environments, on cloud or on premise.
• Good familiarity with the wider data ecosystem and open source frameworks.
• Experience with orchestration (Airflow/Composer) and workflow automation.
• Understanding of DevOps for data systems: CI/CD, containers, Kubernetes and automated testing.
• Knowledge of security for big data systems including IAM, encryption, and cluster level controls.
• Basic understanding of monitoring and alerting for distributed systems.
• Knowledge of dimensional modelling (star, snowflake, normalized/denormalized).
• Awareness of data governance, cataloguing and lineage tools.
Data Engineer – GCP Data Products Team · NR Consulting - India