DL
GCP Data Engineer (PySpark & Python) - (Wells Fargo)
Diverse Lynx India
🇮🇳 India
On-site
1 month ago
- GCP
- PySpark
- Python
- SQL
- CI/CD
- Pub/Sub
- Composer
- Airflow
- Cloud Functions
- MongoDB
- JSON
- Parquet
- Apache Avro
- XML
- BigQuery
- Apache Airflow
- Git
- Devops
- Data Modeling
- ETL
- ELT
- Agile
- Scrum
1 month ago
Looking for highly skilledGCP data engineer with 5-7 years of experience in designing, developing and supporting end to end data engineering solutions with Pyspark, Python and SQL skills. The candidate should have strong expertise in both batch & real time data processing, modern CI/CD practices, excellent communication skills and the ability to work directly with client stakeholders.
**Required Skills & Experience:**
- Hands on experience with Google cloud platform (GCP) services including Big Query, Cloud Storage (GCS). Pub/Sub, Data Flow (Apache beam), Cloud composer (Airflow) & DAG development, Cloud functions, Dataproc, Cloud logging and monitoring.
- Strong experience in design, develop and implementing end-end data engineering pipelines
- Experience in real time streaming using Kafka, Pub/sub and data flow.
- Experience in batch ingestion from MongoDB, relational databases, files (CSV, JSON, Parquet, Avro, XML), APIs and other enterprise data sources.
- Strong programming skills in Pyspark, Python & SQL.
- Good understanding of BigQuery optimization, partitioning, clustering, and performance tuning
- Experience with Apache Airflow/Cloud composer, Workflow orchestration and DAG development
- Hands on experience with GIT, CI/CD pipelines, GitHub copilot, automated deployment, release management and DevOps practices.
- Experience with testing, debugging, monitoring, trouble shooting and production support of data pipelines
- Experience on data modeling, ETL/ELT, data quality, and data governance.
- Having Google Professional Data Engineer certification is an added advantage
- Excellent verbal and written communication skills
- Ability to communicate effectively with clients and stakeholders, providing regular status updates and driving technical discussions
- Proactive mindset with strong ownership and accountability
- Ability to identify risks early and recommend innovative and scalable solutions
- Experience working in Agile/scrum teams and collaborating with cross-functional teams"
GCP Data Engineer (PySpark & Python) - (Wells Fargo) · Diverse Lynx India