VI
ETL Developer Kafka
VARITE INDIA PRIVATE LIMITED
- 🇮🇳 India
- On-site
- 10 months ago
- Java
- Scala
- Python
- AWS
- Apache Kafka
- EC2
- CloudWatch
- Kafka Connect
- Kafka Streams
- ACLS
- Apache Spark
- AWS Lambda
- ETL
- IAM
- VPC
- ELT
- Snowflake
- SQL
- Perl
- Core Data
- Azure
- Airflow
- Agile
- Kanban
- Scrum
- Docker
- Kubernetes
- Devops
- Git
- CI/CD
10 months ago
About The Client:
Client is one of the world’s leading professional services firms and the fastest growing Big Four accounting firm
Essential Job Functions:
- Design, implement, and maintain Kafka producers, consumers, and stream processing applications using languages like Java, Scala, or Python.
- Deploy, manage, and optimize Kafka clusters and related applications on AWS services such as Client Managed Streaming for Apache Kafka (MSK), EC2, S3, Lambda, and CloudWatch.
- Develop and manage end-to-end data pipelines involving Kafka Connect, Kafka Streams, and other data integration tools
- Ensure the performance, scalability, and reliability of Kafka-based systems, including cluster tuning, monitoring, and troubleshooting.
- Implement security best practices for Kafka on AWS, including authentication, authorization (ACLs), and data encryption.
- Manage Kafka Schema Registry for data serialization and evolution
- Develop real-time stream processing applications using Apache Spark Streaming, Kafka Streams, or AWS Lambda
- Implement complex event processing (CEP) patterns for real-time analytics
- Experience: 8+ years
- 8+ years of experience in Data warehouse and Data Lake Implementations as a Data Engineer/ETL Developer
- Deep understanding of Kafka architecture, concepts (topics, partitions, clustering,brokers), and related tools (Kafka Connect, Kafka Streams, Schema Registry).
- Strong experience with relevant AWS services for data streaming and infrastructure management (e.g., MSK, EC2, S3, Lambda, IAM, VPC).
- Expertise in one or more programming languages commonly used with Kafka, such as Java, Scala, or Python.
- Strong knowledge and experience in Data Lake and Data Warehouse architecture and concepts in cloud
- Expert in Data Ingestion (ETL and ELT), Dimension Modeling, Data Quality and Data Validation
- Strong working experience in Snowflake using warehouses, stored procedures, streams, snow pipes, tasks, stages, storage integration, ingestion frameworks and tools etc.
- Ability to develop ELT/ETL pipelines to move data to and from Snowflake data store using combination of Python, Advanced SQL and Snowflake SnowSQL.
- Strong hands-on experience with requirements gathering analysis, coding, testing, implementation, maintenance, and review.
- Extensive development experience using Advanced SQL, Python and other scripting languages (Perl, shell etc.)
- Experience in working with core data engineering services in AWS, MS Azure, or other cloud providers.
- Good hands-on experience on workflow management tools such as Airflow, Control M etc.
- Solid experience with application Support & resolving production issues.
- Great customer support skills and adaptable to changing business needs.
- Good project management, experience in Agile methodologies (Kanban and SCRUM), communication, and interpersonal skills.
- Lead project efforts with the technology solutions to perform proof of concept (POC) analysis.
- Experience with containerization technologies (e.g., Docker, Kubernetes)
- Experience with DevOps, Git, CI/CD
- Experience directly managing a team in an onsite/offshore development environment.
- Experience in Service Now Data is preferred.
- Strongly preferred to have experience: 3 to 5 years of experience working with Kafka platforms in below areas
Unlock Rewards: Refer Candidates and Earn.
If you're not available or interested in this opportunity, please pass this along to anyone in your network who might be a good fit and interested in our open positions. VARITE offers a Candidate Referral program, where you'll receive a one-time referral bonus based on the following scale if the referred candidate completes a three-month assignment with VARITE.
Experience Level Bonus Referral:
| 0-2 years | INR 5,000 |
| 2-6 years | INR 7,500 |
| 6+ years | INR 10,000 |
About VARITE: VARITE is a global staffing and IT consulting company providing technical consulting and team augmentation services to Fortune 500 Companies in USA, UK, CANADA and INDIA. VARITE is currently a primary and direct vendor to the leading corporations in the verticals of Networking, Cloud Infrastructure, Hardware and Software, Digital Marketing and Media Solutions, Clinical Diagnostics, Utilities, Gaming and Entertainment, and Financial Services.
Equal Opportunity Employer:
VARITE is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, marital status, veteran status, or disability status.
ETL Developer Kafka · VARITE INDIA PRIVATE LIMITED