Specialist - Data Engineering

Job Summary

Seeking a Specialist with 5 to 7 years of experience skilled in Scala and Apache Spark to design and maintain scalable big data solutions within the Hadoop and Spark ecosystem

Job Description

Design develop and maintain scalable data processing applications using Scala and Apache Spark Collaborate with crossfunctional teams to gather requirements and translate them into efficient big data solutions Implement data ingestion transformation and aggregation pipelines leveraging Spark and SparkSQL Ensure high performance reliability and scalability of data processing workflows Optimize existing data processing jobs for improved efficiency and resource utilization Stay updated with emerging technologies and best practices in big data and functional programming

Roles and Responsibilities

Lead the development and deployment of complex big data applications using Apache Spark and Scala frameworks

Mentor junior team members and provide technical guidance on Scala and big data technologies

Collaborate with data engineers architects and analysts to design endtoend data solutions

Troubleshoot and resolve performance bottlenecks in distributed data processing jobs

Develop and maintain documentation related to architecture design and code

Drive innovation by evaluating and integrating new tools and frameworks within the Hadoop and Spark ecosystem

Ensure adherence to coding standards security policies and compliance requirements