Specialist - Data Engineering

Job Summary

Seeking a Specialist with 5 to 7 years of experience in Scala and Apache Spark to design and implement scalable big data solutions within the Hadoop ecosystem

Job Description 

Extensive handson experience with Apache Spark for largescale data processing and analytics
Proficient in big data technologies within the Hadoop ecosystem Strong programming skills in Scala for data engineering tasks
Deep understanding of distributed computing frameworks and performance optimization techniques
Experience with batch and realtime data processing pipelines Knowledge of data ingestion transformation and storage best practices
Familiarity with data quality assurance validation and troubleshooting in big data environments 
Ability to work with complex data architectures and cloudbased big data platforms 
Skilled in debugging and optimizing Spark jobs to ensure efficient resource utilization 
Understanding of data security compliance and governance within big data solutions

Roles and Responsibilities 

Design develop and maintain scalable and robust data pipelines using Apache Spark SparkSQL Scala and related big data technologies 
Collaborate with cross functional teams to gather business requirements and translate them into technical solutions 
Optimize Spark applications for performance scalability and cost efficiency 
Lead the implementation of data processing workflows ensuring data quality and consistency 
Provide technical mentorship and guidance to junior team members and peers 
Participate in architectural reviews and recommend best practices for big data solution design 
Troubleshoot and resolve complex issues in data processing pipelines within defined SLAs 
Stay updated with emerging big data technologies and incorporate relevant advancements into existing systems 
Ensure adherence to organizational standards for data security compliance and governance