Senior Specialist - Data Engineering

Job Description :Databricks Engineer

We are seeking a highly skilled and experienced Databricks Engineer to join our growing team and play a crucial role in enhancing our data processing and analytics capabilities

As a Databricks Engineer you will be responsible for dealing with the most complex situations related to development design implementation and maintenance of our Databricks environment to ensure optimal performance scalability and reliability while aligning with our audit and security standards You will work closely with crossfunctional teams including data scientists analysts and software engineers to build robust data pipelines and deliver actionable insights

Key Responsibilities

1Databricks Environment Management

Handson experience with API

Set up and configure Databricks clusters to meet specific project requirements

Optimize Databricks configurations for performance and costeffectiveness

Ensure the security and integrity of the Databricks environment

2Data Pipeline Development

Collaborate with data scientists and analysts to understand data requirements

Design and implement scalable and efficient data processing pipelines using Databricks

Troubleshoot and optimize existing data pipelines for improved performance

3Integration and Collaboration

Work closely with crossfunctional teams to integrate Databricks solutions into the overall data architecture

Collaborate with software engineers to incorporate Databricks functionality into custom applications

4Monitoring and Maintenance

Implement robust monitoring and logging solutions for Databricks clusters

Conduct regular maintenance tasks including updates and patches to ensure system stability

Generate review consumption reports identify abnormal consumption and take corrective action

5Documentation and Training

Create and maintain comprehensive documentation for Databricks configurations and processes document best practices and standards to be followed

Provide training and support to internal teams on using Databricks effectively

Qualifications

Bachelors degree in Computer Science Engineering or a related field Masters Degree preferred

Proven experience as a Databricks Engineer or similar role

Strong proficiency in Apache Spark and Databricks

Experience with cloud platforms such as AWS Azure or Google Cloud

Proficient in programming languages such as Python Scala or Java

Solid understanding of data architecture and ETL processes

Preferred Qualifications

Databricks Certification

Experience with big data technologies and distributed computing

Familiarity with machine learning frameworks