Senior Databricks AI Engineer

Job Description: Senior Databricks AI Engineer

• Location: Atlanta, GA

• Duration: 6 Months

• Experience Required: 4–6 Years

• Role: Senior Databricks AI Engineer

Position Summary

• Design, develop, and deploy scalable data and AI solutions using the Databricks Lakehouse Platform.

• Apply expertise in Data Engineering, Machine Learning, Generative AI, and cloud technologies.

• Build enterprise-grade AI applications and data platforms.

• Develop scalable, secure, and efficient data pipelines and AI solutions.

Key Responsibilities

Data Engineering

• Design and develop scalable data pipelines using Databricks, PySpark, Delta Lake, and Spark SQL.

• Build ELT/ETL solutions for structured and unstructured data.

• Implement Medallion Architecture, including Bronze, Silver, and Gold layers.

• Optimize Databricks jobs, clusters, and workflows for performance and cost efficiency.

• Manage data ingestion from multiple cloud and on-premises sources.

• Design, develop, and maintain reliable data pipelines.

• Support data modernization initiatives and enterprise data platform development.

AI & Machine Learning

• Develop and deploy Machine Learning models using Databricks ML and Python.

• Implement MLOps practices for model training, deployment, monitoring, and governance.

• Build predictive analytics solutions and recommendation engines.

• Create feature stores and reusable ML pipelines.

• Implement AI model deployment and lifecycle management.

• Monitor and optimize machine learning models in production environments.

Generative AI

• Design and develop Generative AI applications using:

  • Large Language Models (LLMs).

  • Retrieval-Augmented Generation (RAG).

  • Vector Databases.

  • AI Agents and Multi-Agent Frameworks.

• Integrate various AI technologies to build enterprise AI solutions.

• Develop agentic AI applications and workflows.

• Collaborate with stakeholders to deliver AI-driven business solutions.

Essential Skills

• Strong expertise in Databricks and Data Engineering.

• Proficiency in Machine Learning and AI model deployment.

• Experience with cloud platforms such as Microsoft Azure or AWS.

• Familiarity with data pipeline design, development, and optimization.

• Strong knowledge of Generative AI technologies.

• Experience with PySpark, Spark SQL, and Delta Lake.

• Understanding of ETL/ELT processes and Medallion Architecture.

• Knowledge of AI model lifecycle management and deployment.

Role-Specific Requirements

• Design and develop scalable data pipelines using Databricks.

• Implement machine learning models and MLOps practices.

• Optimize data workflows for performance and cost efficiency.

• Collaborate with stakeholders to deliver AI-driven solutions.

• Develop Generative AI applications and integrate AI technologies.

Essential and Desirable Qualifications

• Experience in healthcare or banking domains is beneficial.

• Familiarity with AI governance and agentic AI is a plus.

• Knowledge of MLOps practices is advantageous.

• Understanding of data modernization programs is helpful.

• Databricks or Machine Learning certification is preferred.

Technical Skills / Keywords

• Digital: Machine Learning

• Digital: Cloud Foundry

• Digital: Databricks

• Digital: PySpark

• Generative AI

Desirable Skills

• Advanced Databricks and Lakehouse Platform knowledge.

• Experience with enterprise AI application development.

• Familiarity with AI governance, model monitoring, and responsible AI practices.

• Experience building RAG pipelines, vector search solutions, and AI agents.

• Knowledge of cloud-based data and AI architecture.