Senior Azure Databricks Platform Engineer

Job Title:Azure Databricks Engineer
Location: Dallas, TX 75202
Work Arrangement: Onsite
Job Type: 12 months Contract
Interview: Three rounds - Video and in person interview

Job Description:

We are seeking an experienced Azure Databricks Engineer to join our team in Dallas, TX. The ideal candidate has strong hands-on experience in designing and building scalable data engineering solutions using Azure Databricks, PySpark, Python, SQL, and Azure Data Services — including leading the modernization and migration of existing Python object-oriented applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks.

Key Responsibilities
•     Design, develop, and implement scalable data pipelines using Azure Databricks.
•     Build and optimize ETL/ELT pipelines using PySpark, Python, and SQL.
•     Develop data solutions using the Medallion Architecture (Bronze, Silver, and Gold layers).
•     Work with Delta Lake, Delta tables, and advanced Databricks optimization techniques.
•     Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault.
•     Develop and manage Databricks Workflows, Jobs, and Notebooks.
•     Implement data quality, monitoring, error handling, and performance optimization.
•     Collaborate with Data Architects, Data Engineers, Data Scientists, and business stakeholders.
•     Establish best practices for CI/CD, Git/version control, code reviews, and automated deployments.
•     Lead technical discussions and provide guidance to junior engineers.
•     Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution.
•     Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
•     Implement Delta Lake solutions using the Bronze, Silver, Gold architecture.
•     Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
•     Implement data quality, reconciliation, validation, and monitoring frameworks.
•     Optimize Spark jobs using partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, and Delta optimization.

Required Skills
•     10+ years of overall Data Engineering experience, including designing and implementing ETL/ELT pipelines with Azure Data Factory and other Azure 
        services.
•     Strong hands-on experience with Azure Databricks; 4+ years of experience across Azure services and Databricks (ADLS, ADF, Azure DevOps, etc.).
•     7+ years of Python development experience, with the ability to design and build reusable libraries.
•     4+ years of experience with Snowflake or SQL (No-SQL experience is a plus).
•     Expert-level knowledge of PySpark and Spark SQL.
•     Strong programming experience in Python and SQL.
•     Experience with Delta Lake and Lakehouse architecture.
•     Strong experience with Azure Data Lake Storage (ADLS Gen2).
•     Experience with Azure Data Factory (ADF) and other Azure data services.
•     Strong understanding of ETL/ELT, data modeling, and large-scale data pipelines.
•     Experience with performance tuning and optimization in Databricks/Spark.
•     Experience with Git, CI/CD, and DevOps practices.

Preferred Skills
•     Databricks certifications.
•     Experience with Unity Catalog and Databricks governance/security.
•     Experience with Terraform or Infrastructure as Code.
•     Knowledge of Azure DevOps.
•     Experience working in a consulting or client-facing environment.
•     Experience designing APIs and integrating with React JS within a cloud platform.