Lead Data Software Engineer

We are building cloud-scale data pipelines and platforms as aLead Data Software Engineer to enable analytics, reporting, and AI-driven initiatives. You will collaborate with Product, Engineering, and Business teams to deliver reliable solutions using Azure Databricks, Spark, Kafka, and modern governance practices.

Responsibilities

• Design scalable batch and streaming data pipelines
• Build and fine-tune Databricks, Spark, and Kafka-based solutions
• Implement data quality, governance, and observability standards
• Develop data models and reporting foundations for analytics and AI initiatives
• Collaborate with Product, Engineering, and Business teams to deliver data-driven solutions
• Support enterprise reporting, dashboards, and self-service analytics platforms
• Leverage AI tools such as Claude Code, GitHub Copilot, and similar platforms for development, testing, and productivity improvements
• Apply MCP, prompt engineering, and multiple LLMs in AI-assisted development workflows

Requirements

• Proven track record with 5+ years of experience in data engineering or a related field
• Hands-on proficiency in SQL, Python, PySpark, Scala, and Java
• Deep expertise in Apache Spark, Spark SQL, Kafka, and Delta Lake
• Strong experience with Azure Databricks, Unity Catalog, and Databricks Asset Bundles (DAB)
• Solid familiarity with Azure Data Factory (ADF), ADLS Gen2, Azure Key Vault, Azure Cosmos DB, and Azure Monitor
• Thorough knowledge of Data Modeling, Data Quality, Data Governance, and PII Handling
• Demonstrated background in Medallion Architecture (Bronze/Silver/Gold), CDC, and SCD Type 1 & Type 2
• Practical competency in Azure DevOps, Git, CI/CD, and Gradle
• English proficiency at B2+ (Upper-Intermediate) level

Nice to have

• Familiarity with Tableau, ThoughtSpot, Cognos, and Data Visualization Best Practices
• Knowledge of Semantic Layer Concepts, AI/BI Dashboards, and Databricks Genie
• Understanding of scalable data platforms, analytics, and reporting solutions