Senior Data Software Engineer/ AI Agents, Azure, Spark

We are looking for an experiencedSenior Data Software Engineer with strong hands-on expertise inAzureandthe Spark ecosystem, primarily focused on building and maintaining data transformation pipelines.

The candidate should be comfortable with any Spark-adjacent technology (PySpark, Spark SQL, Scala Spark, Databricks, Synapse, etc.) rather than being locked into one specific flavor. Experience with Microsoft Fabric is a strong plus but not a requirement.

The ideal candidate combines great technical skills with leadership capability, contributing to architecture, design, development, and mentoring of engineering teams — preferably in complex enterprise or financial services environments.

Responsibilities

• Lead the design, development, and optimization of scalable data engineering solutions on Azure, using Spark-based processing (PySpark, Spark SQL, Scala, or equivalent)
• Own end-to-end data transformation pipelines, including ingestion, transformation, storage, and analytics
• Work with Azure-native data services such as Data Factory, Databricks, and Synapse
• Support high-performance data access patterns using Cosmos DB (NoSQL API) where applicable
• Collaborate with data scientists, AI engineers, and product stakeholders to enable data-driven analytics and insights
• Mentor and guide junior engineers, setting coding standards and best practices
• Ensure data quality, security, governance, and performance across platforms
• Contribute to technical decision-making and solution architecture discussions

Requirements

• 3+ years of experience in data engineering roles, preferably within complex enterprise or financial services environments
• Expertise in Azure cloud data services, including Data Factory, Databricks, and Synapse
• Proficiency in the Spark ecosystem, including PySpark, Spark SQL, and Scala Spark
• Background in designing and maintaining end-to-end data transformation pipelines covering ingestion, transformation, storage, and analytics
• Skills in ensuring data quality, security, and governance across large-scale platforms
• Capability to contribute to solution architecture and technical decision-making
• Competency in mentoring engineering teams and setting coding standards and best practices
• Understanding of high-performance data access patterns using Cosmos DB (NoSQL API)
• English proficiency at an Upper-Intermediate level (B2) or higher

Nice to have

• Hands-on experience with Microsoft Fabric and OneLake (Delta / OpenLake)
• Familiarity with financial instruments and financial services data
• Exposure to AI-assisted development tools such as GitHub Copilot and awareness of industry-standard LLMs
• Knowledge of Data Science fundamentals and collaboration experience with DS teams