Senior Specialist - Data Engineering

Company:

Description:

Lead Engineer – Vulcan Platform Team

Job Summary

We are seeking a highly capable and experiencedLead Engineerto join theVulcan Platform Team. This role will provide technical leadership in the design, delivery and optimisation of platform and data engineering capabilities that underpin scalable, secure and reliable data products across the organisation.

The successful candidate will bring strong experience incloud-native engineering, with particular depth inAWSandDatabricksor similar Data lakehouse technologies and will be comfortable leading engineers whilst remaining hands-on in architecture, design and delivery. This role is ideal for someone who enjoys shaping engineering standards, improving platform capabilities, and working closely with cross-functional teams to enable high-quality, governed and performant data solutions.

The Lead Engineer will play a key role in advancing the Vulcan platform, ensuring it meets current and future business needs through robust engineering practices, automation, observability, and continuous improvement.

Key Responsibilities

Technical Leadership

• Lead, coach and mentor engineers within the Vulcan Platform Team, fostering a culture of engineering excellence, collaboration and continuous learning.
• Provide hands-on technical leadership across platform design, software engineering practices and delivery execution.
• Set and uphold engineering standards, including code quality, testing, documentation, CI/CD, and operational support.
• Support technical decision-making, architecture reviews and prioritisation of platform improvements in line with strategic objectives.
• Work closely with product, architecture, security and business stakeholders to translate requirements into scalable engineering solutions.

Platform Engineering & Architecture

• Design, build and maintain scalable, resilient and secure platform capabilities usingAWSandDatabricks.
• Lead the development of reusable platform services, patterns and frameworks that accelerate delivery for data and engineering teams.
• Drive the design and implementation of modern data and platform architectures, including lakehouse-oriented and event-driven patterns where appropriate.
• Ensure platform solutions are designed for reliability, performance, maintainability and cost-efficiency.
• Champion automation and infrastructure-as-code approaches to improve consistency, deployment speed and operational control.

Data & Pipeline Enablement

• Oversee the design and implementation of robust data pipelines, ingestion frameworks and transformation workflows.
• Support the development of scalable data processing solutions leveragingDatabricks,Spark, and AWS-native services.
• Ensure high standards of data quality, observability, lineage and operational supportability across platform components.
• Partner with data consumers, product teams and domain teams to enable effective use of shared platform capabilities.
• Contribute to the evolution of platform services that support governed, self-service and reusable data product delivery.

Cloud, Security & Operational Excellence

• Utilise AWS services such asS3, Lambda, Glue, Step Functions, RDS, IAM, CloudWatchand related tooling to build and optimise platform services.
• Ensure security, compliance and governance requirements are embedded into engineering solutions by design.
• Improve monitoring, alerting, logging and incident response processes to strengthen platform reliability and service performance.
• Support cost management and optimisation across cloud infrastructure and workloads.
• Work with enterprise and security teams to ensure alignment with Marsh technology standards and risk controls.

Innovation & Continuous Improvement

• Stay current with developments across AWS, Databricks and modern engineering practices, identifying opportunities to improve platform capability.
• Evaluate and introduce tools, frameworks and practices that enhance productivity, reliability and scalability.
• Lead initiatives to improve developer experience, release processes, testing strategies and operational maturity.
• Promote a culture of continuous improvement through retrospectives, feedback loops, metrics and engineering best practice.

Qualifications

Education

• Bachelor’s degree in Computer Science, Engineering, Information Technology or a related discipline, or equivalent practical experience.

Experience

• Significant experience in software engineering, platform engineering or data engineering roles, with demonstrable time spent in a senior or lead capacity.
• Strong hands-on experience designing and delivering solutions onAWS.
• Strong experience working withDatabricksor similar Data lakehouse technologies in enterprise data or platform environments.
• Proven experience building and supporting scalable data pipelines, distributed processing workloads, and cloud-native platforms.
• Experience leading engineers and driving technical delivery across cross-functional teams.
• Experience with platform automation, CI/CD pipelines and infrastructure as code.
• Experience operating within governed enterprise environments, with an understanding of security, risk and compliance requirements.

Technical Skills

• Strong knowledge ofAWSservices, particularly in storage, compute, orchestration, monitoring and security.
• Strong experience withDatabricksor similar Data lakehouse technologies, including notebooks, workflows, cluster management and data processing patterns.
• Strong programming capability inPythonandSQL.
• Good understanding ofSparkand distributed data processing concepts.
• Experience with data architecture, data modelling and modern lakehouse principles.
• Familiarity with infrastructure-as-code and automation tooling such asTerraform,CloudFormationor similar.
• Experience with DevOps and engineering tooling such asGit, CI/CD platforms, containerisation, and workflow orchestration tools.
• Understanding of observability practices including logging, metrics, monitoring and alerting.

Skills & Competencies

• Strong leadership and stakeholder management skills.
• Excellent problem-solving and analytical ability, with a pragmatic approach to delivery.
• Strong communication skills, with the ability to explain technical concepts clearly to both technical and non-technical audiences.
• Ability to influence engineering direction and drive adoption of standards and best practices.
• Comfortable operating in a fast-paced environment with multiple priorities and evolving requirements.

Preferred

• Experience in financial services, insurance, pensions or other regulated environments.
• Experience supporting shared platforms used by multiple product or domain teams.
• Exposure to data governance, metadata, lineage and access control frameworks.
• AWS certifications such asAWS Certified Solutions Architect,AWS Certified Data Engineer, or similar.
• Databricks certification is an advantage.

What Success Looks Like

• The Vulcan platform is reliable, scalable and secure, with clear engineering standards and strong operational performance.
• Engineers are supported and mentored effectively, with improved delivery quality and team capability.
• Platform users are able to build and run data products more efficiently through reusable, well-governed capabilities.
• AWS and Databricks services are used effectively to balance scalability, maintainability and cost.
• Continuous improvement is embedded into the team’s culture and engineering practices.