Lead Data Engineer - Assistant Vice President

Lead Data Engineer – Databricks, Python, PySpark & AWS

Job Summary

We are seeking aSenior Data Engineer / SME to join theSSIM Business Architecture vertical in ahands-on Individual Contributor (IC) role.

The role requires strong expertise inDatabricks, Python, PySpark, AWS, and SQL to design, build, and support scalable, production-grade data pipelines and cloud data solutions.Snowflake and enterprise data integration experience are desired.

The successful candidate will demonstrate strongtechnical ownership, problem-solving skills, and engineering best practices across the data ecosystem.

Role location : Bangalore

• Business Architecture: The role sits within theSSIM Business Architecture vertical and requires close collaboration with Architecture, Security, engineering, platform, and business technology stakeholders.
• Hybrid working: The role requires4 days per week working from the office, in line with the applicable workplace expectations.
• Working hours: The primary working window will be12:00 PM to 9:00 PM India time, providing alignment withUS stakeholders and teams.
• Flexibility: Given the enterprise and production nature of the role, the individual must beflexible to work outside standard hours and, where required, over weekends to support critical deliveries, production issues, releases,DR or other business needs.

Key Responsibilities

• Design, develop, and maintain scalableETL/ELT, data ingestion, and transformation pipelines.
• Build and optimize data processing solutions usingPython and PySpark.
• Develop and support production-grade workloads and pipelines onDatabricks.
• Design and implement cloud-native data engineering solutions onAWS.
• Work with large-scale structured and unstructured datasets.
• Optimize Spark and Databricks workloads forperformance, scalability, reliability, and cost efficiency.
• Develop reusable data engineering components, frameworks, and engineering patterns.
• Implement data quality, validation, reconciliation, and monitoring controls.
• Troubleshoot complex production issues, perform root cause analysis, and implement sustainable fixes.
• SupportCI/CD, testing, deployment, monitoring, and operational reliability.
• Act as aData Engineering SME, providing technical guidance, design reviews, and code reviews.
• Collaborate with architecture, application, cloud, platform, and business teams to deliver enterprise data solutions.
• Drive automation, standardization, and continuous improvement across the data engineering landscape.

Required Skills

• Strong hands-on experience inData Engineering.
• Strong hands-on experience withDatabricks.
• Advanced programming skills inPython and PySpark.
• Strong experience withAWS cloud-based data engineering.
• Strong proficiency inSQL.
• Experience designing and supporting enterpriseETL/ELT and data pipelines.
• Strong understanding ofApache Spark and distributed data processing.
• Experience withdata lake and lakehouse architecture.
• Strong understanding of data modeling, data integration, and data quality principles.
• Experience withCI/CD and modern engineering practices.
• Experience supporting business-critical production data platforms.
• Strong troubleshooting, debugging, and root cause analysis skills.
• Strong communication and stakeholder-management capabilities.

Desired Skills

• Experience withSnowflake data engineering and integration.
• Knowledge of Snowflake databases, schemas, warehouses, security, and performance optimization.
• Experience integratingDatabricks/AWS data pipelines with Snowflake.
• Experience withGoldenGate, AWS DMS, or similar data replication technologies.
• Knowledge ofdata governance, lineage, metadata management, and auditing.
• Experience with data quality and reconciliation frameworks.
• Exposure to enterprisemonitoring and observability solutions.
• Experience working infinancial services or other regulated environments.

Experience Requirements

• 10+ years of overall experience in software engineering, data engineering, or related technology roles.
• 8+ years of hands-on data engineering & Devops experience.
• 7+ years of relevant experience acrossDatabricks, PySpark/Spark, and cloud-based data engineering preferred.
• Proven experience designing, developing, and supportingenterprise-scale production data pipelines and platforms.
• Demonstrated ability to operate as ahands-on technical SME in complex enterprise environments.

Education Requirements

• Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical discipline.
• Master’s degree in a relevant discipline is desirable.

Technology Stack

Core / Mandatory:
Databricks | Python | PySpark | AWS | SQL | Apache Spark | ETL/ELT | Data Pipelines | Data Lake/Lakehouse

Desired:
Snowflake | GoldenGate | AWS DMS | Data Governance | Data Lineage | Data Quality | Observability

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.

We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

Discover more information on jobs atStateStreet.com/careers

Read ourCEO Statement