Lead II - Data Engineering

Experience: 7 to 9 years. Design and develop API-driven systems to govern, manage, and monitor large-scale batch Big Data applications. Build scalable backend services and data engineering solutions that support data processing and operational workflows. Develop and maintain data transformation processes using PySpark, SQL, Hive, Python, Scala, and related technologies. Work with AWS cloud services such as S3, EC2, EMR, Lambda, DynamoDB, and API Gateway to build cloud-native data and backend solutions. Build and enhance workflow orchestration using Apache Airflow, including advanced DAG design, dependency management, scheduling, monitoring, and failure handling. Define technical scope, objectives, and implementation approaches by participating in requirements gathering, technical research, and process definition. Participate in architecture reviews, code reviews, performance tuning, and operational readiness activities. Contribute to test planning and validation for application integrations, functional areas, and project deliverables. Partner with product owners, data engineers, backend engineers, QA, DevOps, and other cross-functional teams to deliver high-quality solutions in an Agile/Scrum environment. Troubleshoot production issues, improve system reliability, and drive automation across data and backend workflows. Experience in the financial domain or understanding of investment banking workflows.