DL
CGEMJP00359372- Morgan stanley Pyspark
Diverse Lynx India
- Location not stated
- 1 day ago
- PySpark
- SQL
- ETL
- Python
- Data Modeling
- Oracle
- DB2
- Snowflake
- Git
- GitHub
- Unix
- Linux
- Airflow
- CI/CD
- Agile
- Scrum
1 day ago
Location - PUNE
Brokerage is seeking a highly motivated **Data Engineer** with strong expertise in **PySpark, SQL, and large-scale data processing**. The ideal candidate will be responsible for building scalable data pipelines, optimizing data processing frameworks, and supporting critical financial data platforms. The role requires hands-on experience with big data technologies, ETL development, performance tuning, and data quality management. This JD is aligned with enterprise Data Engineering requirements emphasizing PySpark and SQL expertise.
### Key Responsibilities
- Design, develop, and maintain scalable data pipelines using PySpark and SQL.
- Process and transform large volumes of structured and semi-structured data.
- Develop and optimize ETL workflows for high-performance data processing.
- Perform data cleansing, reconciliation, validation, and quality checks.
- Tune Spark jobs and SQL queries for optimal performance and resource utilization.
- Collaborate with business analysts, data consumers, and upstream/downstream application teams.
- Analyze production issues and support batch processing activities.
- Develop reusable frameworks and utilities for data ingestion and transformation.
- Ensure compliance with data governance, security, and regulatory requirements.
- Participate in code reviews and contribute to engineering best practices.
### Required Technical Skills
Primary Skills (Must Have)
- Strong experience in **PySpark**
- Advanced SQL development and query optimization
- Python programming
- Data Engineering and ETL development
- Large-scale data processing and distributed computing
- Data validation, reconciliation, and data quality frameworks
PySpark Expertise
- Spark DataFrames and Spark SQL
- Joins, aggregations, window functions
- Partitioning and bucketing
- Performance tuning and optimization
- Handling large datasets (millions of records)
- Error handling and data recovery mechanisms
SQL Expertise
- Complex joins and subqueries
- CTEs and Window Functions
- Query tuning and performance optimization
- Stored procedures and data modeling concepts
- Data reconciliation and validation queries
Database Technologies
- Oracle / SQL Server / DB2 / Snowflake
- Data warehousing concepts
- Structured and unstructured data processing
Tools & Technologies
- Git/GitHub
- Unix/Linux Shell Scripting
- Autosys/Airflow (preferred)
- CI/CD pipelines
- Agile/Scrum methodology
### Experience
-**5–8 yearsof overall IT experience**
- Minimum **3–5 years of hands-on PySpark development**
- Strong experience in SQL development and optimization
- Experience working on enterprise-grade data platforms
### Interview Focus Areas
1. PySpark Transformations & Actions
2. Spark Performance Tuning
3. SQL Query Optimization
4. Data Modeling & ETL Design
5. Partitioning, Caching, and Memory Management
6. Data Reconciliation & Validation
7. Python Coding
8. Batch Processing & Scheduling
9. Financial Data Processing Concepts
10. Problem Solving & Debugging
CGEMJP00359372- Morgan stanley Pyspark · Diverse Lynx India