DL
PySpark Developer
Diverse Lynx India
🇮🇳 India
On-site
Senior
3 weeks ago
- PySpark
- ETL
- SQL
- Unix
- AWS
- Kinesis
- Azure
- Python
- Pandas
- Databricks
- Unity Catalog
- Airflow
3 weeks ago
Role Overview
Highly skilled Senior PySpark Developer with strong expertise in big data processing, ETL development, and distributed systems. The ideal candidate should have hands-on experience in PySpark, SQL, and Unix scripting, along with exposure to modern cloud and data engineering platforms.
Must-Have Skills
- Strong PySpark expertise – architecture understanding, job execution, script development and optimization
- Advanced SQL skills – CTEs, Window Functions, Aggregations
- Experience with cloud platforms – AWS (Kinesis/Glue)/Azure (Kafka)
- Python (Pandas) – large data handling and performance optimization
- Unix Shell scripting – ability to read, write, and modify scripts
- Hands-on experience with ETL pipelines and data processing frameworks
- Databricks experience – Unity Catalog, Row/Column level security, Data masking
- Experience with batch and streaming data processing
- Knowledge of job scheduling tools such as Autosys, Control-M, or Airflow
- Experience in requirement gathering, design documentation, and implementation
PySpark Developer · Diverse Lynx India