Description:
Location: Hyderabad, India
Technologies: Python, PySpark, SQL, Spark, Hadoop Ecosystem
Role Overview
A Senior Python and PySpark Developer leads the design, development, and delivery of scalable Python and PySpark-based solutions. The role ensures application quality, performance, maintainability, and efficient processing of large-scale datasets while guiding the engineering team and collaborating with stakeholders.
Key Responsibilities
- Lead design and development of Python and PySpark applications
- Develop scalable data processing pipelines using PySpark
- Design and optimize ETL/ELT solutions for large-volume datasets
- Define coding standards, best practices, and development guidelines
- Review and optimize application and Spark job performance
- Handle complex business logic, integrations, and data transformations
- Work with structured and unstructured data across multiple sources
- Implement data quality, validation, and reconciliation processes
- Collaborate with Data Architects, Data Engineers, and Business stakeholders
- Mentor junior and mid-level developers on Python, PySpark, and data engineering best practices
- Support release planning, production stability, troubleshooting, and performance tuning
- Ensure adherence to security, governance, and compliance standards
Required Skills
Python
- Expert-level Python development experience
- Strong understanding of Object-Oriented Programming (OOP), design patterns, and coding best practices
- Experience in building scalable and maintainable applications
- Strong debugging, performance tuning, and code review skills
PySpark & Big Data
- Strong hands-on experience with PySpark and Apache Spark
- Experience developing distributed data processing applications
- Good understanding of Spark architecture, executors, partitions, and cluster computing concepts
- Experience with Spark SQL, DataFrames, Window Functions, UDFs, and optimization techniques
- Knowledge of Spark performance tuning including partitioning, caching, broadcasting, and resource management
- Experience handling large-scale datasets in distributed environments
- Familiarity with data formats such as Parquet, Avro, ORC, and JSON
- Experience integrating Spark with cloud storage and relational databases through JDBC
SQL & Data Engineering
- Strong SQL development and query optimization skills
- Experience designing ETL/ELT pipelines and data transformation frameworks
- Understanding of data warehousing concepts and dimensional modeling
- Experience working with databases such as SQL Server, Oracle, PostgreSQL, DB2, or Snowflake
|
