ES
Sr. Software Engineer(PYTHON + pySPARK)
Epsilon Solutions LTD
Location not stated
Remote
Senior
4 months ago
- Python
- PySpark
- Apache Spark
- Java
- Hadoop
- Parquet
- Flink
- OOP
- AWS
- Kubernetes
- Docker
4 months ago
Work location: Remote anywhere in Canada
Job Description:
About the job you're considering:
As a Senior Software Engineer, you will work on one of the world's largest social media platforms, handling multiple petabytes of data processed daily. You will be part of a self-organized R&D team working in a challenging and innovative environment for our client. You will investigate, design, and implement solutions for complex technical challenges using cutting-edge technologies, including building and enhancing data processing platforms that power software used by hundreds of millions of users.
Your role:
• Receive tasks from the Project Lead or Team Lead (TL), prepare functional and design specifications, and align them with all stakeholders.
• Ensure assigned areas are delivered on time and meet quality expectations.
• Provide effort estimations, agree on timelines with the manager, and contribute to project planning.
• Evaluate existing data systems and identify improvement opportunities.
• Develop best practices for data coding to ensure system-wide consistency.
• Update and optimize local and metadata models.
• Assess implemented data systems for discrepancies, variances, and performance efficiency.
• Troubleshoot and optimize data infrastructure and pipelines.
• Understand business drivers and analytical use cases, translating them into reliable data products.
• Design, implement, and maintain cloud-based pipelines that deliver business-critical data efficiently.
• Identify area-level risks and implement mitigation plans.
• Report readiness/quality status and escalate critical issues appropriately.
• Take responsibility for resolving crisis situations within the assigned area of responsibility (AOR).
• Initiate and conduct code reviews, and help establish coding standards, conventions, and guidelines.
• Propose technical and functional improvements to enhance the product.
• Continuously upgrade professional knowledge and skills.
• Collaborate closely with cross-functional teams.
Your skills and experience:
• 5+ years of experience as a Software Engineer with strong Python development skills, including hands-on PySpark and Apache Spark implementation.
• Mid-level Java proficiency (minimum 2.5/5), with practical experience writing, debugging, and maintaining Java code.
• 3+ years of experience building, operating, and supporting large-scale, scalable Big Data platforms or infrastructure.
• Strong knowledge of Big Data technologies, including:
o Apache Spark
o Hadoop ecosystem
o Parquet / ORC file formats
o Apache Flink (or similar stream-processing frameworks)
• Proven troubleshooting and production support skills, including the ability to handle SEV-1 / critical production incidents under pressure.
• Strong ability to organize, document, and communicate solutions, including technical design, root cause analysis (RCA), and operational runbooks.
• A passion for streamlining systems and processes to make the difficult trivial
• Strong OOP skills
• Effective communication (oral & written), collaboration, and interpersonal skills
• Result oriented approach
Good to have:
• Experience with AWS, Kubernetes (K8s), and Docker.
Sr. Software Engineer(PYTHON + pySPARK) · Epsilon Solutions LTD