L
Specialist - Data Engineering
LTM
🇨🇦 Canada | 🇺🇸 United States
On-site
5 months ago
- PySpark
- SQL
- Devops
- Agile
- Apache Spark
- Linux
- Unix
- ETL
- Azure Databricks
- Hadoop
- Hive
- Airflow
- Python
5 months ago
Job Title: Big data Developer
Work Location : Mississauga-Ontario-CA
Job Summary
- Design develop and maintain data pipelines using PySpark for largescale data processing
- Perform data transformations aggregations and validations using Spark SQL DataFrames
- Write and optimize SQL queries for data extraction reconciliation and reporting
- Perform data quality checks profiling and consistency validations
- Support existing data pipelines by troubleshooting failures and performing root cause analysis
- Work closely with data engineers analysts QA and DevOps teams in Agile delivery models
- Optimize performance of Spark jobs for efficiency and scalability
- Maintain technical documentation workflows and runbooks
- Follow enterprise standards for coding deployment and change management26 years of experience in Big Data Data Engineering roles
- Strong Handson experience with PySpark Apache Spark
- Solid understanding of SQL and relational database concepts
- Experience working in Linux UNIX environments
- Understanding of ETL concepts and largescale data processing
- Strong analytical and debugging skills
- Experience supporting production data pipelines Exposure to Azure Databricks ADF Synapse or similar cloud data platforms
- Experience with Hadoop ecosystem Hive HDFS
- Knowledge of orchestration tools eg Airflow
- Python scripting beyond Spark use cases
- Familiarity with Agile DevOps delivery practices
- Domain exposure eg Banking BFSI Analytics is an advantage Strong problem-solving and ownership mindset
- Ability to work collaboratively in Cross functional teams
- Good communication and documentation skills
- Ability to adapt quickly in dynamic project environments
Specialist - Data Engineering · LTM