LEAD SOFTWARE ENGINEER

Azure Databricks ConsultantLocation: PCS CA, TorontoYears of Experience: 7-10 YearsJob Summary: We are seeking an experienced Azure Databricks Consultant with a strong background in building ETL pipelines and data engineering solutions. The ideal candidate will have hands-on experience with Azure Databricks or Delta Lake, and a deep understanding of data warehousing concepts, data governance, and security practices. This role requires a problem-solver who can create end-to-end environments and work collaboratively in a consultancy setting.Responsibilities:Design and implement ETL pipelines using Azure Databricks, including batch (autoloader) and Spark structured streaming.Utilize Unity Catalog for managing data governance and access control.Create and manage catalogs, schemas, tables, materialized views, functions, and volumes in Databricks.Develop and maintain Slowly Changing Dimensions (SCD1 and SCD2) on dimension tables.Build Change Data Capture (CDC) pipelines to track data changes.Implement Lakehouse federation and create foreign catalogs to integrate data from external sources.Optimize data partitioning and apply Liquid clustering techniques for performance improvement.Collaborate with cross-functional teams to ensure data quality and integrity.Participate in CI/CD pipeline development and adhere to DevOps practices.Mandatory Skills:Hands-on experience with Azure Databricks or Delta Lake.Strong working knowledge of Unity Catalog.Proficiency in building ETL pipelines and data transformation techniques.Experience with SQL and data modeling.Strong understanding of data warehousing concepts.Experience with cloud platforms, particularly Azure.Strong analytical and problem-solving skills.Preferred Skills:Familiarity with Apache Spark (PySpark/Scala).Knowledge of performance tuning in Spark.Exposure to data governance and security practices.Experience with CI/CD pipelines and DevOps methodologies.Qualifications: Bachelor's degree in Computer Science, Information Technology, or a related field.