ES
Principal Data Engineer – Databricks
Epsilon Solutions LTD
🇨🇦 Canada
On-site
Staff / Principal
2 months ago
- Databricks
- Hive
- HBase
- Delta Lake
- PySpark
- SQL
- ELT
- dbt
- AWS
- Azure
- CI/CD
- Git
- Terraform
- Jenkins
- Azure DevOps
- Airflow
- SAP
- Oracle
- S/4HANA
- AI/ML
2 months ago
Full Time
Toronto, ON (Onsite 4 days/week)
Key Requirements
- 12–18 years of overall data engineering experience
- 8+ years of experience in enterprise Data Warehouse and Data Lake platforms
- 5+ years of hands-on experience with Databricks and Spark at scale
- Strong experience in modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse
- Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake
- Refactor legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines
- Ensure data reconciliation, audit integrity, and consistency during migration
- Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures
- Implement layered architecture including Raw/Landing, Curated/Conformed, and Semantic/Consumption layers
- Modernize traditional EDW platforms into scalable lakehouse architectures
- Strong experience in finance and risk data models including General Ledger, Sub-ledger, financial hierarchies, and risk exposure models (credit, liquidity, market risk)
- Enable reporting use cases including aggregation, drill-down, and drill-back capabilities
- Build and manage semantic/consumption layers for BI, reporting, and analytics
- Define business metrics, dimensions, hierarchies, and KPIs
- Experience with Databricks SQL, Delta tables, and dbt or similar frameworks
- Develop and optimize large-scale data pipelines using PySpark, Spark SQL, and Delta Lake
- Implement Medallion architecture (Bronze, Silver, Gold layers)
- Optimize workloads using Z-ORDER, OPTIMIZE, caching, and cluster configurations
- Implement data governance, data quality frameworks, reconciliation controls, and exception handling
- Establish data lineage and metadata management
- Ensure data security, access control, and compliance standards
- Experience with cloud platforms such as AWS or Azure
- Experience with CI/CD pipelines using Git, Terraform, Jenkins, or Azure DevOps
- Familiarity with orchestration tools such as Airflow or Databricks Workflows
- Experience with dbt is a plus
- Act as a technical authority and lead architecture decisions
- Mentor and guide senior engineers and establish engineering standards
- Strong stakeholder management with finance, risk, analytics, and governance teams
- Ability to translate complex data structures into business-ready insights
- Experience in BFSI, Capital Markets, or regulatory reporting
- Exposure to SAP Finance, Oracle Financials, or S/4HANA
- Experience supporting AI/ML workloads
- Databricks or cloud certifications
* Lead Cloudera to Databricks transformation initiatives
* Shape enterprise finance and risk data platforms
* Support regulatory, management, and analytical reporting systems"
Principal Data Engineer – Databricks · Epsilon Solutions LTD