TC
Data Engineer
TechDigital Corporation
🇺🇸 United States
On-site
7 months ago
- SQL
- Azure Databricks
- PySpark
- Azure Cloud
- Azure Data Factory
- Azure SQL
- Databricks
- Unity Catalog
- Python
- Cassandra
- Airflow
- ETL
- ELT
- Azure
- Entra
- GitHub Actions
- Delta Lake
- Agile
7 months ago
Skills
- 5+ years' experience in SQL (Expert)
- 4+ years of experience in Azure Databricks with PySpark
- 4+ years of experience in Azure Cloud platform
- 3+ years of experience in ADF (Azure Data Factory), ADLS Gen 2 and Azure SQL
- 2+ years of experience in Databricks workflow & Unity catalog
- 2+ years of experience in Python programming & package builds
- Experience in data ingestion, cleansing, and transformation processes from various structured and unstructured data sources like Cassandra & Mark Logic and on-prem Mainframe sources using Databricks/PySpark supporting batch and near-real-time ingestion, transformation, and processing.
- Manage job scheduling, orchestration, and monitoring (e.g., using Azure Data Factory, Airflow, or Databricks Workflows).
- Ability to design modular, reusable workflows using tasks, triggers, and dependencies.
- Skilled in using dynamic expressions, parameterized pipelines, custom activities, and triggers.
- Familiarity with integration runtime configurations, pipeline performance tuning, and error handling strategies.
- Strong understanding of ETL/ELT design patterns, data warehousing, and data lakehouse architectures.
- Good to have Azure Entra/AD skills and GitHub Actions
- Good to have experience working on event-driven architectures using Kafka, Azure Event Hub
- Design develop and optimize scalable data pipelines leveraging Databricks (Spark, PySpark, SQL, Delta Lake) to support enterprise level data processing and analytics
- Write clean maintainable and efficient PySpark and Python code to support data ingestion transformation
- Integrate Azure Databricks with various Azure data services to build robust and scalable data platforms
- Implement and maintain ETL workflows for scalability cost effective and operational efficiency
- Collaborate with data analysts and stakeholders to gather requirements and deliver scalable data solutions
- Participate actively in design discussions code reviews and agile ceremonies to foster a collaborative and high performing team environment.
Data Engineer · TechDigital Corporation