Role - Senior Data Engineer (Databricks)

Experience - 8+ years

Location - Remote

Required Skills:-

· 8+ years in Data Engineering project development/design, handling large volumes of data.

· At least 5+ years using cloud data engineering services (preferably on AWS).

· 5+ years as a Big Data architect/solution architect on Big Data platforms.

· 3+ years of proven experience as an architect/solution architect on the

Databricks platform.

· Designed and implemented at least 2-3 end-to-end projects in Databricks.

· Expertise in E2E architecture of unified data platforms covering data lifecycle aspects:

· Data Ingestion, Transformation, Serving, and Consumption.

· Experience in composable architecture to fully leverage Databricks

capabilities.

· Implemented and configured Databricks environments (clusters, notebooks, libraries) for optimal performance and resource utilization.

· Experience in integrating data from various sources (structured, semi-structured, and (unstructured) into Databricks for processing and analysis

Technical Skills:

· Designed and developed scalable batch and streaming data pipelines, data lake architectures, and data warehousing solutions on the Databricks platform using Spark and Delta Lake.

· Knowledgeable in the Databricks Lakehouse concept and its enterprise implementation.

· Strong understanding of data warehousing, governance, and security standards related to Databricks.

· Skilled in cluster optimization, integration with various cloud services, and performance optimization to improve efficiency and reduce costs.

· Proficient in writing unit and integration tests, and setting best practices for Databricks CI/CD.

Qualifications:

· Proven experience as an Architect/Solution Architect on Databricks.

· Hands-on experience with AWS Databricks platform for data processing, warehousing, and analytics solutions.

· Strong background in cloud data engineering, ETL, data integration, and data governance.

Responsibilities:

· Enforce adherence to architectural standards, global product-specific guidelines, and usability design standards.

· Proactively guide engineering methodologies, standards, and leading practices.
Identify, communicate, and mitigate risks, assumptions, issues, and decisions throughout the full lifecycle.

· Provide guidelines for best practices and repeatable methodologies in Cloud Data Engineering, Data Storage, ETL (Extract, Transform, Load), Data Integration & Migration, Data Warehousing, and Data Governance.

TechHostServices.com remote jobs

HR Recruiter

Sr Data Engineer

Sr Data Engineer