Databricks Architecture

Solution ArchitectDatabricks Lakehouse MigrationAI Modernization

Raritan NJ Onsite

We are seeking an experienced Solution Architect to lead the migration and modernization of enterprisescale data lakes from Azure Databricks to AWS Databricks while transforming the platform into an AIready Lakehouse architecture The role involves defining targetstate architecture migration strategy governance frameworks modernization roadmaps and AIGenAI enablement across largescale data ecosystems

The ideal candidate will have deep expertise in Databricks cloud migration modern data architecture and AIdriven data platforms with experience leading complex enterprise transformation programs involving thousands of data assets and multiple downstream applications

Key Responsibilities

ArchitectureStrategy

Define endtoend migration strategy from Azure Databricks to AWS Databricks

Design scalable Lakehouse architecture using Medallion principles Bronze Silver Gold

Develop Day1 Business Continuity and Day2 ModernizationAI Enablement migration roadmaps

Perform application and dependency assessment across upstream and downstream systems

Establish architecture patterns standards and best practices for cloudnative data platforms

MigrationModernization

Lead migration of data pipelines notebooks Delta tables workflows and data products

Modernize legacy ETLELT frameworks using Databricks native capabilities

Optimize performance scalability security and operational efficiency

Define migration approach for CDC batch streaming and realtime workloads

Drive technology rationalization and tool consolidation initiatives

AIReady Data Platform

Build AIready data foundations leveraging Delta Lake and Unity Catalog

Design data architectures supporting ML GenAI RAG feature stores and advanced analytics

Enable governed data access for data scientists and AI applications

Define semantic models metadata strategies lineage and data discovery frameworks

Establish frameworks for AI governance and responsible AI adoption

Data GovernanceSecurity

Architect enterprisewide governance using Unity Catalog

Design data lineage auditing access controls masking and compliance standards

Implement data quality observability and monitoring frameworks

Ensure platform adherence to security and regulatory requirements

Stakeholder Management

Collaborate with business data engineering analytics AI security and cloud teams

Lead architecture review boards and executivelevel discussions

Mentor solution architects technical leads and engineering teams

Drive estimation roadmap planning risk assessment and governance reviews

Required Experience

12 years in DataAnalytics

5 years as a Solution Architect or Lead Architect

Proven experience in largescale cloud migration programs

Handson experience with Azure Databricks and AWS Databricks

Experience migrating enterprise Data LakesLakehouses across cloud platforms

Strong understanding of Medallion Architecture and Delta Lake

Experience with Modern Data Platforms supporting AIML workloads

Expertise in enterprise integration and largescale data ecosystems

Technical Skills

DatabricksData Engineering

Databricks Workflows

Delta Lake

Delta Sharing

Unity Catalog

Delta Live Tables

SparkPySpark

Databricks SQL

Structured Streaming

Cloud Platforms

Azure Databricks

AWS Databricks

ADLS Gen2

Amazon S3

IAM

VPC

Glue

Lambda

Redshift good to have

Data Integration

Informatica PowerCenterIDMC

Azure Data Factory

Kafka

CDC Frameworks

API Integration

Eventdriven Architectures

AIAdvanced Analytics

GenAI Architecture

RAG Frameworks

Vector Databases

Feature Stores

MLflow

Databricks AIML

Azure OpenAIAmazon Bedrock

DevOpsGovernance

CICD

Git

Terraform

Infrastructure as Code

Data Quality Frameworks

Data Observability

SecurityCompliance

Preferred Qualifications

Databricks Certified Professional

AWS Solution Architect Certification

Azure Solution Architect Certification

Experience in Life Sciences Supply Chain Manufacturing or Healthcare domains

Experience leading migration programs involving 1000 datasets and multiple business applications

Key Success Metrics

Successful migration with zero business disruption

Reduction in platform complexity and technical debt

Improved performance and cost optimization

Enterprise adoption of AIready data products

Increased governance lineage and compliance coverage

Accelerated delivery of analytics and GenAI use cases

This JD is tailored for the kind of Azure DatabricksAWS Databricks modernization and AIready lakehouse transformation programs reflected in your recent DPSCDL migration initiatives"