PL
Databricks Administrator
PeopleNTech LLC
🇺🇸 United States
On-site
Manager or above
12 months ago
- Databricks
- Azure
- AWS
- CI/CD
- Apache Spark
- PySpark
- Azure SQL
- Azure Data Factory
- Azure Databricks
- IaC
- Terraform
- Terragrunt
- Bicep
- Python
- PowerShell
- Devops
- Azure DevOps
- GitHub Actions
- SQL
- Data Modeling
- ETL
- Kubernetes
- AKS
12 months ago
Location:Warren, NJ
Duration: 6 Months Contract
Bill rate: $85-90.00 per hour
Client: Capgemini/Everest
Summary:
The Databricks Administrator will be responsible for managing, configuring, optimizing, and maintaining the Databricks environment. The ideal candidate will have strong experience in a modern data platform environment and will work closely with data engineers, data scientists, and business analysts to ensure the platform is secure, cost-effective, and highly performant.
Key responsibilities typically include:
- Managing Databricks workspaces, clusters, and jobs.
- Implementing access controls and enforcing security measures.
- Monitoring and optimizing performance and cost.
- Supporting platform upgrades and automation.
- Integrating Databricks with cloud services (e.g., Azure,AWS).
- Providing technical support and developing documentation.
- Collaborating on CI/CD pipelines.
Qualifications often required:
- Experience as a Databricks Administrator or Engineer.
- Experience with Apache Spark and PySpark.
- Understanding of cloud infrastructure, particularly Azure.
- Experience with Databricks workflows and orchestration.
- Familiarity with data governance.
- A Bachelor's or Master's degree in a relevant field.
Position: Azure Datasite Cloud Engineer (Hybrid, 3 days onsite)
Location:Warren, NJ
Duration: 6 Months Contract
Client: Capgemini/Everest
Bill Rate: $85.00 - $90.00 per hour
Summary:
Azure Datasite Cloud Engineer - Developer would likely involve designing, building, and maintaining Azure data platforms for the company's global insurance and reinsurance business. This specific role would combine data engineering, cloud infrastructure, and development responsibilities.
Core responsibilities
- Data platform architecture and development: Design and implement modern data and analytics platforms using various Azure Data Services, such as Azure SQL, Azure Data Lake, Azure Data Factory, and Azure Databricks.
- Data integration: Create data pipelines to integrate data from various sources and orchestrate data flows for analytics and business intelligence (BI) systems.
- Infrastructure as Code (IaC): Automate the provisioning and management of cloud resources using tools like Terraform, Terragrunt, or Bicep to ensure consistent and scalable deployments.
- Application development: Write code and scripts, likely in Python or PowerShell, to automate tasks and build solutions. Some roles may involve full-stack development, though with a focus on data and backend services.
- DevOps and CI/CD: Implement continuous integration and continuous deployment (CI/CD) pipelines for both infrastructure and data-related code, potentially using Azure DevOps or GitHub Actions.
- Collaboration: Work closely with data scientists, DevOps engineers, and business analysts to deliver data products and ensure seamless integration and operation.
- Security and governance: Implement security measures, monitor data permissions, and ensure compliance with data governance policies.
- Optimization: Tune and optimize data storage and retrieval, including performance tuning for SQL databases and managing storage in Azure Data Lake Storage.
- Mentorship and guidance: Senior roles would involve mentoring and providing technical leadership to a team of engineers.
Required skills and qualifications
- Extensive Azure experience: Deep expertise in Azure services and the broader ecosystem, particularly in data-related services like Azure Data Lake,Databricks, and Synapse Analytics.
- Proficiency in IaC: Strong skills with Terraform and experience with other IaC tools like Bicep or ARM templates.
- Development and scripting: Experience with scripting languages like Python and PowerShell, with advanced SQL knowledge.
- DevOps knowledge: Understanding of DevOps practices and experience building CI/CD pipelines.
- Data handling: Knowledge of data modeling, ETL processes, and data warehousing concepts.
- Containerization: Familiarity with Kubernetes (AKS) is often preferred for modern workloads.
- Problem-solving: Strong troubleshooting and problem-solving skills for resolving issues within the Azure environment.
Databricks Administrator · PeopleNTech LLC