H
Senior Data Engineer – Databricks
HireTalent
🇺🇸 United States
Hybrid
Senior
1 month ago
- Databricks
- PySpark
- Python
- SQL
- Apache Spark
- Delta Lake
- AI
- Data Architecture
- CI/CD
- Fabric
- ETL
- ELT
- Data Modeling
- Unity Catalog
- AWS
- Azure Cloud
- AWS Glue
- Azure Data Factory
- Snowflake
- REST API
- Terraform
- IaC
- AI/ML
- Machine Learning
1 month ago
• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL.
• Build and optimize data processing workflows using Apache Spark and Delta Lake.
• Develop reliable data ingestion, transformation, validation, and enrichment processes across multiple data sources.
• Work with large and complex datasets and optimize pipelines for performance, scalability, and cost efficiency.
• Build reusable data products and data services that support analytics, investment workflows, and AI applications.
• Implement data quality, validation, reconciliation, and monitoring processes to ensure data accuracy and reliability.
• Contribute to data architecture and lakehouse design decisions.
• Work with data governance, metadata, lineage, and access-control requirements.
• Collaborate with Data Architects, Product Managers, Software Engineers, Data Scientists, and business stakeholders to translate requirements into scalable data solutions.
• Integrate data from APIs, databases, files, and third-party data providers.
• Develop and maintain automated testing and data-quality frameworks.
• Support CI/CD practices and automated deployment of data engineering workloads.
• Troubleshoot complex data and pipeline issues and continuously improve platform reliability.
• Contribute to the evolution of Rimes’ AI-ready data infrastructure and intelligence fabric.
Required Qualifications
• 6+ years of experience in data engineering or a related discipline.
• Strong hands-on experience with Databricks.
• Strong programming experience with PySpark and Python.
• Advanced SQL skills, including query optimization and performance tuning.
• Strong experience building production-grade data pipelines and ETL/ELT workflows.
• Experience with Delta Lake / Delta tables and modern lakehouse architectures.
• Strong understanding of data modeling, data integration, and distributed data processing.
• Experience working with large-scale datasets in a cloud environment.
• Experience with data quality, validation, monitoring, and troubleshooting.
• Strong understanding of software engineering practices, including version control, testing, and CI/CD.
• Excellent problem-solving and communication skills.
Preferred Qualifications
• Experience with Databricks Unity Catalog and data governance.
• Experience with Delta Sharing or data-sharing technologies.
• Experience with Databricks Workflows, Jobs, or similar orchestration capabilities.
• Experience with AWS or Azure cloud platforms.
• Experience with technologies such as AWS Glue, S3, Azure Data Factory, ADLS, or similar cloud data services.
• Experience with Snowflake or other modern cloud data platforms.
• Experience working with REST APIs and third-party data providers.
• Experience with Terraform or infrastructure-as-code.
• Experience implementing data lineage, metadata management, and data governance.
• Financial services, investment management, capital markets, or market-data experience.
• Exposure to AI/ML data pipelines or GenAI applications.
What Success Looks Like
In this role, you will help Rimes:
• Build highly scalable and reliable data pipelines.
• Improve the quality, accessibility, and usability of investment data.
• Create governed and reusable data products.
• Improve data-processing performance and operational efficiency.
• Strengthen the foundation required for AI-powered investment workflows.
• Help transform complex capital-markets data into trusted, actionable intelligence.
All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender, identity, national origin, disability, or protected veteran status.
Senior Data Engineer – Databricks · HireTalent