A
Sr. Data Engineer
Apolis
- 🇺🇸 United States
- Hybrid
- Senior
- 13 hours ago
- $62 – $71 / hour
- Databricks
- Azure
- GCP
- PySpark
- SQL
- Azure Databricks
- Machine Learning
- Delta Lake
- Unity Catalog
- RAG
- Key Vault
- Agile
- Python
- Pandas
- Git
- Integration Testing
- MLOps
- IaC
- Terraform
- Bicep
13 hours ago
Warren, MI or Austin, TX 4 days per week.
Contract to Hire
We can only submit candidates do not require sponsorship now or in the future to work in the US.
Required: Strong experience in Databricks on either Azure or GCP. Candidates must have very strong, hands on experience with Databricks, PySpark and SQL.
The interview process will be 1 technical video interview and then an offer will be made.
Job title
Senior Data Engineer / ML Platform Engineer (Azure Databricks & GenAI)
● Design and build production data pipelines on Databricks (PySpark/SQL, Delta Lake, Unity Catalog, Workflows/Jobs) with idempotency, testing, monitoring, and clear lineage.
● Model bronze / silver / gold layers and data contracts (schemas, SLAs, quality checks) for analytics and downstream applications.
● Develop ML-oriented data products: feature stores or curated tables, embedding generation, chunking/indexing for RAG, batch classification/enrichment, and evaluation of retrieval/data quality.
● Integrate with Azure services (ADLS Gen2, Key Vault, managed identities, private connectivity) following enterprise security and compliance patterns.
● Partner with application and ML teams to expose stable APIs, tables, and volumes that power agentic workflows (tool-backed assistants, multi-step orchestration) without owning full UI or real-time serving unless scoped.
● Document data dictionaries, runbooks, and handoff materials; participate in code review and agile ceremonies with full-time staff.
Required qualifications
● 5+ years in data engineering, including 2+ years production Databricks on Azure.
● Strong Python (PySpark, pandas); SQL; experience operationalizing jobs at scale.
● Hands-on Delta Lake and Unity Catalog (schemas, grants, environments, promotion patterns).
● Experience building pipelines that feed ML/GenAI use cases: embeddings, vector or hybrid search inputs, labeled datasets, batch inference outputs.
● Git, CI-friendly repo layout (notebooks + packages), unit/integration testing for data pipelines.
● Strong communication; effective in staff augmentation models (embedded with internal leads, shared standards).
Preferred qualifications
● GenAI / agentic systems: RAG design, retrieval evaluation, metadata for grounding, guardrails-aware data prep.
● Multi-source integration: documents, PLM/BOM-style structured data, files in object storage, APIs, spreadsheets—normalized into a catalog.
● Exposure to containerized or microservice backends (for collaboration with app teams, not primary ownership). Nice to have (role-dependent)
● MLOps on Databricks (Model Serving, Feature Serving, experiment tracking).
● Stream or near-real-time ingestion (Structured Streaming, event-driven triggers).
● Infrastructure-as-code literacy (Terraform/Bicep)—often owned internally; ability to consume outputs is enough.
● Graph or highly relational domains (requirements traceability, product structure, hierarchies).
● Automotive or complex manufacturing data (helpful, not required).
What we provide
● Platform expertise, Databricks/Azure standards, architecture review, and access to tooling and repos per assignment.
● Clear priorities from product/engineering owners; expectations to align with existing team patterns rather than invent a separate stack.
Sr. Data Engineer · Apolis