Role Summary
Hands-on Foundry specialist who can designontology-Financial products, engineerhigh-reliability pipelines, andoperationalize them into secure, observable, and reusable building blocks used by multiple applications (Workshop/Slate, AIP/Actions). You'll own the full lifecycle: from raw sources togoverned, versioned, materialized datasets wired intooperational apps andAIP agents.
Core Responsibilities
- Ontology & Data Product Design: ModelObject Types, relationships, and semantics; enforceschema evolution strategies; defineauthoritative datasets withlineage andprovenance.
- Pipelines & Materializations: BuildCode Workbook transforms (SQL, PySpark/Scala), orchestratemulti-stage DAGs, tunecluster/runtime parameters, and implementincremental + snapshot patterns with backfills and recovery.
- Operationalization: Configureschedules,SLAs/SLOs,alerts/health checks, anddata quality tests (constraints, anomaly/volume checks); implementidempotency,checkpointing, andgraceful retries.
- Governance & Security: ApplyRBAC, object-level permissions,policy tags/PII handling, andleast-privilege patterns; integrate with enterprise identity; documentdata contracts.
- Performance Engineering: Optimize joins/partitions, caching/materialization strategies,file layout (e.g., Parquet/Delta), andshuffle minimization; instrument withruntime metrics and cost controls.
- Dev Productivity & SDLC: UseGit-backed code repos,branching/versioning,code reviews,unit/integration tests for transforms; templatize patterns for reuse across domains.
- Applications & Interfaces: Expose ontology-backed data toWorkshop/Slate apps wireActions andAIP agents to governed datasets; publishclean APIs/feeds for downstream systems.
- Reliability & Incident Response: Ownon-call for data products, runRCAs, createrunbooks, and drive preventive engineering.
- Documentation & Enablement: Produceplaybooks, data product specs, and runbooks; mentor engineers and analysts on Foundry best practices.
Required Qualifications
- 7+ years in data engineering/analytics engineering with4+ years hands-on Palantir Foundry at scale.
- Deep expertise inFoundry Ontology, Code Workbooks, Pipelines, Materializations, Lineage/Provenance, andobject permissions.
- StrongSQL andPySpark/Scala in Foundry; comfort withUDFs,window functions, andpartitioning/bucketing strategies.
- Provenoperational excellence: SLAs/SLOs, alerting, data quality frameworks, backfills, rollbacks, blue/green or canary data releases.
- Fluency withGit,CI/CD for Foundry code repos, test automation for transforms, andenvironment promotion.
- Hands-on withcloud storage & compute (AWS/Azure/GCP), file formats (Parquet/Delta), andcost/perf tuning.
- Strong grasp ofdata governance (PII, masking, policy tags) andsecurity models within Foundry.
Nice to Have
- BuildingWorkshop/Slate UX tied to ontology objects; authoringActions and integratingAIP use cases.
- Streaming/event ingestion patterns (e.g.,Kafka/Kinesis) materialized into curated datasets.
- Observability stacks (e.g.,Datadog/CloudWatch/Prometheus) for pipeline telemetry;FinOps/cost governance.
- Experience establishingplatform standards: templates, code style, testing frameworks,domain data product catalogs.
Success Metrics (90–180 Days)
- &Client;99.5% pipeline success rate, with documentedSLOs and activealerting.
- <20% runtime/cost reduction via optimization and materialization strategy.
- Zero P1 data incidents and ≤4h MTTR with playbooks and automated remediation.
- 3+ reusable templates (ingestion, CDC, enrichment) adopted by partner teams.
- Ontology coverage for priority domains with versioned contracts and lineage.
Example Work You'll Own
- Stand upincremental CDC pipelines with watermarking & late-arrivals handling; backfill historical data safely.
- Definebusiness-ready ontology for a domain and wire it toWorkshop apps andAIP agents that triggerActions.
- ImplementDQ gates (null/dup checks, distribution drift) that fail fast and auto-open incidents with context.
- Buildpromotion workflows (dev → staging → prod) with automated tests on transforms and compatibility checks for ontology changes.
|
