M
Data Pipeline Engineer
Mindlance
- 🇺🇸 United States
- On-site
- 1 day ago
- $100 / hour
- Data Modeling
- Python
- SQL
- Airflow
- Trino
- Snowflake
- FFmpeg
- OpenCV
- AI/ML
- TensorFlow
- PyTorch
1 day ago
Updated Job Description
Years of Experience
- 8 years of software development in data modeling, database design & optimization, scalable data pipelines for image/video (multimedia) data, and full-stack development using Python and SQL
- 5 years of development on data warehouse and analytics solutions (e.g., Airflow, Trino, Spark, Snowflake)
- 5 years of development on cloud-based data processing pipelines
- Demonstrated software development skills using Python
- Preferred: Agentic AI development experience
Bachelor's or Master's degree (or above) in Computer Science, Software Engineering, or a related field
Responsibilities
- Unified Metadata & Schema Design: Design and build standardized schema that unify file indexing, metadata, and enriched metadata across heterogeneous storage platforms (e.g., a document-store-based file platform, a relational per-protocol file platform, and other data collection platforms) and multiple dataset with differing file/metadata structures.
- Data Pipeline Engineering for Multimedia Data: Build and maintain efficient, scalable, monitored data pipelines that process image and video data at scale within a unified data platform, and support downstream consumer dataset generation.
- Database Design & Optimization: Create and maintain database schemas across SQL, vector, and unstructured data stores; apply normalization and denormalization appropriately; tune queries and storage for performance, reliability, and query efficiency at scale, addressing the known challenge of indexing/querying file- and frame-level metadata beyond classic DB solutions.
- Dashboards & Data Quality Monitoring: Develop dashboards to track data ingestion progress, surface metadata gaps, detect invalid/inconsistent metadata, and fix data issues early.
- Federated Ecosystem Enablement: Support the broader goal of a federated data ecosystem where independently contributed tables are connected, discoverable, and queryable through one consistent data schema interface.
- Full-Stack Development: Build front-end and back-end components of data-driven applications/dashboards, develop APIs, and integrate data from multiple sources/technologies into a single consumer-facing interface.
- Data Security & Compliance: Ensure data handling meets security and privacy compliance requirements across platforms and studies.
- Agentic AI Prototyping (stretch/preferred): Prototype and develop agentic AI capabilities to assist with data analysis and processing workflows on top of the unified data interface.
- Proficiency with multimedia processing tools (FFmpeg, GStreamer, OpenCV) and familiarity with AI/ML frameworks (TensorFlow, PyTorch) for video analysis
- Experience with Spark, Airflow, Trino, Snowflake, Iceberg-based lakehouse/data platforms
- Experience designing schema layers that separate interface from implementation to allow backward-compatible schema evolution
- Experience with config-driven/declarative pipeline and schema design
- Agentic AI development experience
Data Pipeline Engineer · Mindlance