DL
Data Scientist
Diverse Lynx India
🇮🇳 India
On-site
2 months ago
- AWS
- ETL
- ELT
- Apache Airflow
- Athena
- Redshift
- EMR
- AI/ML
- IAM
2 months ago
Develop and manage ETL/ELT workflows for batch and near real‐time data processing.
Implement data ingestion, transformation, and curation across diverse structured and semi‐structured data sources.
Orchestrate workflows using Apache Airflow, including DAG design, dependency handling, and operational monitoring.
Build and optimize data solutions using AWS-native services such as S3, Glue, Athena, Redshift, EMR, and Lambda.
Enable ML workflows using Amazon SageMaker, including feature engineering and automated data preparation pipelines.
Ensure data quality, lineage, observability, and monitoring using appropriate tools and frameworks.
Collaborate with data scientists, analytics, and application teams to support data-driven and AI/ML initiatives.
Apply data security, governance, and compliance standards, including IAM, encryption, access control, and data protection.
Apply strong data engineering design principles, covering storage formats, partitioning, schema evolution, and data distribution.
Make informed decisions balancing scalability, performance, cost efficiency, and operational simplicity.
Design for resilience and reliability, incorporating idempotency, retry behavior, and fault‐tolerant patterns.
Select appropriate data processing patterns (event-driven, batch, micro‐batch) aligned with business and latency needs.
Adopt a domain-driven, data‐product mindset, ensuring solutions align with domain boundaries and reusability.
Participate in design reviews and architectural discussions, ensuring adherence to engineering best practices
Data Scientist · Diverse Lynx India