DL
Data Engineer + AI
Diverse Lynx India
Location not stated
2 months ago
- Python
- SQL
- Apache Spark
- PySpark
- ETL
- ELT
- Data Modeling
- Kinesis
- Azure Databricks
- Azure Data Factory
- Azure Synapse
- Microsoft Fabric
- Unity Catalog
- CI/CD
- Azure DevOps
- AI
- Machine Learning
- AI/ML
- MLOps
- RAG
- OpenAI
- Azure OpenAI
- LangChain
- Semantic Kernel
- Databricks
- Delta Lake
- Snowflake
- Docker
- Kubernetes
2 months ago
Strong experience in Python and SQL.
Hands-on experience with Apache Spark / PySpark.
Experience building ETL/ELT pipelines.
Expertise in data modeling, data warehousing, and data lake architectures.
Experience with streaming technologies such as Kafka, Event Hubs, or Kinesis.
Cloud & Data Platforms
Strong experience with Azure Data Platform:
Azure Databricks
Azure Data Factory
Azure Data Lake Storage (ADLS)
Azure Synapse Analytics
Microsoft Fabric (preferred)
Experience with Unity Catalog and Data Governance frameworks.
Familiarity with CI/CD and Azure DevOps.
AI & Machine Learning
Understanding of Machine Learning and Generative AI concepts.
Experience working with AI/ML pipelines and MLOps frameworks.
Knowledge of Vector Databases, RAG (Retrieval-Augmented Generation), and LLM integrations.
Experience with OpenAI, Azure OpenAI, LangChain, Semantic Kernel, or similar AI frameworks.
Exposure to Feature Engineering and Feature Stores.
Data Governance & Quality
Data quality management and observability.
Metadata management and lineage tracking.
Experience implementing governance policies and security controls.
Preferred Qualifications
Experience with Databricks Delta Lake, Delta Live Tables, and Declarative Pipelines.
Knowledge of AI Agent frameworks and enterprise AI architectures.
Experience with Snowflake or other modern cloud data warehouses.
Familiarity with Docker, Kubernetes, and containerized deployments.
Experience in aviation, retail, finance, or revenue optimization domains is an added advantage.
Data Engineer + AI · Diverse Lynx India