DL
Data Engineer+AI
Diverse Lynx India
Location not stated
2 months ago
- AI
- AI/ML
- ETL
- ELT
- MLOps
- Devops
- Python
- SQL
- Apache Spark
- PySpark
- Data Modeling
- Kinesis
- Azure Databricks
- Azure Data Factory
- Azure Synapse
- Microsoft Fabric
- Unity Catalog
- CI/CD
- Azure DevOps
- Machine Learning
- RAG
- OpenAI
- Azure OpenAI
- LangChain
- Semantic Kernel
- Databricks
- Delta Lake
- Snowflake
- Docker
- Kubernetes
2 months ago
We are seeking a highly skilled Data Engineer with strong expertise in modern data platforms and Artificial Intelligence (AI) technologies. The ideal candidate will be responsible for designing, building, and optimizing scalable data pipelines, enabling AI/ML workloads, and supporting enterprise data and analytics initiatives. The candidate should possess hands-on experience across cloud platforms, big data technologies, and AI-driven data engineering solutions.
Key Responsibilities
Design, develop, and maintain scalable batch and real-time data pipelines.
Build and optimize data lakes, lakehouses, and enterprise data platforms.
Enable AI/ML initiatives by preparing, transforming, and managing large-scale datasets.
Integrate structured and unstructured data from multiple enterprise sources.
Develop data ingestion frameworks, ETL/ELT pipelines, and data quality processes.
Collaborate with Data Scientists, AI Engineers, and Business Stakeholders to deliver AI-powered solutions.
Implement data governance, security, lineage, and compliance standards.
Optimize data processing performance, scalability, and cost efficiency.
Support MLOps and AI platform initiatives by enabling feature stores, vector databases, and model data pipelines.
Automate deployment, monitoring, and maintenance of data solutions using DevOps practices.
Required Skills
Data Engineering
Strong experience in Python and SQL.
Hands-on experience with Apache Spark / PySpark.
Experience building ETL/ELT pipelines.
Expertise in data modeling, data warehousing, and data lake architectures.
Experience with streaming technologies such as Kafka, Event Hubs, or Kinesis.
Cloud & Data Platforms
Strong experience with Azure Data Platform:
Azure Databricks
Azure Data Factory
Azure Data Lake Storage (ADLS)
Azure Synapse Analytics
Microsoft Fabric (preferred)
Experience with Unity Catalog and Data Governance frameworks.
Familiarity with CI/CD and Azure DevOps.
AI & Machine Learning
Understanding of Machine Learning and Generative AI concepts.
Experience working with AI/ML pipelines and MLOps frameworks.
Knowledge of Vector Databases, RAG (Retrieval-Augmented Generation), and LLM integrations.
Experience with OpenAI, Azure OpenAI, LangChain, Semantic Kernel, or similar AI frameworks.
Exposure to Feature Engineering and Feature Stores.
Data Governance & Quality
Data quality management and observability.
Metadata management and lineage tracking.
Experience implementing governance policies and security controls.
Preferred Qualifications
Experience with Databricks Delta Lake, Delta Live Tables, and Declarative Pipelines.
Knowledge of AI Agent frameworks and enterprise AI architectures.
Experience with Snowflake or other modern cloud data warehouses.
Familiarity with Docker, Kubernetes, and containerized deployments.
Experience in aviation, retail, finance, or revenue optimization domains is an added advantage.
Data Engineer+AI · Diverse Lynx India