Azure Data Engineer

Role Name: Data Engineer for Rialto

Experience: 6 to 8 Years

Primary Location: Mumbai
Other Acceptable Locations: Kolkata, Pune, Hyderabad, Bangalore, Chennai, Noida

Key Skills: Python, PySpark, Microsoft Fabric, AI & Gen AI – Products & Tools, Azure Data Factory

Must-Have Technical/Functional Skills
Proficiency in Python, TensorFlow, PyTorch, and other popular libraries for AI development.

Experience Required

Conversational AI and LLM Experience — Hands-on experience with conversational AI systems, including Retrieval-Augmented Generation (RAG) models, fine-tuning large language models (LLMs) at an enterprise scale, and developing dialogue management systems to handle dynamic user interactions effectively.

NLP and Language Model Specialization — Proficiency with NLP techniques and a deep understanding of language models like GPT, BERT, T5, and other transformer-based models.

Data Engineering and Analytics — Substantial experience in data engineering and analytics, with strong expertise in Databricks and PySpark. Hands-on knowledge of Apache Spark fundamentals, including Spark Architecture, its APIs, and how to leverage it for data processing and analysis.

Programming and Query Skills — Proficiency in PySpark, Python, and SQL is essential for data manipulation, analysis, and transformation.

Cloud Platforms — Demonstrated experience with cloud platforms (AWS, Azure, Google Cloud) and their integration with Databricks.

Data Solutions Architecture — Experience as a data solutions architect is a significant advantage, demonstrating the ability to design comprehensive data solutions.

Soft Skills — Excellent communication and team collaboration skills.

Roles & Responsibilities

Machine Learning and Deep Learning Expertise — Strong foundation in ML and DL concepts, including model architectures such as transformers, RNNs, and LSTMs, along with training techniques like supervised, unsupervised, and reinforcement learning.

Strategic Data Solution Development — Serve as a primary contributor to the development of data solutions with a strategic outlook, focused on building a data platform that serves as the single source of truth data lakehouse.

Design and Development — Architect and manage data solutions, ensuring integration of Databricks with existing data systems. Design ELT processes and data models in line with business requirements.

Data Management and Optimization — Optimize data storage and processing in Databricks. Implement data lakehouse solutions, ensuring performance and scalability.

Data Engineering Expertise — Lead the development of data tables tailored to specific use cases by engineering critical elements from multiple data domains. Ensure ingestion of third-party data is well-structured, compliant with data quality standards, and traceable from the consumption layer back to the raw data layer.

Strategic Integration and Security — Partner with the Information Security and Infrastructure teams to streamline data integration, maintain data security, and follow access best practices. Contribute to the creation of end-to-end data analytics solutions.