DL
Data Engineer (Spark/Scala)
Diverse Lynx India
Location not stated
Mid level
1 month ago
- Scala
- Apache Spark
- Databricks
- PySpark
- SQL
- Python
- JSON
- Parquet
- Excel
- Apache Avro
- Airflow
- Azure
1 month ago
About the Role:
• We are seeking an experienced Data Engineer to design, build, and optimize complex data workflows across on-premises and cloud environments. This role requires deep hands-on expertise in Apache Spark, Databricks, and Scala/PySpark, along with strong SQL and Python skills, to build robust, high-performance data pipelines. You will work extensively on complex on-prem workflows, integrating data across multiple file systems and formats, migrating and modernizing legacy processes, and ensuring efficient, reliable data movement across heterogeneous environments.
Key Responsibilities:
• Design, develop, and maintain large-scale data pipelines using Apache Spark, Databricks, Scala Spark, and PySpark
• Build and support complex on-premises data workflows, including migration/hybrid on-prem-to-cloud integration patterns
• Integrate data across diverse file systems (on-prem file shares, NAS, HDFS, S3) and formats JSON, Parquet, Fixed-Length, CSV, Excel, Avro
• Write efficient, optimized SQL for data extraction, transformation, and loading across relational databases
• Connect to and extract data efficiently from various source databases, tuning queries and pipelines for performance at scale
• Develop and maintain workflow orchestration using Airflow (or similar schedulers) for reliable, monitored pipeline execution
• Write clean, production-grade Python code for data processing, automation, and tooling
• Build and maintain unit/integration tests for data pipelines to ensure data quality and reliability
• Create and maintain clear technical documentation for pipelines, data flows, and system architecture
• Troubleshoot and resolve data pipeline failures, performance bottlenecks, and data quality issues in complex, multi-system workflows
• Collaborate with cross-functional teams (data science, analytics, application engineering) to support downstream data consumption
• Support cloud integration efforts, particularly with Azure, as workloads evolve from on-prem to hybrid/cloud architectures
Data Engineer (Spark/Scala) · Diverse Lynx India