SS
Mid-level Big Data Engineer
Sky Systems, Inc
🇮🇳 India
Hybrid
Mid level
1 week ago
- AWS Cloud
- Scala
- Python
- PySpark
- Hadoop
- EMR
- Microservices
- AI
- Apache Spark
- Hive
- AWS
- EC2
- ECS
- EKS
- Airflow
- Step Functions
- API Gateway
- DynamoDB
- Aurora
- RDS
- Linux
- Data Modeling
- Agile
- Scrum
- Machine Learning
- Cursor
- Large Language Models
- RAG
- AI/ML
- Devops
- CI/CD
- Git
- GitHub
- GitLab
- Bitbucket
- Jenkins
- Maven
- Gradle
- Artifactory
- Docker
- Kubernetes
- Databricks
1 week ago
Position Type:Full-Time Contract (40hrs/week)
Contract Duration: 6 months + extendable
Work Schedule: 8 hours/day (Mon-Fri)
Location:Hybrid - Wed/Thursday to be in onsite in Hyderabad, India
About the role:
- Design, develop, test, and deploy scalable Big Data solutions on AWS Cloud.
- Build and maintain batch and streaming data pipelines using Scala, Python, Spark, and PySpark.
- Analyze, process, and transform large volumes of structured and unstructured data.
- Develop and support distributed data processing frameworks and data ingestion platforms.
- Integrate new data sources and tools into existing big data ecosystems.
- Optimize performance of Spark, Hadoop, EMR, and distributed computing workloads.
- Design and implement cloud-native APIs and microservices supporting data products and operational platforms.
- Collaborate with product owners, architects, QA teams, and business stakeholders throughout the software development lifecycle.
- Automate data workflows, deployments, monitoring, and operational processes.
- Evaluate emerging Big Data, AI, and cloud technologies and prototype innovative solutions.
- Ensure adherence to security, governance, reliability, and software engineering best practices.
Required Skills:
- Bachelor's degree in Computer Science, Computer Engineering, Information Technology, or equivalent experience.
- 5-8 years of experience in software engineering and Big Data application development.
- Strong hands-on experience with: Scala, Python, Apache Spark / PySpark, Hadoop ecosystem (HDFS, MapReduce, Hive), Kafka or similar event-streaming technologies
- Strong AWS experience with services such as: EMR, S3, EC2, ECS/EKS, Airflow (MWAA preferred), Step Functions, API Gateway, Lambda, DynamoDB and/or Aurora/RDS
- Experience building and supporting large-scale batch and API-based data processing solutions in AWS.
- Strong Linux and shell scripting skills.
- Experience with data ingestion, transformation, data modeling, schema design, and data lifecycle management.
- Strong understanding of performance tuning for Spark, Hadoop, EMR, and distributed systems.
- Experience with Agile/Scrum development methodologies.
- Understanding of secure software development practices and cloud security fundamentals.
- Strong analytical, troubleshooting, and problem-solving skills.
- Excellent written and verbal communication skills.
Preferred Skills
- Experience with Generative AI, AI-assisted development tools (GitHub Copilot, Amazon Q, Cursor, etc.), and AI-driven automation.
- Understanding of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Vector Databases, and AI/ML solution integration.
- Experience building AI-enabled data platforms and data pipelines supporting ML workloads.
- Experience with DevOps and CI/CD tools such as Git, GitHub, GitLab, Bitbucket, Jenkins, Maven, Gradle, and Artifactory.
- Experience with containerization technologies such as Docker and Kubernetes.
- Knowledge of automated testing frameworks and data quality validation tools.
- Experience working in enterprise-scale cloud environments.
- AWS certification(s) and/or Databricks certification(s) are a plus.
Mid-level Big Data Engineer · Sky Systems, Inc