EI
Data Engineer
Expert In Recruitment Solutions
🇺🇸 United States
On-site
2 months ago
- AWS Glue
- IAM
- Redshift
- Python
- SFTP
- SNS
- CI/CD
- Jenkins
- Databricks
- Snowflake
- Data Modeling
- AWS
- AWS Cloud
- Apache Spark
- PySpark
- ETL
- Git
2 months ago
hybird - malvern pa
2 openings
assessment required
Job Description
Technical Stack & Requirements:
- Core Competencies: AWS, Glue, S3, IAM, Redshift, Python, Spark.
- Development Environment: Experience with Amazon SageMaker is mandatory, as this is our primary environment for code development before production deployment.
- Pipeline Management: Experience building end-to-end data pipelines, managing SFTP connections, and integrating with diverse data sources (Data Lakes, flat files, etc.).
- Infrastructure & Automation: Experience with SNS notifications and CI/CD tools (Jenkins) to ensure process efficiency.
- Nice-to-Have Skills: Experience with Databricks or Snowflake is highly desirable as we transition toward these platforms.
- Data Modeling: Proficiency in designing data models and creating multi-layered data structures to feed executive and operational reports.
Key Responsibilities:
- Manage priorities across multiple workstreams and handle production support as needed.
- Maintain robust pipelines with automated checks, particularly for event-driven jobs.
- Communicate effectively with external vendors and internal business stakeholders to gather requirements and provide solutions.
- Collaborate with senior engineering staff to ensure the security and integrity of our AWS development environments.
- 5+ years of Data Engineering or IT experience.
- 4+ years of AWS cloud development experience.
- Expert Python programming skills.
- Experience with Sagemaker.
- Robust hands-on experience with Apache Spark / PySpark.
- Experience building cloud-native ETL pipelines.
- Robust understanding of distributed computing.
- Experience designing scalable data models.
- Experience with Git and CI/CD pipelines.
- Robust analytical and problem-solving skills.
- Build scalable and auto recoverable pipelines.
- Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline.
- Working directly with business SME's and stakeholders.
Good to have
- Experience with Databricks tool.
Data Engineer · Expert In Recruitment Solutions