TC
AWS Data Engineer
TechDigital Corporation
Location not stated
Remote
6 months ago
- Python
- ETL
- Event-Driven Architecture
- AWS
- Amazon EventBridge
- SNS
- SQS
- AWS Cloud
- Pandas
- NumPy
- NoSQL
- MongoDB
- Cassandra
- DynamoDB
6 months ago
- Lead the team technically to complete milestones on time.
- Understand complete requirement, create Architecture and update all stakeholders.
- Create POCs
- Manage delivery/release to customer
- Develop Services to enable data ingestion from and synchronization with system which exposes required data access mechanisms ensuring near-real-time updates
- Ingest data from multiple sources using the python and any other ETL tools
- Design and implement an event-driven architecture using AWS EventBridge, Kafka, or SNS/SQS for real-time data streaming
- Design, implement, and maintain scalable data pipelines that integrate both on-prem and AWS cloud environments.
- Develop efficient Python scripts and applications using libraries like pandas, NumPy, etc., to handle and process large datasets.
- Work with various NoSQL databases (e.g., MongoDB, Cassandra, DynamoDB) to support high-performance data storage and retrieval.
- Develop and deploy applications in a cloud-native architecture, leveraging modern cloud technologies for scalability and resilience.
- Continuously monitor data workflows and systems, troubleshoot issues, and optimize performance for reliability and scalability
- Transition existing pipeline to MSSQL server
AWS Data Engineer · TechDigital Corporation