TC
AWS Data Engineer
TechDigital Corporation
Location not stated
7 months ago
- AWS
- SNS
- SQS
- Python
- SQL
- ETL
- Event-Driven Architecture
- Amazon EventBridge
- AWS Cloud
- Pandas
- NumPy
- NoSQL
- MongoDB
- Cassandra
- DynamoDB
- Data Architecture
7 months ago
| Skills | Years of Experience | Last Worked in Which Year | Rating 1-10 |
| AWS | |||
| Glue | |||
| SNS/SQS | |||
| Python | |||
| Py-Spark | |||
| Data Lake | |||
| Cloud Watch | |||
| Cloud Trail | |||
| DB Design | |||
| SQL |
• Lead the team technically to complete milestones on time.
• Understand complete requirement, create Architecture and update all stakeholders.
• Create POCs
• Manage delivery/release to customer
• Develop Services to enable data ingestion from and synchronization with system which exposes required data access mechanisms ensuring near-real-time updates
• Ingest data from multiple sources using the python and any other ETL tools
• Design and implement an event-driven architecture using AWS EventBridge, Kafka, or SNS/SQS for real-time data streaming
• Design, implement, and maintain scalable data pipelines that integrate both on-prem and AWS cloud environments.
• Develop efficient Python scripts and applications using libraries like pandas, NumPy, etc., to handle and process large datasets.
• Work with various NoSQL databases (e.g., MongoDB, Cassandra, DynamoDB) to support high-performance data storage and retrieval.
• Develop and deploy applications in a cloud-native architecture, leveraging modern cloud technologies for scalability and resilience.
• Continuously monitor data workflows and systems, troubleshoot issues, and optimize performance for reliability andscalability, Transition existing pipeline to MSSQL server
• Collaborate with the business application owner on the existing data architecture, including data ingestion, data pipelines, business logic, data consumption patterns, and analytics requirements
• Design and document the target data architecture, pipelines, processing and analytics architecture
• Identify opportunities for optimization and consolidation
• Collaboration with data team on decomposition of business logic and data transformation patterns
AWS Data Engineer · TechDigital Corporation