TC
Data Engineer
TechDigital Corporation
- 🇺🇸 United States
- On-site
- 10 months ago
- Python
- SQL
- AWS S3
- Redshift
- EC2
- EMR
- Apache Spark
- Pandas
- PySpark
- Airflow
- PostgreSQL
- MySQL
- NoSQL
- DynamoDB
- ETL
- AWS Glue
- Apache Airflow
- Git
- Devops
- CI/CD
- IaC
- Terraform
- CloudFormation
- AWS
10 months ago
• Programming Languages: Python (primary), SQL
•Cloud Platforms: AWS (S3, Glue, Lambda, Redshift, EC2, EMR)
•Data Tools: Apache Spark, Pandas, PySpark, Airflow
•Databases: PostgreSQL, MySQL, NoSQL (e.g., DynamoDB)
•ETL & Workflow Orchestration: AWS Glue, Apache Airflow
•Version Control: Git
•DevOps & CI/CD: Basic understanding of CI/CD pipelines and infrastructure as code (e.g., Terraform, CloudFormation)
Job Summary
•Data Pipeline Development - Design, build, and maintain scalable and reliable data pipelines to ingest, process, and transform data from various sources.
• Data Integration & Management - Integrate structured and unstructured data from internal and external systems.
•Ensure data quality, consistency, and availability across platforms.
• Cloud-Based Data Engineering- Leverage AWS services (e.g., S3, Lambda, Glue, Redshift, EMR) to build cloud-native data solutions.
•Optimize cloud resources for performance and cost-efficiency.
• Programming & Automation - Use Python for data manipulation, ETL workflows, and automation of data tasks.
•Develop reusable scripts and modules for data processing.
• Collaboration & Stakeholder Engagement
•Work closely with data scientists, analysts, and business teams to understand data needs.
•Translate business requirements into technical solutions.
• Monitoring & Optimization - Monitor data pipelines and troubleshoot issues proactively.
•Continuously improve performance, scalability, and reliability of data systems.
Data Engineer · TechDigital Corporation