A
Data Engineer
Apolis
- 🇺🇸 United States
- Hybrid
- Mid level
- 1 day ago
- $50 – $55 / hour
- Databricks
- PySpark
- Python
- SQL
- ETL
- ELT
- Delta Lake
- CI/CD
- Apache Spark
- Data Modeling
- Azure
- AWS
- GCP
- Git
- Unity Catalog
- Azure Data Factory
- Apache Airflow
1 day ago
- Location:- Wilmington, DE (5 days office mandatory).
- No of Position – 1
- Job title:- Data Engineer (Preferred Ex Banking candidates)
- Experience – 8 years
JD:-
We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks, PySpark, Python, and SQL. The candidate will be responsible for designing, developing, and optimizing scalable data pipelines and data-processing solutions in a cloud-based data environment.
The candidate should be a hands-on Data Engineer who has personally built and supported Databricks solutions, rather than someone with only theoretical exposure. They should be able to explain real examples of PySpark transformations, ETL/ELT pipelines, Delta Lake implementations, performance optimization, data-quality controls, and production troubleshooting.
Key Skills:- Databricks | PySpark | Python | SQL | ETL/ELT | Delta Lake | Spark | Cloud | Data Engineering.
Responsibilities- Design and develop scalable ETL/ELT data pipelines using Databricks.
- Develop data-processing and transformation solutions using PySpark and Python.
- Write complex SQL queries for data transformation, validation, and analysis.
- Build and maintain solutions using Delta Lake and Lakehouse architecture.
- Integrate data from multiple sources, including databases, APIs, files, and cloud storage.
- Optimize Spark jobs, Databricks workloads, and SQL queries for performance and cost efficiency.
- Implement data-quality, validation, reconciliation, monitoring, and error-handling mechanisms.
- Troubleshoot production data issues and perform root-cause analysis.
- Collaborate with business analysts, architects, and application teams to translate business requirements into technical solutions.
- Follow engineering best practices for testing, source control, CI/CD, documentation, and production deployment.
Required Skills & Experience- 5–8 years of overall Data Engineering experience.
- Strong hands-on experience with Databricks.
- Strong hands-on development experience using PySpark / Apache Spark.
- Strong Python programming skills and advanced SQL skills.
- Experience developing production-grade ETL/ELT pipelines.
- Experience with Delta Lake and Lakehouse concepts.
- Experience processing large-volume and complex datasets.
- Good understanding of data modeling, data quality, and data governance principles.
- Strong troubleshooting and performance-tuning skills.
- Experience with at least one cloud platform: Microsoft Azure, AWS, or Google Cloud.
- Experience with Git and CI/CD practices.
Good to Have Skills- Databricks Unity Catalog experience.
- Databricks Workflows / Jobs and cluster optimization experience.
- Experience with Azure Data Factory or Apache Airflow.
- Experience with Kafka or Spark Structured Streaming.
- Knowledge of cloud-native storage and data services.
- Databricks or cloud certification.
- Experience in banking or financial-services data environments.
Data Engineer · Apolis