RC
Data Architect / Data Engineering Consultant
Reuben Cooley, Inc.
🇺🇸 United States
On-site
1 week ago
- Python
- PySpark
- Hive
- SQL
- ETL
- Data Architecture
- Databricks
- Snowflake
- ELT
- Apache Spark
- Linux
1 week ago
The consultant will be responsible for understanding end-to-end data flows, developing and troubleshooting data pipelines, optimizing distributed processing workloads, performing SQL-based data reconciliation, and supporting production data platforms.
Required Qualifications
- 12-15 years in data architecture, data engineering, or enterprise architecture.
- 7 or more Hands-on development background.
- Experience with Databricks and Snowflake. And/or-
- Strong expertise in data warehousing, Lakehouse architecture, ETL/ELT, data modelling, and event-driven integration.
- Experience in regulated financial services.
- Design, develop, enhance, and troubleshoot ETL/ELT data pipelines.
- Develop and maintain data processing solutions using Python and PySpark.
- Work extensively with Apache Spark, including performance tuning and optimization.
- Analyze Spark jobs to identify performance bottlenecks related to partitions, shuffles, joins, data skew, caching, serialization, and resource utilization.
- Work with distributed data processing concepts and large-volume datasets.
- Develop complex SQL queries for data transformation, validation, reconciliation, and troubleshooting.
- Perform source-to-target reconciliation and investigate data discrepancies.
- Work with Hive and Impala for querying and processing large datasets.
- Troubleshoot production ETL failures, data quality issues, performance problems, and batch-processing failures.
- Perform root-cause analysis and implement permanent fixes for recurring production issues.
- Understand and troubleshoot end-to-end data flows, from source systems through ETL processing to downstream consumers.
- Work with Linux environments, shell commands, batch processing, and job scheduling.
- Collaborate with engineering, application, and business teams to resolve complex data issues.
- Participate in technical design discussions and provide recommendations for scalable and maintainable data solutions.
- Document technical designs, data flows, troubleshooting procedures, and production resolutions
Data Architect / Data Engineering Consultant · Reuben Cooley, Inc.