NC
Data Architect-Pyspark
NR Consulting - India
- 🇮🇳 India
- On-site
- 13 hours ago
- PySpark
- Python
- Apache Spark
- Delta Lake
- Parquet
- Trino
- Linux
- Airflow
- MinIO
- Kubernetes
- Git
- Azure DevOps
- ADO
- Visual Studio
- VS Code
- Windows
13 hours ago
Location: Bangalore
Exp: 8+ Yrs
Job Description:
Required Skills
PySpark and Python
Apache Spark internals and distributed systems architecture
Delta Lake, Parquet, and Trino
Linux and vi commands
Workflow orchestration using Airflow
MinIO
Containerized deployment environments using Kubernetes
Git and Azure DevOps (ADO)
Visual Studio Code (VS Code)
GitHub Copilot
Preferred / Nice-to-Have Skills
Trade surveillance domain experience
Financial services data engineering experience
Time-series analytics
Market data processing
Ideal Candidate Profile
The ideal candidate is a Spark architecture expert with proven real-world experience in:
Distributed computing
Memory and execution optimization
Performance tuning at scale
Framework development
Scalable data platform engineering
The candidate should thrive in high-scale environments, combine deep technical execution with leadership, and deliver robust Spark systems for mission-critical workloads.
Key Responsibilities
1. Platform and Application Engineering
Design and develop enterprise-scale distributed data processing applications using PySpark.
Build reusable Spark frameworks and libraries that can be shared across multiple projects.
Architect scalable batch and near-real-time processing pipelines.
Develop modular, configurable, and highly maintainable Spark applications.
Establish coding standards and Spark engineering best practices across the team.
2. Performance Engineering
Own end-to-end Spark performance optimization, including:
Memory and CPU utilization optimization
Shuffle reduction
Stage and DAG optimization
Data skew handling
Job parallelism tuning
Executor sizing
Garbage Collection (GC) optimization
Identify and resolve performance bottlenecks using:
Spark UI
Explain plans
Event logs
Executor- and stage-level metrics
3. Distributed Processing Expertise
Demonstrate strong expertise in:
Spark execution model
Driver and Executor architecture
Lazy evaluation
Catalyst Optimizer
Lineage and fault tolerance
Adaptive Query Execution (AQE)
4. Data Partitioning and Data Layout
Design and implement effective partitioning strategies.
Optimize partition pruning.
Implement static and dynamic partitioning.
Improve data layout for efficient distributed processing.
5. Join and Window Optimization
Implement and optimize large-to-large dataset joins.
Work with window joins, time-series joins, time-based joins, and interval joins.
Select appropriate join strategies based on data volume, data skew, cluster behavior, and resource profiles.
Implement advanced window processing, including:
Running aggregations
Sliding and rolling windows
Ranking
Look-back calculations
Time-based analytics
Complex business-rule implementation
6. Spark Optimization Techniques
Apply and optimize the following techniques:
Cache and Persist
Checkpointing
Compression
Predicate pushdown
Bucketing
Z-ordering
7. Data Engineering Practices
Design and optimize Delta Lake solutions.
Implement schema evolution and data governance practices.
8. Technical Leadership
Lead architecture discussions for distributed Spark platforms.
Conduct Spark code reviews and design reviews.
Establish and enforce engineering standards and quality practices.
Mentor junior and senior engineers.
Drive technical decisions across the data platform.
Troubleshoot critical production incidents and lead their resolution.
Data Architect-Pyspark · NR Consulting - India