Databricks, Apache Spark

Job Description:

• Lead the end-to-end migration of data pipelines and workloads from Teradata to Databricks.
• Analyze existing Teradata workloads to identify optimization opportunities and migration strategies.
• Design and implement scalable, efficient, and cost-effective data pipelines using PySpark/Scala on Databricks

Skills Required:

• Strong experience withDatabricks,Apache Spark, andDelta Lake.
• Proficiency inPySpark,SQL, and optionallyScala.
• Strong understanding of Spark, Databricks performance tuning techniques
• Understanding ofTeradata utilities, BTEQ scripts, stored procedures
• Experience with cloud platforms (Azure, AWS, or GCP), preferablyAzure Databricks.
• Familiarity with data modeling, data warehousing concepts, and big data best practices.
• Strong problem-solving and communication skills.