Tech Stack

Description

You will design and build high-performance batch and stream data processing infrastructure to support ML operations, develop foundational data models and pipelines using tools like AWS EMR Spark, Kafka, and Snowflake, and collaborate with ML and data science teams to drive the data roadmap.

Requirements

• 5+ years experience as senior/software engineer
• Experience with Python, Golang, Java, or C++
• Experience with cloud infrastructure (AWS/GCP/Azure)
• Experience with relational/non-relational databases (Postgres, AWS DataLake/S3, DynamoDB, Snowflake)
• BS or higher in Computer Science or related field

Responsibilities

• Design and build batch and stream data processing infrastructure for ML operations
• Develop data models, data warehouse, and real-time/offline pipelines using AWS EMR Spark, Kafka, Athena, Snowflake, Airflow, Apache HUDI
• Collaborate with ML and data science teams to understand data needs and influence roadmap
• Build data governance platform for secure data management including cataloging, lineage, audit, deletion, masking
• Build and operate orchestration platform with Temporal and Airflow for other teams' workflows