Tech Stack

Description

You will architect and scale batch and streaming pipelines using Spark and Flink, build ML job execution frameworks, and maintain internal API servers and developer tools on Kubernetes. Collaborate with data scientists and engineers to deliver reliable ML platform capabilities while promoting best practices and LLM-based tools.

Requirements

• 4+ years in data systems, ML infrastructure, or platform engineering
• Strong coding proficiency in Python and/or Java
• Experience with Spark, Flink, Kubernetes, Terraform, and Helm
• Experience with high-throughput data infrastructure like ClickHouse or PostgreSQL
• Proven ability to use LLM-based tools like Copilot or ChatGPT

Responsibilities

• Architect and scale batch (Spark) and streaming (Flink) pipelines for ML training and evaluation
• Design and operate ML job execution frameworks for training, inference, and post-processing
• Build and maintain internal API servers and tools to orchestrate ML jobs on Kubernetes
• Design and monitor data infrastructure using ClickHouse and PostgreSQL
• Ensure high availability and observability with Prometheus and Grafana