Tech Stack

Description

You will lead development of DataHub's Platform framework, building scalable ingestion systems, clean APIs, and event-driven architectures for real-time metadata processing. Your work will directly power the next generation of AI systems at global scale.

Requirements

• 8+ years building production-grade distributed systems
• Advanced Python and API design expertise
• Experience with high-scale data processing or integration frameworks
• Strong systems knowledge and distributed architecture experience
• Proven track record solving complex technical challenges
• Built and maintained online applications serving live traffic at scale (100+ QPS)
• Set up monitoring and alerting for services
• Designed indexing, storage, and data architectures to make large-scale data accessible to online services
• Designed and scaled distributed systems
• Hands-on experience developing in a tight loop with LLMs and applying best practices for scalable LLM development
• Proficiency in one of Java/Scala/Kotlin/C#/Go

Responsibilities

• Build scalable, fault-tolerant ingestion systems for enterprise-scale metadata
• Create clean, intuitive APIs for our connector ecosystem
• Develop event-driven architectures for real-time metadata processing
• Implement schema mapping between diverse systems and DataHub's unified model
• Build versioning systems for AI assets (training data, model weights, embeddings)