Tech Stack

Description

You will join the ML subteam within IntelliJ AI to design evaluation frameworks, diagnose model performance issues, experiment with contexts and lightweight models, act as ML liaison for product teams, build helper models (re-rankers, classifiers, embedding models), collaborate across teams, and mentor junior engineers.

Requirements

• 5+ years experience as ML Engineer with production-grade ML systems (especially LLMs and agent architectures)
• Experience with LLM evaluation methods and frameworks
• Ability to design and run end-to-end experiments (hypotheses, metrics, data collection, analysis)
• Skilled in context-aware pipelines or conversational/agent systems
• Strong Python programming skills

Responsibilities

• Design and drive evaluation frameworks for AI features including metrics, experiments, and agent trace analysis
• Diagnose model performance issues (prompt drift, context mismatches, latency/quality trade-offs) and translate findings into improvements
• Experiment with contexts and lightweight models to continuously develop the ML system
• Act as ML liaison for product teams across JetBrains, adapting AI capabilities in IDEs
• Build and maintain small helper models (re-rankers, classifiers, embedding models) for domain-specific tasks