Tech Stack

Description

You will own the end-to-end loop of analyzing how AI coding agents fail on Kotlin, building evaluation infrastructure, and implementing methods to improve agent behavior. Your work directly shapes how millions of developers experience Kotlin through AI coding agents.

Requirements

• Hands-on experience building evaluation or analysis pipelines for LLMs or AI coding agents
• Strong Python engineering skills (at least 3 years)
• Experience with data analysis at scale (SQL, data pipelines, statistical analysis)
• Ability to own projects end-to-end
• Familiarity with Kotlin or strong willingness to develop deep Kotlin expertise

Responsibilities

• Build tools for agentic error analysis and observability pipelines over agentic traces
• Design, implement, and maintain evaluation pipelines measuring Kotlin code generation quality
• Research post-training techniques (SFT, DPO, GRPO) and context engineering to improve agent behavior
• Collaborate with model providers to translate Kotlin-specific findings into model improvements
• Design and build open-source benchmarks for AI coding agent performance on Kotlin tasks