Tech Stack

Description

You will partner with engineers, product, and design to productionize early-stage AI features into high-quality, performant features that delight users at scale. Focus on quality and safety testing, devising evaluation frameworks, refining prompts and pipelines, and optimizing model choices for cost, latency, accuracy, tone, and safety. Build the shared AI platform powering products like AI Interviewer and AI Fraud Signals.

Requirements

• 5+ years in Data Science or ML engineering with a strong focus on ML or NLP systems.
• 1+ year focused on Gen-AI or LLM systems.
• Strong Python and SQL skills.
• Experience creating automated evaluation suites for LLM outputs (accuracy, safety, bias, tone, style).
• Knowledge of prompt engineering, RAG techniques, vector search, embeddings, fine-tuning, and model selection across multiple providers.

Responsibilities

• Design and own comprehensive evaluations measuring accuracy, completeness, style, hallucination rate, bias, and safety across every release.
• Tune and iterate on RAG pipelines, prompt chains, conversation loops, provider selections, and fine-tunes until quality bars are met or exceeded.
• Build reusable data and evaluation pipelines, a shared semantic layer, and monitoring dashboards for product teams to ship reliable AI quickly.
• Optimize for cost and latency, continuously benchmarking models and negotiating trade-offs between performance and spend.
• Implement robust data governance and lineage practices for enterprise compliance and AI bias audit support.