Tech Stack

Description

You will define and own the AI quality testing standards at Roofr, building LLM eval frameworks and integrating quality gates into CI/CD pipelines. You'll work horizontally across teams to embed best practices and ensure AI integrations are reliable, consistent, and safe.

Requirements

• 5-8 years of software engineering or QA experience
• Hands-on experience building eval frameworks for LLM-powered features
• Strong engineering fundamentals with ability to write code and build tooling
• Experience integrating automated quality checks into CI/CD pipelines
• Familiarity with LLM APIs (e.g., Anthropic Claude, OpenAI) and agent frameworks

Responsibilities

• Define AI testing standards and patterns for the whole engineering org
• Build and own Roofr's LLM eval framework using tools like Promptfoo, DeepEval, or Braintrust
• Integrate quality gates into CI/CD pipelines to catch AI regressions before production
• Design human-in-the-loop review processes for AI outputs
• Coach QA engineers and developers on AI eval patterns and raise quality bar across teams