AI Engineer, Quality

Tech Stack

About This Role

You'll own evaluation infrastructure for AI agents at Fieldguide, a vertical AI company for audit and advisory. You'll design unified evaluation platforms, observability systems, and automated pipelines that ensure enterprise-grade reliability. This role offers high-impact visibility as your work directly informs leadership and customers on AI quality.

What You'll Do

• Design and build a unified evaluation platform for AI agents
• Build observability systems to trace agent behavior and failures
• Create automated pipelines for rapid model evaluation
• Partner with product and ML engineers to integrate evaluation requirements

Requirements

• Production software experience in complex real-world systems
• Experience with TypeScript, React, Python, and Postgres
• Built and deployed LLM-powered features serving production traffic
• Implemented evaluation frameworks for model outputs and agent behaviors

Benefits & Perks

• 🏖️ Flexible PTO
• 📈 Meaningful ownership
• 💼 401k
• 🧘 Wellness benefits (free therapy sessions)
• 💻 Work-from-home reimbursement

Heads Up

• Role listed as remote but emphasizes in-person collaboration at SF office
• Experience requirement says '1+ years' but also asks for 'multiple years' of experience