💼 About This Role

You'll validate AI-generated outputs and agent behaviors, ensuring quality and reliability. You'll design evaluation frameworks and work with finance SMEs to define acceptance criteria. This role is critical for building benchmark scenarios and identifying failure patterns that traditional testing misses.

🎯 What You'll Do

• Design and execute structured evaluations for AI-enabled features
• Assess outputs for groundedness, consistency, and risk
• Identify hallucinations and unsupported assertions
• Build and maintain benchmark datasets and scorecards

📋 Requirements

• 4+ years in QA, analytics, or AI evaluation
• Strong writing and pattern-recognition skills
• Experience evaluating outputs against nuanced criteria
• Ability to work with structured rubrics and evidence-based reviews

✨ Nice to Have

• Experience with finance, accounting, or FP&A
• Bachelor's degree preferred

🎁 Benefits & Perks

• 💰 Bonus included in compensation
• 🏖️ Hybrid schedule (remote 2 days/week)
• 🏢 Office locations in major US cities

📨 Hiring Process

Estimated timeline: 2-4 weeks · AI estimate

1. 1Recruiter Screen· 30 min
2. 2Technical Interview· 60 min
3. 3Hiring Manager· 45 min

🚩 Heads Up

• Role mixes QA with AI evaluation, which may have unclear expectations
• No detailed tech stack listed for an AI-focused position