Data Scientist
- Design and implement end-to-end evaluation frameworks to assess performance, reliability, and safety of multi-agent AI systems
- Lead experimentation and A/B testing efforts to systematically test hypotheses, validate model improvements, and track performance across agent iterations
- Curate and maintain high-quality ground truth datasets to enable accurate, reproducible evaluation of multi-agent outputs
- Identify and address reliability and accuracy gaps across agent workflows, failure modes, and edge cases in production-like environments
- Stay current on emerging research in agentic AI, LLM evaluation, and multi-agent coordination to continuously improve framework design
Technical Skills
- Proficiency in Python and ML frameworks
- Hands-on experience with LLM APIs and agentic frameworks (Lang Chain, Llama Index, Semetic Kernal I)
- Familiarity with evaluation tooling (Ragas, Deep Eval, Lang Smith, or similar)
- Experience with data pip...