Quality Engineer - AI Agents, Evals & Integrations

tsWhat You Will

Do● Define and maintain an evaluation strategy for AI-powered workflows, including age

ntbehavior, output quality, accuracy, completeness, safety, and business-rule complianc

e.● Build automated AI evaluations using tools such as DeepEval, Ragas, Promptfo

o,LangSmith, Langfuse, or similar framework

s.● Create and maintain representative evaluation datasets, golden examples, regressi

onsuites, synthetic test data, and edge-case scenario

s.● Combine deterministic checks, business rules, structured assertions, LLM-as-a-jud

getechniques, human-review workflows, and production evidence where appropriat

e.● Integrate evaluations into CI/CD pipelines so meaningful AI regressions are detect

edbefore releas

e.● Establish practical quality thresholds, release signals, and investigation workflows f

orchanges to prompts,...