AI QA and Test Engineer

Quick Facts

• Work Policy: Remote (the listing shows Atlanta, Georgia)
• Job Type: Contract
• Salary Range: $50 per hour
• Industry: Information technology services (infrastructure services client in Georgia)
• Core Tools: Arize or similar AI observability platforms, Azure DevOps
• Job ID: 26-02326

About the Role

SGA is looking for an AI QA and Test Engineer to test, evaluate, and validate AI-powered agent experiences across customer-facing and internal business workflows. The role focuses on prompt and agent quality assurance, running evaluation frameworks, managing defects, and AI observability.

The ideal candidate is curious, analytical, and detail-oriented, with hands-on experience testing LLM-based systems, designing evaluation criteria, and using AI observability tools. You’ll work with product managers, software engineers, and business stakeholders to find quality gaps and improve agent reliability, accuracy, and user experience.

What You’ll Do

Prompt and Agent Evaluation

• Test and evaluate existing prompts, system instructions, and agent workflows for accuracy, consistency, groundedness, and user satisfaction
• Run structured prompt experiments and analyze A/B test results (the role covers execution and analysis, not writing prompts)
• Document evaluation findings, failure patterns, and suggested refinements for engineers to implement

AI Quality Assurance and Testing

• Design and run test plans covering AI agents, web applications, APIs, and backend services
• Build evaluation frameworks and scoring methods for AI output quality
• Write test cases for accuracy, hallucination detection, safety, tone and style adherence, instruction following, context retention, and response consistency
• Perform regression testing for prompt and model changes
• Track, prioritize, validate, and retest defects in Azure DevOps

AI Evaluation and Observability

• Use Arize or similar platforms to monitor model and agent performance
• Analyze user interactions and evaluation results to find trends and improvement opportunities
• Create quality metrics and dashboards covering accuracy, relevance, latency, user satisfaction, defect rates, and prompt performance
• Investigate production issues and recommend prompt, workflow, or architecture improvements

Documentation and Collaboration

• Maintain documentation for evaluation methods, testing procedures, and defect logs
• Take part in sprint planning, backlog refinement, and other Agile ceremonies
• Communicate findings and recommendations to technical and non-technical stakeholders
• Share best practices and lessons learned across the team

Ideal Candidate

• Hands-on experience testing LLM-based systems
• Experience designing evaluation criteria
• Experience using AI observability tooling
• Curious, analytical, and detail-oriented

Application Notes

By applying, you consent to SGA processing your application for this and future opportunities under its Privacy Policy, and you agree to be contacted by SGA’s AI recruiter by email, text, or phone. AI is not used to make hiring decisions, and choosing not to engage with AI will not hurt your consideration for placement.

About SGA

Software Guidance & Assistance (SGA) is a women-owned technology and resource solutions provider that takes a personal, boutique approach to large IT problems and makes more than 1,000 engagements a year. Its core values are customer service, employee development, and quality and integrity. SGA is an Equal Opportunity Employer and provides reasonable accommodation for people with disabilities.