QA Engineer, Voice AI Agents

Quick Facts

• Work Policy: Remote (US); remote or onsite options also listed for the New York City and Boca Raton offices
• Job Type: Full Time
• Salary Range: $180,000 – $200,000 base, plus a competitive equity package
• Experience Level: 3 to 5 years in QA, support engineering, technical operations, or a similar production software role

About the Role

The QA Engineer reports to the Head of Product and investigates calls and platform issues on a voice AI platform. You’ll review conversations, trace failures through transcripts, logs, and campaign configuration, identify root causes, and work with engineering and deployment teams on fixes. The job combines hands-on investigation, structured QA processes, and a growing amount of automation.

What You’ll Do

Call and issue debugging. You’ll investigate problems on live campaigns, such as dropped calls, incorrect agent responses, failed handoffs, wrong scheduling outcomes, and data mismatches. You’ll trace each problem across recordings, transcripts, logs, configuration, and integration data, write reproducible bug reports with severity, member impact, and evidence, and triage issues to the right owner, whether that is agent behavior, configuration, data, or a platform bug.

Campaign QA. You’ll run pre-launch testing on new campaigns and workflow changes, covering scripts, edge cases, escalation paths, and nurse or live-staff handoffs. You’ll carry out ongoing conversation QA against quality standards and flag trends early, and verify that campaigns behave as designed, including targeting, sequencing, disposition logic, and outcome capture.

Process and automation. You’ll build and maintain the QA framework, including test cases, scoring rubrics, regression suites, and release checklists. You’ll automate repetitive checks such as transcript analysis, configuration validation, and outcome reconciliation, and track quality metrics so patterns get reported and issues are fixed at the source.

• Investigate live-campaign issues: dropped calls, incorrect agent responses, failed handoffs, wrong scheduling outcomes, and data mismatches
• Trace problems across recordings, transcripts, logs, configuration, and integration data
• Write reproducible bug reports with severity, member impact, and evidence
• Triage issues to the right owner (agent behavior, configuration, data, or platform bug)
• Run pre-launch testing on new campaigns and workflow changes, including scripts, edge cases, escalation paths, and nurse or live-staff handoffs
• Conduct ongoing conversation QA against quality standards and flag trends early
• Verify that campaigns behave as designed, including targeting, sequencing, disposition logic, and outcome capture
• Build and maintain the QA framework: test cases, scoring rubrics, regression suites, and release checklists
• Automate repetitive checks such as transcript analysis, configuration validation, and outcome reconciliation
• Track quality metrics and report patterns so issues are fixed at the source

What You’ll Need

• 3 to 5 years in QA, support engineering, technical operations, or a similar production software role
• Strong debugging instincts
• Comfort with logs, SQL, APIs, and JSON or YAML configuration
• Experience writing test plans and bug reports that engineers trust
• Clear written communication and precise documentation
• Good judgment on severity and prioritization

Nice to Have

• Experience with voice AI, conversational AI, IVR, or contact center platforms
• Python or similar scripting skills
• Exposure to LLM evaluation, prompt testing, or conversation quality scoring
• Healthcare experience, especially payer or provider outreach, with HIPAA and PHI familiarity
• Experience with observability or call analytics tools

Benefits

Competitive salary and equity, generous PTO, health, vision, and dental insurance, life and disability insurance, Simple IRA matching, a professional development budget, and wellness programs.