Quality Engineer, AI & Test Automation

Quick Facts

• Work Policy: Remote (United States)
• Job Type: Full Time, day job
• Experience Level: 5 to 8+ years in quality engineering, software testing, test automation, SDET, software engineering, AI quality, or related roles
• Core Tech Stack: Python, JavaScript/TypeScript, Java, SQL; Playwright, Cypress, Selenium, Appium, PyTest, Postman/Newman, JUnit, TestNG
• Category: Technology

About the Role

The Quality Engineer, AI & Test Automation helps modernize Baylor Scott & White Health’s quality approach by moving beyond traditional manual testing toward automation-first, engineering-integrated, and AI-enabled quality practices. The role designs, builds, runs, and maintains automated test suites, validation utilities, evaluation assets, and quality reporting that improve release confidence and delivery speed across digital products, enterprise applications, and AI-enabled use cases.

You’ll partner closely with Product, Engineering, Architecture, Data & Analytics, AI Foundation, Ontology, Security, Privacy, Clinical, Operations, and other delivery teams to set quality expectations early and build quality practices into the whole development lifecycle. The role combines strong testing discipline with technical skills such as automation, API validation, data validation, CI/CD integration, quality metrics, defect triage, and emerging AI-enabled testing practices. It reports to the Director of Quality Engineering and helps establish reusable practices that support safe, reliable, efficient, and scalable delivery.

What You’ll Do

Quality Engineering and Test Strategy

• Develop test strategies, test plans, acceptance criteria, and quality approaches for software and AI-enabled capabilities with Product, Engineering, Architecture, and business stakeholders
• Translate requirements, user stories, workflows, APIs, data dependencies, and operational expectations into clear test scenarios and quality validation plans
• Identify quality risks early and recommend validation approaches, including automated, manual, exploratory, integration, regression, performance, accessibility, or AI-specific evaluation methods
• Give clear release-readiness input based on test evidence, defect trends, quality metrics, risk assessment, and stakeholder expectations

Test Automation and Engineering Practices

• Design, build, maintain, and improve automated test suites for API, UI, integration, end-to-end, regression, data, and workflow validation
• Integrate automated tests into CI/CD pipelines and delivery workflows so quality signals arrive earlier and more consistently
• Create reusable test data, utilities, fixtures, scripts, and automation patterns that multiple teams and products can use
• Improve test reliability, maintainability, execution time, coverage, and signal-to-noise ratio by reducing brittle scripts and eliminating repetitive manual validation where practical

AI-Enabled Testing and Evaluation

• Support testing and evaluation of AI-enabled experiences, including agent workflows, generated responses, retrieval behavior, tool and API calls, escalation paths, guardrail behavior, and human-in-the-loop patterns
• Help define evaluation datasets, expected behaviors, failure modes, scoring rubrics, regression scenarios, and quality thresholds for AI-enabled capabilities
• Use AI-enabled testing tools where appropriate to generate test cases, analyze logs, triage defects, find coverage gaps, summarize results, and speed up quality workflows
• Partner with AI Quality, Architecture, Engineering, and Product teams to confirm AI-enabled capabilities are useful, reliable, explainable enough for the use case, and aligned to defined quality expectations

Defect Management, Root Cause and Continuous Improvement

• Identify, document, reproduce, triage, and communicate defects with enough technical detail for engineering teams to resolve them efficiently
• Analyze defect patterns, escaped defects, quality trends, release issues, and production feedback to find root causes and improvement opportunities
• Work with engineering teams to improve testability, observability, logging, error handling, and diagnostic capabilities across applications and AI-enabled workflows
• Contribute to retrospectives, quality reviews, and process improvements that reduce rework, improve release confidence, and increase delivery velocity

Cross-Functional Partnership and Release Readiness

• Work closely with Product Managers, Senior AI Product Managers, Software Engineers, AI Engineers, Data Product Owners, Architects, Designers, Clinical and Operational stakeholders, and vendors and partners to align on quality expectations and release readiness
• Ensure quality criteria are built into requirements, design reviews, backlog refinement, development, testing, implementation, and post-release monitoring
• Support production validation, monitoring handoffs, post-release assessment, incident follow-up, and quality reporting for assigned products and use cases
• Communicate quality status, risks, defects, test results, and recommendations clearly to technical and non-technical stakeholders

Quality Standards, Documentation and Enablement

• Create and maintain test documentation, automation standards, quality playbooks, release-readiness checklists, defect triage practices, and reusable testing patterns
• Help teams adopt automation-first and AI-enabled quality practices while clarifying where manual validation is still needed because of risk, ambiguity, or workflow complexity
• Share knowledge with quality, product, engineering, and business stakeholders to raise the organization’s overall quality engineering capability
• Stay current on modern testing approaches, automation frameworks, AI evaluation practices, test data management, observability, and quality engineering methods

Key Success Factors

• Demonstrated ability to build and maintain meaningful automated tests rather than relying mainly on late-stage manual testing
• Strong technical judgment across API testing, UI testing, integration testing, data validation, regression testing, defect triage, CI/CD integration, and release readiness
• Ability to test AI-enabled systems whose behavior may be probabilistic, workflow-dependent, or context-sensitive rather than purely deterministic
• Ability to use AI-enabled productivity tools responsibly for test generation, analysis, defect triage, and quality reporting while keeping human judgment over release quality
• Strong collaboration with product and engineering teams to define quality expectations early and build quality into delivery rather than inspecting it at the end
• Strong analytical and communication skills, including explaining quality risks, evidence, tradeoffs, and release recommendations to technical and non-technical stakeholders
• Comfort working in a regulated or high-trust environment where safety, privacy, security, compliance, auditability, and customer trust are critical

Minimum Requirements

Education

• Bachelor’s degree in Computer Science, Engineering, Information Systems, Data Science, or a related technical field
• An equivalent combination of education and relevant technical experience may be considered

Experience

• 5 to 8+ years in quality engineering, software testing, test automation, SDET, software engineering, AI quality, or related technology delivery roles
• Hands-on experience designing and implementing automated tests for APIs, web or mobile applications, workflows, data validation, integrations, or end-to-end product experiences
• Experience working with engineering teams in agile, DevSecOps, CI/CD, or modern product-delivery environments
• Experience with defect triage, root-cause analysis, test evidence documentation, release readiness, and quality metrics
• Exposure to AI-enabled products, machine learning systems, large language models, conversational experiences, agentic workflows, automation, or decision-support technologies is preferred
• Healthcare, life sciences, financial services, or other regulated-industry experience is preferred but not required

Required Technical Expertise

• Strong understanding of software testing concepts, including test design, test coverage, functional, regression, integration, and exploratory testing, data validation, and release validation
• Hands-on experience with test automation frameworks, scripting, test data, CI/CD integration, and defect management tools
• Ability to validate APIs, backend services, user interfaces, data flows, workflows, and enterprise integrations
• Programming, scripting, or data analysis skills, preferably Python, JavaScript/TypeScript, Java, SQL, or comparable languages used to build or maintain automated test assets
• Familiarity with AI-enabled testing and evaluation concepts, including prompt and response validation, model-output evaluation, RAG or retrieval testing, agent workflow testing, hallucination risk, guardrails, and human-in-the-loop workflows
• Ability to use logs, traces, telemetry, monitoring, dashboards, test artifacts, and production feedback to investigate quality issues

Preferred Qualifications

• Experience as an SDET, Quality Engineer, Test Automation Engineer, Software Quality Engineer, AI Quality Engineer, or a similar technically oriented quality role
• Experience building or maintaining automation frameworks with tools such as Playwright, Cypress, Selenium, Appium, PyTest, Postman/Newman, JUnit, TestNG, or comparable frameworks
• Experience testing AI agents, copilots, chatbots, LLM-powered applications, workflow automation, RAG systems, tool-calling workflows, or other AI-enabled capabilities
• Experience creating evaluation suites, golden datasets, scoring rubrics, regression tests, synthetic test data, or quality dashboards for AI-enabled systems
• Experience with cloud, APIs, microservices, event-driven systems, data platforms, analytics, observability tools, or modern application architectures
• Experience supporting quality engineering transformation, reducing dependence on manual testing, or introducing AI-enabled testing practices into product teams

Benefits

• Immediate eligibility for health and welfare benefits
• 401(k) savings plan with a dollar-for-dollar match up to 5%
• Tuition reimbursement
• PTO accrual beginning on day 1

Benefits may vary based on position type and level.