Senior Software Development Test Engineer (SDET), AI & Automation

Quick Facts

• Work Policy: 100% Remote (United States), with a hybrid option for candidates in the El Dorado Hills, California or Chicago, Illinois metro areas
• Job Type: Full Time (Exempt)
• Experience Level: 5+ years in SDET, software development, test automation, or a similar code-intensive QA role

About the Role

Keeper Security is hiring a Senior SDET to join its AI team and build an automated approach to quality for AI-powered products and services. You’ll work as an embedded quality partner alongside engineers and product managers, defining test strategy, acceptance criteria, and automated coverage during development rather than after it. The role requires testing both deterministic software and non-deterministic AI/ML systems, where outputs can vary from run to run but still be acceptable. The overall goal is to move Keeper toward continuous quality engineering instead of end-of-cycle QA.

What You’ll Do

You’ll partner with AI engineers, software engineers, and product managers on test strategy, acceptance criteria, and automation requirements, and design and maintain automated test frameworks for AI applications, backend services, APIs, and related systems. A distinctive part of the work is developing testing approaches that cover both deterministic behavior and non-deterministic AI outputs, which includes defining measurable evaluation criteria, tolerances, and quality thresholds for LLM-driven features.

You’ll build regression, integration, API, and end-to-end test coverage alongside feature development, create harnesses and evaluation workflows for prompts, models, agents, tool-calling, and AI-integrated apps, and validate AI behavior across different models, prompts, inputs, and configurations. You’ll write automated test utilities in languages such as Python, Java, JavaScript/TypeScript, or Rust, and add validation and quality gates to CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI, or similar tools.

Day to day, you’ll maintain test datasets, mocks, fixtures, and synthetic data so validation is repeatable, and analyze failures to separate real defects from expected model variation, data issues, and test instability. You’ll assess AI outputs for reliability, consistency, safety, and alignment with product behavior, help engineering teams improve testability, observability, and diagnostics, and track test cases, results, and coverage in TestRail or similar tools. When issues arise, you’ll use logs and telemetry to find root causes and write reproducible defect reports, reduce flaky tests, and improve suite speed, stability, and maintainability. You’ll also contribute to quality standards and AI testing practices across the AI organization, and use AI-assisted tools such as Claude, ChatGPT, and GitHub Copilot for automation, test generation, debugging, and documentation.

• Partner with AI engineers, software engineers, and product managers on test strategy, acceptance criteria, and automation requirements
• Design and maintain automated test frameworks for AI applications, backend services, APIs, and related systems
• Develop testing approaches for both deterministic behavior and non-deterministic AI outputs
• Define measurable evaluation criteria, tolerances, and quality thresholds for LLM-driven features
• Build regression, integration, API, and end-to-end test coverage alongside feature development
• Create harnesses and evaluation workflows for prompts, models, agents, tool-calling, and AI-integrated apps
• Validate AI behavior across models, prompts, inputs, and configurations
• Write automated test utilities in languages such as Python, Java, JavaScript/TypeScript, or Rust
• Add validation and quality gates to CI/CD pipelines (GitHub Actions, Jenkins, GitLab CI, or similar)
• Maintain test datasets, mocks, fixtures, and synthetic data for repeatable validation
• Analyze failures to separate real defects, expected model variation, data issues, and test instability
• Assess AI outputs for reliability, consistency, safety, and alignment with product behavior
• Help engineering teams improve testability, observability, and diagnostics
• Track test cases, results, and coverage in TestRail or similar tools
• Use logs and telemetry to find root causes and write reproducible defect reports
• Reduce flaky tests and improve suite speed, stability, and maintainability
• Contribute to quality standards and AI testing practices across the AI organization
• Use AI-assisted tools (such as Claude, ChatGPT, and GitHub Copilot) for automation, test generation, debugging, and documentation

What You’ll Need

• 5+ years in SDET, software development, test automation, or a similar code-intensive QA role
• Hands-on experience testing AI/ML, LLM-powered, or model-integrated systems
• Understanding of how deterministic testing differs from non-deterministic AI validation
• Experience defining evaluation criteria, expected behaviors, tolerances, or scoring methods for AI features
• Strong programming skills in Python, Java, JavaScript/TypeScript, Rust, or another general-purpose language
• Significant experience building and maintaining automated test frameworks
• Experience with functional, integration, API, and end-to-end test coverage
• Experience testing REST APIs and distributed workflows with tools like Postman or REST Assured
• Experience integrating tests into CI/CD pipelines
• Experience in Agile teams working closely with QA, engineering, and product
• Understanding of shift-left testing, continuous quality, and testability
• Experience with TestRail or similar platforms
• Strong debugging skills using logs, APIs, telemetry, and development tools
• Experience with Git workflows, code review, and modern engineering practices
• Familiarity with AI-assisted development and testing tools
• Strong analytical skills for evaluating variable AI behavior beyond exact-match results
• Strong written and verbal communication skills
• Bachelor’s degree in computer science, engineering, or a related field, or equivalent practical experience
• Because of FedRAMP compliance requirements, candidates must be a U.S. person (U.S. citizen or lawful permanent resident)

Preferred Qualifications

• Experience testing LLMs, agentic systems, RAG, tool calling, or prompt-driven workflows
• Experience building model evaluation, benchmarking, or regression frameworks
• Familiarity with AI evaluation metrics for non-deterministic outputs
• Production or test-automation experience with Rust
• Experience with performance, reliability, security, or adversarial testing of AI systems
• Experience using AI to generate tests or synthetic test data
• Familiarity with Docker, Kubernetes, or cloud-based test environments
• Experience testing cybersecurity, identity, PAM, or other security-sensitive software
• Experience building shared automation frameworks or internal testing tools

Benefits

Medical, dental, and vision coverage, including domestic partnerships; employer-paid life insurance, with optional supplemental life for employees, spouses, and children; voluntary short- and long-term disability insurance; a 401(k) with Roth and traditional options; and a PTO plan that increases with seniority, plus paid bereavement and jury duty leave.