Lead QA Engineer

Quick Facts

• Work Policy: Remote (US), with occasional travel to client sites and team offsites
• Job Type: Full Time
• Experience Level: 7+ years in QA, including at least 2 years testing AI or ML systems
• Core Tech Stack: Python, pytest, Playwright or Selenium, Azure, REST APIs, CI/CD

About Titan

Titan builds AI software for banks, including purpose-built small language models, a banking ontology, and AI bankers. Its customers include community banks, credit unions, and large regional institutions.

Why This Role Exists

Titan does not yet have a formal QA function, evaluation framework, regression baseline, or CI/CD quality gate. This role is hands-on and individual-contributor first, and QE engineers will be hired later to scale the practice. You’ll build and test rather than manage.

What You’ll Own

AI evaluation. You’ll design and run evaluation frameworks for LLM and agentic outputs across Foundry, Agent Builder, and client-deployed instances, including assertions, behavioral contracts, and regression baselines for model behavior.

Test coverage. You’ll build and maintain end-to-end, integration, and regression suites for APIs, document ingestion, AI inference, and the frontend. You’ll own performance and load testing, enforce CI/CD quality gates, and write regression tests for production bugs.

Compliance and client quality. You’ll produce SOC 2 Type II test artifacts, audit logs, and process documentation, and work with Forward Deployed Engineering on client-side validation.

• Design and run evaluation frameworks for LLM and agentic outputs across Foundry, Agent Builder, and client-deployed instances
• Create assertions, behavioral contracts, and regression baselines for model behavior
• Build and maintain end-to-end, integration, and regression suites for APIs, document ingestion, AI inference, and the frontend
• Own performance and load testing
• Enforce CI/CD quality gates
• Write regression tests for production bugs
• Produce SOC 2 Type II test artifacts, audit logs, and process documentation
• Work with Forward Deployed Engineering on client-side validation

Who You Are

• Seven or more years in QA, with at least two years testing AI or ML systems
• Fluent in Python
• Experience with pytest, Playwright, or Selenium
• Hands-on with evaluation tools such as RAGAS, DeepEval, or LangSmith
• Comfortable with Azure, async systems, REST APIs, and CI/CD
• Fintech or regulated-industry experience is a strong advantage

What Success Looks Like

• 90 days: A test coverage diagnostic, an evaluation framework running on one AI workflow, and quality gates live in CI/CD
• Six months: Regression baselines for model behavior, audit-ready SOC 2 artifacts, and a test suite that runs on every release
• One year: The QA function is staffed, and quality is a first-class input to deployment decisions

Compensation

Competitive base salary and meaningful equity. No salary range is listed.