Bask Health
Job Description
Quick Facts
- Work Policy: Remote (United States)
- Job Type: Full Time
- Experience Level: 3+ years of professional software engineering experience
- Core Tech Stack: TypeScript, JavaScript, Python, APIs, Relational Databases, Automated Testing Frameworks, CI/CD, LLMs & AI Evaluation Infrastructure, Git
About Bask Health
Bask Health provides an integrated software platform that empowers healthcare businesses, physicians, and entrepreneurs to launch, operate, and scale direct-to-consumer telehealth offerings. Unifying commerce tools, pharmacy operations, clinical workflows, and patient experiences in a single ecosystem, Bask Health operates with an entrepreneurial, AI-first, builder-led engineering team that utilizes modern tools such as Claude Code and Cursor while upholding rigorous standards for software quality, testing, and system reliability.
About the Role
Bask Health is hiring a Software Engineer, AI Quality & Evaluation to architect and build the core engineering infrastructure used to test, evaluate, and refine AI-driven applications across development and production lifecycles. Rather than evaluating isolated models manually, you will develop reusable backend services, developer utilities, evaluation execution engines, dataset management tools, and automated regression pipelines that help engineering teams measure how nondeterministic AI systems behave. Collaborating closely with backend, machine learning, and applied AI engineers, you will embed AI evaluation directly into everyday engineering workflows while ensuring all evaluation systems enforce strict tenant isolation, access permissions, and healthcare data privacy safeguards.
What You’ll Do
- Architect, build, and maintain core software infrastructure supporting automated AI quality and evaluation across Bask Health’s platform.
- Develop production backend services and asynchronous job processing systems—complete with error handling, timeouts, retries, job scheduling, monitoring, and logging—to execute large-scale evaluation workloads reliably.
- Build APIs enabling engineering teams to trigger evaluation runs and retrieve structured outcomes, alongside internal dashboards and inspection tools for analyzing AI responses, test histories, and failures.
- Create reusable software libraries and testing tools for Large Language Model (LLM) applications, covering structured model outputs, tool-calling behavior, and Retrieval-Augmented Generation (RAG) workflows.
- Develop systems to manage test cases, expected outcomes, and evaluation datasets, as well as automated workflows for comparing application configurations, prompts, and models.
- Implement automated AI regression testing infrastructure and integrate evaluation checks directly into CI/CD pipelines.
- Optimize evaluation infrastructure costs, execution speed, and reliability while troubleshooting service issues and maintaining thorough developer documentation.
- Collaborate with machine learning and AI evaluation specialists to translate testing needs into scalable developer tools, enforcing strict data privacy, tenant boundaries, and access controls for sensitive healthcare information.
- Participate actively in architecture discussions, technical planning, and code reviews, owning assigned AI quality engineering initiatives from initial design through production deployment and ongoing improvement.
What You’ll Need
- At least 3 years of professional software engineering experience, ideally within developer tooling, infrastructure, platform, or backend engineering roles.
- Proven background developing and maintaining production backend services, APIs, and service integrations.
- Strong programming capabilities in Python, JavaScript, TypeScript, or comparable languages, writing clean, maintainable, and well-tested production code.
- Hands-on experience with relational databases, structured application data, background job systems, asynchronous processing, and distributed systems reliability.
- Solid familiarity with automated testing frameworks, automated testing pipelines, CI/CD workflows, release processes, and configuration management.
- Practical understanding of Large Language Models, modern AI application development, and the unique challenges of testing nondeterministic AI systems.
- Working knowledge of secure service communication, authorization, authentication, API error handling, retries, failure recovery, logging, monitoring, and production diagnostics.
- Experience using Git in collaborative engineering workflows, paired with strong analytical problem-solving skills and the ability to independently own projects from technical design to deployment.
Nice to Have
- Direct experience building machine learning testing infrastructure, automated benchmark systems, evaluation execution services, or internal developer platforms and SDKs.
- Hands-on familiarity with Anthropic, OpenAI, or similar model APIs, as well as evaluation tools like promptfoo, Ragas, DeepEval, or LangSmith and LLM-as-a-judge methodologies.
- Experience with AI request tracing, model observability, vector databases, embeddings, RAG architectures, and prompt comparison or AI regression workflows.
- Familiarity with Node.js and TypeScript backend development, React or Next.js for internal interfaces, and tools such as Drizzle, Prisma, or tRPC.
- Background working with serverless cloud architectures and infrastructure-as-code tools, including SST and AWS services (EventBridge, SQS, Lambda).
- Understanding of data isolation in multi-tenant SaaS systems, healthcare privacy and HIPAA considerations, and AI-assisted development tools such as Claude Code or Cursor.
To apply for this job please visit apply.workable.com.