Weekday AI
Job Description
Quick Facts
- Work Policy: Remote (India; hiring for a Weekday AI client)
- Job Type: Full Time
- Salary Range: INR 1,000,000 – 2,500,000 per year (INR 10 – 25 LPA)
- Experience Level: 4+ years in AI or data science QA environments
- Core Tech Stack: Python, pytest, REST API Testing (Requests, RestAssured), GenAI, LLMs, Agentic AI, RAG, CI/CD (Jenkins, GitHub Actions, GitLab CI)
About the Role
Weekday AI is recruiting a remote QA Engineer in India on behalf of one of its client organizations to design, build, and execute end-to-end testing strategies for advanced Large Language Model (LLM) and Generative AI platforms. Partnering closely with product managers, machine learning engineers, and data scientists, you will translate complex, non-deterministic AI behaviors into structured, measurable test suites covering Retrieval-Augmented Generation (RAG) and Agentic AI workflows.
Alongside building automated Python and pytest validation pipelines integrated into CI/CD, you will track model performance metrics, detect output anomalies, maintain quality dashboards, and promote scalable AI QA standards across engineering teams.
What You’ll Do
- Design and execute comprehensive test plans tailored for RAG, Agentic AI, LLM, and Generative AI systems.
- Architect, build, and maintain automated test suites utilizing Python and pytest.
- Create and run automated API tests validating microservices, data pipelines, and model inference endpoints.
- Conduct performance, regression, and functional testing across AI-powered services.
- Embed automated test runs into CI/CD workflows using platforms such as GitLab CI, GitHub Actions, or Jenkins.
- Evaluate and document model fairness, bias, and output quality metrics.
- Partner with machine learning teams to build dedicated test harnesses for upcoming model releases.
- Develop and maintain mock services, test fixtures, and curated test datasets.
- Log software and model defects, track fixes through resolution, and coach junior QA engineers on AI testing best practices.
What You’ll Need
- At least 4 years of professional QA engineering experience within data science or AI-focused environments.
- Strong programming and testing proficiency in Python, pytest, and API testing libraries or frameworks (such as RestAssured or Requests).
- Practical, hands-on experience applying testing methodologies for RAG, Agentic AI, LLM, and GenAI applications.
- Working familiarity with test automation tooling and CI/CD pipelines (such as GitHub Actions or Jenkins).
- Solid grasp of model explainability, bias mitigation, and AI ethics principles.
- Strong analytical thinking and problem-solving abilities with keen attention to detail.
- Capability to distill complex AI system behaviors into clear, verifiable test requirements.
- Effective communication skills for cross-functional collaboration, backed by a track record of on-time, high-quality software releases and a passion for emerging AI testing trends.
To apply for this job please visit jobs.workable.com.