Job Description AgileEngine is one of the Inc. 5000 fastest-growing companies in the US and a top-3 ranked dev shop according to Clutch. We create award-winning custom software solutions that help companies across 15+ industries change the lives of millions. If you like a challenging environment where you’re working with the best and are encouraged to learn and experiment every day, there’s no better place - guaranteed! :) WHAT YOU WILL DO - Architect and oversee end-to-end test strategies across different layers in the testing pyramid; - Define, evaluate, and monitor model validation criteria, including accuracy threshold, confidence scoring, fallback behaviors, etc.; - Drive the automation of QA processes, ensuring robust, scalable processes for data ingestion, extraction, and classification; - Lead the validation integration points between AI components and business logic, identifying risks and ensuring seamless operation; - Own, maintain, and evolve the test infrastructure, specifically the CI/CD quality gates, data generation, and automated validation scripts; - Establish and monitor key quality metrics aligned with the business goals, including turnaround time, confidence scores, and user corrections; - Act as a strategic partner to Product and Engineering, and advocate for quality from design through delivery; - Keep up with the latest AI tech and how to best apply the latest advancements to make the previously impossible possible for our customers; - Sense of ownership, take ownership of code quality through exploratory and automated tests. MUST HAVES - Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field experience; - 5+ years in AQA, Software Test Engineering, or related roles, including leadership responsibilities; - Experience in AQA Javascript; - Solid knowledge of software development life cycle (SDLC), Agile methodologies, and Quality Assurance or Quality Engineering methodologies; - Proven experience building and scaling test automation frameworks (e.g. Pytest, Cypress, Selenium, Postman, etc.). Strong technical skills with REST APIs, data, pipelines, and backend services; - Strong technical background with APIs, CI/CD pipelines, backend services, and cloud environments (AWS, Azure); - Exceptional attention to detail, with a mindset for building robust systems that catch issues early and protect the customer experience; - Demonstrated fast, proactive learning skills, especially crucial in the rapidly changing field of AI; - Uppper-Intermediate English level. NICE TO HAVES - Experience testing AI/ML in production environments; - Experience with GenAI model evaluation metrics; - Experience setting up proactive monitoring and alerting systems to catch issues like model drift, performance degradation, or workflow failures. THE BENEFITS OF JOINING US - Professional growth: Accelerate your professional journey with mentorship, TechTalks, and personalized growth roadmaps. - Competitive compensation: We match your ever-growing skills, talent, and contributions with competitive USD-based compensation and budgets for education, fitness, and team activities. - A selection of exciting projects: Join projects with modern solutions development and top-tier clients that include Fortune 500 enterprises and leading product brands. - Flextime: Tailor your schedule for an optimal work-life balance, by having the options of working from home and going to the office – whatever makes you the happiest and most productive. Your application doesn't end here! To unlock the next steps, check your email and complete your registration on our Applicant Site . The incomplete registration results in the termination of your process. Requirements Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field experience; 5+ years in AQA, Software Test Engineering, or related roles, including leadership responsibilities; Experience in AQA Javascript; Solid knowledge of software development life cycle (SDLC), Agile methodologies, and Quality Assurance or Quality Engineering methodologies; Proven experience building and scaling test automation frameworks (e.g. Pytest, Cypress, Selenium, Postman, etc.). Strong technical skills with REST APIs, data, pipelines, and backend services; Strong technical background with APIs, CI/CD pipelines, backend services, and cloud environments (AWS, Azure); Exceptional attention to detail, with a mindset for building robust systems that catch issues early and protect the customer experience; Demonstrated fast, proactive learning skills, especially crucial in the rapidly changing field of AI; Uppper-Intermediate English level.