AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards. Key Responsibilities: - Architect and oversee end-to-end test strategies across different layers of the testing pyramid; - Define, evaluate, and monitor model validation criteria, including accuracy thresholds, confidence scoring, fallback behaviors, and more; - Drive automation of QA processes, ensuring robust, scalable solutions for data ingestion, extraction, and classification; - Lead validation of integration points between AI components and business logic, identifying risks and ensuring seamless operation; - Owenmaintain, and evolve test infrastructure, specifically CI/CD quality gates, data generation, and automated validation scripts; - Establish and monitor key quality metrics aligned with business goals, including turnaround time, confidence scores, and user corrections; - Act as a strategic partner to Product and Engineering teams, advocating for quality from design through delivery; MUST HAVES: - Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent experience; - 2–3 years of experience in AQA, Software Test Engineering, or related roles, including leadership responsibilities; - Experience in AQA JavaScript; - Solid knowledge of the software development life cycle (SDLC), Agile methodologies, and QA/QE best practices; - Proven experience building and scaling test automation frameworks (e.g., Pytest, Cypress, Selenium, Postman); - Strong technical skills in working with REST APIs, data pipelines, backend services, and cloud platforms (e.g., AWS, Azure); - Exceptional attention to detail and a mindset for building robust systems that catch issues early and protect the user experience; NICE TO HAVES: - Experience testing AI/ML systems in production environments; - Familiarity with GenAI model evaluation metrics; - Experience setting up proactive monitoring and alerting systems to detect model drift, performance degradation, or workflow failures.