Principal AI Quality Developer

Toronto, ON
Permanent
Full-time

2 months ago

Job Requisition ID #26WD95110Position OverviewWe are hiring a principal quality engineering leader to design, build, and operationalize testing and evaluation for applied AI and ML systems across Autodesk products, with a focus on the Fusion Industry Cloud. This role is responsible for validating the quality, safety, reliability, and performance of AI features and agentic workflows shipped by the applied AI engineering team. You will create repeatable evaluation approaches for predictive and generative AI systems, build automated test harnesses integrated into CI/CD, and partner closely with Applied AI engineers, Product, Security, and Platform teams to ensure AI capabilities meet defined acceptance thresholds and customer expectations.Autodesk is a global leader in 3D design, engineering, and entertainment software, empowering innovators everywhere to imagine, design, and make a better world. Our software is used by professionals and hobbyists alike, across industries from architecture and manufacturing to media and entertainment. We are committed to diversity, flexibility, and continuous learning, offering a collaborative environment where you can grow your skills and make a real impact.You will lead the quality strategy for applied AI experiences that directly impact product experience and customer outcomes. This role offers the opportunity to define how we test and trust AI features at scale, partner with senior engineering leaders, and raise the bar for reliability and responsible AI in a high-impact product environment.ResponsibilitiesDefine and drive the quality strategy for applied AI and agentic features within Autodesk products and services with a focus on Fusion Industry CloudTest planning, acceptance criteria, and release readiness for AI-powered capabilitiesDesign and build automated evaluation frameworks for LLM and ML outputs, including curated gold sets, synthetic test sets, adversarial test cases, and regression suites that run in CI/CDCreate repeatable evaluation metrics and scoring approaches for AI behavior, including relevance, faithfulness, hallucination rate, toxicity, bias, robustness, latency, and cost, and partner with Engineering and Product to define pass or fail thresholdsValidate retrieval-augmented generation workflows, tool use, and agent behavior across edge cases, including prompt sensitivity, context recall, and retrieval accuracyLead AI safety and abuse testing in partnership with Security, including prompt injection and jailbreak scenarios, data leakage risks, and misuse cases relevant to product integrationsEstablish production monitoring and post-deployment validation for AI features, including drift detection, failure taxonomy, and alerting tied to customer impact and SLAsPerform deep issue triage and root-cause analysis for AI-related defects across data, prompts, retrieval, model versions, and integration logic, and drive fixes with Applied AI engineersMentor engineers and QA practitioners on AI testing approaches, improve team testing standards, and contribute to hiring and interview loops for quality rolesCollaborate with cross-functional partners to ensure responsible AI practices are embedded into development workflows, including documentation of test methodology, evaluation results, and release sign-off criteriaMinimum Qualifications7+ years of experience in quality engineering, test automation, or software engineering, with demonstrated ownership of test strategy and automation for complex systems, including services, platforms, or product integrationsStrong programming skills in Python and experience with test frameworks such as pytest or unittest and building automated test harnesses integrated with CI/CDHands-on experience testing AI or ML systems, including model output evaluation, error analysis, regression testing, and monitoring of quality over timePractical experience validating LLM-based systems, including prompt testing, retrieval-augmented generation workflows, hallucination detection, and robustness testing across edge casesExperience defining measurable acceptance criteria and working with product and engineering teams to translate requirements into evaluation metrics and release gatesProven ability to debug complex cross-stack issues spanning application logic, APIs, data flows, model behavior, and evaluation pipelines.Strong communication skills and ability to influence without direct authority in a collaborative, matrixed environmentBachelor’s degree in Computer Science, Engineering, or related field, or equivalent experiencePreferred QualificationsExperience with adversarial testing or AI security testing, including prompt injection, jailbreak testing, data leakage assessments, and abuse case modelingFamiliarity with observability tooling and telemetry pipelines, including tracing and structured logging for multi-step AI workflowsExperience with cloud platforms, AWS preferred, and containerized test environments using Docker and KubernetesExperience building or maintaining evaluation datasets, including gold, synthetic, and adversarial sets, and implementing automated scoring for quality and safetyBackground in responsible AI testing, including bias evaluation, explainability approaches, and privacy considerationsDomain knowledge in AEC, design, or manufacturingThe Ideal CandidateA technical quality leader who sets high standards, raises the bar through operational rigor, and delivers repeatable testing practices that scale with the organizationComfortable operating in a fast-paced, collaborative, and matrixed environment, partnering closely with engineers, product partners, and security stakeholdersDemonstrates One ORBIT values by being optimistic and solutions-focused, relentless about outcomes and quality, brave in surfacing risks early, ingenious in building new evaluation methods, and trusted in follow-through and transparent communicationLearn MoreAbout AutodeskWelcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made.We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world.When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us!Salary transparency Salary is one part of Autodesk’s competitive compensation package. For Canada-BC based roles, we expect a starting base salary between $99,000 and $145,200. Offers are based on the candidate’s experience and geographic location, and may exceed this range. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.Diversity & Belonging
We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here:Are you an existing contractor or consultant with Autodesk?Please search for open jobs and apply internally (not on this external site).

Autodesk

Apply Now