TTC Logo

TTC

AI Tester

Reposted 14 Days Ago
Be an Early Applicant
Canada
Mid level
Canada
Mid level
The AI Tester will identify and mitigate risks in applications using Large Language Models by designing and executing comprehensive test strategies. Responsibilities include functional, performance, security, and compliance testing, as well as bias and fairness assessments.
The summary above was generated by AI

Job Summary: 
As an AI Tester, you will be responsible for identifying risks to the business and investigating the quality, accuracy, and fairness of applications built on top of Large Language Models. Working closely with data scientists, engineers, and business stakeholders, you will design and execute test strategies that address functional and non-functional requirements. Your expertise in AI testing will be critical in ensuring the application performs as intended, that the guardrails implemented mitigates risks, and aligns with regulatory and ethical standards. 

 

Key Responsibilities: 

Test Planning and Strategy: 

  • Develop comprehensive test plans and strategies for the LLM based application implementation, covering functional, performance, security, and ethical considerations. 
  • Collaborate with stakeholders to understand use cases and key business processes the AI agent will support. 

Application Validation and Functional Testing: 

  • Test the accuracy, consistency, and reliability of the LLM’s outputs across various scenarios and use cases. 
  • Validate the model’s adherence to prompts, contextual understanding, and user intent. 
  • Identify and address potential failure modes in conversational flows. 

Bias and Fairness Testing: 

  • Assess the model for biases and unfair treatment in outputs, ensuring compliance with ethical guidelines and company policies. 
  • Collaborate with data teams to test fairness across diverse customer demographics. 

Performance and Scalability Testing: 

  • Measure the response times, throughput, and scalability of the application under different load conditions. 
  • Test integrations with APIs, backend systems, and other enterprise tools. 

Security and Compliance Testing: 

  • Identify vulnerabilities and ensure the LLM meets security and data protection standards. 
  • Validate compliance with industry regulations such as GDPR, CCPA, or other industry standards & regulations. 

Automation and Tooling: 

  • Develop and maintain automated test suites for model retraining and updates. 
  • Use AI-specific testing tools and frameworks to improve efficiency and accuracy in validation efforts. 

Defect Reporting and Resolution: 

  • Document issues, collaborate with developers to resolve them, and retest to confirm fixes. 
  • Maintain clear and detailed test artifacts to support audits and reviews. 

 

Qualifications: 

Required: 

  • Bachelor’s degree in Computer Science, Engineering, or related field. 
  • 3+ years of experience in software testing, including at least 1 year of AI/ML testing experience. 
  • Familiarity with Large Language Models, NLP (Natural Language Processing), and AI frameworks. 
  • Strong understanding of testing methodologies and frameworks for AI systems, including bias testing, performance testing, and security testing. 
  • Programming skills in Python, JavaScript, or another language relevant to test automation. 
  • Excellent analytical and problem-solving skills with attention to detail. 
  • Experience working in agile development environments. 

Preferred: 

  • Experience with cloud-based AI services (e.g., AWS, Azure, Google Cloud AI). 
  • Knowledge of insurance industry processes and systems. 
  • Certification in AI/ML or software testing (e.g., ISTQB, Certified Machine Learning Specialist). 
  • Experience with testing tools such as LangTest, DeepChecks, or similar AI-focused platforms. 

 

Why Join Us? 

  • Be a part of transformative AI initiatives. 
  • Collaborate with a diverse, innovative team. 
  • Access opportunities for professional growth and upskilling in cutting-edge technologies. 
  • Competitive salary, benefits, and flexible work arrangements. 

If your experience or qualifications is similar to our ideal of a successful candidate, please consider applying. Experience comes in many ways; skills may be transferred, but passion for your career can't be substituted. At TTC, we understand the importance of diversity and how much value it brings to the table. Diversity brings about creativity and new perspectives, which is why we beckon everyone to apply. 

Top Skills

Ai Frameworks
AWS
Azure
Deepchecks
Google Cloud Ai
JavaScript
Langtest
Nlp
Python

Similar Jobs

Mid level
Blockchain • Internet of Things • Payments • Cryptocurrency • Web3
As a Site Reliability Engineer, you will optimize and manage cloud infrastructure costs, enhance service efficiency, collaborate with teams on cloud spending, and provide insights on cost trends while ensuring high operational standards.
Top Skills: ArgocdAWSGCPGithub ActionsGrafanaKubernetesTerraform
8 Hours Ago
Hybrid
Toronto, ON, CAN
Mid level
Mid level
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
As a QA Engineer at Capco, you will work on mission-critical projects for financial services clients, applying innovative solutions to enhance client success.
8 Hours Ago
Hybrid
Toronto, ON, CAN
Senior level
Senior level
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
The QA Engineer will conduct functional and performance testing, verify API accuracy, and develop automation scripts. Responsibilities include executing test cases and resolving API-related defects.
Top Skills: InsomniaPostmanSoapui

What you need to know about the Ottawa Tech Scene

The capital city of Canada and the nation's fourth-largest urban area, Ottawa has proven a rapidly growing global tech hub. With over 1,800 tech companies, many of which are leaders in their sectors, the city's tech talent now makes up more than 13 percent of its total workforce. This growth is driven not only by the big players like UL Solutions and Dropbox, but also by a thriving startup ecosystem, as new businesses emerge to follow in the footsteps of those that came before them.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account