Own model observability across auto claims models, building monitoring infrastructure, dashboards, alerts, health metrics, and remediation processes for traditional ML and GenAI systems. Detect drift, performance degradation, data-quality issues, and anomalous behavior; partner with data scientists, engineers, business stakeholders, and governance teams. Produce audit-ready reporting, evaluate model outcomes against business KPIs, support MLOps/LLMOps integration, and communicate model health findings to technical and non-technical audiences.
Description
The Auto Physical Damage (APD) Data Science team builds and deploys data science products that power faster, more consistent, and more accurate claims outcomes. Our portfolio spans both traditional machine learning models and Generative AI systems (e.g., document summarization, LLM-driven decision support, and unstructured-data extraction). As our model footprint grows, ensuring these systems remain accurate, reliable, and trustworthy in production is mission-critical.
We are seeking a Model Monitoring Analyst to design, build, and operate the systems that keep our production models healthy. You will be the owner of model observability across the APD portfolio - establishing how we detect performance degradation, data drift, and anomalous behavior for both classical ML and GenAI systems. This is a highly visible role that partners closely with data scientists, ML engineers, claims business partners, and model governance teams.
**Candidates who live within 50 miles of Boston, MA; Portsmouth, NH; Seattle, WA; Columbus, OH; or Plano, TX will follow a hybrid schedule, coming into the office two days per week. Otherwise, this role is remote with occasional travel.**
Key Responsibilities
The ideal candidate will have:
Additionally:
Qualifications
About Us
Pay Philosophy: The typical starting salary range for this role is determined by a number of factors including skills, experience, education, certifications and location. The full salary range for this role reflects the competitive labor market value for all employees in these positions across the national market and provides an opportunity to progress as employees grow and develop within the role. Some roles at Liberty Mutual have a corresponding compensation plan which may include commission and/or bonus earnings at rates that vary based on multiple factors set forth in the compensation plan for the role.
At Liberty Mutual, our goal is to create a workplace where everyone feels valued, supported, and can thrive. We build an environment that welcomes a wide range of perspectives and experiences, with inclusion embedded in every aspect of our culture and reflected in everyday interactions. This comes to life through comprehensive benefits, workplace flexibility, professional development opportunities, and a host of opportunities provided through our Employee Resource Groups. Each employee plays a role in creating our inclusive culture, which supports every individual to do their best work. Together, we cultivate a community where everyone can make a meaningful impact for our business, our customers, and the communities we serve.
We value your hard work, integrity and commitment to make things better, and we put people first by offering you benefits that support your life and well-being. To learn more about our benefit offerings please visit: https://www.libertymutualgroup.com/about-lm/careers/benefits
Liberty Mutual is an equal opportunity employer. We will not tolerate discrimination on the basis of race, color, national origin, sex, sexual orientation, gender identity, religion, age, disability, veteran's status, pregnancy, genetic information or on any basis prohibited by federal, state or local law.
Fair Chance Notices
$83,000.00 - 157,000.00
The Auto Physical Damage (APD) Data Science team builds and deploys data science products that power faster, more consistent, and more accurate claims outcomes. Our portfolio spans both traditional machine learning models and Generative AI systems (e.g., document summarization, LLM-driven decision support, and unstructured-data extraction). As our model footprint grows, ensuring these systems remain accurate, reliable, and trustworthy in production is mission-critical.
We are seeking a Model Monitoring Analyst to design, build, and operate the systems that keep our production models healthy. You will be the owner of model observability across the APD portfolio - establishing how we detect performance degradation, data drift, and anomalous behavior for both classical ML and GenAI systems. This is a highly visible role that partners closely with data scientists, ML engineers, claims business partners, and model governance teams.
**Candidates who live within 50 miles of Boston, MA; Portsmouth, NH; Seattle, WA; Columbus, OH; or Plano, TX will follow a hybrid schedule, coming into the office two days per week. Otherwise, this role is remote with occasional travel.**
Key Responsibilities
- Build monitoring infrastructure for production models, covering both traditional ML and GenAI/LLM systems, including automated pipelines, dashboards, and alerting.
- Define and track model health metrics - for ML: accuracy, precision/recall, AUC, calibration, feature and prediction drift. For GenAI: output quality, hallucination/grounding checks, relevance, latency, token/cost usage, and guardrail adherence.
- Detect and diagnose issues such as data drift, concept drift, performance decay, and data-quality breaks, then triage and escalate to the appropriate model owners.
- Establish thresholds and alerting that balance early detection with alert fatigue, and document expected behavior and remediation runbooks.
- Partner with data scientists and ML engineers to integrate monitoring into the model deployment lifecycle (CI/CD, MLOps/LLMOps).
- Support model governance and compliance by producing monitoring evidence, audit-ready reporting, and documentation aligned with enterprise model risk management standards.
- Analyze production outcomes against business KPIs to surface opportunities for model improvement or retraining.
- Communicate findings clearly to both technical and non-technical stakeholders through reporting and periodic model health reviews.
The ideal candidate will have:
- Bachelor's degree in a quantitative field (Statistics, Data Science, Computer Science, Engineering, Economics, or related), or equivalent experience.
- 3+ years of experience in data analytics, data science, ML engineering, or a related analytical role.
- Proficiency in SQL and Python for data manipulation and analysis.
- Solid understanding of machine learning concepts and model performance evaluation.
- Experience building dashboards and reports (e.g., Streamlit, Tableau, or similar).
- Strong analytical, problem-solving, and communication skills, with attention to detail.
Additionally:
- Graduate degree in a quantitative field (Statistics, Data Science, Computer Science, Engineering, Economics, or related), or equivalent experience.
- Experience with model monitoring / observability tooling
- Experience with A/B testing or experiment design to test impact of solutions
- Familiarity with GenAI/LLM evaluation concepts - prompt/response quality, hallucination detection, retrieval-augmented generation (RAG), guardrails, and LLM cost/latency monitoring.
- Exposure to cloud platforms (AWS, Azure, or GCP) and MLOps/LLMOps practices.
- Knowledge of the auto claims or insurance domain.
Qualifications
- Bachelor's Degree plus a minimum 3 years, typically 4 or more years of experience, or equivalent, is required.
- Mathematics, Economics, Statistics or other quantitative field are preferred fields of study.
- Advanced knowledge of data sources, tools, statistical principles and methodologies, and techniques.
- Advanced proficiency in Excel (VBA, macros, scripts, formulas, data visualization, etc.), PowerPoint, and statistical software packages (SAS, Emblem).
- Must have good planning, analytical, decision-making and communication skills. Solid understanding of business to improve business outcomes.
About Us
Pay Philosophy: The typical starting salary range for this role is determined by a number of factors including skills, experience, education, certifications and location. The full salary range for this role reflects the competitive labor market value for all employees in these positions across the national market and provides an opportunity to progress as employees grow and develop within the role. Some roles at Liberty Mutual have a corresponding compensation plan which may include commission and/or bonus earnings at rates that vary based on multiple factors set forth in the compensation plan for the role.
At Liberty Mutual, our goal is to create a workplace where everyone feels valued, supported, and can thrive. We build an environment that welcomes a wide range of perspectives and experiences, with inclusion embedded in every aspect of our culture and reflected in everyday interactions. This comes to life through comprehensive benefits, workplace flexibility, professional development opportunities, and a host of opportunities provided through our Employee Resource Groups. Each employee plays a role in creating our inclusive culture, which supports every individual to do their best work. Together, we cultivate a community where everyone can make a meaningful impact for our business, our customers, and the communities we serve.
We value your hard work, integrity and commitment to make things better, and we put people first by offering you benefits that support your life and well-being. To learn more about our benefit offerings please visit: https://www.libertymutualgroup.com/about-lm/careers/benefits
Liberty Mutual is an equal opportunity employer. We will not tolerate discrimination on the basis of race, color, national origin, sex, sexual orientation, gender identity, religion, age, disability, veteran's status, pregnancy, genetic information or on any basis prohibited by federal, state or local law.
Fair Chance Notices
- California
- Los Angeles Incorporated
- Los Angeles Unincorporated
- Philadelphia
- San Francisco
$83,000.00 - 157,000.00
Similar Jobs at Liberty Mutual Insurance
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Owner of model observability for APD models: build monitoring pipelines, dashboards, and alerts for ML and GenAI systems; detect and triage drift, performance decay, and anomalies; integrate monitoring into deployment CI/CD; produce audit-ready governance evidence; analyze production outcomes versus business KPIs; communicate findings to technical and business stakeholders.
Top Skills:
AWSAzureEmblemExcel (Vba)GCPGenerative AiLlmMlopsPowerPointPythonSASSQLStreamlitTableau
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Handles inbound calls and warm leads to understand customers’ insurance needs, recommend appropriate Property and Casualty coverage, and convert prospects into policyholders. The role includes paid training and licensing, customer relationship development, sales closing, and remote collaboration. Representatives must work four weekdays and one weekend day, maintain a dedicated home workspace, and use wired high-speed internet. Two years of sales or service experience is required, with a Property and Casualty license required after hire.
Top Skills:
Cable InternetDslFiber InternetPc
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Handle inbound calls and warm leads, consult customers on insurance needs, recommend appropriate Property and Casualty coverage, and convert prospects into policyholders. The role includes paid training and licensing, commission opportunities, customer service, schedule flexibility, and remote work. Representatives must work four weekdays and one weekend day, complete 16–18 weeks of training, and maintain a dedicated home workspace with high-speed wired internet.
Top Skills:
PcWired High-Speed Internet
What you need to know about the Ottawa Tech Scene
The capital city of Canada and the nation's fourth-largest urban area, Ottawa has proven a rapidly growing global tech hub. With over 1,800 tech companies, many of which are leaders in their sectors, the city's tech talent now makes up more than 13 percent of its total workforce. This growth is driven not only by the big players like UL Solutions and Dropbox, but also by a thriving startup ecosystem, as new businesses emerge to follow in the footsteps of those that came before them.
