AI Model Assurance Analyst

About the Team

JV AI & Systems Integrity is the independent function inside the TikTok USDS Joint Venture security organization that opines on whether generative AI features shipping to US users meet the JV's safety and content commitments. We operate the approval gates, testing programs, deviation adjudication, and audit-facing evidence pipeline for every JV-covered GenAI surface. We work in close partnership with legal, safety, and engineering teams.

About the Role

The AI Model Assurance Analyst is the operational backbone of our GenAI oversight program. You will be the person who makes our control set (covering pre-launch approval, change management, change and deviation detection, inventory, independent validation, incident response, metrics reporting, and access segregation) run day-to-day: routing and completing intake and assessment tickets, driving evaluation dossiers to decisions, keeping the our risk and safety strategy registries accurate, monitoring performance, coordinating independent validation, and producing the monthly/quarterly oversight reports that go to JV leadership.

This is a hands-on operations role with meaningful policy and technical surface area. You will discuss and opine on release thresholds, develop strategies to catch unnotified changes, drive automation in our end-to-end process, and be the person a launch is waiting on when the evidence isn't good enough yet. We need someone who can hold this operational tempo without losing the thread on independence.

Responsibilities

• Manage the intake queue and progression of GenAI model and feature releases; validate submission completeness and be accountable for driving SLAs. Escalate risk of slippage proactively.
• Classify submissions as material vs. non-material vs. emergency and route them to appropriate evaluation teams
• Prepare approval-decision packages (approve / approve-with-conditions / reject / request more evidence) for the Content Assurance lead and log conditional approvals against post-launch monitoring commitments.
• Operationalize change-detection tooling (e.g., hash / configuration snapshots, deployment-event feed, canary-input probes) and triage unnotified-change alerts
• Codify the performance envelope for each approval node against our standards of independent testing
• Watch alerts routing into the Content Assurance queue in parallel with T&S; run joint triage; distinguish drift from change; drive deviation adjudication to closure and log outcomes
• Support CA incident response; co-author author remediation plans and post-incident reviews in partnership with our incident response team and compliance engineering teams.
• Feed the audit evidence pipeline; own artifact readiness against GRC's evidence request list and track completion percentage weekly.
• Produce the standing JV oversight reports on a regular cadence: adoptions and changes processed, incidents, SLA breaches, validation coverage, open remediation, and other details.
• Drive the weekly joint T&S / AISI triage, the quarterly rotating deep-dive, and the annual program review and threshold recalibration.
• Track program-level KPIs and surface trends and blockers to leadership.
• Contribute to model assurance evaluation refreshes (risk-based eval, neutrality test sets, safety testing framework and SOP) and the "white glove" testing service for partner product teams
• Contribute to the AI Model Assurance Dashboard, model release pipeline automation, and notification loops across all cross-functional partners for an integrated source of truth on independent testing status

Minimum Qualifications

• Bachelor's degree in a technical, policy, or risk-adjacent field (e.g., Computer Science, Statistics, Information Security, Public Policy, Trust & Safety, Compliance), or equivalent practical experience.
• 3+ years of operations, program management, GRC, T&S policy operations, or model-risk / model-ops experience, including at least one role owning a recurring cross-functional workflow with hard SLAs.
• Direct working knowledge of at least one of: generative AI / LLM systems, content moderation classifiers, trust & safety operations, model-risk management, or AI/ML evaluation.
• Ability to read and reason about an evaluation dossier (precision / recall, refusal rate, false-positive rate on protected categories, red-team results) and to challenge materiality classifications on the substance rather than only the paperwork.
• Fluency operating ticketed intake and approval workflows (Jira / Meego / equivalent), maintaining registries of record, and producing structured evidence packages for audit or regulatory review.
• Demonstrated ability to drive cross-functional partners (engineering, policy, legal, security) to a decision on a fixed clock.
• Written communication strong enough to draft SOP language, decision memos, and monthly oversight reports without heavy editing.

Preferred Qualifications

• Experience running or supporting a third-party pre-assessment or independent audit — including artifact readiness, control walk-throughs, and finding remediation planning.
• Hands-on familiarity with red-teaming, adversarial test set design, or evaluation harnesses for text, image, audio, or video GenAI systems.
• Basic scripting proficiency (Python / SQL) sufficient to pull metrics from a dashboard, join two tables, or sanity-check a claimed evaluation delta.
• Exposure to internal tooling is a strong plus but not required.