Executive Director, AI and Digital Health Assurance

To accomplish this job successfully, an individual must be able to perform, with or without reasonable accommodation, each essential function satisfactorily. Reasonable accommodations may be made to help enable qualified individuals with disabilities to perform the essential functions.

• Design and launch the offering. Define the service architecture, methodology, delivery model, and commercial structure for ECRI's AI assurance offering, and take it from concept to an operating, revenue-generating practice spanning advisory, independent evaluation, event investigation, and ongoing monitoring.

• Carry commercial accountability. Partner with sales, marketing, and product strategy to build pipeline, serve as the senior technical authority in client pursuits, and own bookings, revenue, margin, and retention for the practice.

• Productize what repeats. Convert recurring elements of the practice into scalable products and tools, including assessment frameworks, benchmark and reference datasets, monitoring instrumentation, and certification-readiness toolkits, in partnership with product strategy and the data and analytics products team.

• Set evaluation strategy and methodology. Establish ECRI's approach to evaluating AI-enabled solutions: evidence requirements, test design, reference and adversarial datasets, data provenance and fitness-for-context, subgroup and fairness performance, acceptance criteria, and reporting standards. Require disparities to be quantified and mitigated, not asserted away. Ensure methods are defensible, reproducible, and published where doing so advances the field.

• Maintain frontier fluency and track adoption. Personally track foundation model releases, agentic architectures, and clinical AI platforms as they ship, distinguishing genuine capability advances from marketing claims. Maintain a current view of how healthcare organizations are actually deploying AI, including the shift of agentic AI from pilot to production and the prevalence of unsanctioned "shadow AI" use. Translate both into updated evaluation criteria and where ECRI's evaluation and monitoring capacity is applied.

• Build and operate the test environment. Direct the design and operation of a secure sandbox for exercising AI systems against curated, edge-case, and adversarial inputs, covering accuracy, robustness, and failure modes, and, for generative and LLM-based tools, non-determinism, confabulation, and prompt sensitivity.

• Stand up AI safety event surveillance. Working with ECRI's Patient Safety Organization and hazard reporting programs, build the intake, taxonomy, and analytic capability to collect and classify AI-related safety events, near misses, and performance failures at scale, and to generate actionable signal for clients, vendors, and the field.

• Lead investigations of suspected AI-related harm. Direct root cause analyses of events in which an AI-enabled tool is implicated, applying systems-based and just culture methods to distinguish model failure, data failure, integration failure, workflow failure, and human-automation interaction failure, and translate findings into corrective action.

• Design post-deployment monitoring and close the loop. Specify how clients detect model drift, data distribution shift, degradation following vendor updates, and erosion of clinical benefit over time, including thresholds, cadence, escalation paths, and triggers for revalidation or decommissioning. Ensure findings from surveillance and investigation systematically update ECRI's evaluation criteria, hazard communications, and published guidance, so real-world field experience continuously strengthens pre-deployment testing.

• Advise senior leaders and track the regulatory landscape. Serve as ECRI's senior authority on AI assurance to boards, executive teams, AI governance committees, and vendor executives, translating technical findings, including limitations, uncertainty, and the recommendation not to deploy, into decisions those audiences can act on. Track and interpret FDA expectations for AI-enabled device software functions, Joint Commission's Responsible Use of AI in Healthcare certification standards, CHAI governance playbooks, NIST AI RMF, and ISO/IEC 42001, ISO 14971, and IEC 62304, and position ECRI's offering to help clients meet them.

• Direct security and threat evaluation of AI-enabled systems. Oversee assessment of AI-specific threats, including prompt injection and adversarial manipulation, agentic and tool-use attack surfaces (e.g., Model Context Protocol-style integrations), model and data supply chain exposure, unsanctioned "shadow AI" use, and access control for connected and EHR-integrated deployments. Reference frameworks such as the OWASP Top 10 for LLM Applications and MITRE ATLAS, in coordination with ECRI's cybersecurity capability.

• Build and lead the team. Recruit, develop, and lead a multidisciplinary team spanning clinical, data science, engineering, human factors, and cybersecurity expertise. Establish standard operating procedures, training, quality control, and peer review so that output is consistent, reproducible, and defensible at scale.

• Protect ECRI's independence. Own and enforce the conflict-of-interest and firewall protocols that keep ECRI's evaluations free of vendor influence, including clear separation between advisory work, independent evaluation, and any vendor-facing revenue.

• Represent ECRI in external standards, consensus, and policy forums on health AI assurance, including regulatory dockets, professional societies, and industry coalitions.

• Contribute AI-related content to ECRI's flagship publications and thought leadership, including the annual Top 10 Health Technology Hazards and Top 10 Patient Safety Concerns.

• Advise ECRI's internal AI program on the safe, appropriate, and defensible use of AI within ECRI's own products, services, and operations.

• Other duties, as assigned.

• Design and launch of the AI assurance offering against agreed milestones, including a published methodology and a defined, priced service catalog.

• Bookings, revenue, and margin for the practice; number and caliber of reference clients secured.

• Volume, cycle time, and quality of completed evaluations and investigations, with documented adherence to methodology and peer review standards.

• Growth and analytic yield of ECRI's AI safety event corpus, and demonstrable feedback of findings into evaluation criteria and published guidance.

• Client satisfaction and retention, and evidence of client action taken on ECRI findings.

• Team recruitment, retention, and capability development against plan.

• Integrity of ECRI's independence, evidenced by zero substantiated conflict-of-interest findings.

• Meet or exceed annual organizational, divisional, departmental, and individual goals.

• 10+ years of progressive experience in healthcare technology evaluation, clinical software quality and validation, medical device or health IT safety, or a closely related discipline.

• 3+ years of hands-on leadership or delivery experience specifically involving AI- or ML-enabled systems, including meaningful direct exposure to generative AI, agentic systems, or large language model applications.

• 5+ years leading teams, with demonstrated success building a function, practice, or program from an early or undefined state rather than inheriting a mature one.

• Demonstrated experience shipping an AI-enabled product or feature into production and operating it post-launch, with direct accountability for real-world performance, user adoption, and iteration. Academic research, pilots, or third-party evaluation alone do not substitute for this.

• Deep and current command of AI and machine learning as applied in healthcare, including the distinct evaluation challenges posed by generative, agentic, and large language model systems, sustained through active use of current tools rather than fluency anchored to a single past project.

• Demonstrated experience designing and executing validation or evaluation strategies for clinical software in regulated or highly governed environments, including design controls, traceability, change control, and risk management.

• Experience building or operating technical infrastructure to exercise and test AI systems and their outputs in a controlled sandbox environment, including construction of benchmark and adversarial test sets.

• Proven client-facing experience translating provider, payer, and vendor needs into measurable evaluation plans, and communicating technical and clinical findings, including limitations, uncertainty, and risk implications, in a practical and action-oriented manner.

• Demonstrated commercial contribution: shaping offerings, scoping and pricing engagements, supporting pursuits, and growing client relationships.

• Established credibility with senior healthcare and industry leaders, evidenced by publication, standards or consensus body participation, or speaking at recognized venues.

• Preferred: working knowledge of patient safety event analysis and root cause methodology, or demonstrated capability leading investigations of clinical harm events. Experience within a Patient Safety Organization, risk management, or quality and safety function is a strong advantage.

• Preferred: experience in one or more adjacent domains, such as connected and wearable device validation, EHR-integrated clinical decision support and workflow integration, or familiarity with the health AI assurance frameworks now in market (Joint Commission's Responsible Use of AI in Healthcare certification, CHAI governance playbooks).

• Bachelor's degree required in computer science, biomedical or software engineering, data science, information security, health informatics, or a related technical or scientific field.

• Advanced degree strongly preferred: a master's or doctoral degree in a related technical field, or a clinical doctorate (MD, DO, PharmD, DNP, PhD) paired with substantial informatics, data science, or health technology evaluation experience.

• An equivalent combination of education and directly relevant experience will be considered.

• Advanced, current, hands-on fluency across multiple leading foundation model families (proprietary and open-weight), agent and orchestration frameworks, and model/prompt evaluation and observability tooling, maintained through active use rather than point-in-time familiarity.

• Working knowledge of AI-specific threat and risk frameworks, such as the OWASP Top 10 for LLM Applications and MITRE ATLAS.

• Working familiarity with data analysis environments such as Python, R, or SQL, sufficient to direct technical work and independently interrogate results.

• Proficient with Microsoft 365 (Word, Excel, PowerPoint) and standard collaboration tools.

• Required: None.

• Preferred (AI governance and management systems): ISO/IEC 42001 lead implementer or auditor, or an equivalent AI management system credential.

• Preferred (Medical device and SaMD quality and regulatory): RAPS RAC (US or Global) and/or ASQ CQA or CQE, or an equivalent quality systems credential.

• Preferred (Patient safety): Certified Professional in Patient Safety (CPPS).

This position operates in a remote environment and requires the individual to remain in a stationary position, whether sitting or standing, before a desk or other fixed workspace, most of their workday. In addition, this position requires the individual to occasionally move about their workspace to access and inspect work-related materials, such as file cabinets with physical files and standard office equipment. This position requires the ability to operate standard office equipment, including, but not limited to, a laptop, keyboard, mouse, webcam, and phone, as well as effectively communicate information and ideas to a wide variety of audiences in written and oral form.