Clinical AI Governance
The AI vendor agreement cleared legal, cleared procurement, and landed with a signature. Six months into deployment, the questions start. Who owns liability when a clinical decision goes wrong and the tool contributed? What triggers a vendor obligation when model performance degrades in your specific patient population? The answers are in the contract. Or they are not.
Here is what most health system leadership does not know about that document: it was written by technology lawyers working for a technology company. Every clause was drafted to minimize the vendor's exposure, not yours. The indemnification language, the performance definitions, the termination conditions, the data portability terms. All of it is optimized for the party that wrote it.
And almost all of it is negotiable.
The problem is not that legal teams are incompetent. It is that most legal teams reviewing clinical AI agreements have deep expertise in contract law and limited direct experience with what happens clinically when a deployed AI tool underperforms in a specific patient population at month seven. The technical lawyers on the other side have spent years refining these agreements. The operational and clinical consequences of their clause choices do not show up in their inbox.
They show up in yours.
Most clinical AI agreements are drafted by technology lawyers for technology companies. The document that arrives for your signature is optimized for the vendor's exposure, not yours. Almost everything in it is negotiable. Almost nobody negotiates it.
What follows is not a comprehensive contract review guide. It is a focused breakdown of five contractual positions that have consistently determined whether organizations maintain meaningful control over their clinical AI deployments, or hand that control to the vendor at signature.
Before the clauses: the structural problem that produces gaps in the first place.
Clinical AI procurement typically routes through a sequence: clinical champion identifies the tool, IT does a technical security review, procurement handles commercial terms, legal reviews the agreement. Each step adds a filter, but the filters are domain-specific. Clinical expertise is heaviest at the front. Legal expertise is heaviest at the back. The specific intersection of clinical consequence and contractual obligation rarely has a named owner in the room at signature.
The result is that legal reviews clinical AI agreements the same way it reviews SaaS agreements, because the document looks like a SaaS agreement. Standard indemnification, standard SLAs covering uptime and data security, standard termination for convenience. All of that is appropriate for software that manages your scheduling or billing. It is structurally insufficient for software that contributes to clinical decision-making in a specific patient population with a specific risk profile.
The vendor knows this. The clauses below are not industry secrets. They are simply the positions that most procurement teams do not know to ask for, so vendors do not volunteer them.
A vendor's published accuracy figures come from validation datasets. Those datasets may share nothing demographically with your patient population. A tool validated on a predominantly urban academic medical center population that deploys into a rural critical access hospital is operating outside its evidence base from day one. The published accuracy number still appears in your procurement documentation.
The contract should specify minimum performance thresholds validated on data representative of your patients, with a defined methodology for establishing that baseline and defined remediation timelines if thresholds are missed post-deployment. This is not an exotic ask. It is the clinical equivalent of what any well-run formulary committee would require before approving a new medication for a specific population.
"Best efforts" is not a performance standard. It is legal language that means the vendor will try, and that trying is sufficient regardless of outcome. Do not accept it in any clause that governs clinical tool performance.
"Vendor warrants that Tool performance on Organization's patient population will meet or exceed [X]% on [defined metric] within [N] days of go-live, validated via a retrospective analysis of Organization's de-identified patient records prior to deployment. If performance falls below threshold at any 30-day measurement window post-deployment, Vendor will provide a written remediation plan within 15 days and restore threshold performance within 60 days, or Organization may terminate for cause without penalty."
Clinical AI tools are not static deployments. Models retrain. Algorithms update. Output logic shifts in ways that may not be visible in the user interface or in standard uptime monitoring. A model that performed within your validated parameters at go-live may be operating on substantially different logic by month eight. Without a contractual obligation for advance disclosure, you will learn about material changes to clinical performance characteristics from your own incident reports.
The clause needs two components. First, mandatory advance written notice before any update that alters clinical output logic, with 30 days as the floor. Notice should include: a description of what changed, the expected direction of impact on accuracy and output behavior, and updated validation data if available. Second, an organizational right to pause deployment during the review window without triggering any contract penalty or support degradation. The right to pause is only meaningful if exercising it costs you nothing contractually.
Vendors will sometimes push back on 30 days for security patches. That is a legitimate carve-out. Negotiate it explicitly: emergency security patches may deploy on 72 hours notice, but any update touching clinical output logic requires the full window. The distinction matters because "security patch" is a category that can be stretched to cover almost anything if the contract does not define it narrowly.
"Vendor will provide written notice no fewer than 30 calendar days prior to any model update, algorithm change, or modification to clinical output logic. Notice will include a plain-language description of changes and updated validation results. Organization retains the right to delay implementation of any such update for up to 60 days for internal review, without service degradation or penalty. Emergency security patches are exempt from the 30-day requirement but exempt only if the patch does not alter clinical output logic, as attested in writing by Vendor's Chief Medical Officer or equivalent."
This is the clause most legal teams negotiate least aggressively, and the one with the sharpest clinical consequence.
Standard AI vendor agreements contain indemnification language that places liability for clinical decisions entirely on the health system, regardless of the tool's documented contribution to the outcome. The framing is usually something like: the tool provides clinical decision support, and all clinical decisions remain the responsibility of the licensed clinician. That framing is legally coherent and operationally dishonest.
A clinician who reviews an AI-generated recommendation and acts on it has made a clinical decision. But so has the vendor when it trained the model, validated it on a dataset that did not represent that patient, deployed it without adequate population-specific calibration, and pushed a model update without notice two months before the adverse event. The standard indemnification language erases the vendor's contribution entirely.
The goal here is not to eliminate your organization's liability. That is not achievable, and it should not be. The goal is to establish the vendor's obligation to participate in accountability when documented tool failure contributed to a patient outcome, and to prevent that participation from requiring litigation to initiate.
"In the event of an adverse clinical outcome in which Tool output is identified as a contributing factor in the organization's internal review, Vendor agrees to: (a) provide full documentation of Tool's performance logs, model version, and output record for the relevant clinical encounter within 10 business days of written request; (b) cooperate with any internal, regulatory, or external investigation, including providing technical witnesses; and (c) participate in a joint root cause analysis within 30 days of written request. Nothing in this section limits Vendor's right to assert its own defenses in any proceeding."
A vendor that refuses to discuss this clause before signature is communicating something useful about how they expect the relationship to function when problems arise.
Default termination clauses in clinical AI agreements protect the vendor. They govern termination for non-payment, termination for material breach, termination at contract expiration. What they rarely address is termination triggered by performance failure, which is the exit scenario most likely to matter.
Two pieces belong in this clause. First, objective performance triggers that activate an exit process: specific, measurable conditions under which the organization can terminate for cause without penalty, without waiting for contract expiration, and without needing to prove vendor fault in a legal proceeding. The performance standards negotiated in Clause 01 are the input. The exit triggers are the output.
Second, data portability with teeth. The contract should specify a guaranteed right to export all organizational data, including training contributions and inference logs, in a defined machine-readable format, within a defined window (30 days is a reasonable floor), at no additional charge, upon any termination type. Without this, your exit conversation begins from a position of structural dependence on a vendor you are trying to leave.
"Organization may terminate this Agreement for cause, without penalty, if Tool performance falls below the thresholds defined in Section [X] for two consecutive 30-day measurement windows, or if Vendor materially modifies Tool clinical output logic without providing the notice required in Section [Y]. Upon any termination, Vendor will provide Organization with a complete export of all Organization data, including de-identified inference logs and any Organization-contributed training data, in [CSV/HL7 FHIR/defined format] within 30 calendar days, at no charge."
This is the clause that most health systems discover after the fact, and the one with the longest tail.
Many clinical AI agreements include language permitting the vendor to use organizational data, de-identified and aggregated, to improve their models. This is often buried in a data use or privacy section and framed as a standard practice that benefits the product. What it means operationally: your clinical data, generated by your patient encounters, is improving a commercial product that the vendor will license to your competitors at full market rate.
The question is not whether to permit this categorically. Some organizations may find genuine value in contributing to shared model improvement. The question is whether the clause is explicit, whether it requires affirmative opt-in rather than opt-out, whether it provides any form of recognition or compensation for the contribution, and whether it survives termination. Most default clauses permit this indefinitely, including after the contract ends.
At minimum, the training data clause should require affirmative written consent for each use category, prohibit use of organizational data to train models marketed to direct competitors without organizational approval, and include a defined sunset on training rights upon contract termination. The sunset provision matters. Without it, data contributed during a five-year contract continues improving the vendor's product for the life of the model.
"Vendor may use Organization's de-identified data to improve Tool only with Organization's prior written consent for each defined use category. Vendor may not use Organization's data to train models marketed to organizations in Organization's defined competitive market without separate written agreement. All training rights terminate 12 months after expiration or termination of this Agreement. Vendor will provide an annual data use report documenting all instances in which Organization's data contributed to model training."
Most vendors will negotiate most of these clauses. Not all of them, and not without pushback, but the opening position in a clinical AI agreement is not the final position. The organizations that successfully negotiate better terms share one characteristic: they raise the clauses before procurement pressure is fully engaged, while the vendor still needs the deal more than the organization needs that specific vendor.
The sequence matters. Raising liability language after heads of terms are agreed and the contracting team is focused on closing is a much harder conversation than raising it in the initial term sheet discussion, when the vendor's legal team has not yet invested weeks in the current draft.
Practical note on the liability clause specifically: many vendors will accept the documentation and cooperation language without accepting shared financial liability. That is a reasonable outcome. The documentation and cooperation piece is what actually matters in most adverse event scenarios. Litigation over shared financial liability is slow, expensive, and uncertain. Clear contractual access to the vendor's documentation and technical personnel is what makes your internal review process function.
One vendor response worth knowing: if the vendor's legal team comes back with a unilateral refusal on any of the first four clauses, with no counter-offer and no explanation, that response is itself useful clinical governance information. It tells you how this vendor defines partnership when something goes wrong. That is a legitimate procurement input before the agreement is signed.
Contract negotiation is one layer. The governance structure that activates those clauses is what most organizations never build, and that gap is where the contract language becomes irrelevant in practice.
Three questions that should be answered in writing before any clinical AI tool goes live:
The pattern in organizations that exit bad AI tool relationships cleanly: they wrote the exit criteria and the exit process before the tool went live, when the vendor was cooperative, the clinical team was engaged, and nobody had a personal stake in proving the deployment succeeded. That is when specificity costs nothing.
The pattern in organizations that spend twelve months in a vendor dispute: they relied on the contract language to do work that only a functioning governance structure can do.
If your organization deployed a clinical AI tool in Q1 or Q2 of this year, you are now in the window where the early adoption energy has worn off and the real performance picture is coming into focus. This is when the governance gaps that were present at contract signing become operationally visible.
The most common scenario: performance is mixed, the contractual benchmarks were never clearly defined, the exit criteria were not written, and the vendor's roadmap is increasingly the frame through which performance is being evaluated. The sunk cost framing is available to everyone in the room, and it is doing work that governance documentation should be doing.
If that description is accurate, the contract renegotiation window is not closed. Addenda and amendments are normal, and vendors who want long-term relationships will often engage on structural improvements mid-contract if the ask is specific and the relationship is functional. The five clauses above translate directly into an amendment conversation.
Episode 18 of The Clinical Realist goes live Wednesday, May 13. The topic: what clinical AI accountability actually looks like in practice, the third component most organizations have not built, and what the gap looks like from the inside of an adverse event review.
The advisory work in this zone: contract review against the five-clause framework, governance structure documentation, vendor conversation scripts, and the executive brief that makes the case at the board level when performance is not meeting expectations.
Book a Discovery and Clarity Session →Prefer to start with a structured framework?
AI Vendor Due Diligence course →