On Vital Signs for Medical Economics earlier this month, I covered the questions you should be asking about AI scribing tools. The frame was direct: here are five questions to ask your AI scribing vendor, here is what the answer tells you, here is the red flag if the vendor hedges. This post is the procurement-side written companion, the deeper read for the executive who has to actually run the procurement and own the decision afterward. Yes, you have heard Question 1 before. Q2 through Q5 are the ones that get skipped under deal pressure. The five questions are the floor of vendor evaluation, not the ceiling. If your vendor cannot answer them clearly, that is the answer.
AI scribing has become the most common point of clinical AI procurement in 2026. Most health systems and large practice groups now have at least one ambient documentation tool in evaluation, pilot, or production. The vendor field has consolidated to roughly a dozen credible players, and the demos are good. The demos are very good. The demo is also not the product, and the validation slide deck is not deployment performance.
The five questions below are the ones that separate vendors who are confident in their product from vendors who are optimizing for the sale. They are the questions a sophisticated buyer asks. They are not, in my experience, the questions most procurement scorecards include.
Most clinical AI vendor scorecards built before 2024 weight three categories: feature parity, integration cost, and pilot performance. Each of those categories is reasonable. None of them, individually or together, predicts whether the vendor is a competent long-term partner for a clinical product.
Feature parity is table stakes; the vendors at the top of the field all have substantially the same feature surface. Integration cost is a procurement variable, not a clinical variable. Pilot performance is the vendor's best foot forward, in a controlled cohort, with vendor support resources fully attached. None of these answer the question that actually matters in Year 2 of deployment: is this vendor going to perform when the support team is no longer leaning in, when the patient population shifts, when the EHR upgrades, when the algorithm updates without a notification you can act on.
"The vendors who answer these questions clearly are ready for a real relationship. The ones who hedge are optimizing for the sale."
The vendor demo dataset rarely looks like your patient population. If the validation cohort skews younger, healthier, English-dominant, and academic-medical-center, the performance metrics on the slide deck do not predict performance on your actual patients. Specialty mix matters too: a tool validated on primary care visits has not been validated on orthopedic post-op visits or behavioral health intakes.
The right ask is a scoped validation in a representative subset of your population before procurement closes. A serious vendor will offer this or explain a credible path to it. If the answer is "we have not done that," or "our published research covers this," the vendor has just told you the answer is no. That is a procurement-stage red flag, not a deployment-stage red flag. The remediation cost is much lower if you raise it now.
The contract is where this lives. Look for two specific provisions. First, indemnification language that names the vendor for clinical errors traceable to the model. Second, a defined limitation-of-liability ceiling that is not capped at the contract value. Most vendor contracts default to a limitation of liability equal to 12 months of fees. For a clinical AI tool that could contribute to patient harm, that ceiling is not adequate insurance. The order of magnitude is wrong by a factor of 100 or more.
If the vendor will not negotiate these provisions, you are not buying a clinical product. You are buying a software license with clinical exposure, which is a different category of purchase with a different governance and insurance overhead. Your general counsel and your risk management team need to know that distinction before procurement closes.
Published research is the validation cohort. Implementation data is the production cohort. They are not the same. Validation studies are run with vendor support, in defined populations, with cleaning and adjudication processes that do not exist in routine production. Implementation data shows what happens after the vendor walks away from the install.
Ask for performance data from at least two reference deployments in environments structurally similar to yours: similar EHR, similar specialty mix, similar volume, similar clinical workflow. The vendor should be able to anonymize and produce this. The format can be a redacted deck, a structured walk-through with an existing customer, a reference call with the clinical owner at a peer site. If the answer is "we cannot share that for confidentiality reasons" with no follow-up offer, there is a reason. The reason is usually that the implementation data does not support the validation deck.
The right answer is a documented day-one operator guide, a defined escalation path, and a named clinical owner on the vendor side who is reachable when something goes sideways. The wrong answer is "we handle all the training, you do not need to worry about it." The wrong answer outsources clinical safety to the vendor. Your team is the one operating the tool in your clinical environment. They need to be able to run it, recognize when it is failing, and escalate without going through a vendor support queue that may not be staffed at clinical hours.
Day-one operator readiness is also a forcing function on the vendor's documentation quality. Vendors who can produce a clear day-one guide have done the work to understand how their tool actually behaves in clinical hands. Vendors who cannot have not, and the gap will surface in your environment within the first 90 days of go-live, while the implementation team is still on the floor.
Most vendor support contracts are 24 to 36 months. At month 30, the vendor's renewal team starts a conversation that is not about whether the tool is performing. It is about whether you will renew. If you do not renew, what happens to the tool? If the vendor sunsets the product, gets acquired, or pivots away from your use case, what is your transition plan?
The right time to negotiate the off-ramp is before you sign, not 30 months in. Specific provisions to ask for: a defined data export path for any documentation, audit logs, or model outputs you need to retain; a transition assistance window with defined hours and deliverables; a contractual right to receive performance data and integration documentation in a format your next vendor can ingest. If the answer is "we have not thought about it," you will be building that answer yourself, on the vendor's clock, when your only negotiating position is the threat of leaving.
Three structural shifts.
None of these shifts requires new technology. They require organizational will, explicit assignment of ownership, and a procurement scorecard that asks the questions the demo will not surface on its own.
It is not "which AI scribing vendor should we choose."
It is "is our governance structure built to ask these questions and act on the answers." In most health systems I work with, the procurement process and the clinical evaluation process are run by different teams with different incentives. Procurement wants the lowest cost and the cleanest contract. Clinical leadership wants the best performance and the cleanest clinical fit. Neither team owns vendor readiness as a strategic dimension. That gap is where the bad decisions get made, not by individuals making bad calls, but by an organizational structure that does not assign anyone to ask.
The advisory work in this zone is structural: building the evaluation scorecard, designing the cross-functional committee that owns vendor readiness, and producing the executive briefing that defends the decision at the board level when it gets challenged a year in.
The full 10-question version of this framework, with a fillable Due Diligence Checklist, the contract provisions to negotiate before signing, and the procurement scorecard template. Built for health system executives, governance committees, and operators running clinical AI vendor selection.
Get the course at drsarahmatt.com/course-ai-due-diligenceIf your health system is in active vendor evaluation right now, or you are designing the procurement scorecard for one, that is the conversation I have every week.
Book a call: calendly.com/sarahmattmdTHE SARAH MATT BRIEFING
The board-level read does not live on this blog.
Each week, subscribers get the field specifics, vendor questions, and practitioner tools that go beyond the public post. This week: the C-suite frame on the five-question vendor evaluation, plus a preview of Episode 16 of The Clinical Realist on the Q2 governance problem.
Weekly. No filler. Unsubscribe anytime.
Subscribe at drsarahmatt.com