Skip to main content
Back to blog

AI Procurement Checklist for Healthcare CIOs

A practical checklist for evaluating AI vendors and AI projects in healthcare — the questions to ask before money moves and the red flags to watch for in vendor responses.

HealthcareProcurementHIPAAAI

AI procurement in healthcare is harder than software procurement was a decade ago. The data path is more complex, the regulatory exposure is broader, and the vendor ecosystem includes everything from established health IT firms to two-person startups with a Cohere wrapper.

This is the checklist we wish more CIOs had when we walk into a procurement conversation. It is organized by the questions auditors and Security Officers actually ask, not by the marketing categories vendors use.

Data path

  1. Where does PHI go? Trace every system the data touches: ingestion, embedding, vector storage, model invocation, post-processing, logging, analytics. Each is on the BAA or it is not.
  2. What model is the vendor using, and does it have a BAA? "We use AI" is not an answer. The specific model and the specific provider matter. AWS Bedrock under the AWS BAA, Azure OpenAI under Microsoft's BAA, and Google Cloud's Vertex under Google's BAA are different vendors with different documentation. Get specifics.
  3. Where does the model run? A model "deployed in your VPC" is different from a model "accessed via the vendor's API endpoint." Both can be appropriate. Mixing them up is not.
  4. Does the vendor train on customer data? Default policy at major LLM providers is that enterprise tier customers' data is not used for training, but verify in the contract, not the marketing page.

Tenant isolation

  1. How is your data separated from other customers' data? Index per customer, namespace per customer, or shared resources with metadata filters. Each has different security implications. Get specifics.
  2. Can you produce evidence of isolation? Vendor diagrams are not evidence. Test data, configuration, and logs are.
  3. What happens if another customer is breached? A vendor whose architecture has a single shared resource is one breach away from being yours. A vendor with hard isolation is not.

Audit logging

  1. What does the vendor log? Every model invocation, with prompts, retrieved context, and outputs, is the right answer. "Aggregate metrics" is not.
  2. Can you access the audit log? Some vendors keep the AI audit log internal. For HIPAA work, you need to be able to query "every interaction touching this patient" without a support ticket.
  3. How long is the audit log retained? Six years is the HIPAA minimum. Some vendors retain shorter; some retain forever (which has its own discovery exposure).
  4. What is in the audit log for tool calls? If the AI calls back into your systems, every tool invocation needs a log entry with parameters and results.

Right-to-deletion

  1. If a patient asks for deletion, what is the path? PHI in source systems, PHI in the vector index, PHI in audit logs. Each needs a path. "We can delete records on request" without specifics is not enough.
  2. If you terminate the contract, what happens to your data? Data return, data destruction, attestation. Spell out the timeline.

Access control

  1. How are users authenticated to the AI system? Federation with your IdP (Entra, Okta, Cognito) is the default expectation. Vendor-managed user accounts are a flag.
  2. How does role-based access work? A clinician seeing patient AI summaries should see only their patients. A pharmacist seeing prior-auth recommendations should see only the cases assigned to them. Test the access model with realistic scenarios.

Outputs

  1. Is every output cited? Free-form answers without source citations are not auditable. Compliance teams should reject them.
  2. What happens when retrieval comes back empty? A safe system tells the user it does not know. A dangerous system makes something up.
  3. What is the human-in-the-loop boundary? For any decision with clinical or regulatory consequence, there should be a human review step. Confirm where it sits and how it is logged.

Operations

  1. Who can change the prompts and retrieval logic? Vendor-side changes that alter system behavior should produce audit entries. Silent prompt updates that change AI behavior are a regulatory liability.
  2. What is the model upgrade path? When the underlying model changes — and it will — does behavior change? Is there an evaluation harness? Will you be notified before deployment?
  3. What is the SLA for incident response? AI systems break in ways traditional software does not. Hallucinations, retrieval failures, prompt injection. The vendor's response capability matters.

Contract terms

  1. BAA signed, with all sub-processors named. The LLM provider, the vector database, the embedding endpoint, the logging vendor — each is a sub-processor.
  2. Indemnification for breach caused by vendor's AI. Carve-outs for "AI-generated content" that shift liability back to you should be flagged.
  3. Audit rights. You should be able to audit the vendor's controls without their permission, on reasonable notice.

Red flags

A few responses that should slow procurement down:

  • "We use the latest models" without specifying which.
  • "Your data is secure" without explaining how.
  • "We have HIPAA compliance" without naming services and BAAs.
  • Vague answers to "what is in the audit log."
  • An inability to demonstrate tenant isolation.
  • Refusal to answer detailed technical questions in front of a Security Officer.

These are not always disqualifying. They often indicate a vendor whose security posture is less mature than they realize. A good procurement process surfaces them before contract.

What this checklist does not cover

Outcomes. The checklist above ensures the vendor can be deployed responsibly. Whether the vendor solves the problem you bought them to solve is a separate question. We have seen vendors clear every item on this list and still produce a system that does not work for the intended workflow. The checklist is necessary, not sufficient.

The right pairing: this checklist for the security and compliance review, plus a pilot deployment with measurable success criteria for the operational evaluation. Both questions need an answer before procurement closes.

Where we fit

If you are working through this checklist for a vendor evaluation, or if you are deciding between buying and building, an architecture review is the lowest-risk way to get a second opinion. We do this for healthcare CIOs regularly. Get in touch if you have a procurement decision in front of you.

Architecture Review