AI or You? Who’s on the hook?
Your Hospital Bought an AI Tool. Who's Actually Responsible When It Goes Wrong?
March 18, 2026
I've had some version of the same conversation probably a dozen times in the last year.
A healthcare organization has deployed an AI tool. Something goes sideways. Maybe a diagnostic recommendation gets followed without enough human review. Maybe the model starts producing outputs that don't make clinical sense. Maybe there's a data exposure. And then someone at the table asks the question everyone was hoping to avoid:
"Who's responsible for this?"
And what follows is one of the most uncomfortable silences in healthcare compliance.
The Accountability Gap Nobody Wants to Talk About
Here's what happens: clinical staff assume it's an IT problem. IT assumes it's a legal or compliance problem. Legal assumes the vendor is on the hook. The vendor's contract says something like "not for diagnostic purposes" or "not intended as medical advice." And the patient - or the regulator - doesn't care about any of that.
This is the accountability gap. And it's not hypothetical anymore. Healthcare organizations are deploying AI tools for clinical decision support, patient triage, documentation, coding, and supply chain, often without a clear internal answer to a very basic question: if this fails, who owns it?
The answer matters legally, operationally, and clinically. And getting it right before something goes wrong is about a thousand times easier than figuring it out after.
Why "The Vendor is Responsible" Isn't a Real Answer
I understand why organizations default to pointing at the vendor. The contract is there. There are indemnification clauses. It feels like coverage.
But vendor agreements, especially for AI tools, are written to limit liability, not distribute it. Most will include language making clear that the tool is a decision-support aid, that clinical judgment remains with the licensed practitioner, and that the organization accepted responsibility when it signed the BAA and deployed the product.
More importantly: when OCR comes knocking after a breach, or a plaintiff's attorney is building a negligence case, they're not going to sue your SaaS vendor. They're coming to your organization. They'll want to know what your governance process looks like. Whether anyone reviewed the model before deployment. Whether clinicians had clear guidance on when to override or escalate. Whether there was a human in the loop and whether that human understood their role.
The vendor contract doesn't answer any of those questions.
What Actual Accountability Looks Like
I'm not suggesting every AI deployment needs a 200-page governance framework before anyone can move forward. Healthcare has enough policy documents that nobody reads. What's useful is being able to answer six questions before you go live:
1. Who is the named owner of this AI system - not the team, a specific person accountable for what it does?
2. Does the AI system have a default "stop" setting before the decisions it makes take effect? Is there an engineering setup that can pause action to allow human interface?
3. Under what conditions does a human need to review, override, or stop the system? Is that documented anywhere, or just understood?
4. If the system produces a bad output and a patient is harmed, what is your incident response path? Does your existing IR process even contemplate AI?
5. What does the vendor actually own contractually, and have you read that section of your agreement recently?
6. Does your staff know who to call when something looks wrong? Not the vendor's support line but internally, who do they escalate to?
If you can answer all six of those clearly, you're ahead of most organizations I've seen. If you can't answer two or three of them, that's where to start. You likely don't need a new policy but have a conversation that hasn't happened yet.
The Security Angle Healthcare Often Misses
Most accountability conversations in healthcare AI focus on clinical risk, which makes sense. But there's a security dimension that's equally important and often treated as a separate track.
AI tools expand your attack surface. They introduce new data flows, often involving PHI. They may involve third-party APIs and model providers whose own security posture you've never evaluated. They can create new pathways for data exfiltration that your existing SIEM and DLP rules weren't built to catch.
When I work with healthcare organizations on AI governance, I ask: has security operations seen this tool? Not just the privacy team, not just compliance, but has someone actually looked at the data flows, the API integrations, the logging capabilities, the vendor's breach notification obligations?
Because if there's an incident and the answer is no, that's a problem that compounds quickly.
This Is a Solvable Problem
I want to be clear - I embrace digital transformation - and I am not trying to slow down AI adoption in healthcare. AI has real potential to improve patient outcomes, reduce administrative burden, and address serious workforce constraints. Organizations can do this well.
The ones doing it well aren't the ones with the longest governance policies. They're the ones that took the time to answer the hard ownership questions before deployment, built escalation paths that clinical staff use, and treated security review as a requirement rather than an afterthought.
That's a reasonable bar. It's achievable. It just requires someone at the organization who's willing to ask the uncomfortable questions before something goes wrong, rather than after.
If your organization is working through this and needs a sounding board, feel free to reach out. It's exactly what I work on.