Healthcare AI Agents Need to Prove Two Things: Value and Control

Recent findings on unapproved AI deployments raise a practical question for healthcare technology teams: can they demonstrate both measurable value and control over what their systems do?

Healthcare AI teams face two questions when they ask an organization to adopt their product: will it improve the workflow, and can the organization govern it once it is running?

Recent research on unapproved AI deployments makes the second question harder to leave until later.

In a survey commissioned by Imprivata and conducted by Vanson Bourne, 72% of 250 U.S. healthcare leaders responsible for identity security or AI strategy reported AI tools or agents being deployed without formal IT approval. Imprivata’s accompanying analysis describes this as happening at least occasionally. The research was released on September 15, with Axios covering the issue on October 5.

The finding needs careful interpretation. It covers AI tools and agents, comes from a vendor-commissioned survey, and does not establish how many deployments caused harm. It should not be read as evidence that 72% of hospitals are running unauthorized autonomous agents.

Its relevance is more specific: some AI use is bypassing the process intended to make it visible and accountable.

For companies building healthcare AI, that raises the standard for what a convincing product must demonstrate.

The signal: approval and autonomy are separate questions

Shadow AI describes tools used outside an organization’s approval or oversight process. Agentic AI describes systems that can carry out tasks through actions, potentially across multiple tools.

These categories can overlap, but they are not interchangeable. An unapproved writing assistant may only generate text. A formally approved agent may have permission to update records, send messages, or move work between queues.

Both need appropriate oversight. The agent also needs explicit limits on its authority.

That distinction becomes important when a product evolves. A feature that originally summarized information might later prepare actions, execute them with approval, and eventually handle selected steps independently. Each expansion changes what the organization is trusting the system to do.

Product teams should treat those transitions as changes in responsibility that require fresh evidence and review.

Why it matters: a correct answer can still lead to an unauthorized action

Consider a hypothetical referral assistant.

Its first version reads incoming documents and identifies missing information. Staff review their suggestions and decide what to do next.

A later version can contact patients, update referral statuses, and route cases automatically. The underlying extraction capability may be similar, but the consequences of a mistake have changed.

An incorrect suggestion can be caught before someone acts. An incorrect status update may move a referral out of an active queue before staff notice.

Even a factually correct interpretation does not establish permission to act. The system might correctly identify a missing document but send a message that requires staff approval. It might correctly recognize a duplicate record but lack the authority to merge it.

This is why model quality and operational control need separate acceptance criteria. Healthcare teams need evidence that the system understands the task and that its actions remain within the authority granted to it.

The first test: prove value across the workflow

The original proof problem remains. Healthcare AI needs to demonstrate benefits in the setting where people will use it.

Research on ambient AI scribes offers a useful example. In a randomized trial involving 238 outpatient physicians, Nabla reduced measured time spent writing notes by 9.5% relative to the control group. DAX did not show a statistically significant change on that measure. Some secondary findings suggested potential benefits, but the authors called for confirmation in larger, multicenter trials.

The study evaluated documentation tools, not autonomous agents. Its relevance here is methodological: outcomes depend on the product, the measure, and the setting. A general claim that “AI saves time” cannot replace that detail.

For a referral assistant, document-processing speed would be only one part of the evaluation. The more useful questions concern the completed workflow:

• Do referrals reach the correct team sooner?

• How much time do staff spend checking and correcting the system’s work?

• Are incomplete cases easier to identify?

• Do errors create additional work downstream?

• Which cases still require manual handling?

A product can accelerate one step while increasing the total effort needed to complete the task.

Teams should establish a baseline and define success before deployment. Review effort and exception handling belong in that calculation from the beginning.

The second test: prove that authority is bounded

Once AI can take action, control becomes a product requirement.

For the referral assistant, an organization might permit document classification and draft preparation while requiring approval before patient communication. Other actions, such as changing clinical priority, may remain outside the system’s scope.

Those decisions need to be reflected in what the software can actually execute. Instructions to seek approval should be supported by permissions and workflow controls that prevent execution until approval is recorded.

Activity must also be traceable. When something goes wrong, the team should be able to establish which agent acted, which version was running, what information it used, what approval it received, and what changed.

That record does not need to expose every piece of sensitive data to every administrator. Logging itself requires decisions about access, retention, and the information needed for investigation.

Control also needs an operational response. Someone must own alerts, be able to suspend the system, and know how staff will continue the work.

NIST’s research on post-deployment monitoring reinforces the importance of evaluating AI in real settings and identifying unexpected behavior and consequences after release. It also notes that monitoring methods remain an evolving field.

Monitoring is therefore an ongoing responsibility, with defined actions attached to what the team observes.

What does this change for healthcare product teams

For founders and product leaders, the implication is straightforward: the case for adoption should include how customers will operate and supervise the product.

A demonstration should show the successful workflow, then show what happens when information is incomplete, permission is denied, or an integration fails.

Can staff see which actions were completed and which are still proposed? Does a failed request trigger a retry that could create duplicate work? Can a reviewer understand the case without reconstructing it across several systems?

These questions affect architecture, interface design, integrations, and support. They should influence the roadmap before a customer asks them during the deployment review.

They also affect how responsibility is shared. The technology provider can supply permissions, logs, monitoring capabilities, and approval mechanisms. The healthcare organization still needs to assign owners, authorize use cases, and decide who responds when intervention is necessary.

Those responsibilities need to be met in a working process.

BeKey has previously explored why healthcare AI pilots struggle to reach production. Agentic capabilities make that discussion more specific: every additional action needs an owner, an operating boundary, and evidence that the surrounding workflow can support it.

What to check before scaling AI agents

Before extending an agent to another department or granting additional permissions, teams should be able to answer five questions with evidence.

The depth of these checks should match the consequences of the task. A tool preparing an internal summary and an agent changing a patient-facing workflow require different levels of assurance.

The expansion decision should reflect that difference.

What Healthcare AI Teams Need to Demonstrate Next

For healthcare software teams, measurable value and operational control belong in the same delivery plan.

Workflow discovery should establish where AI can help and where judgment must stay with people. Product design should make approvals and exceptions understandable. Engineering should enforce permissions and preserve traceability. Evaluation should continue as the system and its environment change.

The recent findings on unapproved AI use make these responsibilities more visible. They also sharpen the question healthcare AI companies need to answer: can a customer use the product effectively while retaining meaningful authority over its behavior?

A strong case for scaling shows a worthwhile improvement in the workflow and tested control over the actions that produce it.

Planning an AI-enabled healthcare workflow? Talk to BeKey about your product, integration requirements, and development needs.