Key Takeaways
- FDA Pathway Established — UpDoc’s clearance demonstrates a viable regulatory pathway for SaMD products incorporating patient-facing LLMs, though future approvals will depend on each product’s intended functions, safety and efficacy.
- Human Oversight Remains Critical — Clinical AI will likely continue to function as a support tool for clinicians rather than make independent diagnostic or treatment decisions due to “human-in-the-loop” and state law practice-of-medicine requirements.
- Comprehensive Compliance Required — Organizations developing or deploying clinical AI must address a wide range of legal concerns, including HIPAA and data privacy and security, data governance, use case limitations, human-in-the-loop requirements and related practice of medicine concerns, monitoring the safety and efficacy of such tools, and complying with the rapidly evolving AI regulatory landscape.
On June 25, 2026, UpDoc Inc., a clinical AI company, announced that it received clearance from FDA for what the company describes as the first Software as a Medical Device (SaMD) that uses patient-facing large language models (LLMs). UpDoc describes its platform as an enterprise-ready agentic clinical AI platform designed to enable clinicians to deploy AI agents that are integrated with a provider’s electronic health record and other workflows to perform certain care-related tasks.
UpDoc was issued a 510k clearance letter on Dec. 23, 2025, for its type 2 diabetes medication management software, which implements a healthcare provider-specified treatment plan. The summary attached to the letter describes a conversational data collection module that allows patients to enter data through voice or chat interfaces with the UpDoc agent and receive new treatment plan instructions.
While this development highlights a potential pathway for SaMD products to include LLMs, the clearance and approval of future clinical AI products will depend on the intended functions, safety and efficacy of such products. Clinical AI will likely continue to serve as a support tool for clinicians due to “human-in-the-loop” and state law practice-of-medicine requirements, rather than be approved to make independent diagnostic or treatment decisions.
Organizations developing clinical AI platforms should carefully assess the specific FDA clearance/approval pathway applicable to the platform based on its intended use, functionality, safety and efficacy, including 510(k), De Novo and PMA pathways. Developers should also monitor FDA’s evolving expectations for AI-enabled devices, including post-market surveillance, transparent labeling and risk communication, cybersecurity, data integrity and other controls, and predetermined change control plans.
Clinical AI tools may also raise a complex set of legal and operational questions for healthcare providers, including how organizations should implement and manage the use of such tools, validate AI-supported interventions, integrate such tools into healthcare workflows and address related liability, licensure and ethical concerns. Healthcare providers should carefully define the boundaries between autonomous AI action, such as implementing physician-defined protocols, and required physician intervention. They also should determine how human oversight will be implemented and documented, assess whether and how payors and regulators permit the use of AI agents, and review malpractice coverage and vendor contracts to address liability concerns.
LLMs with access to patient data also raise HIPAA, privacy, cybersecurity, consent and data governance concerns. Accordingly, healthcare providers should carefully vet potential vendors and partners to ensure compliance with applicable laws, data security and, as applicable, enact data use restrictions and redisclosure limitations.
McGuireWoods’ AI practice team includes attorneys from the FDA & Life Sciences group and the Healthcare Compliance, Regulatory & Policy group that understand the unique healthcare regulatory and compliance issues implicated by the development and deployment of AI products in the healthcare industry. Our multidisciplinary teams are well positioned to assist life sciences and healthcare companies with the development and deployment of AI products, including addressing FDA regulatory strategy, healthcare compliance, data privacy and security, reimbursement, enforcement risk, policy developments, and the quickly evolving federal and state AI regulatory landscape. For assistance with development or deployment of healthcare AI products, contact one of the authors.