AI in Medtech

Seldom does a day pass without hearing the acronym "AI" or having our lives touched by artificial intelligence in some way. It is one of the few acronyms that everyone seems to understand. Intentionally or not, we likely all interact with some form of AI in our daily lives—whether through a simple Google search that generates AI-driven results, or by deliberately using AI to reword something we have written.

Like COVID-19, the AI revolution has caught us somewhat off guard. We are having to adjust how we work and live to embrace it. But unlike COVID-19—conspiracy theories aside—AI, as the name implies, is artificial. Arguably, we should be in full control of its seemingly uncontrolled proliferation in society. Yet the reverse appears true: something artificial is trying to control us. So how did we get here?

For context, I am no IT expert—just someone who has worked in the MedTech quality and regulatory space for decades. Like many of you, I am coming to grips with what AI means and what lies ahead. While the term "artificial intelligence" might strike some as an oxymoron, I thought it prudent to explore some background.

First: AI is not new.
It has been around for nearly 90 years. In 1936, Alan Turing conceptualised the "Turing Machine"—a device capable of reading data and solving problems using algorithms. The term "artificial intelligence" didn't enter common vocabulary until the 1950s. In 1955, the Oxford English Dictionary defined it as:

"The capacity of computers or other machines to exhibit or simulate intelligent behaviour; the field of study concerned with this."

This was later expanded to include:

"…software used to perform tasks or produce output previously thought to require human intelligence, especially by using machine learning to extrapolate from large collections of data."

The key word here is intelligence, defined as "the faculty of understanding; intellect." Intellect, in turn, is "that faculty of the mind by which a person knows and reasons; power of thought; understanding; analytic intelligence."

All pretty deep.

Looking at the updated Oxford definition above, it states:

"…software used to perform tasks or produce output previously thought to require human intelligence."

"Previously thought"? Are we now suggesting that AI's tasks aren't truly intelligent but merely computational—driven purely by input?

Reflecting on the deeper definition of intellect, the key terms are reasoning and power of thought. Reasoning and thought rely on knowledge—yet knowledge alone does not equal intelligence. Someone with vast knowledge may simply have a strong memory, not necessarily the ability to critically process, question, or extrapolate. Knowledge serves as the foundation for analysis and justification. However, the ability to retain knowledge, analyse, reason, and process information varies greatly among individuals. It is not one-size-fits-all.

The rise of AI is directly tied to software's ability to analyse information at lightning speed. Tasks that might take us hours or days can now be completed in seconds. But AI is not magic. It is software—rooted in zeros and ones—that detects patterns in data to draw conclusions or make predictions. AI merely simulates and processes existing knowledge. That is arguably not intelligence. It cannot reason or justify.

AI in Healthcare
AI's use in MedTech has also been around for decades. In 2002, during a Master's degree in Medical Diagnostics, I was assigned a project on Artificial Neural Networks. I found it fascinating and saw huge potential—but with my quality and regulatory hat on, I could see clinical risks, particularly in disease diagnosis. I highlighted those risks forcefully. Despite being challenged by a few academics, I earned a very good mark because I could argue my points with reasoning and justification—points grounded in real-world experience as a device developer and user.

Fast forward 20+ years, and AI in MedTech is becoming more prominent. As noted, we are playing catch-up. Recent publications demonstrate this:

• 2019: OECD AI Principles (fairness, innovation, accountability, transparency)
• 2021: EU proposal for the AI Act (entered into force 01 August 2024)
• 2023: BS/AAMI 34971 (applying ISO 14971 to machine learning in AI)
• 2023: ISO 42001 (AI Management System)
• 2023: US Blueprint for an AI Bill of Rights (ethics and bias)
• 2025 (draft): FDA guidance Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations

With the exception of the FDA guidance (specific to medical devices), all documents are non-industry-specific.

Notably, ISO 42001 does not include the word quality, yet it contains many elements of a medical device QMS (document control, internal audit, etc.) and is expected to be integrated within it. Like the OECD document, it promotes transparency, fairness, accountability, and security—but takes a deeper dive into risk management. Rightly so.

BS/AAMI 34971 specifically addresses risks arising from machine learning. It provides guidance on applying ISO 14971 to regulated AI medical technologies, without replacing it.

The FDA draft guidance outlines key information for marketing submissions. One of the FDA's strengths I've always appreciated is its commitment to transparency. The guidance underscores a Total Product Lifecycle (TPLC) approach for AI-integrated devices, highlighting transparency and bias mitigation. It adopts a patient/user-centred focus, requiring details on:

• Intended users and user interface
• Instructions and information for users
• User training requirements
• Device performance and AI training data (including representativeness)
• Post-market surveillance (PMS) and performance monitoring

To further transparency, the FDA also requires a Public Submission Summary, offering stakeholders insight into design and validation.

The EU AI Act
The EU AI Act is comprehensive. It aims to provide a regulatory framework for AI within the EU, ensuring safety while respecting rights and values, using a risk-based approach. Implementation is staged over 6–36 months. As of 02 February 2025, obligations on prohibited AI practices and AI literacy are mandatory. Obligations for high-risk AI systems—including medical devices and IVDs—take effect on 02 August 2026. Only two guidance documents have been published so far; others are expected by August 2026.

Unlike the FDA guidance, the Act places obligations on both providers and deployers of AI systems:

• Provider: develops an AI system or places it on the market under its own name.
• Deployer: uses an AI system under its authority (excluding personal non-professional use).

Article 26 requires deployers to:

1. Ensure use in accordance with instructions for use (IFU)
2. Assign human oversight to high-risk AI systems
3. Monitor use and associated risks
4. Report serious incidents to provider, importer, distributor, and market surveillance authority
5. Undertake a data protection assessment
6. Inform affected workers and representatives before use
7. Inform natural persons how the AI system's output impacts them

These responsibilities aim at transparency—users and patients know AI is being used and how it may affect them. In theory, that is good. But given my years in MedTech, the following concerns arise:

• Use in accordance with IFU: In reality, IFUs for MedTech devices are rarely read in clinical facilities. Do we really expect medical professionals and home users to read them? How will compliance be monitored?
• Human oversight: This implies a "trust but verify" approach. How much oversight? By whom? Who decides? If constant verification is needed, what is the point of the AI?
• Monitor risks: Risk assessment is subjective. User A may assign lower risk than User B for the same product. If the manufacturer monitors itself, is that not marking its own homework while trying to eliminate bias?
• Report serious incidents: Is it realistic to expect users to know who the importer, distributor, and market surveillance authority are? A consultant using an AI device will not be the one reporting incidents. Hospitals are chaotic environments.
• Data protection assessment: Who would do this? How would they know the software's limitations regarding data security? Full disclosure from providers is required, but the more risks they disclose, the less likely they are to sell. Commercial reality.
• Inform workers and representatives: Who determines what information is disclosed? How is this monitored over time as workers and users change?
• Inform natural persons: As above. How will users or patients acknowledge consent to AI use and accept associated risks?

The vision is commendable, but experience tells me reality may differ. Transparency is good, but it relies heavily on labelling and product information—challenging ISO 14971's principle that information for safety is the last resort of risk mitigation (for good reason, as IFUs are rarely read).

To date, those classified as deployers have had no obligation to understand regulatory requirements for bringing a device to market. Any failure when used as intended has largely been on the provider. Do deployer responsibilities now mean a clinical facility may be held accountable under an Act they likely know nothing about? Do we expect nurses, doctors, and patients to read and understand the Act? Having worked in QARA for decades, I can say it is not an easy read. Plenty of caffeine was required.

I have major concerns that placing such obligations on deployers may give providers an easy escape from mitigating risks as far as possible—preferably by design, as required by EU MDR/IVDR. A defence of "they didn't read the IFU" or "they didn't use it as intended" becomes more justifiable after an adverse event.

The FDA takes a more pragmatic approach. They state that transparency is context-dependent. They encourage designers to consider:

• Where will the device be used, and what are the conditions?
• What else might users be doing simultaneously?
• How timely is the application of information?
• In what settings will the device output be viewed?
• Will the users who interpret output be the same as those who operate the device?

This is music to my ears. Anyone who has sat with me in a risk assessment session knows I always emphasise context of use, user profiles, what else users are doing (your device is not their focus!), and environmental limitations. Context is key—not only in AI transparency but in any MedTech risk assessment.

Machine Learning
AAMI TIR66:2017 defines machine learning as "function of a system that can learn from input data instead of strictly following a set of specific instructions." So, like us, AI "learns." Its brain is uploaded with existing knowledge (data) and computes based on input quality.

Everyone reading this can recall good and bad teachers. I think back to algebra class. Everyone received the same teaching input. Some grasped it immediately; most, including myself, did not. Was that intelligence, or just different ways of learning? The input was the teaching. The desired output was understanding. It didn't work.

After class, many of us tried to teach ourselves, with limited success. Then one student went home, and his civil engineer father explained how algebra was used and why—he put it into context. Let's call him Pupil A. Pupil A then explained the reasoning to Pupil B, who grasped it and explained to others. Some grasped it; others did not. We all started with the same knowledge, but our ability to draw conclusions varied. The initial teaching input did not consider context of use or user variability.

This is pertinent to AI in MedTech. Critical factors for safety and effectiveness include:

• Method of training
• Training data
• Understanding user/patient variability (skills, customs, race, predispositions)
• Understanding use environments

Above all, data—the input—is king. All these factors are linked and carry many permutations of risk. For an AI product to work according to its intended use, they must be fully understood and "built in."

Take an AI diagnostic system for melanoma. If the algorithm is trained on white Europeans but destined for a global population, the data is not representative. As a pale Scot who turns beetroot red under a full moon, I differ hugely from my Mediterranean wife. Our skin cancer risk profiles and ease of detection are totally different. Similarly, an AI device for diabetes trained on people in the Far East would not necessarily be appropriate for US or Western European populations.

I am not anti-AI. Far from it. AI has enormous potential in society and MedTech, but it needs careful management. Quality in healthcare AI does not happen by chance. It is only as good as the data used to feed it. It builds on existing quality concepts like risk analysis and training—but here, training is of the algorithm, not people, and the permutations of harm increase profoundly.

MedTech practice is grounded in risk. "First, do no harm" acknowledges that every intervention carries risk. Effective management is essential, yet many companies lack robust understanding of their devices' true clinical and usability risks or the realities of the clinical environment. AI will introduce unprecedented complexity and new risk categories. Consider an AI system that trains continuously during clinical use: Who verifies this learning? Who is accountable? Ultimately, it comes down to controlling inputs, outputs, and variability—factors tied to risk.

Thankfully, BS/AAMI 34971 addresses some concerns by guiding what to consider when identifying risks. It states that people involved in risk assessment must have relevant knowledge of the data used to train, test, and validate the system. But this requires correct input data at the time of use (e.g., ethnicity, age). The algorithm may work, but training data may be erroneous and this will be somewhat out of the developers control.

Curious about numbers of AI-based devices approved/cleared, I ironically used AI to do the task for me. Why? It saved hours of trawling databases, and if the numbers are somewhat inaccurate, no one gets harmed.

In an unscientific study, I asked three AI platforms (Deepseek, Grok, ChatGPT): "How many AI-enabled medical devices have been cleared by the US FDA to date?" and "How many have been approved under EU MDR/IVDR?"

For FDA clearance:

• ChatGPT: 1400–1450 (to April 2026)
• Grok: 1430–1451 (to mid-April 2026)
• Deepseek: 1356 (to end of March 2026)

Deepseek and Grok provided further detail (radiology predominant). ChatGPT provided most information on trends over time.

For the EU, unsurprisingly, no platform could provide exact numbers due to the lack of an operational public database. Deepseek and ChatGPT stated numbers were unavailable and gave reasons. Grok provided estimates from multiple sources.

The same query yielded different results because algorithms and training differ. Using AI saved me hours or days of work, and I was grateful. It replaced a manual, tedious task that didn't require much intellect—just knowing where to look (knowledge). Time was the winner. Despite differing outputs, I was impressed. But these results also show that AI in MedTech is growing—even the FDA now uses AI in technical reviews.

Should we be scared? To a point, yes. At a recent MedTech expo, many expressed concerns about the speed of AI's takeover, especially in MedTech. The human reasoning element seems absent. Computational power is quick and efficient, but reasoning power is debatable. We will rely less on highly trained clinicians for diagnosis or procedures. It is not all binary zeros and ones—there are 0.27s, 0.35s, 0.52s, 0.86s. Those grey areas rely on clinical oversight, justification, reasoning, and thought.

Multidisciplinary team meetings have been common practice in medicine for centuries. They discuss grey areas, gather opinions that matter, include those who have seen something before that triggers alarm bells or offers alternative perspectives. Clinicians know the risks—they spent years training, are bound by the Hippocratic oath, and undergo periodic peer assessments. Should AI replace that?

I gathered a clinician's viewpoint from consultant gynaecologist Dr. Maria Vella:

"We increasingly hear how, in a cash-strapped health service, AI can save money, increase productivity, and improve accuracy—especially in diagnostics. AI will undoubtedly help in many clinical scenarios, but its use needs careful regulation and governance.
Areas like breast imaging (screening mammography) could deploy AI successfully, processing large volumes at lower cost. Dermatology is another trial area.
The downside: human bodies have subtle differences. The experienced clinician's eye is essential to differentiate normal variants from early pathological changes. Using AI in diagnostics could, at best, increase clinician workload (anything not 'box standard' normal requires investigation, increasing patient anxiety). At worst, subtle early changes—critical for identifying disease—could be ignored. Any process deploying these systems needs careful review and appropriate governance before becoming the default service."

To conclude: The potential for AI in MedTech is enormous. It will save time, save money, and I hope save lives. But we need to proceed with caution—a lot of it. The AI Act, FDA guidance, and others are steps in the right direction, but we seem to be running before we can walk. My fear is that society is being driven by AI—it is controlling us, not the other way around. The appeal of time and cost savings in cash-strapped health systems is real, but so are the risks. We must not lose sight of the bigger picture.

We may be ready for AI to provide recommendations in MedTech that are verified by trained clinicians. But there are genuine concerns about AI devices that train on the job with no human validation check, and those that make final clinical decisions or perform surgical procedures. Lives are at stake. I hope that in the rush to adopt AI in MedTech, there is not a disaster that forces us to reassess its use.

Technological advancement only creates possibility. We cannot just drop AI into practice and assume a positive clinical impact. Positive impact requires deliberate design, thorough risk management, and a fully weighted benefit-risk assessment throughout the entire lifecycle—especially for AI.

The tech revolution is here. But unless we close the gaps, we will miss out on the value.