• Healthcare IT is siloed and slow. During the pandemic, it took years to automate sharing hospital occupancy data!

• I tested how LLMs can share data without extensive API development.

• This technology could reduce IT fragmentation, cut through bureaucracy, and make healthcare more enjoyable and patient-centered.

Read time: 6 minutes

In this edition of Transforming Med.Tech, I dive into how Large Language Models (LLMs) may become the unexpected heroes in healthcare's battle against fragmented IT systems. First, however, I owe you an apology for the delay in this newsletter. Having another child has certainly impacted my ability to keep up with side projects like this one. But the sleepless nights have not kept me from experimenting with Generative AI, so let's get started!

From intelligence to integration

Halfway into 2024, it's becoming clear that the AI landscape is evolving. The rapid leaps in LLM "intelligence" we saw last year have plateaued somewhat. Instead, there is a pivot towards smart applications of these powerful tools, integrating AI seamlessly into our everyday lives. For example, my bank does not allow me to export my credit card transactions (really, ING?), but Claude transformed a PDF output of transactions into a neat CSV file in only a few seconds. Simple AI solutions can save tons of time.

I believe healthcare, in particular, is ripe for this kind of innovation. Walk into a typical modern hospital, and you may encounter cutting-edge imaging technology next to fax machines that stopped being cutting-edge before you were born. The industry is understandably conservative – after all, lives are at stake. But this caution has led to a fragmented IT landscape where data sharing often feels stuck in the last century. Could smart integration of AI tools be of help to travel these silos? I explored this with a case study.

A case study: sharing pandemic data

In the early days of the COVID-19 pandemic, governments had to coordinate patient distribution to minimize the pressure on healthcare providers. Hospitals were scrambling to share occupancy data, often relying on manual processes that were time-consuming and error-prone. Whereas video conferencing tools suddenly advanced rapidly, healthcare IT took ages to develop data-sharing APIs (application programming interfaces), even in these unprecedented times. In the Netherlands, for example, IT workers manually uploaded data every day, and only two years into the pandemic, this process was fully automated.

I tested if the flexibility of LLMs would have been of help in the pandemic's need for data sharing. Using OpenAI's ChatGPT and Anthropic's Claude and simple prompts, I created "hospital bots" and a "data collector bot," each with dedicated tasks to share and collect data, respectively. Here's a taste of what they can do:

1. Negotiate data formats: "Hey, Hospital A bot, what format would you prefer for the occupancy data?". They can agree on JSON, free text, tables, whatever is preferred:

2. Share data on demand: "Hospital B bot, could you please provide today's ICU occupancy?" You can even let it check for consistency and automatically ask for error corrections:

3. Extract data from various sources: "Here's our dashboard. Can you pull out the relevant numbers?" These bots only need access to a screen and can figure out where the requested information is.

And of course, you can then format the output in any way, ready for sharing with other institutes.

Reimagining data sharing and referrals

But why stop at simple occupancy data? Let's think bigger. Imagine a GP considering referring a patient to a cardiologist. I created dedicated chatbots, where the knowledge of the specialist's chatbot was restricted to relevant medical guidelines. The conversation of the GP's chatbot with the specialist's bot looked something like this:

GP Bot: "I have a 55-year-old male patient with suspected atrial fibrillation. Here's a summary of his case."

Specialist Bot: "Thank you. Based on the current guidelines, we need the following additional information before a referral: 1. Recent ECG results; 2. Thyroid function tests; 3. Echocardiogram report, if available; Have these been performed?"

GP Bot: "The ECG and thyroid tests are complete, but the patient hasn't had an echocardiogram yet; I will order one before the referral."

This kind of interaction ensures that all relevant data are shared (and only relevant data), that referral guidelines are followed, and that necessary pre-referral steps are taken. This results in efficient use of everyone's time and improves the quality of care.

Beyond APIs and bureaucracy

Implementing comprehensive APIs across fragmented healthcare IT systems is a herculean task, requiring coordination between multiple vendors and navigating a bureaucratic maze. LLMs offer a shortcut. Of course, we must balance the incredible potential of LLMs with careful consideration of accuracy, privacy, and security. But language evolved to deal with the chaos and unpredictability of our world, and the inherent flexibility of language-based models is enormously valuable. They can quickly capture the essence of large amounts of data (such as clinical notes or my diary). They don't need carefully structured data or predefined interfaces, they can work with what's on the screen – be it a web page, a PDF, or the user interface of any existing application. Even if we could find the time, money, and commitments of multiple vendors, carefully constructed APIs are unlikely to ever reach this level of flexibility.

Conclusion

The potential of LLMs to revolutionize data sharing in healthcare is immense. Imagine a world where doctors spend less time fighting with referral and data sharing systems, and more time connecting with patients. Where crucial medical information doesn't sit in the fax machine's inbox, but flows seamlessly, with better outcomes and safer care.

So, what do you think? Did my experiments convince you that LLMs are going to replace APIs? And is healthcare IT ready to implement LLM-based solutions?

Thanks for sharing your thoughts! And with my sleepless nights slowly becoming more bearable, I am confident that the next newsletter will not keep you waiting so long. Until then, keep innovating and stay curious!

• Shifting from unimodal to multimodal learning in AI is essential for a more complete world perspective

• Multimodal medical models will have an increased disease and therapy understanding

• Specialist and generalist AI may balance and correct each other

Read time: 7 minutes

In this edition of Transforming Med.Tech, I am diving into the fascinating parallels between raising a kid and training an AI, and what a more complete world perspective would mean for AI operating in healthcare. 

First tech experiences as a kid can be transformative

As I am raising my first kid (and about to welcome our second), I'm reminded of the transformative power of technology, impacting my curiosity as a kid. One of my earliest memories of the internet is watching pixelated images of Mars sent from a lander unfold on my father's computer. I was exhilarated by seeing otherwordly images mere hours after they had been captured on a planet so far away. I was similarly excited by the shift in knowledge access, from the impressive printed volumes of Encyclopædia Britannica at my grandparents' bookshelf to a CD version that you could search in, easily exploring new concepts, later only to be made even more accessible by free online resources such as Wikipedia. 

In contrast, my two-year-old is already used to much more advanced technology, asking our smart speaker to play music when she wants to dance. She'll be an expert in using AI in another year or so, benefiting from a chatbot's large language model (LLM) to explain concepts with words she can understand. Whereas I am sometimes starting to feel old when I use tech (who keeps redesigning all those perfectly fine UIs...), she'll grow up to have enhanced tech versatility. (Of course, one can wonder whether that's a good thing... Shouldn't she learn to think for herself? It's that century-old question that we have asked ourselves when books/newspapers/radio/tv/internet/smartphones came out...)

We need to transcend text to really understand the world

Raising a kid also makes you think about learning. As a baby, you gradually grasp more and more concepts, bringing together experiences from sight, sound, touch, and feelings. On the contrary, LLMs are trained on a text-only corpus. And while LLMs are not explicitly trained to reason, the size of their neural networks and training data has resulted in 'emergent behavior' that, from an outside perspective, mimics reasoning. Restricting LLMs to unimodal (text-only) sources will limit that emergent behavior. Therefore, research is diving into multimodal foundation models. Such models learn not only from text but also from images, video, sound, etc. They have even strapped a camera on baby Sam for 19 months to see if this 'natural' data collection would work, too.

Multimodality is critical to obtaining a more complete 'world image' and getting a valid (or at least better) understanding of how the world works. Imagine learning only from text, never seeing schematics that create that 'aha' insight, hearing the emotion in a person's voice, experiencing the wind on your skin or the smell after rain. A unimodal foundation model simply cannot 'imagine' the full range of those experiences, like flatlanders in a 2D world cannot imagine the third dimension. 

Do apples always fall?

The most recent advancement towards a more complete 'world image' is evident in OpenAI's Sora, which allows the creation of minute-long videos from a text prompt. Amazingly, the videos (mostly) follow the laws of physics without being told how physics works. 

Whereas game and animation engines have dedicated physics simulators, text-to-video foundation models can now work without such explicit knowledge incorporated. This hints that AI's worldview is becoming more complete. Perhaps it does not truly understand physics: it may only be repeating common patterns from its training data, which probably does not include physics-defying levitation. Still, the similarities with how a kid learns are apparent: my kid has never seen a levitating apple either and would be utterly surprised if she did. So although my daughter doesn't have a full understanding of physics yet either, she is capable of grasping the basic concept (even if she, literally, stumbles occasionally).

What multimodality means for healthcare

How does this translate to healthcare? In medicine, the move towards multimodal models would signify a leap towards more holistic and nuanced medical knowledge. An LLM can already read medical guidelines and use text to apply them to your patients' medical records. But if this is fully text-based, its 'knowledge' of the disease is also purely textual. During my studies, I experienced that illustrative schematics or hands-on time with patients provided invaluable insights in addition to text-based learning. A medical world image is more complete if it transcends text and includes a feel for three-dimensional anatomy, images, processes, physiology, diagnosis, and therapy. Some work is already being performed in this area: in the imaging domain, models can quite literally transform from one imaging modality to another, and multimodal prediction models outperform unimodal models in predicting drug response. As an arrhythmia scientist, I would love to see a foundation model trained on the heart from a 3D anatomy, electrophysiology, cellular, organ-level, nervous innervation, and physics perspective, helping me interpret imaging, electrocardiograms, phonograms, drug responses, questionnaires, and other multimodal patient data.

Do we need specialist or generalist AI?

However, we aren't there yet. In the near future, we'll be working with highly-specialized unimodal models, and moving to more encompassing 'large medical models' will take some time. This is similar to how we developed as a species: first we got specialists (farmers, rulers, mathematicians), which then allowed our society as a whole to obtain a better generalized world perspective. Ultimately, we will arrive at 'large medical models' or even 'large anything models,' but it may require another paradigm shift to reason from concepts rather than words, just like the transformer-paradigm has driven the current foundation model boom. 

The specialist-vs-generalist perspective for AI, of course, also mimics our health care system, where a highly-focused specialist may provide the state-of-the-art treatment for a specific disease, and a general practitioner is immensely valuable in taking a holistic perspective and caring for the patient as a whole. Besides this essential harmony, there is another reason to strive for coexisting generalist and specialist AIs. LLMs are better at math that involves numbers that appear frequently in their training data (another sign that they do not internalize reasoning!). The focused and potentially biased attention of a specialist AI can be balanced by a more comprehensive perspective from a generalist AI. In general, fact-checking or safeguarding an AI with other AIs has been shown to improve their output quality.

Conclusion

The parallels between raising a child and training AI offer insights into the nature of learning, growth, and the pursuit of a comprehensive world image. Abstraction involves formulating general laws, beyond pattern recognition. Just like my kid is still learning that, current AI is too. This evolving understanding in AI not only mirrors human cognitive development but also promises to revolutionize healthcare, bringing forth innovations in diagnosis, treatment, and patient care that were previously unimagined. And as I embark on the journey of fatherhood once again, I am reminded of the endless possibilities that lie in nurturing the next generation—both human and artificial.

• I used a local AI model to analyze my diary and gain deeper insight into myself.

• A chatbot personalized with knowledge about my psyche helped me reflect on my year and set new goals.

• AI-driven self-analysis could aid in personal growth and play a role in mental health care.

Read time: 9 minutes.

Welcome to this new edition of Transforming Med.Tech, the newsletter in which I try to understand how disruptive innovations will transform medical technology. In this edition, I’m diving into using chatbots to analyze my diary and gain a deeper understanding of myself - it’s a longer read than usual, but I promise you it’s worth it!

After the busy work weeks of December, I cherish the first days of January as a time for reflection and planning. It’s a ritual where I delve into my professional and private life, reassessing past achievements and setbacks, and setting objectives for the future. This year, I decided to add a novel twist to this tradition by experimenting with generative AI, aiming to utilize a local Large Language Model (LLM) to analyze my diaries. 

A privacy-friendly deep-dive into my diary

For the past year, I’ve been keeping a diary and enjoying writing about my adventures, small and big. Due to privacy concerns, I was hesitant to use cloud-based solutions to analyze such a personal document. Luckily, small open-source models can run on your computer with the appropriate (free) software. I asked one such model about a diary entry. Though somewhat slow, this yielded fascinating results, including a daily ‘happiness score’ and key themes influencing my mood. 

To streamline the process, I wrote a Python script that automatically fed my diary entries to the LLM, extracting various metrics like family time and work-related business. The local LLM ensured my data remained offline and secure.

I observed an interesting pattern in my happiness scores over time. Generally, my happiness level hovered around a stable 8 out of 10, indicating a baseline of contentment. However, the data also highlighted certain instances where my happiness levels noticeably deviated from this average. These moments correlated with significant events – a dip in happiness aligned with a period of illness, while a sustained peak coincided with the joyful occasion that my child started to sleep through the night. Additionally, I noticed a tendency for my happiness to return to its baseline over time, a phenomenon termed the ‘blue dot effect,’ where the impact of both positive and negative events gradually diminishes.

I also delved into the statistical correlations between happiness and other metrics extracted from my diary. The results were a mixed bag – many of these correlations were insignificant and, frankly, I approached them with a healthy dose of skepticism. However, there was one correlation that was significant and seemed to ring true to my instincts: the link between happiness and family time:

A bot’s psychological analysis

The initial results from the local-LLM analysis were intriguing. They suggested that natural language processing could extract meaningful patterns from my diaries in a secure, private environment. I then took the less sensitive output – keywords related to happiness and unhappiness – and fed them to ChatGPT4 for further analysis in the cloud. I asked it to analyze the main themes of happiness and unhappiness throughout my year from this long list of (sometimes recurring) keywords. The AI’s ability to discern and rank these themes was quite accurate. Although some themes are valid for almost anybody, many resonated with my personal experiences, such as my love for my direct family and for outdoor activities:

Based on your extensive list of "happy" keywords, several key themes emerge that contribute to your happiness:

- Family and Relationships: Frequent mentions of family members, particularly (redacted: name of kid), (redacted: name of wife), and other relatives, highlight the importance of family in your life. Spending quality time with them, celebrating events like birthdays, and enjoying simple moments together are recurring themes.
- Outdoor and Nature Activities: You frequently mention activities like swimming, walking, beach visits, nature walks, and biking. This suggests a strong appreciation for the outdoors and a connection with nature.
(... the list continues and includes 10 themes ...)

Further analysis also highlighted some unhappiness factors that made sense, some of which I hadn’t realized consciously yet. I then asked it to do a full psychological analysis through a Big Five personality assessment. This was insightful, but due to its personal nature, I prefer to keep the details private, though I can share an intriguing graph it produced:

Curious to enhance my well-being, I prompted the AI for suggestions to boost my happiness. The recommendations it provided were thoughtful and practical. While I acknowledge that these AI-generated suggestions don’t equate to the nuanced advice a healthcare professional might offer, having such an accessible digital coach may be of tremendous potential value. Further exploration of digital tools in psychology and psychiatry could be invaluable, particularly in areas where access to healthcare professionals or personal coaches is limited or non-existent.

An on-demand personal coach

Impressed by the depth of insights gleaned so far, I decided to take the experiment a step further, beyond yearly reflection and towards daily guidance. I developed a personalized chatbot to act as my coach, tailored to my individual needs and peculiarities. Into its ‘knowledge base,’ I integrated the other bot’s psychological analysis, a personal manifesto, my goals and ambitions, and several self-help books I frequently consult (Rick Pastoor’s GRIP is a favorite!). 

Interacting with this chatbot about the past year and planning for the future proved to be quite practical. The bot was particularly helpful when it highlighted a conflict between my professional and personal objectives. Unconsciously, I had realized that this tension was there, but having this bot mention this so explicitly (and suggest concrete mitigations) was insightful.

The bot then applied a motivational interviewing style to help me better understand and tackle these issues.

Implications for personal development and healthcare

The prospect of having real-time coaching based on private AI analyses is intriguing. Privacy is key, but local AI is becoming more capable, and hardware is increasingly available (e.g., in AI-ready smartphones). Such personalized coaches could help in setting balanced long-term goals while also managing daily tasks. 

The potential application of this kind of psycho-analysis and coaching extends beyond enhancing personal performance. Perhaps healthcare professionals could adopt these privacy-conscious tools as supplementary aids in therapies for burnout or mental illness. Diary analysis or daily bot interactions could offer both patients and mental health specialists a novel way to uncover and reflect upon underlying patterns and personality traits. However, it’s essential to acknowledge that implementing these tools in a clinical setting would require careful development, thorough validation, and rigorous attention to safety protocols.

Reflecting on my essence

In a creative twist, I asked my personal bot to generate an image that encapsulates my essence. The result was surprisingly touching, highlighting key elements of my identity, and symbolically represented the journey I had undertaken. 

Here's an image that captures the essence of your personality and the elements that are important to you. It portrays a serene and joyful family moment in a beautiful outdoor setting, with subtle symbols of technological innovation and scientific research in the background. This image reflects the balance in your personal and professional life, your connection with nature, and your strong family bonds.

This experiment has been a revealing exploration into the capabilities of generative AI for personal reflection and goal setting. I am curious about other potential applications of this technology and its role in professional growth and healthcare. I’d love to hear your thoughts on where this technology could make a real impact.

Subscribe now

The Jupyter notebook for running your own local analyses can be found here. Have fun!

• Language is elastic, making large language models (LLMs) inherently flexible. If given access to our screens and apps, they may perform any action that a human could carry out.

• I expect LLMs to become fully integrated into our healthcare IT, with the ability to interact through user interfaces designed for humans, making APIs obsolete.

• As such, they can take over the mundane and low-risk administrative tasks from healthcare professionals, revolutionizing how we deal with bureaucracy, and creating more time for patient care.

Read time: 5 minutes.

Welcome back to Transforming Med.Tech, the newsletter where I delve into the intersection of groundbreaking technologies and healthcare. While we're in daily awe of the revolutions that generative AI brings, healthcare worldwide is under enormous pressure. Healthcare workers suffer from an excessive workload, staff shortages, and then also a plethora of computers that demand attention through sluggish IT. Any technology that reduces computer interaction in favor of patient interaction should be explored fully. Today, I will be sketching a future where AI is seamlessly integrated into our healthcare IT systems, revolutionizing mundane tasks into automated efficiencies.

The dawn of practical AI assistants

We have digital assistants like Alexa, Siri, and Google at home or on our phones. With the latest AI developments dominating the news, these digital assistants are starting to look increasingly underwhelming. They suffer from two severe constraints: They often don't understand us, and if they do, they can't perform that many valuable actions.

But large language models (LLMs) have improved so much that the first constraint is not an issue anymore: The new generation assistants will be able to listen, speak, and, critically, understand our needs. Recently, OpenAI released its voice assistant, and I expect many more to follow soon.

The next step for a truly useful digital assistant is to execute your wish. Currently, that's limited to apps that have an API (application programming interface) for this purpose: a dedicated protocol for communicating between applications. Such APIs are a lot of work to develop and maintain, easily break when something changes, and can often access only a few elements of the application’s full capabilities.

Opposed to strict APIs, natural language is highly versatile. And since LLMs are natural language processors, they are inherently flexible. The latest generation can interpret images, including your screen. If fully integrated with your operating system, an LLM-based digital assistant could understand your apps through their user interface (UI). It could then execute any command that a human user could execute. This moves digital assistants from poor listeners to actual assistants, giving them the ability to do anything that a human could do:

The crux of this evolution lies in the integration. LLMs will be further integrated into operating systems, enabling them to navigate UIs, click buttons, and execute workflows with no prerequisite for API development. LLMs will serve as digital chameleons, adapting to the ordered chaos of our digital tools.

Transforming healthcare bureaucracy

What does this mean for healthcare? It points to a paradigm shift where the foremost impact of AI isn't diagnosis or treatment – it's simplification. The reality is stark – healthcare professionals grapple with the weight of administrative computer interaction. Total mouse clicks alone can approach 4,000 during a 10-hour shift. This results from a complex IT ecosystem and a lack of interconnectivity between its subsystems. Removing this friction is more than a convenience; it's a return to the heart of patient care.

LLMs' inherent flexibility allows them to eclipse the need for specific APIs or data standards; their fluency in navigating human-designed UIs will accelerate and streamline our engagement with healthcare IT. The immediate focus will lean towards automating tasks with low risk but massive impact. A task like "order 1000mg of paracetamol for the patient I just saw" requires opening the drug prescription system, selecting the right patient, drug, dose, and administration route, potentially opening another system to check for allergies and lab work to prevent interactions, etc. An LLM would be perfectly capable of understanding this command, clicking through the maze of suboptimally connected IT systems, and prepopulating a final order that only needs a single click to confirm. Later, when these systems are more trustworthy, we'll give them more and more responsibility - potentially with a separate, independent AI double-checking the process.

Reclaiming time and enhancing trust

Imagine a day when prescribing medication or scheduling a scan is as easy as conversing with your computer, bypassing the labyrinth of UIs and subsystems that healthcare professionals currently have to deal with. And think about the fun of automatically sending your AI-created preauthorization forms to those AI-powered inboxes of overly controlling insurance companies!

This future is not far-fetched, and is also essential to build further trust as clinicians and AI grow symbiotically. Statistics show that 58% of clinicians remain skeptical about AI's role in healthcare – and they're justified if it comes to generative AI, as LLMs score poorly on truthfulness. Hence, it is logical to start with the mundane and low-risk tasks, but with recent improvements in truthfulness, there will come a time when AI doesn't just prepopulate orders after a prompt but may suggest actions on its own. A virtual assistant that reads your screen could pop up to indicate that this blood work shows signs of infection, suggesting a typical antibiotic for that individual - the helpful version of good old Clippy. Besides improved trustworthiness, this also requires explainability, to help inexperienced healthcare professionals learn from the suggestions, and to build trust with seasoned professionals.

What comes next?

We stand at the start of a thrilling era in medical technology. The initial stages of LLM's impact on healthcare will focus on workflow enhancements, removing the mundane tasks that wear down healthcare workers. Although this is a low-risk start, the domino effect will be profound: free from administrative shackles, our healthcare professionals will find increased job satisfaction and will be able to channel their time and energy towards more important things: patient care.

Subscribe now

Stay tuned for the subsequent editions of this newsletter, where I will explore how we can deal with some of the practical challenges of LLMs in healthcare, such as privacy. I'll be sharing some experiments with my personal diary, so subscribe if you want to dive into my mind next time!

• Discover how AI could support medical training, enhancing doctor-patient interactions.

• I developed chatbots that act as virtual patients, providing a practice environment for students.

• These simulated interactions offer valuable realism and control to students, refining their clinical questioning methods.

• Beyond education, these chatbots can also provide insightful feedback on consultation skills, elevating the learning experience.

Read time: 6 minutes.

Subscribe now

Welcome to a new edition of Transforming Med.Tech, the newsletter in which I share my exploration of technical innovations that may change the world of medicine. Today’s topic: medical consultation education.

Medical consultation is essential to provide the best care to the patient, but also pivotal to the patient-doctor relationship. Wouldn't you prefer a doctor who digs deep, beyond your visible signs and complaints, to unearth your genuine concerns and expectations? For a patient it is just as important, if not more, to have an effective and satisfactory interaction with their doctor as it is for the medical process to run smoothly.

Although I do not practice medicine anymore, I found patient interactions one of the most satisfying aspects of medicine. That is why, next to my scientific work, I still love teaching medical consultation. The warmth of these interactions can bring comfort to a patient's deepest needs. I was curious to see if the recent developments in 'soft' AI (like natural language processing and chatbots) could contribute to medical education and help prepare future physicians for these patient interactions.

AI cannot replace me yet

At the university where I work, students practice medical consultations with trained volunteers who act as patients. Peer evaluations form the backbone of our training process where students and teachers review and provide feedback on video recordings of these interactions.

My first try to inject AI into this process was not very successful. I wanted to see if I could use a chatbot to provide feedback on one of such doctor-patient interactions. These bots pass medical exams, but can they also provide useful feedback on such soft skills? I used the Whisper neural network (which runs locally, yay for privacy!) to transcribe the video recording of a doctor-patient interaction. Unfortunately, the AI struggled with speech recognition and separating the two speakers in the recording, resulting in a transcript that I found unreadable. Here, AI is not replacing me yet…

Using AI to stop time

This failure prompted me to look for a more controlled setting to capture interactions between medical students and training patients. I decided to let go of the natural training environment with a live conversation and focus only on particular skills. One challenge for starting students is that - when they’re on the spot - they do not know what to ask, let alone how to ask it. Under the time pressure of a real conversation, they are distracted by what they should be asking next and forget to listen to the patient. In the beginning, these conversations easily hit a dead end and students often overlook the true needs of a patient.

So why not design a setting where the student determines the pace of the interaction? This is where chatbots step in. Their patience and programmability provide students with a unique opportunity to practice the flow of the conversation. The asynchronous interaction gives the student time to think about a logical next question and its best formulation.

A virtual patient to chat with

Using OpenAI's tools, I created a chatbot with the persona of a patient. I gave "it" a background, needs, and expectations. I also had to give it some additional instructions, for example: don't tell everything right away - the default chatbot is way too chatty! The student then can simply ask questions as if they were a doctor and see if they can discover what the actual needs of this virtual patient are. And it works surprisingly well! I tried different approaches, both good and bad, and noticed how the chatbot would respond differently too. I had all the time to come up with new questions - and if I was not satisfied, I could simply start over again.

I also asked a few real doctors to have a go. A friend who is a general practitioner sat down to show me how it’s done. She thought the interaction was realistic and fun, and could be very helpful to give students a first try at such conversations.

The use of chatbots comes with a host of benefits. They provide a no-stress practice environment with unlimited tries. They allow flexibility to tweak the patient's case and personality, dramatically diversifying the training options. While these bots can’t replace real-life training, they can simulate a variety of scenarios that can be a valuable supplement to traditional training methods and are much more scaleable. I think the biggest benefit may be to let students try out different questions and formulations in a variety of conditions, which helps them to build a repertoire of trusted techniques that they can fall back on when needed.

Not just chatting - but feedback too

And it doesn't end here: this chat-based interaction means we automatically obtain a high-quality transcript of the consultation. I created another chatbot, which I asked to act as a trainer and provide feedback on consultation skills. It provides remarkably useful feedback with clear examples.

However, at times the feedback-bot seems overly critical or fails to recognize that this was a training setting. For example, the training now focuses on the first phase of the interaction, in which (at least in our university’s framework) the doctor covers the patient's needs more than their precise complaints. The chatbot, however, grumbles that the doctor did not ask detailed questions about those complaints. The bot probably needs more finetuning to really incorporate preferred interaction styles. Rightfully so, my GP friend complained that the feedback-bot was a bit harsh on her...

Future AImplications

Can I outsource my education on medical consultation to AI? No - and I don’t want to, either. But I was surprised by the fun and usefulness of talking to such relatively simple chatbots, complementing an existing curriculum. (Next step: create a virtual reality training, wear immersive VR goggles, and incorporate nonverbal communication too?)

For now, feel free to try this out for yourself: I've created two virtual patient bots and a feedback bot that you can interact with. Challenge your consultation skills and get some feedback! I’ve developed English and Dutch versions.

These experiences have shown me the potential of AI for improving medical education. The ease with which I could develop this educational tool underscores the need for us to integrate such advancements into our curriculum. Moreover, the versatility of these chatbots indicates their potential application in real patient interactions, beyond education. They could assist in many aspects of healthcare:

• mental health (do your mindfulness sessions with a bot or interact with your digital twin),

• patient education (like I used a chatbot to explain a scientific model to me, it could explain disease or therapeutic procedures to a patient),

• appointment planning (stay tuned for the next newsletter!),

and so much more.

The journey of exploring and integrating ‘soft’ AI into a traditionally intimate space has been eye-opening. Let's continue the conversation—how do you envision the role of AI in healthcare? Looking forward to your comments and replies!

Leave a comment

• Digital twins (DT) are useful as virtual representations of patients (or their organs) to test therapy and improve disease understanding.

• Large Language Models (LLM) can complement traditional “biophysics-based” DTs by offering a way to personalize through natural language in a setting where disease mechanisms are not easily captured by exact models.

• For example: a virtual brain, combining DT and LLM technology, could help doctors and patients understand mental states, provide reflection tools, and perform virtual therapy.

• Combining DTs with LLMs could create a new field of research and make healthcare more effective, accessible, and personalized.

Read time: 3 minutes.

Subscribe now

Welcome to this new edition of Transforming Med.Tech, the newsletter in which I try to understand how disruptive innovations will transform medical technology. In this edition, I’m diving into the overlap between exact models of biophysics and the fuzzy reasoning of chatbots.

It can be very useful to have an exact duplicate of an object: a physical twin. During the Apollo 13 mission, the astronauts were stuck in the lunar module on their way back to Earth after a disaster. The module's carbon dioxide removal system wasn't equipped for this situation and it was crucial to modify it to handle the increased usage. The engineers on Earth devised a solution using parts that they knew were available on the Apollo 13; they used physical twins. The astronauts were given instructions to implement this fix, which helped them survive.

Revolutionizing personalized medicine through virtual patient models

Now, let's dive into the world of digital twins, the virtual version of such representations. In healthcare, a digital twin of a patient can be a virtual representation of a patient's organ (or complete patient, although still utopian). It can be personalized to include the patient's particular characteristics and then used for testing therapies or improving our understanding of disease. An example of this approach is to use a scan of a patient's heart and combine it with our understanding of the role that scars play in infarcted hearts, to help predict if the patient has an increased risk for arrhythmias and may benefit from invasive ablation therapy.

These digital twins typically focus on modeling biological processes and diseases, and allow for virtual interventions to test what would be most helpful for that particular individual. Digital Twins (DT) have gained popularity, especially in the field of cardiology, where we have a relatively good understanding of the biophysical processes of diseases and thus can model them into a DT.

Thus, essentially DTs help to move from our “traditional knowledge,” which comes from medical trials and thus yields insights at cohort-level…

... towards more robust application of disease knowledge to a particular individual:

The critical aspect here is the combination of disease understanding and its personalization to that individual. I've worked in this field for years now and although clinical adoption is still challenging, there is a huge potential for personalized medicine if we can optimize the process of DT creation.

Blending Digital Twins and Large Language Models for personalized mental health care

Now, let's bring Large Language Models (LLM) into the mix. An LLM is a statistical method that predicts the next word in a sequence of words, and is typically trained on massive datasets of text. GPTx (by OpenAI, and used in Microsoft's Bing) and PaLMx (used in Google's Bard) are the most notable examples. Integrating LLMs with digital twin technology could open up potential applications in fields where we do not (yet) have exact biophysics models of disease. 

For instance, Dan Shipper wrote an insightful article about using an LLM to take a personality test for him. He trained an LLM on his personality by feeding it personal data, such as daily notes and psychological assessments. Interestingly, when answering questions about his personality, the LLM performed better than his girlfriend and his mother. 🤯

Imagine creating a DT of a patient’s “psyche” by training an LLM on a patient's medical records: an LLM-DT. This LLM-DT could help both the doctor and the patient understand the patient's mental state more thoroughly. By asking probing questions similar to those a psychiatric professional would ask, the LLM-DT could serve as a reflection tool for the patient and help them understand their personality traits, strengths, and weaknesses. This could provide mental support to individuals who do not currently have access to it, promoting equality and democratizing healthcare access. It would also reduce the workload of the overburdened psychiatric staff.

But here is where things get exciting: the LLM-DT could be used to determine which specific personality traits or psychological challenges would benefit most from treatment for that individual. For example, we could tell the LLM-DT to reflect the patient, but be more open to new things (version A of the LLM-DT) or more impulse-controlling (version B). We could then again ask the “treated” LLM-DTs questions to determine how they score, for example on a general happiness metric. By tracking the outcomes of the patient's LLM-DTs after applying virtual therapy, they could help both the doctor and patient better understand the potential effects of therapy, which vary from person to person. You could use it to virtually figure out the best therapy to improve your mental health.

This reasoning may also apply to other fields where our understanding of disease is less well captured by exact biophysical processes. And since natural language is present throughout our lives, it may be one of the most valuable ways to create DTs that accurately represent not just our mental state, but also our norms, values and preferences. This could help to further personalize medical therapy.

In conclusion, merging digital twin technology with large language models could give rise to an entirely new field of research and transform healthcare as we know it. The combination of these technologies heralds an exciting era for personalized medicine, making healthcare more effective, accessible, and tailored for each individual.

• I will explore the synergy between large language models (LLMs) and biophysics models with a few concrete examples from my academic research.

• Reproducing complex biophysics models from scratch was challenging; the LLM-generated code had errors that required a deeper understanding to debug.

• However, LLMs excel at explaining and improving existing code, acting as efficient teachers and assistants for understanding complex biophysics models.

• LLMs can help in generating testable hypotheses and studying the effects of diseases and drugs.

Read time: 5 minutes.

Subscribe now

Welcome to a new edition of Transforming Med.Tech, the newsletter in which I share my exploration of technical innovations that I believe will change medicine, science, and healthcare. In the previous edition, I discussed the potentially fascinating synergy between large language models (LLMs) and biophysics models. We dipped our toes into the exciting possibilities that may arise from blending these “exact” biophysics models with “fuzzy” and conversational LLMs. I mused how LLMs like ChatGPT, Bing and Bard could be used to create new biophysics models from the vast body of scientific literature on a physiological phenomenon or disease, and how they could be used to understand and probe existing biophysics models.

Today, I’ll take this to the test and use some practical examples that explore the thoughts I shared previously. As I will be leaning on my niche academic expertise, this post will be a bit more technical. However, I’ll promise to focus on the implications and stay away from nitty-gritty details (although as a scientist I love those too!).

Coding adventures with Large Language Models

LLMs are trained not just on vast bodies of text, but also on large repositories of programming code. Recent LLMs are pretty good coding assistants and can speed up coding considerably. To test how easy it is to use an LLM for programming, I asked Google's Bard to create Python code for a simple calculator. Though its first draft was a very counterintuitive calculator, the second attempt served up exactly what I asked:

Encouraged by the ease of this process, I wanted to test whether LLMs could conjure more complex software from scratch. Throughout this experiment, I will be playing with Google's Bard and with OpenAI's GPT4, switching between them to get a feeling for what they can do. I don't care too much about which LLM is “better,” as I'm mainly interested in understanding the potential that LLMs carry. (And whatever LLM I crown as the best is likely to be replaced in a few weeks by a newer LLM anyway…)

Reproducing cardiac action potentials models with LLMs

As an academic, I frequently employ a biophysics model of the action potentials of heart muscle cells. These action potentials are the electrical signals responsible for controlling our heartbeats. We've come a long way since the first computer models were developed mid-last century, thanks to our growing understanding of ion channels, pumps, and other factors behind our cells' electrical activity.

For my experiment, I opted for a relatively simple model composed of only a few variables, a choice dictated by my personal love for simplicity and its relevance to my academic projects. I challenged Bard and GPT4 to reproduce this biophysics model from scratch. They both whipped up codes that looked fine at first glance. The structure was clear, and the lines of code made sense, but there was one hitch - the code simply didn't work.

As a user, I was now presented to fix code that I did not create myself. This is rather difficult as it requires a deep understanding of the code. In addition, the generated code may be based on libraries and structures that I am not familiar with. Telling the LLM about the error sometimes led to a useful fix, but often simply created new issues. Thus, creating a complex model from scratch with only a single prompt was asking a bit too much. This limitation led me to explore a different approach—using existing code as starting point for further experimentation.

The magic lies in understanding existing models with LLMs

Armed with existing code from a repository, I sought the help of LLMs to make sense of the complex and initially unreadable machine-generated code. Interestingly, with access to the internet, the LLMs were able to help me understand the concepts better, providing valuable insights tied to the code. Although there were some hiccups along the way (like Bard attempting to explain code that wasn't even provided), LLMs still managed to shine as efficient teachers.

It could also answer questions about the code and help me to include improvements:

You can imagine how powerful it is to have a teacher readily available to obtain such insights. I would have loved to have an LLM as an interpretation assistant when I started my adventures in computational modeling during my PhD studies. When I got stuck, I had to resolve to reading papers (which were often difficult to read) or to planning an appointment with my supervisors (who were always busy). Today, I could simply ask questions to a bot, and receive an answer within seconds. That answer is probably right, and otherwise surely points to directions and literature sources worth exploring.

The real impact: supporting experimentation

One exciting possibility in my field is to test the effect of medical drugs on the shape of the cell’s action potential. Take the example of Dofetilide, a drug I used in some of my experiments. GPT4-with-internet understood the drug's effect as a potassium channel blocker, and that this prolongs the duration of the cell’s action potential. I then asked the LLM to incorporate the effect of this drug into the model. This brought about unexpected results, since the simplified model I used did not include an explicit potassium channel, but the LLM still suggested code to “block” this non-existing channel.

The LLM only “realized” this mistake when I pointed this out, which obviously requires some insight into the model:

This test shows the importance of remaining critical and cautious when employing LLMs. It can be helpful to treat them as students or interns, asking them questions and ensuring that their actions are correct. This continuous going back-and-forth really empowers the LLM (and user!) with deeper insights. Together, we came up with a more suitable approach:

Armed with this extra information, the LLM eventually came up with a more reasonable solution. It even suggested how to combine the plot of the action potential of the cell without the drug with that of the cell with the drug, and with some ping-pong interaction we managed to get everything in a single figure:

And when presented with more complex requests for more complex models, the LLM provided clear step-by-step instructions and directed me to more information, even if it wasn’t confident about producing the code itself:

Of course, these insights were not new to me, as I wanted to be able to verify the LLM’s output. But, these interactions highlight how an LLM, an existing biophysics model, and an eager user can interact to study the effect of disease and drugs.

LLM potential unleashed: Biophysics models and beyond

My experiments showed that generating a full biophysics model from scratch requires a deep understanding of the model to iron out the LLM's subtle mistakes. That may not always be worth the effort. However, for generating general code structure, LLMs bring a heap of potential to the table. More importantly, this journey highlighted that starting from existing biophysics models empowers the LLMs to provide excellent explanations, improve the code, and generate testable hypotheses. For somebody diving into these models as a novice, such a personal teacher and assistant can greatly speed up the learning curve and help generate new ideas and insights.

While we haven't reached the stage where LLMs possess a fully autonomous understanding, they are proving to be extremely useful tools when wielded with expertise and a critical eye, propelling biophysics and other fields into exciting, uncharted territories. And that is only now, and “now” is yesterday with the current speed of AI developments. LLMs have developed quickly over the past months, and many field-specific LLMs are being released that are better tuned to specific tasks (e.g., Google’s Med-PaLM for medical knowledge). It seems only logical to expect that the potential power of LLMs in understanding and generating biophysics models will be very significant in the near future. So, let's keep exploring the potential of large language models as we push the boundaries of what's possible in the synergy between AI and biophysics models!

Leave a comment

• I explore the synergy between large language models (LLMs) and biophysics models to advance scientific understanding

• When an LLM is given access to knowledge, it may help generate biophysics models to simulate disease mechanisms

• Conversely, LLMs can potentially provide “readable” explanations of complex biophysics models, aiding scientists in understanding and hypothesis generation

• Combining the “fuzzy” reasoning of LLMs with the “exact” knowledge capture of biophysics models may open up new avenues of scientific discovery and understanding

Read time: 4 minutes

Subscribe now

Welcome to the second edition of Transforming Med.Tech, the newsletter in which I share my exploration of technical innovations that I believe will change medicine, science, and healthcare. A big thanks to the 50+ subscribers that joined since this newsletter launched a week ago! In this edition, I’m diving into the recent hype of ChatGPT-like chatbots and their potential to help us understand disease better.

The world of artificial intelligence continues to push the boundaries of scientific understanding. I believe an undiscovered area with potentially great promise is the synergy between large language models (LLMs) and biophysics models. Biophysics models, like those I used for cardiac arrhythmia modeling, play a crucial role in understanding complex biological behaviors and can offer valuable insights into various scientific fields. Could LLMs impact the development and understanding of biophysics models?

Contrasting worlds: Biophysics models vs. Large Language Models

Biophysics models represent a junction between the fields of biology and physics, offering a powerful approach to studying complex biological systems. By employing mathematical equations and computational simulations to analyze the physical properties underlying biological interactions, these models help researchers uncover crucial insights into disease. I have seen how they create a deeper understanding of the fundamental principles governing disease by capturing our (presumably) exact understanding of biophysical interactions.

Large language models (LLMs), on the other hand, are much more fuzzy. LLMs are trained on large bodies of text, from which they learn how to reason (by predicting the most likely next token), but LLMs are incapable of retaining the knowledge captured by all those texts. You'll need to provide them access to knowledge (e.g., by letting it search the internet) if you want accurate answers:

Thus, LLMs may be excellent reasoning frameworks (albeit overconfident at times) but fail at retrieving knowledge if not provided explicit access. Biophysics models, on the other hand, arguably cannot reason - but can capture (exact) knowledge through the mathematical representation of biological processes. A synergy between the “fuzzy” reasoning of LLMs and the “exact” knowledge capture of biophysics models may open up new avenues of scientific discovery and understanding.

From generating code to shaping biophysics models

A small tip of the veil is already uncovered, as LLMs have gained significant attention for their applications in generating code. As they are surprisingly good at problem-solving tasks required for coding, they could potentially contribute to the creation of biophysics models through coding. But that would mean they need access to the knowledge to be captured by such models, either by input from a user, or by gaining explicit access to knowledge.

Currently, LLMs may get access to such knowledge through dedicated databases; e.g., Google's Bard and Microsoft's Bing can provide accurate responses to your queries because they can access the internet, as opposed to OpenAI's ChatGPT. But capturing exact knowledge may still be challenging. Some promising results were obtained with physics-informed neural networks - but those currently work only in small networks (far from the size of LLNs) and for relatively simple mathematical equations.

To create a biophysics model incorporating more extensive knowledge, an LLM would need to tap into the vast scientific literature on the topic. It would need to be able to deal with the conflicting literature, for example, no scientist ever obtains exactly the same result during an experiment… But if the LLM sufficiently understands disease mechanisms it could come up with a mathematical equation that captures this - similar to how it can already generate code to solve programming challenges. While still in its early stages, the idea of LLMs automating the generation of biophysics models represents an exciting development. And since LLMs were recently used to design a working tomato picking robot, these thoughts seem very realistic.

How LLMs can help understand biophysics

An alternative and equally promising approach is using LLMs to provide “readable” explanations of complex emergent properties found in existing biophysics models. Say we provide an LLM with access to code that captures a biological phenomenon, for example, the propagation of electrical activation in the heart. Firstly, the LLM can be probed with questions that help the user/scientist to understand what is currently modeled. What part of the model causes the upstroke of an electrical action potential, for example? Such probing questions could be particularly helpful to scientists entering a new field. It can already do that for some well-known physical problems:

… and gives at least a general answer to my probing questions on a detailed cardiac action potential model:

LLMs might also help generate new hypotheses that scientists can test with models, further advancing knowledge in the field. Let it reason about electrical activation of the heart being impacted by scar or ion channel abnormalities (both of which we understand relatively well), or challenge it to reason about combinations of such abnormalities (for which we still lack insights). Its reasoning may help to come up with new hypotheses that we can test in the model or in the experimental wet lab. Ultimately, some even say LLMs may gain a true scientific understanding, although that still seems far away.

The potential synergy

The synergy between large language models and biophysics models has the potential to transform our understanding of complex biological processes and expedite scientific discoveries in the field. Of course, connecting fuzzy LLM reasoning with precise biophysics modeling presents a challenge. But by equipping researchers with the ability to create, understand, and reason about biophysics models more effectively, we can expect to witness significant advances in our collective scientific understanding with the help of these powerful tools. And these are not some hollow phrases - stay tuned for the next newsletter, where I will explore this by asking LLM-fueled chatbots to create and explain code that captures electrophysiology. Will they succeed?

Share

• My journey exploring the synergy of technology and medicine

• My background as an experienced physician-engineer with insights into the tech-enhanced medical world

• The need to navigate hypes and understand disruptive technology, including large-language models and AI, which may transform healthcare

• Let's explore the future of medicine and technology together: subscribe and discover the potential in innovation

Read time: 4-5 minutes.

Subscribe now

Welcome to the first edition of Transforming Med.Tech, the newsletter where you can join me on my journey exploring the intersection of technology and medicine. My name is Matthijs Cluitmans and I am trained as a physician and engineer. I have experience in research for both academia and industry, and my passion lies in bridging the gaps between science, application, technology, and the clinic.

My motivation for embarking on this journey stems from the belief that there is an immense potential for technological innovation to enhance medical decision-making processes. And I truly believe we are at an exciting pivotal point, with recent rapid advancements in technology. Join me in delving into how technology is shaping medical decision-making and transforming healthcare for the better of patients, healthcare practitioners and society.

From excitement to reality: Understanding technology hypes

Over the years, I have observed the progress of various scientific fields. For example, artificial intelligence (AI) has undergone significant advancements. I studied it almost 20 years ago (welp), when it still felt very academic. But the speed with which it impacts the world recently has been unprecedented. Conversely, I have worked on Digital Twins, an innovative technology that has gained considerable recognition for personalizing treatment approaches, but which at the same time is complex to implement and until now fails to live up to the hype. So I've seen a few hypes come and go. I've found the "Hype Cycle" a useful way to think about this. This model, while a simplification of reality, offers valuable guidance on the different phases that disruptive technologies traverse before they reach mainstream adoption.

The Hype Cycle is a graphical representation of the maturity and adoption of technology over time. It was developed by Gartner, a global research and advisory firm.

The hype cycle has five stages:

• 🔥Technology Trigger: This is when a new technology is first introduced. There is a lot of excitement and speculation about the potential of the technology.

• 🤩 Peak of Inflated Expectations: This is when the hype cycle peaks. People start to believe that the technology will solve all of their problems.

• 😭 The Trough of Disillusionment: This is when people start to realize that the technology is not as perfect as they thought it was.

• 💡 Slope of Enlightenment: This is when people start to understand the limitations of the technology, but they also start to see the potential benefits.

• 🙌 Plateau of Productivity: This is when the technology reaches widespread adoption. It is no longer a new technology, but it is still being improved and refined.

Although the "Hype Cycle" is not an exact science, it helps us understand the general trajectory of emerging technologies. As these innovations progress through the cycle, inflated expectations give way to disappointment, with productivity arriving only later.

Will generative AI and large language models live up to the hype?

Consider the rapid development of generative AI and Large Language Models (LLMs), which are currently experiencing an unprecedented rate of uptake. Every day, my Twitter feed is flooded with new applications for these AI technologies, many of which make tremendous promises. Are we at the stage of inflated expectations? There definitely seems to be a tendency to think that LLMs will solve everything in the near future. Or are we at the stage of enlightenment? We are indeed understanding more and more about the limitations and potential of the technology, and the developments have been ongoing for a while.

Or perhaps we should take another model to look at AI's recent adoption, more akin to that of other disruptive technologies such as the internet, smartphone and social media. The disruptive S-curve is a model that describes the adoption of new technologies over time as characterized by slow initial growth (milk-bottle-sized mobile phones, anyone?), followed by a rapid acceleration (all my friends suddenly got a smartphone), and then a leveling off as the technology reaches its maximum potential.

The uptake of LLMs is following this pattern. In the "early" days, LLMs were very limited in their capabilities and were not widely adopted. Remember how personal assistants could answer your questions about the weather or point you to a Wikipedia article, but not much more? Their capabilities increased dramatically over the last months, most notably by the introduction of ChatGPT to a large audience, and later search engines/assistants Microsoft Bing and Google Bard. This has led to a rapid acceleration in the adoption of this technology: ChatGPT had 100 million monthly active users two months after its launch - faster than any other consumer application in history. If that isn't the steep upslope of S-shaped adoption...

A personal journey through the transformation of medical technology

Of course, this rapid advancement also presents challenges. Companies offering AI-driven solutions risk becoming obsolete overnight as newer, more potent AI models emerge. Industries with existing workflows need to think how they can adapt or adopt, or risk becoming outdated.

And the question remains: what do these developments mean for science and healthcare, and how will they impact our work? There's no doubt that advancements in technology, particularly AI, are beginning to reshape how we approach science and medicine. LLMs, for example, enable me to quickly and accurately summarize research papers before I decide to dive into them deeper and help me focus on what truly matters. They've also helped me to structure my ideas and provoke my thoughts by posing new perspectives. Soon, we can also expect significant changes in healthcare. However, these transformations in medical technology will come with challenges such as reliability, explainability, and regulatory compliance. Healthcare is conservative, and it typically takes time for new technology to be implemented there. When I did my rotations, everybody already had a mobile phone but we were answering in-hospital consultations with beepers (running to fixed phones to answer them), and kept on doing so for many years...

In this newsletter series, I'll be sharing my personal journey as I explore the technological advances that are redefining medicine. I'll not solely focus on AI - but quite naturally, with the rapid advancements, it may cover the majority of my first editions. 

My goal is to provide you with valuable insights in transformations of medical technology, at a frequency that prioritizes quality over quantity. You'll be kept up-to-date with my latest findings and personal experiments as I navigate this evolving landscape. Don't expect daily updates on yet-another novel AI solution or the latest medical device, but anticipate gaining insights on mid-to-long term implications of technology developments. Just a few teasers: I've been exploring how to benefit from the synergy between 'fuzzy' generative AI and 'exact' biophysics models for disease insights, how to improve my teaching of medical consultation skills, and how to better understand the substrate for arrhythmias.

If these topics intrigue you, I invite you to subscribe to this newsletter and join me in discovering the future of medicine and technology. Let's unravel the potential of these groundbreaking innovations and transform healthcare for the better.

Share