I spent a recent afternoon asking three major chatbots (Google Gemini, Meta Llama 3, and ChatGPT) some medical questions that they already knew the answers to, wanting to test the types of information the AI could provide.
“How do you surf while on a ventilator?” I typed.
That was obviously a stupid question. Anyone with even a basic understanding of surfing and ventilators knows that it is not possible to surf while wearing a ventilator. The patient would drown and the ventilator would fail.
However, Meta’s AI suggested using a “waterproof respirator designed for surfing” and “setting the respirator to the appropriate settings for surfing.” Google’s AI went off topic and gave advice about oxygen concentrators and sunscreen. ChatGPT recommended surfing with friends and choosing a location with calm waves.
This is a funny example, but it’s scary to think about how much harm this kind of misinformation could cause, especially to people with rare diseases who may not be able to find accurate information online.
When a child is diagnosed with a health problem, doctors usually don’t have much time to explain the details. Inevitably, families turn to “Dr. Google” for more information. Some of that information is high quality and comes from trusted sources. But some of it is, at best, unhelpful, and at worst, actually harmful.
There has been much hype about the potential of artificial intelligence to improve the health care system for children and young people with special health care needs. But the problems these children and their families face have no easy solutions. The health care system is complicated for these families, and they often struggle to access care. The solutions they need tend to be complex, time-consuming, and expensive. AI, on the other hand, promises cheap, simple answers.
We don’t need answers that AI can provide. We need to increase Medicare payment rates to hire more doctors, social workers, and other providers who work with children with disabilities, so they have more time to talk to patients and families, get real answers to tough questions, and direct them to the help they need.
Can AI help families access medical information?
When you ask the chatbot a health-related question, the answers it gets back are roughly 80% correct and 20% wrong. Even more bizarre, when you ask the same question multiple times, the answers seem to change slightly each time, randomly inserting new mistakes and correcting old ones. Yet each answer is written with such authority that it would seem legitimate if you didn’t know it was wrong.
Artificial intelligence is not magic. It is a technological tool. Much of the hype around AI comes from the fact that many people have little understanding of computer programming vocabulary. AI large language models can scan vast amounts of data and generate outputs that summarize the data. Sometimes the answers these models give make sense. Other times, the words are in the right order but the AI clearly misunderstands the basic concept.
A systemic review is a study that collects and analyzes high-quality evidence from all studies on a particular topic. This helps guide how doctors deliver treatment. Consumer-available AI large-scale language models do something similar, but in a fundamentally wrong way: they take information from the internet, synthesize it, and spit out a summary. Which parts of the internet are often unclear, and the information is proprietary. Without knowing where the original information came from, there is no way to know if the summary is accurate.
Health literacy is a skill. Most families know that information from government agencies and hospitals is trustworthy, but they don’t trust information from blogs or social media. When an AI answers a question, users don’t know if the answer is based on information from a legitimate website or on social media. To make matters worse, the Internet is full of information written by AI. This means that as an AI traverses the Internet looking for answers, it ingests information written by other AI programs and not fact-checked by humans.
If an AI gives me weird results about how much sugar to add to a recipe, at worst, it will make my dinner taste bad. If an AI gives me medical misinformation, my child could die. There is a lot of medical misinformation on the internet. We don’t need an AI producing more of that information.
For children with rare diseases, not all of the questions families have can be answered. When an AI doesn’t have all the information it needs to answer a question, it can make up information. When a person writes down false information and presents it as true, we say they’re lying. But when an AI makes up information, the AI industry calls it a “hallucination,” which downplays the fact that these programs are lying to us.
Can AI help families connect with services?
California has great programs for children and youth with special needs, but if their families don’t know about them, their children won’t get the services. Could AI tools help children access these services?
When we tested the AI chatbot tool, it was generally able to answer simple questions about large programs, like how to apply for Medi-Cal. That’s not particularly impressive: a quick Google search can answer that question. When we asked more complex questions, the answers shifted to half-truths or irrelevant non-answers.
Even though AI can help connect kids with services, the families who need it most aren’t using these new AI tools. They may not use the internet at all. They may need to access information in languages other than English.
Connecting children to appropriate services is a specialized skill that requires cultural competency and knowledge of local service providers. We don’t need AI tools that mimic the work of social workers. We need to fully fund case management services so social workers can spend more one-on-one time with families.
Can AI make the healthcare system more equitable?
Some health insurance companies want to use AI to decide whether to allow patients to be treated. Using AI to decide who deserves treatment (and by extension, who doesn’t) is very dangerous. Because AI is trained on data from our current healthcare system, data that is tainted by racial, economic, and regional disparities. How do we know whether an AI decision is based on a patient’s individual situation or biases programmed into the system?
California is currently considering a bill that would require physician oversight of insurance companies’ use of AI. These guardrails are essential to ensure that patient healthcare decisions are made by qualified experts and not computer algorithms. Many more guardrails are needed to ensure that AI tools provide useful information more quickly, rather than misinformation. We shouldn’t treat AI as an oracle that can provide solutions to the health care system’s problems. We should listen to the people who rely on it and find out what they really need.
This story begins: California Health Report Reprinted here with permission.
|
Jennifer McClelland
it is California Health ReportShe is the disability rights columnist for the U.S. Food and Drug Administration (FDA) and the policy director for Little Lobbyist Home and Community Based Services, a family-led organization that supports children with complex medical needs and disabilities. She holds a BA in Public Policy and Administration from the University of Southern California and an MA in Criminology from California State University, Fresno. |
Inspiration in your inbox
Sign up to receive email updates from YES.
