ChatGPT Was on My Medical Team
My doctors recommended surgery. ChatGPT had a different perspective. As artificial intelligence increasingly joins human experts in decision making spheres, we must be aware of the way in which AI does more than act as a thought partner – it shapes the very environment in which our decisions are made.
Photo by: HANS LUCAS/Reuters
On Monday, September 7, I decided to part ways with my ovaries. It was not clear that anything was wrong with them. My latest ultrasound looked slightly less concerning than the one before it. And yet, after intensive testing and consultations, I decided that my ovaries and I had reached the end of our journey together. It was a significant personal and medical decision, and like decisions of this kind, it had to be made under conditions of uncertainty: weighing risks, family history, quality of life, probabilities, and preferences, and ultimately making a choice.
As I made that decision, I had two human authorities whose judgment I trust deeply: a beloved sister who is a senior physician, though not a specialist in women’s health, and one of Israel’s leading gynecologists. Both believed the surgery was necessary, and I did not have much time. The night after speaking with them, I also turned to ChatGPT. We have been talking for several years now—I was an early adopter—and over the years it has become far more capable, more involved, and better able to remember what I have told it. As someone who has spent years studying algorithmic bias and human reliance on technological systems, I found myself observing not only the decision I was making, but also the way I was making it: whom I trusted, when, and why. That, I discovered, is also one way of coping with anxiety.
ChatGPT went over the test results with me again and again. With nonhuman precision, it analyzed the ultrasound printouts and found meaning in millimeters. It laid out arguments and dismantled them, connected pieces of information with probabilities, gave me a long explanation and then gave it again in simpler terms, and answered the same question for the fifth time without hinting that perhaps it was time for me to go to sleep. I realized that it saw things differently from my sister, and I felt lost. The next morning, I added another member to the panel: my wise, longtime family physician, who has known me for years.
In the end, ChatGPT did not decide whether I would undergo surgery. I did. But I made that decision within something like a “marketplace of trust.” I remembered that the ultrasound technician had been less concerned; the specialist recommended surgery; my sister thought it through with me; the artificial intelligence saw things differently from them; and finally, my family physician joined the discussion as well. There was no mathematical way to combine all the advice I received. Each source had a different “trust coefficient” for me, made up of expertise, familiarity, prior experience, intimacy, availability, and perceived interests. In that sense, artificial intelligence entered the “architecture of trust” through which I decide whom to listen to.
Keeping a human in the loop?
For decades, we have been familiar with the concern known as “automation bias”: the tendency of people to accept the recommendation of an automated system even when they should independently verify it. But the new generation of AI systems is changing the nature of the problem. The “machine” is no longer a warning light in a cockpit. We talk to it. It explains itself. We argue with it and it responds. We ask it to consider an alternative and it does. It remembers context, learns how to explain things to us, and adapts itself to us. Reliance on a machine is beginning to shift from a one-way encounter into a relationship. That fundamentally changes how we need to think about artificial intelligence and decision-making. We have grown accustomed to asking whether a machine is accurate, whether it is biased, whether its output can be explained, and whether it performs better or worse than a human being. But the more interesting question is what happens to our own process of judgment when the machine enters the room.
This dilemma is now emerging wherever consequential decisions are made. Consider, for example, the use of artificial intelligence by judges. Many judges and legal scholars draw an intuitive distinction: a machine can be an efficient legal assistant sitting on a judge’s desk, but it should not be the one sitting in the judge’s chair. But this overlooks the fact that even a machine that sits “only on the desk” can determine which documents the judge sees first, what is included in the summary of a case that runs to thousands of pages and what is left out, which precedent is presented as analogous, and which legal question is framed as central. Even summarizing is not a cognitively neutral act. Whoever organizes the material for us also participates in organizing the way we think about it.
No matter, we reassure ourselves: whenever we design systems like these, we will make sure there is a “human in the loop.” The “human in the loop” has become an almost automatic answer to concerns about handing power over to artificial intelligence: the system may screen job applicants, assess medical risk, or summarize legal materials, so long as a human being makes the final decision. But that solution assumes precisely the thing that unsettled me: we are using human judgment as a safety mechanism even as that very judgment is being shaped by the system it is supposed to supervise.
When we focus too heavily on who makes the final decision, we miss another question: who structured the decision space? Who determined which information would appear important and which would recede into the background; which options would be presented; who would appear authoritative, and which explanations would sound plausible before the human being ever reached the moment of decision? If the machine has already determined what to look at, whom to examine, which risk to emphasize, and which argument appears central, then the slogan “human in the loop” tells only part of the story.
When the tool becomes a teammate
There is another layer as well. The machine influences us not only through the information it provides or the way it organizes that information, but also through the kind of relationship we develop with it. Research in recent years has shown that even framing matters: when the same system is presented to a person as a “teammate” rather than a “tool,” both the level of trust placed in it and the nature of the interaction change. As we move from systems that provide a score or recommendation to systems that converse with us, remember us, explain themselves, and respond to our objections, the question is no longer only what the machine said, but who it has become to us.
I saw something similar in a series of “smart boardroom” simulations we conducted over the past year with senior executives. In one, an AI system ran the meeting: it allocated speaking time, brought participants back to the agenda, and even commented on their behavior. The participants argued constantly with one another, but almost no one challenged the machine’s authority to manage them. In another scenario, the CEO and CFO were each given an AI adviser drawing on the same data but sometimes arriving at different interpretations. Before long, they were no longer talking about “the system,” but about “my adviser,” and even tended to defend the interpretation offered by the adviser assigned to them.
The machine in these simulations did not make any decisions for the humans. It did something subtler: it became an actor within the social system in which decisions were made. It acquired authority, inspired loyalty, changed the language people used with one another, and made certain forms of behavior seem legitimate. In other words, AI does not necessarily cause us to stop thinking or deciding. It shifts the ground on which our thinking takes place. We move from gathering information to verifying it, from solving a problem to choosing a solution that has already been formulated, and from carrying out a task to managing and supervising it. We continue to think and choose, but the machine changes where our thinking begins, what it is directed toward, and with whom we think.
That was exactly what troubled me that night with ChatGPT. It was doing more than placing another piece of information alongside the doctors’ opinions. By then, we had a history of conversations. I knew how to question it, and it knew a great deal about my circumstances. I could challenge its reasoning, and it could defend its position or revise its analysis. This then begged the question of how the relationship itself changed the position from which I weighed everyone else’s advice.
None of this means that we should remove the machine from the room, nor is there any reason to pretend that human judgment is pure, objective, or free from influence. Quite the opposite. Human beings have always made decisions within systems of influence: judges listen to lawyers and experts; doctors consult colleagues and professional literature; all of us rely on people we trust. The institutions we have built—legal procedure, medical ethics, administrative law—all have these influences baked in. But they try to structure them: to make them visible, allow them to be challenged, require reasons, and create the possibility of review.
Perhaps that is also what we will need to do with artificial intelligence. We should not be satisfied with asking whether we left a human being in the loop. We need to understand what kind of relationship is being created between the human and the machine: what authority the machine acquires in the person’s eyes, how trust in it is built, whether it encourages resistance and skepticism or instead weakens the tendency to question, and how its very presence changes the way a person sees the problem before them.
I discovered that I cannot say that artificial intelligence made the decision for me to part ways with my ovaries. But the opposite statement—“I made the decision; the machine merely assisted me”—suddenly seems too simple as well. The decision emerged from a system made up of a woman and a machine, doctors and tests, knowledge and uncertainty, expertise and trust. More and more medical, legal, governmental, and professional decisions will emerge from systems like these. Our challenge is to understand and shape the relationships within the loop, not just to make sure a human is in it.
This article was published in The Times of Israel.
