Study 1

188 adults rated how much quantitative reasoning fifteen medical decisions need, then rated eleven medical recommenders, from a doctor to an AI fitness coach, on trust, quantitative reasoning and intuition. This page gives the main findings and what they mean. Every table and figure is on the next page, rebuilt from the released data.

Chapter 2: Is the gap real, and what is behind it?

The first chapter of the experiments asks a plain question. When a doctor and a medical AI are judged on the same health decision, do people trust the doctor more, and is that true for every kind of decision? Both studies take the view of an ordinary patient, using broad medical decisions rather than one specific scenario.

The chapter also tests a first explanation. Clinical expertise blends quantitative reasoning (data and statistics) with intuition (experience and judgment), and an AI is usually thought to have only the first. If people trust doctors more because they credit them with both, ratings of these two abilities should track trust.

Study 1 (188 adults) rated fifteen decisions and eleven recommenders, from a doctor to an AI fitness coach. Study 2 (106 adults) replicated the gap on twelve concrete decisions, rated a doctor and an AI side by side, and added warmth, competence and mind perception. Together they set up Chapters 3 and 4, where I try to close the gap by making an AI seem more human.

Main findings

A doctor is the most trusted of eleven recommenders, and AI sits near the bottom

188 adults rated how much they would trust eleven recommenders for medical decisions, from a doctor to an AI fitness coach. The doctor came first, then the nurse. Every AI landed below the human experts, close to where people put a layperson.

A doctor who uses AI landed in the middle, which suggests that AI in a doctor's hands is trusted more than AI on its own.

Doctor

7.5

Nurse

6.8

Doctor, remote visit

6.1

Therapist

6.0

Doctor using AI

5.7

Personal trainer

5.3

AI, general

4.7

AI diagnostic tool

4.4

Layperson

4.3

AI fitness coach

4.3

AI chatbot

4.3

Average trust, 0 to 10. Blue: human. Coral: AI. Grey: doctor using AI.

The doctor beat the AI on trust, and on both kinds of thinking

The doctor scored 7.5 on trust against 4.7 for a general medical AI. People also rated the doctor as better at using quantitative reasoning and at using intuition.

The trust gap was the largest of the three. That fits the idea that people credit doctors with both numbers and judgment, and trust them for it.

Doctor AI

Trust

7.5

4.7

Quantitative reasoning

7.2

5.2

Intuition

6.7

4.5

Average ratings, 0 to 10.

Most people trusted humans over AI, but not everyone

I averaged each person's trust in the six human recommenders and in the four AI ones. 126 of 181 people trusted the humans more. 42 trusted the AI more, and 13 rated them the same.

So the gap is a strong tendency, not a rule. About 23 percent of people trusted the AI more.

126 of 181 trusted humans more than AI

Each dot is one person. Blue dots trusted humans more.

People do not treat all medical decisions alike

I asked how much a professional would need numbers versus intuition for each decision. Diagnosing and prescribing leaned on numbers. Prevention, such as advice to stretch or take vitamins, leaned on intuition, as did decisions about mental health.

That suggested the kind of decision might change how people judge an AI, which Study 2 tested.

Prescriptive

61.8

Diagnostic

61.4

Chronic sickness

60.7

Bodily injury

59.7

Acute sickness

57.4

Mental health

55.8

Preventative

52.0

0 means only intuition is needed, 100 means only quantitative reasoning.

What this means

The gap replicates. People trusted a doctor more than a medical AI, as earlier research found, which gave the later studies a foundation to build on. These were medical decisions at their most hypothetical, so the next study used concrete decisions to see whether the gap holds when the decision is specific.

People credit doctors with more of both kinds of reasoning. The doctor was rated higher than the AI on intuition and on quantitative reasoning, so in these two respects the AI starts at a deficit. Ratings of both abilities went with trust, and the two ratings explained about half of the variance in trust (R² from .47 to .51, depending on the recommender). It did not explain all of it, so these abilities look like part of how people evaluate a recommender, not the whole of it.

Medical decisions are not one thing. Preventative decisions and mental health decisions were seen as needing relatively more intuition, and less quantitative reasoning, than the other decisions. That variation suggests the character of a decision may also shape how much a recommender is trusted, which Study 2 put to the test.

Files and methods

study1-ratings.csv, the de-identified ratings
study1-codebook.csv, what every column means
study1-analysis.R, the main tests in R, from the public file
study1-original-analysis.Rmd, my original notebook, run on the raw export
Preregistration (PDF), written before data collection
study1-full-results.py, the code behind the tables and figures on the full analysis page
study1-human-vs-ai.py, the code behind the “Humans versus AI” page

Everything above was computed in Python from the released file. Where the notebook used an R package, I used the closest equivalent: the reliability, factor and descriptive tables follow psych, and the mixed models follow lme4 with a random intercept for each person. The slopes in my mixed models for AI and for human recommenders match the ones saved in the notebook. I have not yet run the R code on the released file to compare the two line by line.

The data, codebook and preregistration are shared under CC BY 4.0; the code under the MIT license. Please do not use the file to try to identify anyone. Please cite the dissertation: Leitao, M. R. (2024). Artificial Intelligence, Humanization, and Health Beliefs. Georgetown University.