Matthew Leitao

Trust in medical AI

Study 3

609 adults were randomly assigned to read a health recommendation from a doctor, a standard AI or a humanlike AI, and then rated how much they trusted and believed it, and how warm and competent its source seemed. This page gives the main findings and what they mean. Every table and figure is on the next page, rebuilt from the released data.

Main findings

Seen alone, a humanlike AI was trusted no more than a standard one

609 adults read the same health advice from one of three sources: a doctor, a standard AI, or an AI that was given a human name and spoke like a person. Each person saw only one. The humanlike AI was rated 0.52 on trust and the standard AI 0.41, on a scale from −3 to 3.

That difference is too small to tell apart from chance (p = .446, d = 0.07). The data are 4.5 times more likely under no difference than under a medium one, moderate evidence that the humanlike framing did nothing here.

d = 0.07

humanlike AI against standard AI, on trust

Zero is no difference. About 0.4 would be a medium one.

The doctor was trusted more than either AI, but by a modest amount

People who saw the doctor rated trust at 0.86, against 0.46 for the two AIs combined. That is a gap of 0.4 points, or d = 0.26.

The gaps in Chapter 2, where each person rated a doctor and an AI side by side, were larger.

+0.4 points

more trust in the doctor than in the AIs

Trust runs from −3 to 3. Study 3, doctor against both AI conditions.

People saw the doctor as warmer and more competent

On warmth, the doctor scored 2.0 and the AIs 1.5 out of 4. On competence, the scores were 2.3 and 1.9.

The humanlike AI did not close these gaps either. It looked no warmer than the standard AI.

Doctor Both AIs

Warmth (0 to 4)

2.0
1.5

Competence (0 to 4)

2.3
1.9

Average ratings, Study 3.

Warmth and competence account for the doctor's edge

When I compared only the three sources, the doctor sat 0.34 points above the humanlike AI. After I accounted for how warm and competent people found each one, that gap fell to 0.02.

Competence carried most of it. Each extra point of competence went with 0.73 more points of trust. That fits Chapter 2, where people who found a recommender more capable also trusted it more.

0.34 to 0.02

doctor advantage over the humanlike AI, before and after accounting for warmth and competence

Points of trust, Study 3.

The humanlike framing may not have registered

I asked people how humanlike the AI seemed, from 0 to 4. The humanlike AI scored 1.64 and the standard AI 1.54. That is no meaningful difference (p = .328).

So the weak result may reflect a weak manipulation rather than a real absence of effect. Study 4 strengthened it by adding a voice.

Humanlike AI Standard AI

How humanlike the AI seemed (0 to 4)

1.6
1.5

Manipulation check, Study 3.

What this means

A human name and humanlike wording were not enough. Describing the AI this way did not raise trust or belief when people judged one recommender alone. The result is a null, and I read it with the manipulation check in mind: people did not rate the humanlike AI as more humanlike. The test also had limited power. With these group sizes the smallest difference it could reliably detect between the two AIs was d = 0.26.

The doctor's advantage was real but modest. It held for trust, belief, warmth and competence in the planned test of the doctor against both AIs. Against the humanlike AI alone, the edge in trust (d = 0.23) and belief did not survive correction for the three pairwise comparisons (Tukey p = .10 and .10). For warmth and competence it did.

Perceived competence does most of the work. Once warmth and competence were in the model, the doctor's edge was close to zero. That matches Chapter 2, where how capable people thought a recommender was explained about half the gap. It suggests that an AI would earn trust by looking capable, not by sounding friendly.

Cautions. People rated one recommender each, so differences between conditions are differences between groups. The advice was also a single piece of advice, so I cannot say how far the result carries to other health decisions. Studies 4 and 5 address the weak manipulation and the single-recommender design.

Files and methods

Everything on these pages was computed in Python from the released file. Group comparisons are Welch t tests, with Cohen's d from the pooled standard deviation and intervals from the noncentral t distribution. The three-group tests are one-way ANOVAs with Tukey adjustment. The released file keeps the 609 people who rated trust. It contains no IDs, locations, timestamps or free text, age appears only in four bands, and the order of rows is shuffled.

The dissertation reports Cohen's d as 2t divided by the square root of the degrees of freedom. These pages use the pooled standard deviation, which gives values a little different from the dissertation's in some tests. The conclusions are the same.

The data, codebook and code are shared under CC BY 4.0 (data) and the MIT license (code). Please do not use the file to try to identify anyone. Please cite the dissertation: Leitao, M. R. (2024). Artificial Intelligence, Humanization, and Health Beliefs. Georgetown University.